-
Notifications
You must be signed in to change notification settings - Fork 984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upstream DeepSpeed breaks HF conversion script #750
Comments
I'm facing a somewhat similar problem, so just want to check if it has the same root-cause before I open a new Issue for it. When I train with NeoX (cloned a few months back) on two nodes with 8 GPUs each, the main node saves checkpoints with files for all layer states, as well as the I am unable to resume training from these checkpoints; in the above state, the second machine skips loading checkpoints as
Do you think #752 will fix this problem? Please let me know if you think this isn't related to the above and I'll make a separate issue! -Vincent |
@VHellendoorn I think you may have intended to leave this comment on a different issue (#732), as this one is about the HF conversion script rather than the checkpoints themselves. It sounds like the issue is that you are writing to a non-shared storage space, and I'm not sure if we've tested in that context. Because we use SLURM and Kubernetes to manage our runs on datacenter compute, we need to save our checkpoints to a storage device that's independent of the compute nodes if we want to actually be able to access them later. It seems plausible that a bug crept in unnoticed that only effects when you are writing to local storage though. If I'm correct about what's going on, can you open an independent issue to discuss? It should be easy to replicate on our end, and if you can confirm that Megatron-DS doesn't have this problem that would be appreciated. |
Yep, happy to discuss in a separate issue! Seconding Stella's point, it sounds like non-shared storage between nodes may be the issue. |
Thanks both for chiming in! @StellaAthena you're right, my bad; I'd clicked through to here from that issue. I'll make a new issue soon-ish then; might try to debug it a bit more now that I have a lead :) |
The tools/convert_to_hf.py script will need to be updated / a different version may need to be created for checkpoints saved with DeepSpeed. Checkpoints are no longer saved layer-by-layer, it seems, and now all weights are in several
mp_rank_{MP_RANK}_model_states.pt
files for each Model Parallel partition.Upstream DeepSpeed checkpoint:
DeeperSpeed checkpoint:
Updating the script shouldn't be too hard at all though.
The text was updated successfully, but these errors were encountered: