Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Gradient Checkpointing #55

Closed
StellaAthena opened this issue Jan 12, 2021 · 2 comments · Fixed by #89
Closed

Implement Gradient Checkpointing #55

StellaAthena opened this issue Jan 12, 2021 · 2 comments · Fixed by #89
Labels
feature request New feature or request good first issue Good for newcomers
Projects
1T or BUST

Comments

@StellaAthena
Copy link
Member

Add the option to checkpoint gradients
@StellaAthena StellaAthena added feature request New feature or request good first issue Good for newcomers labels Jan 12, 2021
@StellaAthena StellaAthena added this to To do in 1T or BUST via automation Jan 12, 2021
@glebshevchukk
Copy link

glebshevchukk commented Jan 12, 2021 •
edited

Useful links for this:
microsoft/DeepSpeed#63
https://github.com/microsoft/DeepSpeedExamples/blob/07d1ce0d26044602a7b8bb289e590a980f14aded/Megatron-LM/mpu/random.py#L273

@sdtblck
Copy link
Contributor

sdtblck commented Jan 12, 2021

@StellaAthena deepspeed handles this implicitly from within the deepspeed config, doesn't need any implementation

@sdtblck sdtblck closed this as completed Jan 12, 2021
1T or BUST automation moved this from To do to Done Jan 12, 2021
@sdtblck sdtblck mentioned this issue Jan 13, 2021
Merged
@StellaAthena StellaAthena linked a pull request Jan 25, 2021 that will close this issue
Fix train pipeline #89
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request good first issue Good for newcomers
Projects
1T or BUST
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix train pipeline EleutherAI/gpt-neox
3 participants
@StellaAthena @glebshevchukk @sdtblck