Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement DeepSpeed Main autotuning for NeoX #739

Merged
merged 101 commits into from
Mar 9, 2023
Merged

Implement DeepSpeed Main autotuning for NeoX #739

merged 101 commits into from
Mar 9, 2023

Conversation

dashstander
Copy link
Contributor

@dashstander dashstander commented Dec 9, 2022 •
edited
Loading

Implements autotuning feature from DeepSpeed, which programmatically explores different combinations of microbatch size, gradient accumulation, and ZeRO sharding options given your model and compute set up.

Dashiell Stander and others added 30 commits September 21, 2022 20:33
Merge branch 'srun' into autotune
26e2255
@dashstander
Add autotuning
07f49ae 
Signed-off-by: Dashiell Stander <[email protected]>
Add autotuning config
c3611e9
@dashstander
Need to add it to deepspeed args
5d5626b
@dashstander
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox int…
80c4d3d 
…o autotune
@dashstander
Do not calculate derived values when autotuning
c26a656
@dashstander
Do not calculate derived values when autotuning
80661a1
@dashstander
Do not calculate derived values when autotuning
b2de9ba
@dashstander
Do not calculate derived values when autotuning
08a6300
@dashstander
Do not calculate derived values when autotuning
f5a35da
@dashstander
Need to set no_ssh_check argument with slurm....
7ea39ee
@dashstander
set master_address for SLURM
ee22677
@dashstander
set master_address for SLURM
eb658e3
@dashstander
let json be a file ending
21e1708
@dashstander
Write configs to json files instead of passing them in as CL arguments
110a31f 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Write configs to json files instead of passing them in as CL arguments
dece01b 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Pass in slurm_comment directly to DeepSpeed
ecd8f8c 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Move slurm_comment to deepspeed args
c390be1 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Move slurm_comment to deepspeed args
8033d35 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Slurm comment
e1d6b92 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Slurm comment
bb0209d 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Slurm comment
0097a39 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Move configs out of \/tmp
669ee08 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get values from ds_config when autotuning
dfbe565 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Get values from ds_config when autotuning
ef94f8d 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge main
091b115 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Pass in autotuning config properly
ccf1fff 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Debug print statement
eb45fc0 
Signed-off-by: Dashiell Stander <[email protected]>
lower mem requirement in tune.sh
7d093e3 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Cursed hack to pass in autotuning config properly
36ca337 
Signed-off-by: Dashiell Stander <[email protected]>
dashstander and others added 15 commits January 16, 2023 17:46
@dashstander
Remove debugging configs
1c2fb3f 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Remove test scripts
00c9df6 
Signed-off-by: Dashiell Stander <[email protected]>
Update NeoXArgs docs automatically
9c1d2fe
@dashstander
Remove test script
2c5dedd 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox int…
917b0b1 
…o autotune
Update NeoXArgs docs automatically
a28f4b8
@dashstander
Clean up
6326ca1 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox int…
08edb21 
…o autotune
Update NeoXArgs docs automatically
6cd80ab
@dashstander
Run pre-commit hooks
0f2e492 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox int…
a59c9ec 
…o autotune
Update NeoXArgs docs automatically
b92e936
@dashstander
base64 error
bc586c9 
Signed-off-by: Dashiell Stander <[email protected]>
@dashstander
Merge branch 'autotune' of https://github.com/EleutherAI/gpt-neox int…
e05c967 
…o autotune
Update NeoXArgs docs automatically
c19b020
@dashstander
Copy link
Contributor Author

Ok, should be ready for review @Quentin-Anthony

@dashstander dashstander mentioned this pull request Jan 17, 2023
Merged
@Quentin-Anthony Quentin-Anthony changed the base branch from main to deepspeed_main February 14, 2023 22:02
Base automatically changed from deepspeed_main to main March 9, 2023 16:55
@Quentin-Anthony Quentin-Anthony merged commit e897c23 into main Mar 9, 2023
@Quentin-Anthony Quentin-Anthony deleted the autotune branch March 9, 2023 17:28
@silverriver silverriver mentioned this pull request May 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Quentin-Anthony Quentin-Anthony Quentin-Anthony approved these changes

@ShivanshuPurohit ShivanshuPurohit Awaiting requested review from ShivanshuPurohit ShivanshuPurohit was automatically assigned from EleutherAI/pm-gptneo

Assignees

@nsarka nsarka

@dashstander dashstander

Labels
None yet
Projects
None yet
Milestone
Release V2
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@dashstander @Quentin-Anthony @nsarka @StellaAthena