Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline Parallel QoL Fixes #63

Merged
merged 37 commits into from
Jan 17, 2021
Merged

Pipeline Parallel QoL Fixes #63

Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
f900fcd
Added reduce_bucket_size argument to optimizer
StellaAthena Jan 13, 2021
7c7cd80
Started to revamp train.py
StellaAthena Jan 13, 2021
3f04d31
revised training function, version 1
StellaAthena Jan 13, 2021
64bc31c
Update gpt_neox.py
StellaAthena Jan 13, 2021
1f00a10
Update train.sh
StellaAthena Jan 13, 2021
a182b80
Update base_deepspeed.json
StellaAthena Jan 13, 2021
9310806
Create train_gpt3small_pipeline.sh
StellaAthena Jan 13, 2021
079980d
Update train_stella.py
StellaAthena Jan 13, 2021
b7b318d
Update train_stella.py
StellaAthena Jan 13, 2021
932a624
adding environment.yml file
sdtblck Jan 13, 2021
07f58a3
Set zero to stage 1
StellaAthena Jan 13, 2021
87c0485
set ZeRO to stage 2
StellaAthena Jan 13, 2021
a026ad1
turned off activation checkpointing
StellaAthena Jan 13, 2021
e039b5f
turn continuous gradients off
StellaAthena Jan 13, 2021
4d83747
turn activation checkpointing on
StellaAthena Jan 13, 2021
6cecc28
removed reduce bucket size config
StellaAthena Jan 13, 2021
090b50c
turn on cont. graadients
StellaAthena Jan 13, 2021
4e7debd
removed extraneous comma
StellaAthena Jan 13, 2021
2a4cfc8
Turn off masking
StellaAthena Jan 13, 2021
f95f2d5
Turned on attention masking
StellaAthena Jan 14, 2021
a0a1221
Change name in prep for merging
StellaAthena Jan 14, 2021
f895554
Rename base_deepspeed.json to deepspeed_zero2.json
StellaAthena Jan 14, 2021
6e7e7dd
Create deepspeed_zero1.json
StellaAthena Jan 14, 2021
2a79738
Updated with new config names
StellaAthena Jan 14, 2021
4ff12fd
Update with new config names
StellaAthena Jan 14, 2021
d23cb1e
Update with new config names
StellaAthena Jan 14, 2021
1fe703e
Update with new config names
StellaAthena Jan 14, 2021
086c48b
Rename train_stella.py to train_pipeline.py
StellaAthena Jan 14, 2021
c92c946
Update README.md
StellaAthena Jan 14, 2021
f997b84
Changed training loop to be consistent with train_enwik8.py
StellaAthena Jan 14, 2021
544d520
Reorder code to match train_enwik8.py
StellaAthena Jan 14, 2021
ae07ae6
Modified transformerblock to pass mask argument
StellaAthena Jan 14, 2021
4fc2184
Add print statement for testing
StellaAthena Jan 14, 2021
e471fd4
Revert changes to avoid pulling dev work into main
StellaAthena Jan 14, 2021
71c7a77
Updates configs to allow for the third failure mode (#64)
StellaAthena Jan 14, 2021
16468d4
Update train_enwik8.sh
StellaAthena Jan 16, 2021
fa72e69
Update deepspeed_zero2.json
StellaAthena Jan 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update README.md
  • Loading branch information
@StellaAthena
StellaAthena committed Jan 14, 2021
commit c92c9466296aa14d1be327977e05ee40491edbf9
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,9 @@ model = GPTNeoX(
)
```

## Contribute

If you want to get involved, check out our repo projects. Anything that is listed as "todo" or has not been assigned to anyone is fair game, but please leave a comment so that we know you're working on it!

## Resources
If you have trouble getting the model to run, consider consulting [this guide](https://gist.github.com/kevinwatkins/232b88bfecbeca8d48d612a3e9cf65e4) to installing in a GCE virtual machine.