Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline Parallel QoL Fixes #63

Merged
merged 37 commits into from
Jan 17, 2021
Merged

Pipeline Parallel QoL Fixes #63

merged 37 commits into from
Jan 17, 2021

Commits on Jan 13, 2021

  1. Added reduce_bucket_size argument to optimizer

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    f900fcd View commit details
    Browse the repository at this point in the history

  2. Started to revamp train.py

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    7c7cd80 View commit details
    Browse the repository at this point in the history

  3. revised training function, version 1

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    3f04d31 View commit details
    Browse the repository at this point in the history

  4. Update gpt_neox.py

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    64bc31c View commit details
    Browse the repository at this point in the history

  5. Update train.sh

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    1f00a10 View commit details
    Browse the repository at this point in the history

  6. Update base_deepspeed.json

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    a182b80 View commit details
    Browse the repository at this point in the history

  7. Create train_gpt3small_pipeline.sh

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    9310806 View commit details
    Browse the repository at this point in the history

  8. Update train_stella.py

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    079980d View commit details
    Browse the repository at this point in the history

  9. Update train_stella.py

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    b7b318d View commit details
    Browse the repository at this point in the history

  10. adding environment.yml file

    @sdtblck
    sdtblck committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    932a624 View commit details
    Browse the repository at this point in the history

  11. Set zero to stage 1

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    07f58a3 View commit details
    Browse the repository at this point in the history

  12. set ZeRO to stage 2

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    87c0485 View commit details
    Browse the repository at this point in the history

  13. turned off activation checkpointing

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    a026ad1 View commit details
    Browse the repository at this point in the history

  14. turn continuous gradients off

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    e039b5f View commit details
    Browse the repository at this point in the history

  15. turn activation checkpointing on

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    4d83747 View commit details
    Browse the repository at this point in the history

  16. removed reduce bucket size config

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    6cecc28 View commit details
    Browse the repository at this point in the history

  17. turn on cont. graadients

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    090b50c View commit details
    Browse the repository at this point in the history

  18. removed extraneous comma

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    4e7debd View commit details
    Browse the repository at this point in the history

  19. Turn off masking

    @StellaAthena
    StellaAthena committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    2a4cfc8 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2021

  1. Turned on attention masking

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    f95f2d5 View commit details
    Browse the repository at this point in the history

  2. Change name in prep for merging

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    a0a1221 View commit details
    Browse the repository at this point in the history

  3. Rename base_deepspeed.json to deepspeed_zero2.json

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    f895554 View commit details
    Browse the repository at this point in the history

  4. Create deepspeed_zero1.json

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    6e7e7dd View commit details
    Browse the repository at this point in the history

  5. Updated with new config names

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    2a79738 View commit details
    Browse the repository at this point in the history

  6. Update with new config names

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    4ff12fd View commit details
    Browse the repository at this point in the history

  7. Update with new config names

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    d23cb1e View commit details
    Browse the repository at this point in the history

  8. Update with new config names

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    1fe703e View commit details
    Browse the repository at this point in the history

  9. Rename train_stella.py to train_pipeline.py

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    086c48b View commit details
    Browse the repository at this point in the history

  10. Update README.md

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    c92c946 View commit details
    Browse the repository at this point in the history

  11. Changed training loop to be consistent with train_enwik8.py

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    f997b84 View commit details
    Browse the repository at this point in the history

  12. Reorder code to match train_enwik8.py

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    544d520 View commit details
    Browse the repository at this point in the history

  13. Modified transformerblock to pass mask argument 

    Minor tweak to `forward` to align better with the demo code
    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    ae07ae6 View commit details
    Browse the repository at this point in the history

  14. Add print statement for testing

    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    4fc2184 View commit details
    Browse the repository at this point in the history

  15. Revert changes to avoid pulling dev work into main 

    Forgot about the open PR and pushed dev code to this branch.
    @StellaAthena
    StellaAthena committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    e471fd4 View commit details
    Browse the repository at this point in the history

  16. Updates configs to allow for the third failure mode (#64) 

    * Pipeline + Checkpoint - ZeRO 2

This illustrates the third combination: pipeline parallelism, activation checkpoints, and ZeRO Stage 1. Again, this works. Upgrading ZeRO to Stage 2 causes it to fail.

Co-authored-by: Shivanshu Purohit <[email protected]>
    @StellaAthena @ShivanshuPurohit
    StellaAthena and ShivanshuPurohit committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    71c7a77 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2021

  1. Update train_enwik8.sh

    @StellaAthena
    StellaAthena committed Jan 16, 2021
    Configuration menu
    Copy the full SHA
    16468d4 View commit details
    Browse the repository at this point in the history

  2. Update deepspeed_zero2.json

    @StellaAthena
    StellaAthena committed Jan 16, 2021
    Configuration menu
    Copy the full SHA
    fa72e69 View commit details
    Browse the repository at this point in the history