Model sampling/text generation #160
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # Dockerfile # requirements.txt
commit 43be6ce Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 10:19:12 2021 +0000 Remove debugging commit 450dfb9 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 10:04:23 2021 +0000 Test `input-file` commit 3d0f562 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:52:57 2021 +0000 Skip tokens that don't exist commit 9384ace Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:48:52 2021 +0000 Debug commit 82871dd Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:47:49 2021 +0000 Debug commit 5d1d19e Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:44:50 2021 +0000 Remove debugging commit dc4cf73 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:41:33 2021 +0000 Debug commit a4d2bf3 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:33:29 2021 +0000 Force activation checkpointing to be disabled commit 901aa73 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:29:27 2021 +0000 Debugging commit c9706d1 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 09:19:41 2021 +0000 Debugging commit a4835b1 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 08:55:17 2021 +0000 Debugging commit 747b9a9 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 08:46:48 2021 +0000 Debugging commit 36137c1 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 08:28:54 2021 +0000 Debugging commit f58ea20 Author: Josh Levy-Kramer <[email protected]> Date: Tue Mar 9 08:22:35 2021 +0000 Debugging commit 4ff6300 Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 18:47:04 2021 +0000 Change port based on rank commit be4b715 Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 18:45:13 2021 +0000 Change port based on rank commit 63a1be0 Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 18:43:52 2021 +0000 Change port based on rank commit 1b8f6b9 Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 18:16:58 2021 +0000 pycharm debugger commit 1110bee Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 18:05:47 2021 +0000 pycharm debugger commit 90bb5d4 Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 17:51:46 2021 +0000 Test: try manhole commit 6fbcc1f Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 17:50:08 2021 +0000 Test: try manhole commit a272a59 Author: Josh Levy-Kramer <[email protected]> Date: Mon Mar 8 17:46:50 2021 +0000 Test: try manhole
|
Ok, this is now working with pipeline parallel size > 1. The merge conflicts will be quite intense (since rotary pos emb means the number of args passed between pipe stages can differ). I'll sort these soon and we can merge this. |
|
Ok so sampling is now working with all pipeline parallel sizes. It's the most awful, hacky code ever, you need to update your deeperspeed branch to the latest commit to get it to work. This should be ready to be merged now imo. |
Bring all dependencies up to their latest stable versions.
ShivanshuPurohit
approved these changes
Apr 19, 2021
ShivanshuPurohit
previously approved these changes
Apr 19, 2021
sdtblck
previously approved these changes
Apr 22, 2021
ShivanshuPurohit
previously approved these changes
Apr 22, 2021
sdtblck
approved these changes
Apr 22, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Text generation is working with no output (unconditional) and with an input file. By "working" I mean it is loading a model, creating output and saving it to file. The output is currently utter gibberish as I'm testing it on basically a randomly initialised model.
Steps to test:
Generate a model:
save-intervalto 10 or something small in the model config../deepy.py pretrain_gpt2.py -d configs small.yml eleutherai_cluster.ymland wait for a checkpoint to save.To test
unconditionaltext generation:"text-gen-type": "unconditional"is set in thetext_generation.ymlconfig./deepy.py text_gen_gpt2.py -d configs small.yml eleutherai_cluster.yml text_generation.yml/mnt/ssd-cluster/output/text_generation.txtTo test
input-filetext generation:"text-gen-type": "input-file"is set in thetext_generation.ymlconfig/mnt/ssd-cluster/output/sample_input.txt./deepy.py text_gen_gpt2.py -d configs small.yml eleutherai_cluster.yml text_generation.yml/mnt/ssd-cluster/output/sample_output.txtTodo: