-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wip: Test llama fsdp #225
Closed
Closed
wip: Test llama fsdp #225
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit
Hold shift + click to select a range
c2b0174
add testing script
thejaminator 79adfcf
fix tensor
thejaminator 91e0503
add fsdp set up
thejaminator e6b865c
add num gpus
thejaminator 4c755ac
cache the input_ids
thejaminator 16a0eb1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 71139a3
change the inference speed
thejaminator e7064f7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] c5f0eea
fix inference speed estimation
thejaminator 5d3fe93
add the cache so its less annoying
thejaminator 45073f4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 25852a8
add device map
thejaminator 12e6542
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 5087dbe
add tqdm
thejaminator 21bac7a
set lm head to 0
thejaminator 2dd0a33
add_default
thejaminator dcea7fe
add shuffling
thejaminator 74fd8a5
add threads
thejaminator e107f7d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] efec1d0
add 8bit option
thejaminator 2ae9525
add init empty weights
thejaminator 5a8fdd8
add llamaattention to not be split
thejaminator 3d2002d
try by instantiating the actual thing
thejaminator 32ace27
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 19bce58
split properly
thejaminator 5836bb3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 395bec6
add device map manually
thejaminator 1fff07f
print the auto map
thejaminator 6561b28
set override
thejaminator 7568b22
put the attention block on device 0
thejaminator 53c09a5
add overwrite for 30b and try to make it set to 1 device for lm head
thejaminator 7a83e0f
add overwrite for 65b
thejaminator 02e4f4f
add batching by padding
thejaminator bcb9efb
print batch size
thejaminator f36989f
fix device
thejaminator 51c4730
put it on device 0 first
thejaminator 048202b
pre batch before sending to worker
thejaminator fb1899e
add batch print
thejaminator 7eabb3d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] c9f0f0f
fix worker args
thejaminator 63a2ed4
add sort before batching
thejaminator 173bd3c
more layers to the first device
thejaminator e6f0d6f
add torch compile option
thejaminator 6594712
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] c33a406
add strategy for shaarding
thejaminator 5df0623
add option to ignore override
thejaminator 68a07e6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] fce2f47
disable device map
thejaminator a37bae2
remove torch compile
thejaminator 75b7c12
correct arg parse
thejaminator 80eb3eb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 7333096
fix not splitting
thejaminator 43c3ada
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 7da5630
specify the max memory properly
thejaminator 3fcaf0e
fix memory
thejaminator c10e2f1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] e57bf4d
put the dummy model somewhere else
thejaminator 6f13c8b
Merge remote-tracking branch 'origin/test-llama-fsdp' into test-llama…
thejaminator ffdef0e
remove device map for init empty
thejaminator 54bee5d
try out fsdp sharding
thejaminator File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
add threads
- Loading branch information
commit 74fd8a59cebbf2eba3df8b9f2fe5ec6c2e15cdd9
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test 65b with
you can up the batch_size up to 4