Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wip: Test llama fsdp #225

Closed
wants to merge 60 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
c2b0174
add testing script
thejaminator Apr 30, 2023
79adfcf
fix tensor
thejaminator Apr 30, 2023
91e0503
add fsdp set up
thejaminator Apr 30, 2023
e6b865c
add num gpus
thejaminator Apr 30, 2023
4c755ac
cache the input_ids
thejaminator Apr 30, 2023
16a0eb1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 30, 2023
71139a3
change the inference speed
thejaminator Apr 30, 2023
e7064f7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 30, 2023
c5f0eea
fix inference speed estimation
thejaminator Apr 30, 2023
5d3fe93
add the cache so its less annoying
thejaminator Apr 30, 2023
45073f4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 30, 2023
25852a8
add device map
thejaminator May 1, 2023
12e6542
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 1, 2023
5087dbe
add tqdm
thejaminator May 1, 2023
21bac7a
set lm head to 0
thejaminator May 1, 2023
2dd0a33
add_default
thejaminator May 1, 2023
dcea7fe
add shuffling
thejaminator May 1, 2023
74fd8a5
add threads
thejaminator May 1, 2023
e107f7d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 1, 2023
efec1d0
add 8bit option
thejaminator May 1, 2023
2ae9525
add init empty weights
thejaminator May 1, 2023
5a8fdd8
add llamaattention to not be split
thejaminator May 1, 2023
3d2002d
try by instantiating the actual thing
thejaminator May 1, 2023
32ace27
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 1, 2023
19bce58
split properly
thejaminator May 1, 2023
5836bb3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 1, 2023
395bec6
add device map manually
thejaminator May 1, 2023
1fff07f
print the auto map
thejaminator May 1, 2023
6561b28
set override
thejaminator May 1, 2023
7568b22
put the attention block on device 0
thejaminator May 1, 2023
53c09a5
add overwrite for 30b and try to make it set to 1 device for lm head
thejaminator May 1, 2023
7a83e0f
add overwrite for 65b
thejaminator May 1, 2023
02e4f4f
add batching by padding
thejaminator May 1, 2023
bcb9efb
print batch size
thejaminator May 1, 2023
f36989f
fix device
thejaminator May 1, 2023
51c4730
put it on device 0 first
thejaminator May 1, 2023
048202b
pre batch before sending to worker
thejaminator May 1, 2023
fb1899e
add batch print
thejaminator May 1, 2023
7eabb3d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 1, 2023
c9f0f0f
fix worker args
thejaminator May 1, 2023
63a2ed4
add sort before batching
thejaminator May 1, 2023
173bd3c
more layers to the first device
thejaminator May 1, 2023
e6f0d6f
add torch compile option
thejaminator May 1, 2023
6594712
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 1, 2023
c33a406
add strategy for shaarding
thejaminator May 1, 2023
5df0623
add option to ignore override
thejaminator May 2, 2023
68a07e6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 2, 2023
fce2f47
disable device map
thejaminator May 2, 2023
a37bae2
remove torch compile
thejaminator May 2, 2023
75b7c12
correct arg parse
thejaminator May 2, 2023
80eb3eb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 2, 2023
7333096
fix not splitting
thejaminator May 2, 2023
43c3ada
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 2, 2023
7da5630
specify the max memory properly
thejaminator May 2, 2023
3fcaf0e
fix memory
thejaminator May 2, 2023
c10e2f1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 2, 2023
e57bf4d
put the dummy model somewhere else
thejaminator May 2, 2023
6f13c8b
Merge remote-tracking branch 'origin/test-llama-fsdp' into test-llama…
thejaminator May 2, 2023
ffdef0e
remove device map for init empty
thejaminator May 2, 2023
54bee5d
try out fsdp sharding
thejaminator May 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
  • Loading branch information
pre-commit-ci[bot] committed Apr 30, 2023
commit e7064f7000c0fbaf6dcb641e408922bf1ad3b8e1
4 changes: 2 additions & 2 deletions llama_inference_speed.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
import torch
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy

from transformers import LlamaTokenizer, LlamaForCausalLM
from transformers import LlamaForCausalLM, LlamaTokenizer
from transformers.models.llama.modeling_llama import LlamaDecoderLayer

# from https://github.com/huggingface/transformers/issues/22687
# torchrun --nproc_per_node=2 --master_port=56718 run_forward.py
model_dir = "huggyllama/llama-13b"
Expand Down