Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load LLMs in FP16 for Faster Inference #10

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
+905 −119
Open

Load LLMs in FP16 for Faster Inference #10

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
acbb67f
Load LLMs in FP16 for faster inference
Kyle1668 Jul 5, 2023
85b45e8
Add error handling for when there is an exception in the pple code
Kyle1668 Jul 15, 2023
5630341
Begin making code parralel
Kyle1668 Aug 7, 2023
f4b02c0
Checkpoint
Kyle1668 Aug 9, 2023
83f2c2a
Created configureble batch sizes
Kyle1668 Aug 9, 2023
96053f7
Fix batch size issue
Kyle1668 Aug 18, 2023
a22fa4c
Updated files to handle new schema
Kyle1668 Sep 14, 2023
f56cf7a
Introduce parallel
Kyle1668 Sep 15, 2023
feda73f
Begin focusing just on ppl
Kyle1668 Sep 15, 2023
f6da96b
Have multiple options
Kyle1668 Sep 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Have multiple options
  • Loading branch information
@Kyle1668
Kyle1668 committed Sep 15, 2023
commit f6da96b9d71bd63bb59b9fbc9d9d8546e33bd535
2 changes: 1 addition & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@


!datasets/eval/Pythia_70m_Deduped_Low_Perplexity_Labeling_Formatted.csv

*.pt
*.zip
.vscode
### Data ###
Expand Down
12 changes: 10 additions & 2 deletions inference_outputs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,16 @@ def run_model_inferences(split_name: str, run_id: str, dataset: str, batch_size:



results = p.map(parse_attn, [t.detach().cpu() for t in outputs.attentions])
print(results)
# results = p.map(parse_attn, [t.detach().cpu() for t in outputs.attentions])
# print(results)

# attentions_table = {}
for i in tqdm(range(len(batch[0]))):
current_example_id = batch[0][i]
current_example_attentions = torch.stack(outputs.attentions)[:, i, :]
# attentions_table[current_example_id] = current_example_attentions
torch.save(current_example_attentions, f"datasets/{run_id}/{dataset}_attentions_{current_example_id}.pt")
# print(current_example_attentions.shape)

# inference_logs = pd.DataFrame({
# "Loss": outputs.loss.detach().cpu().tolist(),
Expand Down
2 changes: 1 addition & 1 deletion inference_sync.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ def get_batch_size(model_name: str) -> int:
"2.8b": 128,
# Large
"6.9b": 64,
"12b": 64,
"12b": 32,
}
model_size = ".".join(model_name.split(".")[1:])
return size_batch_map[model_size]
Expand Down