Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load LLMs in FP16 for Faster Inference #10

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Load LLMs in FP16 for Faster Inference #10

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
acbb67f
Load LLMs in FP16 for faster inference
Kyle1668 Jul 5, 2023
85b45e8
Add error handling for when there is an exception in the pple code
Kyle1668 Jul 15, 2023
5630341
Begin making code parralel
Kyle1668 Aug 7, 2023
f4b02c0
Checkpoint
Kyle1668 Aug 9, 2023
83f2c2a
Created configureble batch sizes
Kyle1668 Aug 9, 2023
96053f7
Fix batch size issue
Kyle1668 Aug 18, 2023
a22fa4c
Updated files to handle new schema
Kyle1668 Sep 14, 2023
f56cf7a
Introduce parallel
Kyle1668 Sep 15, 2023
feda73f
Begin focusing just on ppl
Kyle1668 Sep 15, 2023
f6da96b
Have multiple options
Kyle1668 Sep 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Next Next commit
Load LLMs in FP16 for faster inference
  • Loading branch information
@Kyle1668
Kyle1668 committed Jul 5, 2023
commit acbb67fc8044d496062127261ad69def63635c2a
3 changes: 1 addition & 2 deletions inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def load_model(split_name):
isDeduped = split_name.startswith("deduped")
model = split_name.split("duped.")[-1]
corresponding_model = f"EleutherAI/pythia-{model}{'-deduped' if isDeduped else ''}"
return GPTNeoXForCausalLM.from_pretrained(corresponding_model, device_map="auto")
return GPTNeoXForCausalLM.from_pretrained(corresponding_model, device_map="auto", torch_dtype=torch.float16)


def calculate_perplexity(logits: torch.Tensor, labels: torch.Tensor) -> torch.float64:
Expand Down Expand Up @@ -293,7 +293,6 @@ def parse_cli_args():
"--models",
type=str,
help=models_arg_help,
choices=models_args_default,
default=models_args_default,
)

Expand Down