Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[API] Add octoai back-end #936

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

[API] Add octoai back-end #936

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a74fa56
add octoai front-end
vvchernov Oct 17, 2023
cc2613b
small fixes
vvchernov Oct 19, 2023
bbfc019
add temperature and top_p parameters to octoai request
vvchernov Oct 25, 2023
56b26c9
add url arg to octoai
vvchernov Nov 8, 2023
4151c04
transfer message to call_octoai_inference for a moment
vvchernov Nov 8, 2023
e98b465
implementation of loglikelihood in parallel to greedy_until
vvchernov Nov 10, 2023
4f13d0e
refactor and clean code. Two runner for greedy_until and loglikelihoo…
vvchernov Nov 10, 2023
b29849e
update octoai logprob API for loglikelihood calculations
vvchernov Nov 13, 2023
dada5ad
implement abstract method
vvchernov Nov 13, 2023
b1ce0bb
add response check methods
vvchernov Nov 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add temperature and top_p parameters to octoai request
  • Loading branch information
@vvchernov
vvchernov committed Oct 25, 2023
commit bbfc0190b8f5cb69f3b49030198ed646ca6b28de
12 changes: 8 additions & 4 deletions lm_eval/models/octoai_llms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ def __init__(
model_name="llama2-7b-chat-mlc-q0f16",
batch_size=1,
max_batch_size=None,
device=None):
device=None,
top_p=0.0,
temperature=0.0,):
"""
:param model_name: str
Model name from the list of models supported by OctoAI
Expand All @@ -52,9 +54,9 @@ def __init__(
self._device=device
# TODO(vvchernov): check that model name is supported

self.init_remote()
self.init_remote(top_p, temperature)

def init_remote(self):
def init_remote(self, top_p, temperature):
# Get the API key from the environment variables
api_key=os.environ["OCTOAI_API_KEY"]

Expand All @@ -78,7 +80,9 @@ def init_remote(self):
}
],
"stream": False,
"max_tokens": 256
"max_tokens": 256,
"top_p": top_p,
"temperature": temperature,
}

@property
Expand Down