-
Notifications
You must be signed in to change notification settings - Fork 460
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LLM] Example for Serving Gemma #3207
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, @Michaelvll! Sending some comments first while it's running in background.
MODEL_NAME: google/gemma-7b | ||
HF_TOKEN: <your-huggingface-token> # TODO: Replace with huggingface token | ||
|
||
resources: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Azure is taking a long time without success (quota failover), so I changed this to
resources:
accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB}
ports: 8000
disk_tier: best
any_of:
- cloud: aws
- cloud: gcp
and got
File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/serve/core.py", line 88, in up
raise ValueError(f'Got multiple clouds: {requested_cloud} and '
ValueError: Got multiple clouds: GCP and AWS in different resources. Please specify single cloud instead.
for sky serve up
. Is this expected?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be an issue that needs to be fixed @cblmemo @MaoZiming ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! Fixed in #3226. Thanks!
Co-authored-by: Zongheng Yang <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works like a charm on by launch
and serve up
. LGTM.
llm/gemma/README.md
Outdated
|
||
### Prerequsite | ||
|
||
1. Apply for the access to the Gemma model |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1. Apply for the access to the Gemma model | |
1. Apply for access to the Gemma model |
Co-authored-by: Zongheng Yang <[email protected]>
Co-authored-by: Zongheng Yang <[email protected]>
Co-authored-by: Zongheng Yang <[email protected]>
Co-authored-by: Zongheng Yang <[email protected]>
* Add serve for gemma and fix mixtral dependency * Add hf token * fix model len * Add comment * Serve your private gemma * fix serve yaml * readme * Remove chat completion due to the wrong template * add readme * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * address comments * Update README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Change to it * Add chat API * use HF_TOKEN env * typo --------- Co-authored-by: Zongheng Yang <[email protected]>
* initial commit * newline * comments * run linter * reminder for down * tentatively done with example * formatting * yapf * [Storage] Storage mounting tool permissions fix (#3215) * fix permissions * fix permissions * [LLM] Example for Serving Gemma (#3207) * Add serve for gemma and fix mixtral dependency * Add hf token * fix model len * Add comment * Serve your private gemma * fix serve yaml * readme * Remove chat completion due to the wrong template * add readme * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * address comments * Update README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Change to it * Add chat API * use HF_TOKEN env * typo --------- Co-authored-by: Zongheng Yang <[email protected]> * [LLM] Add logo for Gemma (#3220) * Minor fixes for release 0.5.0 (#3212) * when removing cudo credential, sky check fails * remove tips * minor hint fix * fix cluster version for k8s * fix typo * [Docker] Add retry for docker pull due to daemon not ready (#3218) * Add retry for docker pull due to daemon not ready * longer wait time * longer wait time * retry earlier * add retry for retries as well * longer wait time * change wait time * format * Add comment * Fix * Fix indent for azure docker config * Fix docker login config * Fix comments * More robust docker login config * Add retry for docker check * minor fix * Add additional test for stop and start with docker * Fix cancelled * added comments * quick fix * finished pip issues * fix * fix storage error message, add example link to docs --------- Co-authored-by: Sheth <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: Zhanghao Wu <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]>
* initial commit * newline * comments * run linter * reminder for down * tentatively done with example * formatting * yapf * [Storage] Storage mounting tool permissions fix (#3215) * fix permissions * fix permissions * [LLM] Example for Serving Gemma (#3207) * Add serve for gemma and fix mixtral dependency * Add hf token * fix model len * Add comment * Serve your private gemma * fix serve yaml * readme * Remove chat completion due to the wrong template * add readme * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * address comments * Update README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Change to it * Add chat API * use HF_TOKEN env * typo --------- Co-authored-by: Zongheng Yang <[email protected]> * [LLM] Add logo for Gemma (#3220) * Minor fixes for release 0.5.0 (#3212) * when removing cudo credential, sky check fails * remove tips * minor hint fix * fix cluster version for k8s * fix typo * [Docker] Add retry for docker pull due to daemon not ready (#3218) * Add retry for docker pull due to daemon not ready * longer wait time * longer wait time * retry earlier * add retry for retries as well * longer wait time * change wait time * format * Add comment * Fix * Fix indent for azure docker config * Fix docker login config * Fix comments * More robust docker login config * Add retry for docker check * minor fix * Add additional test for stop and start with docker * Fix cancelled * added comments * quick fix * finished pip issues * fix * fix storage error message, add example link to docs * logging for SSH when doing kubernetes provision * romil edits * took out todo commnt * removed extra file * renamed file * restored right version of file * simplify things * newline * more formatting * formatting * minor fixes * set x and logging * fixes * docstr --------- Co-authored-by: Sheth <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: Zhanghao Wu <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]>
* initial commit * newline * comments * run linter * reminder for down * tentatively done with example * formatting * yapf * [Storage] Storage mounting tool permissions fix (#3215) * fix permissions * fix permissions * [LLM] Example for Serving Gemma (#3207) * Add serve for gemma and fix mixtral dependency * Add hf token * fix model len * Add comment * Serve your private gemma * fix serve yaml * readme * Remove chat completion due to the wrong template * add readme * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * address comments * Update README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Update llm/gemma/README.md Co-authored-by: Zongheng Yang <[email protected]> * Change to it * Add chat API * use HF_TOKEN env * typo --------- Co-authored-by: Zongheng Yang <[email protected]> * [LLM] Add logo for Gemma (#3220) * Minor fixes for release 0.5.0 (#3212) * when removing cudo credential, sky check fails * remove tips * minor hint fix * fix cluster version for k8s * fix typo * [Docker] Add retry for docker pull due to daemon not ready (#3218) * Add retry for docker pull due to daemon not ready * longer wait time * longer wait time * retry earlier * add retry for retries as well * longer wait time * change wait time * format * Add comment * Fix * Fix indent for azure docker config * Fix docker login config * Fix comments * More robust docker login config * Add retry for docker check * minor fix * Add additional test for stop and start with docker * Fix cancelled * added comments * quick fix * finished pip issues * fix * fix storage error message, add example link to docs * changed error message if default nc installed on mac * refactored check_port_forward_mode_dependencies function * update comment --------- Co-authored-by: Sheth <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: Zhanghao Wu <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]>
TODO (future):
Tested (run the relevant ones):
bash format.sh
sky launch -c gemma llm/gemma/serve.yaml --cloud gcp --env HF_TOKEN="xxx"
curl
the model with command in readmesky serve up -n gemma serve.yaml --env HF_TOKEN
pytest tests/test_smoke.py
pytest tests/test_smoke.py::test_fill_in_the_name
bash tests/backward_comaptibility_tests.sh