Yi-VL Model #112

BabyChouSr · 2024-01-28T03:26:22Z

Adding support for the Yi-VL Model: https://huggingface.co/01-ai/Yi-VL-6B

Note, since the original repo does not have a very friendly format, I moved the files and created my own config which makes it more compatible with the SGLang codebase. This allows us to load the model, tokenizer, and processor without much code change.

To test, simply call:

runtime = sgl.Runtime(model_path="BabyChou/Yi-VL-6B",
                      tokenizer_path="BabyChou/Yi-VL-6B")

Link to huggingface repo compatible with this commit:
6B model: https://huggingface.co/BabyChou/Yi-VL-6B
34B model: https://huggingface.co/BabyChou/Yi-VL-34B

exceedzhang · 2024-01-31T04:52:04Z

@BabyChouSr I ran python test_openai_server.py --test-image,
test_chat_completion_image(args) error, following error:

Server Error:

merrymercy

Looks good to me!

Could you also convert the 34B version?
Could you upload the conversion script?
Could you add an example similar to this one https://github.com/sgl-project/sglang/blob/main/examples/quick_start/srt_example_llava.py, but for Yi-VL?

python/sglang/srt/utils.py

paulcx · 2024-01-31T10:04:22Z

try an example in srt_example_llava.py

state = image_qa.run( image_path="images/cat.jpeg", question="What is this?", max_new_tokens=64) then

    File "xx/sglang/python/sglang/srt/models/llava.py", line 63, in pad_input_ids
        offset = input_ids.index(self.config.image_token_index)
ValueError: 64002 is not in list

aliozts · 2024-01-31T11:43:09Z

@paulcx I had the same issue, it's because of the prompting, you need to include the <image> from my understanding. I'd suggest referring to #41

paulcx · 2024-01-31T12:07:21Z

@paulcx I had the same issue, it's because of the prompting, you need to include the <image> from my understanding. I'd suggest referring to #41

Should it be <image_holder>?

aliozts · 2024-01-31T12:25:05Z

I don't know the placeholder for Yi models. it can be different but the key part is the respective token being missing most probably since the error is caused by input_ids.index(self.config.image_token_index) means that the respective token(64002)'s text version is missing.

BabyChouSr · 2024-01-31T20:23:35Z

@paulcx I had the same issue, it's because of the prompting, you need to include the <image> from my understanding. I'd suggest referring to #41

Should it be <image_holder>?

I believe that the sglang frontend language (using the example from srt_example_llava but changing llava for yi-vl model) will automatically add the image token which is <image_placeholder>

paulcx · 2024-01-31T20:26:37Z

@paulcx I had the same issue, it's because of the prompting, you need to include the <image> from my understanding. I'd suggest referring to #41

Should it be <image_holder>?

I believe that the sglang frontend language (using the example from srt_example_llava but changing llava for yi-vl model) will automatically add the image token which is <image_placeholder>

If so, why does "64002 is not in list" happen?

BabyChouSr · 2024-01-31T20:30:08Z

@paulcx I had the same issue, it's because of the prompting, you need to include the <image> from my understanding. I'd suggest referring to #41

Should it be <image_holder>?

I believe that the sglang frontend language (using the example from srt_example_llava but changing llava for yi-vl model) will automatically add the image token which is <image_placeholder>

If so, why does "64002 is not in list" happen?

Are you using the following for your runtime?

runtime = sgl.Runtime(model_path="BabyChou/Yi-VL-6B",
                      tokenizer_path="BabyChou/Yi-VL-6B")

paulcx · 2024-01-31T20:32:46Z

@paulcx I had the same issue, it's because of the prompting, you need to include the <image> from my understanding. I'd suggest referring to #41

Should it be <image_holder>?

I believe that the sglang frontend language (using the example from srt_example_llava but changing llava for yi-vl model) will automatically add the image token which is <image_placeholder>

If so, why does "64002 is not in list" happen?

Are you using the following for your runtime?
runtime = sgl.Runtime(model_path="BabyChou/Yi-VL-6B",

                      tokenizer_path="BabyChou/Yi-VL-6B")

I used the runtime of endpoint. It failed with same error using cli above.

BabyChouSr · 2024-01-31T20:41:11Z

@paulcx I posted some new changes, try running:

python3 srt_example_yi_vl.py

paulcx · 2024-01-31T20:46:15Z

@paulcx I posted some new changes, try running:
python3 srt_example_yi_vl.py

will try later. btw i found yi-vl architecture and model class were not found in model runner by hf config only. so i changed with hard code for now.

paulcx · 2024-02-01T00:50:25Z

@paulcx I posted some new changes, try running:
python3 srt_example_yi_vl.py

It works.

Also works with:

curl http:https://localhost/generate -H "Content-Type: application/json" -d '{"text": "### Human: <image_placeholder>\n图片里有什么？\n### Assistant:", "image_data": "xxxx", "sampling_params": {"max_new_tokens": 64, "temperature": 0, "stop": "### "}}'

merrymercy

@BabyChouSr Ready to be merged?

BabyChouSr · 2024-02-01T07:30:02Z

@BabyChouSr Ready to be merged?

Yup!

loveunk · 2024-02-06T14:10:15Z

It looks like this PR has been merged to SGLang v0.1.11.
However, still encountered the ValueError: 64002 is not in list issue while performing inference with llava-1.6-34B and SGLang v0.1.11.

BabyChouSr · 2024-02-06T19:26:23Z

What is your model_path and tokenizer_path? One reason I can think of is that the 34B LLava uses Yi-Chat as the language model so the image token index is 64002, but vicuna-based language models will have image token index of 32000, thus causing the mismatch.

merrymercy · 2024-02-06T20:20:43Z

@loveunk llava-1.6-34B is not the same as the Yi-VL in this PR, although they used the same base model Yi-34B.

If you use sgl.user/sgl. assistant, our current chat templates only correctly handle Yi-VL and Llava-1.5.
For Llava 1.6, we need some additional handling. We can fix it soon or you can help us fix it. The related code is

sglang/python/sglang/lang/chat_template.py

Lines 171 to 172 in ee1df26

 if "llava" in model_path.lower(): 

 return get_chat_template("vicuna_v1.1")

.

For now, you can follow https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#Demo to use llava 34B with SGLang. They handled chat template correctly in their interface.

Lzhang-hub · 2024-03-11T06:29:04Z

sglang=0.1.12 vllm=0.3.0

python3 srt_example_yi_vl.py with yi-vl-34B and tp_size=2
get error:

Traceback (most recent call last):
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/rpyc/core/protocol.py", line 369, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/rpyc/core/protocol.py", line 863, in _handle_call
    return obj(*args, **dict(kwargs))
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 62, in exposed_init_model
    self.model_runner = ModelRunner(
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 275, in __init__
    self.load_model()
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 308, in load_model
    model.load_weights(
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/sglang/srt/models/yivl.py", line 85, in load_weights
    self.language_model.load_weights(
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 320, in load_weights
    weight_loader(param, loaded_weight)
  File "/home/work/l20/envs/sglang-env/lib/python3.10/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 89, in weight_loader
    assert loaded_weight.shape[parallel_dim] == self.org_vocab_size
AssertionError

BabyChouSr added 6 commits January 28, 2024 03:24

Initialize model

71ffb1c

Fix the tokenizer to use LLAMA tokenizer

4686f6c

Remove more print statements

4abdb35

Use huggingface model path

4307c3e

Remove print statements and test using custom model path

c77ba93

Remove commented out portion of test

52402c1

merrymercy mentioned this pull request Jan 30, 2024

Support Yi-VL-6B/34B #91

Closed

BabyChouSr added 2 commits January 30, 2024 18:35

Remove model_path as argument to get_tokenizer and get_processor

4964c4e

Remove more references to model_path

41212ba

BabyChouSr marked this pull request as ready for review January 30, 2024 18:44

BabyChouSr changed the title ~~[WIP] Yi-VL Model~~ Yi-VL Model Jan 30, 2024

merrymercy reviewed Jan 31, 2024

View reviewed changes

python/sglang/srt/utils.py Outdated Show resolved Hide resolved

Add example usage

3269964

BabyChouSr added 3 commits January 31, 2024 22:32

Add processing scripts

eee337b

Fix 6B script

4ff620d

Use model.path

4272de0

merrymercy approved these changes Feb 1, 2024

View reviewed changes

merrymercy merged commit 8644253 into main Feb 1, 2024

merrymercy deleted the yi-vl branch February 1, 2024 16:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yi-VL Model #112

Yi-VL Model #112

BabyChouSr commented Jan 28, 2024 •

edited

Loading

exceedzhang commented Jan 31, 2024

merrymercy left a comment

paulcx commented Jan 31, 2024

aliozts commented Jan 31, 2024

paulcx commented Jan 31, 2024

aliozts commented Jan 31, 2024

BabyChouSr commented Jan 31, 2024

paulcx commented Jan 31, 2024

BabyChouSr commented Jan 31, 2024

paulcx commented Jan 31, 2024 •

edited

Loading

BabyChouSr commented Jan 31, 2024

paulcx commented Jan 31, 2024

paulcx commented Feb 1, 2024

merrymercy left a comment

BabyChouSr commented Feb 1, 2024

loveunk commented Feb 6, 2024

BabyChouSr commented Feb 6, 2024

merrymercy commented Feb 6, 2024 •

edited

Loading

Lzhang-hub commented Mar 11, 2024

Yi-VL Model #112

Yi-VL Model #112

Conversation

BabyChouSr commented Jan 28, 2024 • edited Loading

exceedzhang commented Jan 31, 2024

merrymercy left a comment

Choose a reason for hiding this comment

paulcx commented Jan 31, 2024

aliozts commented Jan 31, 2024

paulcx commented Jan 31, 2024

aliozts commented Jan 31, 2024

BabyChouSr commented Jan 31, 2024

paulcx commented Jan 31, 2024

BabyChouSr commented Jan 31, 2024

paulcx commented Jan 31, 2024 • edited Loading

BabyChouSr commented Jan 31, 2024

paulcx commented Jan 31, 2024

paulcx commented Feb 1, 2024

merrymercy left a comment

Choose a reason for hiding this comment

BabyChouSr commented Feb 1, 2024

loveunk commented Feb 6, 2024

BabyChouSr commented Feb 6, 2024

merrymercy commented Feb 6, 2024 • edited Loading

Lzhang-hub commented Mar 11, 2024

BabyChouSr commented Jan 28, 2024 •

edited

Loading

paulcx commented Jan 31, 2024 •

edited

Loading

merrymercy commented Feb 6, 2024 •

edited

Loading