Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace skip_embed with input_embeds #222

Merged
merged 1 commit into from
Mar 11, 2024
Merged

Conversation

TideDra
Copy link
Contributor

@TideDra TideDra commented Feb 23, 2024

This PR replace the arg skip_embed: bool with input_embeds: torch.Tensor in the forward functions of all the language models.

This fixes the bug raised by LogitsProcessor when using llava in Extend mode and return log_prob, such as using gen(choices=[...]) with llava, where LogitsProcessor need to receive input_ids and uses it to get the token logprob. However, the input_ids given to LogitsProcessor by llava is actually the input embedding, which will cause a Tensor dimension mismatch error.

@TideDra
Copy link
Contributor Author

TideDra commented Feb 23, 2024

With this fix, we can use gen(choices=[...]) with llava in text-only mode but can not give images, such as:

@function
def test(s,img_path,question):
    s += user(image(img_path)+question+" Please answer yes or no.")
    s += assistant(gen('answer',choices=['yes','no']))

This will raise Runtime error, and it seems to be caused by the cache system not working well with image input.

@aliencaocao
Copy link

Same observation with runtime error for image inputs

@merrymercy merrymercy merged commit 64fe311 into sgl-project:main Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants