Allow input_ids in the input of the /generate endpoint
#363
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
merrymercy
reviewed
May 12, 2024
| if cur_output_tokens[-1] == self.tokenizer.eos_token_id: | ||
| recv_obj.meta_info[i]['output_tokens'] = cur_output_tokens[:-1] | ||
| else: | ||
| recv_obj.meta_info[i]['output_tokens'] = cur_output_tokens |
There was a problem hiding this comment.
Can you explain why this is needed?
There was a problem hiding this comment.
Oh sry I just saw your comment. I think this piece of code just tries to make sure the token output is aligned with text output. If I remember correctly, when I tested last time, if the last token is EOS, the text output does not include the <|endoftext|> token while the <|endoftext|> token id will be included in output_tokens without this code. But perhaps I am hallucinating on this since I did the test a while ago...
bc6703a
to
48a78a5
Compare
|
@lolipopshock Thanks for the contribution. It is merged. I do not understand this comment (#363 (comment)) so I reverted it. Feel free to reopen |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As of now,
GenerateReqInputonly allowstextin the input; this PR enables specifying eithertextorinput_idsas the request input. Supplementinginput_idsallows more precise token operations in the input; and hopefully this simple patch won't affect other functionality.