Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix local completion huggingface tokenizer #1975

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

okdshin
Copy link

@okdshin okdshin commented Jun 17, 2024

Fixes related to the tokenizer in the constructor of the OpenaiCompletionsLM class

Added the huggingface_tokenizer_trust_remote_code argument to the __init__ method, allowing the trust_remote_code option to be set when initializing the tokenizer. This enables control over whether remote code execution is allowed.
Updated the retrieval of the tokenizer's vocabulary size to use self.tokenizer.vocab_size instead of self.tokenizer.vocab. This ensures that the correct vocabulary size is obtained.

@CLAassistant
Copy link

CLAassistant commented Jun 17, 2024

CLA assistant check
All committers have signed the CLA.

@okdshin
Copy link
Author

okdshin commented Jul 18, 2024

@haileyschoelkopf @lintangsutawika
Hi there,
Just checking in on the PR I submitted a month ago. Any thoughts or feedback would be greatly appreciated. If it's not a good fit, please let me know and I can close it.
Thanks for your time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants