Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix tokenization space cleanup #215

Merged
merged 1 commit into from
Aug 9, 2023
Merged

Fix tokenization space cleanup #215

merged 1 commit into from
Aug 9, 2023

Conversation

gsarti
Copy link
Member

@gsarti gsarti commented Aug 9, 2023

Description

Fixes #198 by forcing clean_up_tokenization_spaces=False at in HuggingfaceModel.decode, ensuring consistency between input_texts and generated_texts for decoder-only models.

@gsarti gsarti added the bug Something isn't working label Aug 9, 2023
@gsarti gsarti added this to the v0.5 milestone Aug 9, 2023
@gsarti gsarti merged commit f8e55f8 into main Aug 9, 2023
4 checks passed
@gsarti gsarti deleted the fix-tok-space-cleanup branch August 9, 2023 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
v0.5
Development

Successfully merging this pull request may close these issues.

Error of input text not matching decoded output of tokenizer
1 participant
@gsarti