New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add diminish (ending soon) token to TokenizerLazy #20

Merged

loubbrad merged 2 commits into EleutherAI:main from loubbrad:add_dim_tok

Aug 9, 2023

Contributor

loubbrad commented Aug 9, 2023

In this PR we make the following changes:

Add a diminish tok "" to TokenizerLazy. This signils that "" will be see in ~100 tokens from now.
Update ROADMAP.md and make small changes to other files

loubbrad added 2 commits

August 9, 2023 11:16


add diminish tok to lazy tokenizer

85820bc


small changes

cc34a99

loubbrad merged commit 7a41d66 into EleutherAI:main

1 check passed

loubbrad deleted the add_dim_tok branch

August 9, 2023 10:21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment