New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add composer prefix tokens to TokenizerLazy #23

Merged

loubbrad merged 3 commits into EleutherAI:main from loubbrad:tokenizer_composer

Aug 12, 2023

Contributor

loubbrad commented Aug 12, 2023

In this PR we make the following changes:

Refactor the way that prefix tokens work for TokenizerLazy. Now they take the form ("prefix", type, data).
Add composer prefix tokens to TokenizerLazy.
Add functionality for including composer prefix tokens in tokenize_midi_dict according to metadata.

loubbrad added 3 commits

August 12, 2023 12:19


added composer prefix tokens

049e0ed


fix composer name config usage

3e69b34


fix roadmap

8c22f07

loubbrad merged commit 1786de2 into EleutherAI:main

1 check passed

loubbrad deleted the tokenizer_composer branch

August 12, 2023 11:34

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment