New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change prefix token implementation - BREAKING #21

Merged

loubbrad merged 2 commits into EleutherAI:main from loubbrad:meta_toks

Aug 10, 2023

Contributor

loubbrad commented Aug 10, 2023 •

edited

Loading

In this PR we make the following changes:

Change the form that prefix tokens take. The token "piano" will now be ("prefix", "piano").
Fix duplicate drum messages in TokenizerLazy.
Fix drum messages from being decoded over the incorrect channel (1 instead of 0).
Make small improvements to detokenize_midi_dict in TokenizerLazy.
Fix a bug with dim_tok causing a data test to fail.

Note that as we are changing TokenizerLazy, it will no longer work with old model checkpoints.

loubbrad added 2 commits

August 10, 2023 22:05


draft meta tok refactor

a5eb4a3


small fixes

82e5c44

loubbrad merged commit 12a1812 into EleutherAI:main

1 check passed

loubbrad deleted the meta_toks branch

August 10, 2023 22:41

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment