New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor max_seq_len from Tokenizer into TokenizedDataset #15

Merged

loubbrad merged 1 commit into EleutherAI:main from loubbrad:refactor_max_seq_len

Aug 7, 2023

Contributor

loubbrad commented Aug 7, 2023

In this PR we make the following changes:

Refactor so that tokenizers don't require a max_seq_len argument
Change MidiDict.from_midi() so that it accepts a path instead of a mido.MidiFile object
Add doc strings and make small corrections
Update test_training.py & fix small bug in training.pretrain()
Add asserts and logging statements


refactored tokenizer max_seq_len

6f23177

loubbrad merged commit f8fac32 into EleutherAI:main

1 check passed

loubbrad deleted the refactor_max_seq_len branch

August 7, 2023 17:37

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment