New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve loading/saving for datasets #11

Merged

loubbrad merged 3 commits into EleutherAI:main from loubbrad:add_dataset_streaming

Jul 21, 2023

Contributor

loubbrad commented Jul 21, 2023

Made the following changes:

Changed the file formats for MidiDataset and TokenizedDataset to jsonl
Added the functionality for MidiDataset objects to be built directly to a file
Refactored TokenizedDataset so that it dynamically loads sequences using mmap
Added tests for MidiDataset and TokenizedDataset

NOTE: Training framework has not been updated to account for these changes.

loubbrad added 3 commits

July 13, 2023 17:49


updated requirements

e33febd


added jsonl save file format

ea910a9


added mmap support for datasets

ddc4faa

loubbrad merged commit fd8714f into EleutherAI:main

1 check passed

loubbrad deleted the add_dataset_streaming branch

July 21, 2023 13:19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment