Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MidiDataset optimizations #68

Merged
merged 11 commits into from
Nov 22, 2023
Merged

MidiDataset optimizations #68

merged 11 commits into from
Nov 22, 2023

Commits on Nov 9, 2023

  1. MidiDataset can initialize with an iterator and only expand when nece… 

    …ssary.
    @honglu2875
    honglu2875 committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    0230715 View commit details
    Browse the repository at this point in the history

  2. reduce some memory overhead (we are starting to have >100k MidiDict a… 

    …nd may get more in the future)
    @honglu2875
    honglu2875 committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c1d170d View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. classmethod+property is better...

    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    deec979 View commit details
    Browse the repository at this point in the history

  2. remove functools import

    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    6e58f09 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'EleutherAI:main' into dev

    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    00cef03 View commit details
    Browse the repository at this point in the history

  4. use separate workers to build dataset instead of process pool

    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    7073a1a View commit details
    Browse the repository at this point in the history

  5. merge

    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    69922b4 View commit details
    Browse the repository at this point in the history

  6. add jsonl.zst support; unit test; fix bug

    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    8bbc14b View commit details
    Browse the repository at this point in the history

  7. receive context length via commandline. It's more convenient than dig… 

    …ging into the config file every time.
    @honglu2875
    honglu2875 committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    95f492d View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2023

  1. fix a minor output format mismatch when grad_checkpoint is true

    @honglu2875
    honglu2875 committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    93301fa View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2023

  1. format and small changes

    @loubbrad
    loubbrad committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    607c438 View commit details
    Browse the repository at this point in the history