Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MidiDataset optimizations #68

Merged
merged 11 commits into from
Nov 22, 2023
Merged

MidiDataset optimizations #68

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0230715
MidiDataset can initialize with an iterator and only expand when nece…
honglu2875 Nov 9, 2023
c1d170d
reduce some memory overhead (we are starting to have >100k MidiDict a…
honglu2875 Nov 9, 2023
deec979
classmethod+property is better...
honglu2875 Nov 10, 2023
6e58f09
remove functools import
honglu2875 Nov 10, 2023
00cef03
Merge branch 'EleutherAI:main' into dev
honglu2875 Nov 10, 2023
7073a1a
use separate workers to build dataset instead of process pool
honglu2875 Nov 10, 2023
69922b4
merge
honglu2875 Nov 10, 2023
8bbc14b
add jsonl.zst support; unit test; fix bug
honglu2875 Nov 10, 2023
95f492d
receive context length via commandline. It's more convenient than dig…
honglu2875 Nov 10, 2023
93301fa
fix a minor output format mismatch when grad_checkpoint is true
honglu2875 Nov 12, 2023
607c438
format and small changes
loubbrad Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reduce some memory overhead (we are starting to have >100k MidiDict a… 
…nd may get more in the future)
  • Loading branch information
@honglu2875
honglu2875 committed Nov 9, 2023
commit c1d170daf4a1b1dbb5fee80e60ff718814fb0dd2
6 changes: 4 additions & 2 deletions aria/data/midi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""Utils for data/MIDI processing."""

import functools
import hashlib
import json
import re
Expand Down Expand Up @@ -114,8 +114,10 @@ def __init__(
}
]

@functools.cached_property
def program_to_instrument(self):
# This combines the individual dictionaries into one
self.program_to_instrument = (
return (
{i: "piano" for i in range(0, 7 + 1)}
| {i: "chromatic" for i in range(8, 15 + 1)}
| {i: "organ" for i in range(16, 23 + 1)}
Expand Down