Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MidiDataset optimizations #68

Merged
merged 11 commits into from
Nov 22, 2023
Merged

MidiDataset optimizations #68

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0230715
MidiDataset can initialize with an iterator and only expand when nece…
honglu2875 Nov 9, 2023
c1d170d
reduce some memory overhead (we are starting to have >100k MidiDict a…
honglu2875 Nov 9, 2023
deec979
classmethod+property is better...
honglu2875 Nov 10, 2023
6e58f09
remove functools import
honglu2875 Nov 10, 2023
00cef03
Merge branch 'EleutherAI:main' into dev
honglu2875 Nov 10, 2023
7073a1a
use separate workers to build dataset instead of process pool
honglu2875 Nov 10, 2023
69922b4
merge
honglu2875 Nov 10, 2023
8bbc14b
add jsonl.zst support; unit test; fix bug
honglu2875 Nov 10, 2023
95f492d
receive context length via commandline. It's more convenient than dig…
honglu2875 Nov 10, 2023
93301fa
fix a minor output format mismatch when grad_checkpoint is true
honglu2875 Nov 12, 2023
607c438
format and small changes
loubbrad Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix a minor output format mismatch when grad_checkpoint is true
  • Loading branch information
@honglu2875
honglu2875 committed Nov 12, 2023
commit 93301fafde4970832c5d6fff0af88181d1a3ade7
2 changes: 1 addition & 1 deletion aria/model/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ def custom_forward(*args):

return custom_forward

hidden_states = torch.utils.checkpoint.checkpoint(
hidden_states, _ = torch.utils.checkpoint.checkpoint(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this end up fixing something?

create_custom_forward(layer),
hidden_states,
preserve_rng_state=True,
Expand Down