Encoding the dataset

[ketan0]

Did you generate one giant TFRecord for the Lakh MIDI dataset? Or did you process the data in shards? If the latter, how exactly does one go about sharding the data with the pipelines that you have in place? I'm finding that generating one giant TFRecord using convert_dir_to_note_sequences is too large to load into memory when doing scripts/generate_song_data_beam.py.

[BenokanDeepBlue]

Hello Ketan,

I had a similar issue and I've managed to solve it by removing ReShuffle from the pipeline to avoid parallelism at the cost of reducing the speed of the process but at least it doesn't try to load entire data to the memory and runs smoothly (but very slowly)