Figured out a bunch of issues, including multichannel synthesis, very importantly also using previous generated sequences as prompts, which was easier than I thought (it was very scary to deal with initially, but just had to be careful about dimensions at every step!) Things are starting to sound okay, especially with smaller hop sizes! Beat detection (and maybe some smarter windowing?) would definitely be a good direction to go towards.

Made a recording as well: https://soundcloud.com/einedzan/cantstop

Update: Generation is totally broken now, prompting is not what I expected, seems like I am getting some weird probabilities returned from the sample function and everything got messed up.

Update 2: seems like I have figured it out, sampling error was caused by temperature equal to zero...

Update 3: generate.py does not actually encode nor use a provided prompt. What have I been thinking?