I view this as not ready for merging based on the current checklist

I started trying this on the Pile, and it seems like fully reconstructing the .bin file from the shards will not be a whole lot faster than just running the tokenization oneself--set to take ~40 hours to reconstruct (maybe I'm doing something stupid and inefficient, but as far as I know to reshard the memmaped numpy array it requires copying all of the 60 10GB memmapped shards into the full-size memmapped .bin array.

(For reference, C4 took 18 hours to tokenize with NeoX)

Tokenization can be non-deterministic (don't ask) so I have a preference for desharding even if it is roughly as time consuming as tokenizing from scratch.

Oh god, is it really nondeterministic on the exact same input string?

I’ll let this run then and we can merge once resharded files are checked for equivalence + once everything’s uploaded.

Running diff against the sharded + unsharded .bin file against the original, the files appear to be the exact same now.

@StellaAthena could you give this a quick review when you get the chance?

Both datasets now being uploaded and the scripts should be ready for merging.

Documentation will come in a separate PR that I'm currently working on.