How was deduplication done?

[theblackcat102]

Specifically what method or library does the deduplication of the Pile carried out?

I have search previous issues and this repo and see no other mention of the methodology for deduplication.

[haileyschoelkopf]

Hi! More detail will be included in the paper that we hope to upload soon, apologies that this is missing from the repository.

This deduplicated copy of the dataset was created using the code from https://github.com/google-research/deduplicate-text-datasets with MinhashLSH and Jaccard similarity thresholded at 0.87, I believe. Let me know if you have further questions, or if this clarifies things!

[theblackcat102]

Okay, I'm excited to see the paper soon!