Commits

History for

Commits on Apr 28, 2022

fix typo in bash call
alistairewj
committedApr 28, 2022

Commits on Mar 11, 2022

Update README to reflect version 1.0 finalization
carlini
committedMar 11, 2022

Commits on Mar 8, 2022

Spellcheck README
carlini
committedMar 8, 2022
Add dedup single file script
carlini
committedMar 8, 2022
Update README for wiki40b, and fix minor code bugs
carlini
committedMar 8, 2022

Commits on Mar 7, 2022

Simplify process
carlini
committedMar 7, 2022

Commits on Mar 3, 2022

Update README to reflect change in datatype sizes
carlini
committedMar 3, 2022
Support suffix arrays that are fewer than 8 bytes per token if we don't need that many
carlini
committedMar 3, 2022
Move version history in README
carlini
committedMar 3, 2022

Commits on Feb 26, 2022

Fill in some TODOs
carlini
committedFeb 26, 2022

Commits on Feb 24, 2022

README typo fixes
carlini
committedFeb 24, 2022
README now is much more readable
carlini
committedFeb 24, 2022
Add example LM1b dedup writeout script
carlini
committedFeb 24, 2022

Commits on Feb 23, 2022

Fixing across-similar script
carlini
committedFeb 23, 2022
Building suffix array works correctly
carlini
committedFeb 23, 2022
Start a v1 rewrite; make individual functions for each command; new argument parsing.
carlini
committedFeb 23, 2022

Commits on Jul 27, 2021

Add CSVs to replicate approximate deduped datasets
daphnei
committedJul 27, 2021

Commits on Jul 19, 2021

Revert header formatting change
daphnei
committedJul 19, 2021
Added a "why deduplicate" section.
daphnei
committedJul 19, 2021

Commits on Jul 18, 2021

Update README.md
daphnei
committedJul 18, 2021
Updated README, fixing some typos.
daphnei
committedJul 18, 2021

Commits on Jul 16, 2021

Initial commit
carlini
committedJul 16, 2021

Loading