Merge branch 'master' of github.com:EleutherAI/The-Pile

glee2429 · Dec 21, 2020 · 05ab1d3 · 05ab1d3
2 parents 4a1d42b + cefc0ec
commit 05ab1d3
Showing 1 changed file with 12 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -46,7 +46,18 @@ Install:
 pip install -e .
 ```
 
-To download all data:
+### To replicate pile
+
+```
+python the_pile/pile.py --interleave_output 30 --using pile_reprod
+```
+
+Use the pass 2 script [here](https://github.com/EleutherAI/The-Pile/tree/master/processing_scripts) to complete shuffling.
+
+
+### Other
+
+To force download all data:
 ```
 python the_pile/pile.py --force_download
 ```