Adds a script to convert HF checkpoints to NeoX 2.0 with mp and pp sh…

…arding (EleutherAI#907) * added HF to NeoX 2.0 conversion script with mp and pp sharding * (1) added missing curly brace to pythial/1-4B config; (2) fixed a bug related to a hardcoded value withing the conversion script (3) fixed possible bugs in the conversion script wrt the mp sharding convention --------- Co-authored-by: Quentin Anthony <[email protected]>
Jarvis-LLM · May 19, 2023 · b70d004 · b70d004
1 parent 03f4f77
commit b70d004
Show file tree

Hide file tree

Showing 2 changed files with 507 additions and 1 deletion.
diff --git a/configs/pythia/1-4B.yml b/configs/pythia/1-4B.yml
@@ -81,6 +81,5 @@
  "log_interval": 10,
  "steps_per_print": 10,
  "wall_clock_breakdown": true,
-
  "tokenizer_type": "HFTokenizer"
  }