Skip to content

Latest commit

 

History

History

pretrain

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
124M_10B_log
124M_10B_log
 
 
log_12_12_12_6e-4
log_12_12_12_6e-4
 
 
log_12_12_4_6e-4
log_12_12_4_6e-4
 
 
log_12_12_6_6e-4
log_12_12_6_6e-4
 
 
log_16_12_4_6e-4
log_16_12_4_6e-4
 
 
log_16_12_6_6e-4
log_16_12_6_6e-4
 
 
log_24_12_6_6e-04
log_24_12_6_6e-04
 
 
log_24_16_8_6e-04
log_24_16_8_6e-04
 
 
log_mix_16_layer(10)_attenH(8)_kvH(8)_acoef(0.02)_b(32)_gradacc(2)_step(38146)dropout(0.0)_jitter(0.0)exp(8)_exppt(2)_lr3e-03
log_mix_16_layer(10)_attenH(8)_kvH(8)_acoef(0.02)_b(32)_gradacc(2)_step(38146)dropout(0.0)_jitter(0.0)exp(8)_exppt(2)_lr3e-03
 
 
log_mix_16_layer(12)_attenH(12)_kvH(4)_acoef(0.02)_b(32)_gradacc(1)_step(38146)dropout(0.0)_jitter(0.0)exp(8)_exppt(2)_lr3e-03
log_mix_16_layer(12)_attenH(12)_kvH(4)_acoef(0.02)_b(32)_gradacc(1)_step(38146)dropout(0.0)_jitter(0.0)exp(8)_exppt(2)_lr3e-03
 
 
README.md
README.md
 
 
fineweb.py
fineweb.py
 
 
hellaswag.py
hellaswag.py
 
 
mixtral_result.ipynb
mixtral_result.ipynb
 
 
play_with_llama.ipynb
play_with_llama.ipynb
 
 
train_deepseek.py
train_deepseek.py
 
 
train_gpt2.py
train_gpt2.py
 
 
train_llama.py
train_llama.py
 
 
train_llama_fsdp.py
train_llama_fsdp.py
 
 
train_mixtral.py
train_mixtral.py
 
 

README.md

pretrain

Inspired by Andrew's https://github.com/karpathy/build-nanogpt

  • fineweb.py: Constructs fineweb training data.
  • hellaswag.py: Evaluates using the Hellaswag dataset.
  • train_gpt2.py: Trains a nano GPT-2 model from scratch.
  • train_llama.py: Trains a nano LLaMA-3 model from scratch.
  • train_mixtral.py: Trains a nano Mixtral 8x model from scratch.
  • train_deepseek.py: Trains a nano Deepseek Moe model from scratch.
  • play_with_llama.ipynb: Compares the training results.
  • mixtral_result.ipynb: Compares the MoE training results.