the loss of pythia training

[Wangpeiyi9979]

Hi, Is there any information about the loss of Pythia pre-training process like LLaMA?

[haileyschoelkopf]

https://wandb.ai/eleutherai/pythia?workspace=user-schoelkopf

Hi! We have a public wandb board (linked above) for our training runs. if you filter this by runs without "crashed" status and runs that lasted longer than 1 hour, then runs named "v2-MODELSIZE" should be what you want (I can help point out specific runs if needed, it is on my todo list to clean this up.)

[Wangpeiyi9979]

Thanks

[xiaoda99]

Hi, I can only find runs for 160M model in the above wandb link. And loading the many pages is very slow. Could you provide a cleaned versoin of runs for v2 6.9B and 12B (withou dedup)? I'm doing experiments on some new transformer architectures and want to compare my training loss results with Pythia's