Pythia v.s. GPT-neox

[chuanli11]

Hi, thanks for the awesome work.

I notice there are Pythia models and GPT-neox models on Eleuther's HF hub, both seem to be very recent. Wonder what are the the main differences between them, and if there is any preference on one of them.

Thank you.

[haileyschoelkopf]

Hi,

The Pythia models and GPT-NeoX-20b model were all trained in the same library, and use a very similar architecture.

The Pythia models are primarily intended for research as a consistent set of models. If you’re only looking for performance, then the GPT-J-6B and GPT-NeoX-20b models have been trained on more data and will outperform Pythia models, though the Pythia models should have standard performance for models of their size!

[chuanli11]

Thank you for the clarification!