About gpt-neox-20B model hyperparameter

[peiyingxin]

A confuse about 20B model hyperparameter with hidden-size=6144, num-attention-heads=64, num-layers=44, but LLaMA or GPT model have different model hyperparameter, LLaMA-65B hidden-size=8192, num-attention-heads=64, num-layers=80, it seems that LLaMA deeper and gpt-neox-20B wider? which model hyperparameter is better?
Thank you~

[StellaAthena]

There is no compelling evidence for what precise width-to-depth ratio is best.