Pythia 12b flash config

[jvendrow]

The pythia 12b config has:

"attention-config": [[["flash"], 40]],

However, in the gpt-neox repo the 40 is replaced by 36, and in the file:

https://huggingface.co/EleutherAI/neox-ckpt-pythia-12b-v1/blob/main/12B.yml

This value of 36. Is this a mistake? Also, the attention config line at:

pythia/models/12B/pythia-12b.yml

Line 19 in 1ff5ade

"attention-config": [[["flash"], 40]]

seems to be missing a comma?

Edit: The num-layers value also seems off, 36 v. 40.