Do pythia untied embedding and unembedding matrics?

[Life-0-1]

In the Pythia paper, it is stated that embedding and unembedding matrics are untied.

But I find that config files of all kinds of pythia models are with the no_weight_tying remaining a default value of False, which results a tying between embed and unembed layers.

Is this a bug? What setting did you actually use while training pythia?

[haileyschoelkopf]

Which config files are you looking at? the ones in the Pythia repo and the ones here https://github.com/EleutherAI/gpt-neox/blob/main/configs/pythia/1-4B.yml use the value “no-weight-tying”: true as expected.

[Life-0-1]

Which config files are you looking at? the ones in the Pythia repo and the ones here https://github.com/EleutherAI/gpt-neox/blob/main/configs/pythia/1-4B.yml use the value “no-weight-tying”: true as expected.

Sorry, I Made a mistake. I searched config files and found no no_weight_tying() , but there is no-weight-tying.

"-" and "_" are different from readme and config files.

[StellaAthena]

@Life-0-1 what readme are you talking about? The one in gpt-neox/configs uses -

[Life-0-1]

@Life-0-1 what readme are you talking about? The one in gpt-neox/configs uses -

gpt-neox/configs/neox_arguments.md this one.

[StellaAthena]

Opened an issue to fix the inconsistency #920