position embedding is not saved

[eghbalhosseini]

I am trying to recover position embedding for a trained model (rotary embedding) from the checkpoint folder, however its not in any of the pt files. how can I save position embedding?

[StellaAthena]

Rotary positional embeddings are calculated in advance. They do not change during training and therefore there's no need to save them. To obtain the desired positional embeddings, you can simply calculate them using the formula.

[eghbalhosseini]

thanks,
is there a tweak I can do in the code to save these during a checkpoint? or recreate them afterward. also is absolute positional embedding implemented in the pipeline ?

[StellaAthena]

Yes absolute positional embeddings is an option. You can check out configs/neox_arguments.md details on all currently supported configurations and related information.