Wrong rotary embedding result between transformers structure and Megatron structure

[GGGGGGXY]

Describe the bug

In FP16 inference:
In transformers

cos_cached and sin_cached is a FloatTensor very beginning. And they will not convert to HalfTensor.

But in Megatron

cos and sin recompute into HalfTensor. And it became a bias between transformers and Megatron.
Sometimes we get total different logits.

[StellaAthena]

@GGGGGGXY Why do you think that their way of doing this is more desirable than ours? Ours is the first implementation to support mixed precision training and so arguably is "canonical"... does it introduce NaNs in fp16 or something?

[GGGGGGXY]

@GGGGGGXY Why do you think that their way of doing this is more desirable than ours? Ours is the first implementation to support mixed precision training and so arguably is "canonical"... does it introduce NaNs in fp16 or something?

Sorry, I think the way in EleutherAI/gpt-neox if more desirable. I just want to show the different between transformers and Megatron. Most of the users using transformers on downstream task. And this difference produce completely different result in some way。