Parameter count on GPT-2 medium

[Heimine]

According to the paper, the lora parameter count for GPT-2 medium is 0.35M, but since the hidden dimension is 1024, and the model have 24 layers, with a rank=4, the correct parameter count should be 1024 * 2[for B,A] * 4[for rank=4] * 2[for Q,V] * 24[num. of layers] = 393216 = 0.39M.

I also checked the code for the above calculation.

Do I miss something here?

Thank you very much!

[HaiminZhang]

I think your counting is correct. Have you figured it out how 0.35M is counted?