Why GGML_F16_STEP is 32?

[xshen053]

Seems GGML_F16_STEP is an arbitrary value?
I feel it can be16, 64 as well

[AngryLoki]

F16_STEP on AVX CPUs is set to 32 because these CPUs basically can't perform half-float math operations (other than convert to F32) due to lack of instructions. As a result, unrolled loops of 32 half-floats unroll into 32 floats = 1024 bits, which takes 4 ymm registers. And because most of operations have 2 sources, 4 destination + 8 source ymm registers fit total of 16 ymm registers. It may be not optimal for unary or binary in-place operations, but it is decent approximation.

For other platforms GGML_F16_STEP is not always 32, as you can see in the code.

[xshen053]

Thanks for the reply, actually on arm it is stil 32, I am testing on arm

[ggerganov]

The value is whatever was optimal during testing on my machine. Probably there is a proper way to calculate the optimum value (as @AngryLoki explained), but in this case it was just trial and error