Investigate the performance (speed and perplexity) of Q4_0 with 2x F16 factors

[ggerganov]

The current Q4_0 uses a single F32 floating-point scaling factor.

An idea was proposed by @ikawrakow to change this to use 2x F16 factors instead of 1x F32: 679e1cb
Initial results indicate that this might be as accurate as Q4_1 and hopefully as fast as current Q4_0.

The goal of this task is to try to implement efficiently this data format (quantization, dequantization and dot product), measure the speed and perplexity and decide if this is viable. Depending on the results, we can think about updating the current Q4_0 data format and potentially dropping support for Q4_1.

SIMD implementation progress

• ARM NEON
• AVX
• WASM

I plan to work on the ARM NEON implementation.
If you want to help with any of the implementations, propose an implementation + results in a PR, summarizing the inference speed and the obtained perplexity of your implementation.

Related

• Investigate alternative approach for Q4 quantization  #397
• More accurate Q4_0 and Q4_1 quantizations #896

[ggerganov]

This approach resulted in the new Q4_2 and Q4_3 which improve the perplexity results and maintain similar inference speeds as the original Q4_0 and Q4_1 approaches.

The remaining bits and pieces to complete this task will be summarized together with other things in a separate issue