5.5x more CUDA performance with 5 minutes of work

[JohannesGaessler]

So I've spent the entire day trying to get better performance for my Pascal cards and nothing seemed to work. Initially I assumed that the problem was that the per-byte dot product has bad performance on Pascal but I eventually found out that the problem has to do with reading in data. I thought that the problem was memory alignment so I tried reordering the data but that made no difference either. As it turns out the problem was that I didn't add the __restrict__ qualifier to the function signatures to tell the compiler that the data represented by the pointers does not overlap. It does not make a difference for my RTX 3090 but for my P40 there is a dramatic performance increase just from adding that qualifier:

GPU	Model	Test	t/s master	t/s PR	Speedup
P40	7b q4_0	tg128	16.51	46.41	2.81
P40	13b q4_0	tg128	8.97	25.48	2.84
P40	33b q4_0	tg128	3.64	10.82	2.97
P40	7b q4_1	tg128	14.34	46.04	3.21
P40	7b q5_0	tg128	11.17	35.37	3.17
P40	7b q5_1	tg128	10.71	41.44	3.87
P40	7b q8_0	tg128	8.45	28.47	3.37
P40	7b f16	tg128	8.58	19.01	2.22
P40	7b q2_K	tg128	6.27	28.88	4.61
P40	7b q3_K_S	tg128	8.70	25.34	2.91
P40	7b q4_K_S	tg128	4.59	25.26	5.50
P40	7b q5_K_S	tg128	7.61	22.99	3.02
P40	7b q6_K	tg128	4.42	22.27	5.04

With the exception of Maxwell cards it should now be possible to get good quantized dot product performance for all NVIDIA GPUs supported by CUDA 12. My next goal will be to integrate this into my matrix multiplication PR #2043 .