k-quants #1684
k-quants #1684
Conversation
I think it is better to have quantization separate from ggml. For now just adding the k-quants there, but it would be better to also factor out the existing ggml quantizations.
CUDA is not ideal - ~50% slower than Q4_0 for single token prediction, about the same in batch mode (perplexity). CPU single token is ~55 ms (on Ryzen 7950X).
It is now ~22.5 ms/token on my GPU, so ~30% slower than Q4_0.
Single token is now 20.5 ms/token (~20% slower than Q4_0). Perplexity is on par with Q4_0.
Performance is the same or perhaps very slightly better than Q4_0 on the CPU. On the GPU, single token prediction is ~10% better than Q4_0, batch mode (perplexity is about the same).
Performance is ~40% lower compared to Q4_K on the CPU. This is to be expected, considering that we are memory bound on the CPU and the 6-bit model is ~44% larger than the 4-bit. On the GPU, single token prediction is ~6% lower than Q4_0, batch mode (perplexity) is even closer (but still slower).
Performance is ~20% lower compared to Q4_K on the CPU. This is to be expected, considering that we are memory bound on the CPU and the 5-bit model is ~22% larger than the 4-bit. On the GPU, single token prediction is about the same as Q4_0 for both, single token and batch prediction.
It is 22% slower than Q4_K, despite the smaller model size. On x86_64, where we are memory bound, the Q3_K model is quite a bit faster than Q4_K.
Token prediction is pretty good - about 15.5 ms on a RTX 4080. Perplexity is about the same as Q4_K.
About the same performance as Q4_K.
Single token prediction is now ~36 ms on M2 Max. The code is much simpler too.
Stranegly enough, for the few prompts I tried with the 7B model the responses looked perfectly reasonable. Only realized something is not quite right when I tried the larger models and started getting nonse back. In any case, Q2_K single token evaluation time on an RTX 4080 in a Ryzen7950X box iusing CUDA and model fully loaded on the GPU are ~15.5 ms for 7B, ~25.4 ms for 13B, and ~55.8 ms for 30B. The max number of layers that fit in VRAM for The 65B is 32. With that, we get ~330 ms per token, which is not that much faster than just running on the CPU (~470 ms per token).
Q3_K is now running at ~18.5 ms / token on CUDA, so the gap to Q4_0 is only 10%. It seems memory acccess pattern is more important for performance than the amount of computation the kernel does.
For perplexity, where we are less memory bound, time per pass drops by ~5%. Barely measurable difference for single token prediction.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
We cannot possibly be expecting rmse < 0.002 for 2- and 3-bit quantization variants.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
Separate Q - comparing ggml k_m to GPTQ:
|
The last time I checked (on Linux) llama.cpp is faster than AutoGPTQ and about the same speed as ExLlama for token generation. On Windows llama.cpp performance is currently worse by a factor of ~2 but that is caused by the much higher kernel launch overhead on Windows. Prompt processing using llama.cpp is currently noticeably slower than ExLlama (on all OSs) but this is not due to the quantization format because currently by default both llama.cpp and ExLlama dequantize the entire weight matrix only once per matrix matrix multiplication. More generally, I've tried rearranging the quantized data to spatially separate the quantized values and the scales (which I think is how the data is stored in GPTQ) but this did not improve performance. I interpreted this as cache locality being more important than memory alignment (or I just did it wrong). |
Yeah I was inferencing using transformers in Colab with T4. That may have caused specific issues as doesn't leverage llamaccp |
That's a bit hard to believe. Do you have data to back this up? If so, is there any way to circumvent these slowdowns? Most people are running Windows after all. |
I'll get to expanding the GPU section of my blog soon and I'll put numbers there.
You can go to settings -> System -> Graphics Settings and enable hardware-accelerated GPU scheduling which somewhat helps but honestly the best solution is to just use Linux. |
|
Also the CUDA rework by slaren will apparently reduce kernel launch overhead: #2230 (comment) |
Just to address this:
I don't see why ggml couldn't take this approach too: #2585 (comment) |
I also did some testing of a similar setup in llama2.scala, to figure out whether being able to use |
|
@ikawrakow Hi, I have a question about the naming convention. Do |
|
Yes |
|
I would like to produce a k-quantized version of some files (the distil-whisper models, but the question applies to any LLM). Is there a reference implementation of the quantizer somewhere? I could look at the |
|
amzing work. |
This graph is gorgeous. Any hope to have the raw numbers to replot it? |
mofosyne
added
Tensor Encoding Scheme
|
So, as a bottom line, is an L quant of a lower bit depth better or worse than an S quant of a higher one? Like Q4 L vs Q5 S. |
What
This PR adds a series of 2-6 bit quantization methods, along with quantization mixes, as proposed in #1240 and #1256. Scalar,
AVX2,ARM_NEON, andCUDAimplementations are provided.Why
This is best explained with the following graph, which shows perplexity on the
wikitextdataset as a function of model size:Note that the x-axis (model size in GiB) is logarithmic. The various circles on the graph show the perplexity of different quantization mixes added by this PR (see details below for explanation). The different colors indicate the LLaMA variant used (7B in black, 13B in red, 30B in blue, 65B in magenta). The solid squares in the corresponding color represent (model size, perplexity) for the original
fp16model. The dashed lines are added for convenience to allow for a better judgement of how closely the quantized models approach thefp16perplexity. As we can see from this graph, generation performance as measured by perplexity is basically a fairly smooth function of quantized model size, and the quantization types added by the PR allow the user to pick the best performing quantized model, given the limits of their compute resources (in terms of being able to fully load the model into memory, but also in terms of inference speed, which tends to depend on the model size). As a specific example, the 2-bit quantization of the 30B model fits on the 16 GB RTX 4080 GPU that I have available, while the others do not, resulting in a large difference in inference performance.Perhaps worth noting is that the 6-bit quantized perplexity is within
0.1%or better from the originalfp16model.Another interesting observation is that the relative quantization error (as measured by perplexity) does not decrease with increasing number of weights in the base model, as one might hypothesize based on the lower quantization error observed at 13B compared to 7B (see, e.g., this table on the main page). The 13B model is indeed somehow better amenable to quantization, but relative quantization error goes back to the 7B level for the 30B and 65B models. This is illustrated with the following graph, which represents an alternative view of the data in the above graph, by showing the relative difference to the
fp16model in percent. Note that now the x-axis, being the ratio of the quantized size to thefp16model size, is linear, while the y-axis (percent error) is logarithmic.How (Details)
In the existing
ggmlquantization types we have "type-0" (Q4_0,Q5_0) and "type-1" (Q4_1,Q5_1). In "type-0", weightsware obtained from quantsqusingw = d * q, wheredis the block scale. In "type-1", weights are given byw = d * q + m, wheremis the block minimum. I use this to describe the quantizations being added by this PR.The following new quantization types are added to
ggml:GGML_TYPE_Q2_K- "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using2.5625bits per weight (bpw)GGML_TYPE_Q3_K- "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using3.4375bpw.GGML_TYPE_Q4_K- "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using4.5bpw.GGML_TYPE_Q5_K- "type-1" 5-bit quantization. Same super-block structure asGGML_TYPE_Q4_Kresulting in5.5bpwGGML_TYPE_Q6_K- "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using6.5625bpwGGML_TYPE_Q8_K- "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existingQ8_0is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.This is exposed via
llama.cppquantization types that define various "quantization mixes" as follows:LLAMA_FTYPE_MOSTLY_Q2_K- usesGGML_TYPE_Q4_Kfor theattention.vwandfeed_forward.w2tensors,GGML_TYPE_Q2_Kfor the other tensors.LLAMA_FTYPE_MOSTLY_Q3_K_S- usesGGML_TYPE_Q3_Kfor all tensorsLLAMA_FTYPE_MOSTLY_Q3_K_M- usesGGML_TYPE_Q4_Kfor theattention.wv,attention.wo, andfeed_forward.w2tensors, elseGGML_TYPE_Q3_KLLAMA_FTYPE_MOSTLY_Q3_K_L- usesGGML_TYPE_Q5_Kfor theattention.wv,attention.wo, andfeed_forward.w2tensors, elseGGML_TYPE_Q3_KLLAMA_FTYPE_MOSTLY_Q4_K_S- usesGGML_TYPE_Q4_Kfor all tensorsLLAMA_FTYPE_MOSTLY_Q4_K_M- usesGGML_TYPE_Q6_Kfor half of theattention.wvandfeed_forward.w2tensors, elseGGML_TYPE_Q4_KLLAMA_FTYPE_MOSTLY_Q5_K_S- usesGGML_TYPE_Q5_Kfor all tensorsLLAMA_FTYPE_MOSTLY_Q5_K_M- usesGGML_TYPE_Q6_Kfor half of theattention.wvandfeed_forward.w2tensors, elseGGML_TYPE_Q5_KLLAMA_FTYPE_MOSTLY_Q6_K- uses 6-bit quantization (GGML_TYPE_Q8_K) for all tensorsNot mentioned explicitly above is the fact that with this PR, all quantization variants use 6-bit quantization for the
output.weighttensor. This lowers the perplexity of, e.g.,Q4_0by about0.03at 7B.The code is quite lengthy, so it is added via separate files
k_quants.h, k_qunats.cinstead of being added toggml.c. I think that it would be better to also factor out all other quantization types fromggml.c, but that is up to @ggerganov to decide.Performance
The following table summarizes the performance results (perplexity, model size, run time for single token prediction). It is basically designed after the corresponding table on the main page).
I realize the above table is not easy to read, so adding a shortened version that shows a subset of the above data: