MPT quantize does not include quantization version

[philpax]

Hi there!

The concept of quantization versions was recently introduced by this commit, which encodes the version in the model's ftype: effcfa6

However, it looks like this change didn't make it over to the MPT quantizer:

ggml/examples/mpt/quantize.cpp

Line 89 in 2a75bd4

fout.write((char *)&ftype, sizeof(hparams.ftype));

This means that the MPT models recently uploaded by @TheBloke at https://huggingface.co/TheBloke/MPT-7B-GGML unfortunately do not include the quantization version. The effect of this is mitigated by the MPT example not currently checking the version.

We're checking the version in llm, which is why this has come up. We may add an option to disable the check or override the perceived version, but I figured that it was worth reporting upstream before it proliferates.

[TheBloke]

Thanks for the heads up. I'm happy to re do them with the fixed quantization code.

[TheBloke]

Going to bed now but as soon as the fix is in I can re do them tomorrow

[lukasmoellerch]

Yes, this was merged while I was working on the mpt model integration - I'll contribute a fix

[philpax]

Also worth noting is the code that loads the model needs to account for this (e.g. it needs to modulo the ftype by GGML_QNT_VERSION_FACTOR)

[marella]

@lukasmoellerch I sent #165 to remove global variables. Can you also please include those changes in your fix.

Also max_seq_len is read from model file but doesn't seem to be used anywhere. Is that expected or is it supposed to be used as n_ctx?

[ggerganov]

Added quantization version support to MPT and Replit models

[TheBloke]

Thanks very much for the fix. I've updated my three repos:
https://huggingface.co/TheBloke/MPT-7B-GGML
https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML
https://huggingface.co/TheBloke/MPT-7B-Storywriter-GGML