Cerberas 2.7B yields garbage tokens after quantizing to 4bits

[lxe]

I'm getting garbage-looking tokens (&>,32>G$F7"=%0.173)@++*$16*:=!32%;:2@$5")0!!DGDA(:F*G$!")=9&9D69C9H-4.>&<A+1>.;6D7^C) after quantizing an f16 Cerberas model like this:

../../build/bin/gpt-2-quantize ./cerebras-gpt2.7b-alpaca-sp/ggml-model-f16.bin ./cerebras-gpt2.7b-alpaca-sp/ggml-model-int4.bin 2

Example:

(llama-lora) lxe@lxepc:~/ggml/examples/gpt-2$ ../../build/bin/gpt-2 -m cerebras-gpt2.7b-alpaca-sp/ggml-model-int4.bin -p "Human: How old is the Sun?\nAssistant:"
main: seed = 1680242724
gpt2_model_load: loading model from 'cerebras-gpt2.7b-alpaca-sp/ggml-model-int4.bin'
gpt2_model_load: n_vocab = 50257
gpt2_model_load: n_ctx   = 2048
gpt2_model_load: n_embd  = 2560
gpt2_model_load: n_head  = 32
gpt2_model_load: n_layer = 32
gpt2_model_load: f16     = 2
gpt2_model_load: ggml ctx size = 2957.55 MB
gpt2_model_load: memory size =  1280.00 MB, n_mem = 65536
gpt2_model_load: model size  =  1677.45 MB
main: prompt: 'Human: How old is the Sun?\nAssistant:'
main: number of tokens in prompt = 15, first 8 tokens: 20490 25 1374 1468 318 262 3825 30

Human: How old is the Sun?\nAssistant:&>,32>G$F7"=%0.173)@++*$16*:=!32%;:2@$5")0!!DGDA(:F*G$!")=9&9D69C9H-4.>&<A+1>.;6D7^C

The f16 model loads and works fine.

[pikalover6]

It’s a known bug, ggerganov tweed about it.

[lxe]

Does this happen only for GPT2-based models?

[pikalover6]

I think it is just an issue with Cerebras but I am not sure.

[elephantpanda]

I am using Cerebras too. It would be great if this could be fixed. The Cerebras are excellent models.

[LostRuins]

I think I have figured out this issue, the f16 to f32 tables were not properly initialized in the quantize examples.

This can be fixed by adding this code to main() in quanitize.cpp

    {
        struct ggml_init_params params = { 0, NULL };
        struct ggml_context * ctx = ggml_init(params);
        ggml_free(ctx);
    }

Please refer to my PR #77