convert fine tuned model to ggml?

[silvacarl2]

we are trying to convert a fine tuned gpt-j model to ggml.

but it always comes back with this error:

python convert-h5-to-ggml.py /home/silvacarl/Desktop/models/gpt-cmd 1

[2023-06-23 15:54:20,338] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/silvacarl/Desktop/ggml/examples/gpt-j/convert-h5-to-ggml.py:113 in │
│ │
│ 110 │ fout.write(text) │
│ 111 │
│ 112 for key in encoder_added: │
│ ❱ 113 │ text = bytearray([byte_decoder[c] for c in key]) │
│ 114 │ fout.write(struct.pack("i", len(text))) │
│ 115 │ fout.write(text) │
│ 116 │
│ │
│ /home/silvacarl/Desktop/ggml/examples/gpt-j/convert-h5-to-ggml.py:113 in │
│ │
│ 110 │ fout.write(text) │
│ 111 │
│ 112 for key in encoder_added: │
│ ❱ 113 │ text = bytearray([byte_decoder[c] for c in key]) │
│ 114 │ fout.write(struct.pack("i", len(text))) │
│ 115 │ fout.write(text) │
│ 116 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: ' '

any ideas what we are missing?

[LLukas22]

This seams to be an error with the tokenizer you are using, i encountered similar issues and decided to refactor and combine many of the conversion scripts for our rustformers\llm project which also uses the ggml file format. If you want you could give our python wrapper a try which implements a Convert function that produces a fp16 ggml model which you can use with the original example/quantize function of this repo.

[silvacarl2]

so something very interesting happened. just for fun, i copied over all of our fine tuend files but deleted the .bin. then i copied over the GGML .bin to the same directory. For GGML, copying over all of the files from gpt-cmd and removing the .bin and just having the converted ggml bin worked to load it.

but, it acted strange:

test prompt:
Prompt = What is metformin used for?
Answer: What is metformin used for? What if it's not good enough. I want something else to take my mind off of stress and get a better job with the doctor on

Prompt = my birth date is august 8, 1942.\n
Fine tuned Answer should be: 8/8/1942
Fine tuned Answer came back as: ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ熷

any ideas? 8-)

[LLukas22]

The included GGML tokenizer is very lossy, in rustformers\llm we support HugginfaceTokenizers which aren't supported in the ggml implementation. Maybe give those a try, if that doesn't work maybe review the converter script in the GGML repo and check what gives you that error.

[silvacarl2]

cool will do!