How to deal with model without tokenizer.json?

[sanmeow]

Hello! Thanks for the community for this great work!

I came to a problem that some model can't transfer to GGML.

When the GPT-NeoX model creator made tokenizer with sentencepiece,
examples/gpt-neox/convert-h5-to-ggml.py
it will say I don't have tokenizer.json

for example:
https://huggingface.co/rinna/japanese-gpt-neox-3.6b-instruction-sft

I got it half solved by use
https://github.com/togethercomputer/redpajama.cpp

and works! So there is some way to solve the sentencepiece problem.(Study a bit that tokenizer was base on T5 not GPTNeoX)

So is there anyway to modify it to this GGML? There are other model facing similar problem.

Thanks!