Gibberish text generation after converting to Huggingface.

[kanwatchara-k]

Hi,
I am having trouble converting my checkpoints to Huggingface format. The model works fine when using Deepspeed + Megatron (example).

generate_samples_from_prompt(neox_args, model, ['สวัสดีครับ' for i in range(1)],temperature=0.9, top_k=40)
>>  {'context': 'สวัสดีครับ', ## means 'hello'
  'text': 'เพื่อนๆครับ มีเพื่อนๆ คนไหนที่ทำงานแล้ว หรือกำลังทำงานแล้ว แล้วได้ลาออกจากงานไปแล้วแต่ยังหางานอยู่บ้างครับ พอดีอยากทราบวิธีหางานหรือแนะนำบริษัท ที่ให้เงินเดือนดี และน่าเชื่อถือหน่อยครับ'}

However, it becomes gibberish when converted into Huggingface format (example).

pipe = TextGenerationPipeline(model, tok, device=0)
pipe("สวัสดีครับ",max_new_tokens=50, top_k=40,do_sample=True, temperature=0.9)
>>  [{'generated_text': 'สวัสดีครับ ค.. ดี. แรง 1-". และทำ<|endoftext|>'}]

I have tried multiple conversion scripts so far (e.g., this and this) without success.

All the related files (weights, config, and tokenizer) are in my google drive.

Any help is greatly appreciated!

[StellaAthena]

@haileyschoelkopf

[haileyschoelkopf]

Hey! Looking into this to see if it's the case on my end!

[haileyschoelkopf]

Oh, @kanwatchara-k would you be willing to send what the exact command you ran to run https://github.com/EleutherAI/gpt-neox/pull/701/files#diff-fff7e2d700e82c3e6027c575c1cd96830ba839ff44fa6b82abf2cb21b029d55c was? There’s a chance that your discrepancy is due to my current script not accepting multiple config files like you used for training.

[kanwatchara-k]

@haileyschoelkopf Of course. Though I did make some changes to the code (the modified version is here). To be specific, I hard-coded the vocab file path and the tokenizer type. I also combined the two config files in the code (with the paths also hard-coded).

With the modified code, I just ran the command
python tools/convert_to_hf.py --input_dir checkpoints/global_step300000/ --output_dir ./

Thanks

[haileyschoelkopf]

Thank you!! I’ll try this to convert your checkpoint as soon as I can, hopefully later today or early tomorrow!

[haileyschoelkopf]

Still working on finding the possible issue here--I'll keep you posted!

[haileyschoelkopf]

@kanwatchara-k so sorry for the delay on my end. What fixed the issue on my end, where I had a model that also had this problem, was:

1. pip install --upgrade transformers to the latest version, to include this PR: Add a use_parallel_residual argument to control the residual computing way huggingface/transformers#18695
2. changing config.json in the HF model to have use_parallel_residual: false

The issue here was that your and my models which wouldn't convert properly use a layer setup different from GPT-NeoX-20b, which is controlled by gpt_j_residual: true in the 20b config file. setting this value in the HF config allows the model to run in the same way it was trained. Hope this helps and works for you! I'll update my conversion script to take this into account.

[kanwatchara-k]

@haileyschoelkopf Thank you so much! It works properly now!