-
Notifications
You must be signed in to change notification settings - Fork 983
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
'attention.bias' and 'attention.masked_bias' not in hf_layer.state_dict()
when converting gpt-neox model to huggingface
#1013
Comments
Use
You can find missing 'attention.bias', 'attention.masked_bias' before |
Thanks so much @shuheikurita, I just made a PR to update the transformers version |
Describe the bug
A clear and concise description of what the bug is.
I encounter the following error when I am converting GPTNeoX models to Huggingface using the
tools/convert_module_to_hf.py
script.Expected behavior
Successful conversion.
Proposed solution
If you comment out lines 230 and 231, the script will run through. From an eyeballing of the results, it doesn't seem like language modelling performance seriously degraded. Could this be some code that was supposed to be taken out?
Additional context
This is for a model trained with the config
configs/pythia/410m.yml
The text was updated successfully, but these errors were encountered: