-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pytorch加载simbert问题 #12
Comments
torch版本的权重的 embedding的layernorm部分,size是d.state_dict()['embeddings.LayerNorm.weight'].size() sorry, 上面2个图贴错了,实际上一个是tiny 312,一个是small 384的,出现的情况是一样的。 |
small版和tiny版的embedding层像albert一样用了低秩分解,base版本正常。应该是这个转换脚本可以考虑到低秩分解。 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
hello 苏神 请问pytorch加载simbert问题。
直接用transformers的包加载,维度会不一致
![image](https://user-images.githubusercontent.com/31469418/114361725-0de7b780-9ba9-11eb-8908-492cdeec04e2.png)
看起来tf的权重是embedding部分多了一个linear层,torch版本没有这一层[bert/encoder/embedding_hidden_mapping_in/]
![image](https://user-images.githubusercontent.com/31469418/114362058-7c2c7a00-9ba9-11eb-8bfa-5ad8867eb6d6.png)
为什么开源权重多了这部分,是不是,torch版本的model的embedding部分加上这个linear,然后加载即可。
The text was updated successfully, but these errors were encountered: