-
Notifications
You must be signed in to change notification settings - Fork 26.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bad Results with Albert #2609
Comments
Hi. Many people are reporting unstable results or just unexpected results. You can search for issues in this library, and even in other ones (e.g. deepset-ai/FARM#202 (comment)). It seems that ALBERT is very sensitive to hyperparameters and even then... For now there seems to be no solution. It is probably best to stick to another model. I'd recommend RoBERTa but it depends on your use-case. |
@BramVanroy I tried with roberta-base as well, the token level similarity is coming out very bad. |
Can you share a repo to your full code? |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
❓ Questions & Help
Trying to understand why is the cosine similarity between tokens with Albert way bad in comparison to DistilBert.
Any inferences on the same would be helpful.
Thanks in advance.
Embeddings constructed for a token by summing the last 4 encoded layers.
Distance metric: cosine
Results with DistilBert
![Screenshot 2020-01-22 at 4 05 06 PM](https://user-images.githubusercontent.com/25073753/72887172-2e6feb80-3d31-11ea-876b-0ba8eac22234.png)
![Screenshot 2020-01-22 at 4 12 27 PM](https://user-images.githubusercontent.com/25073753/72887650-25334e80-3d32-11ea-8255-26109f604c84.png)
Results with Albert
![Screenshot 2020-01-22 at 4 04 56 PM](https://user-images.githubusercontent.com/25073753/72887227-447dac00-3d31-11ea-8797-9873c8439879.png)
![Screenshot 2020-01-22 at 4 10 16 PM](https://user-images.githubusercontent.com/25073753/72887665-2bc1c600-3d32-11ea-95aa-83b3192e9d49.png)
The text was updated successfully, but these errors were encountered: