Bad Results with Albert

[chikubee]

❓ Questions & Help

Trying to understand why is the cosine similarity between tokens with Albert way bad in comparison to DistilBert.
Any inferences on the same would be helpful.
Thanks in advance.

Embeddings constructed for a token by summing the last 4 encoded layers.
Distance metric: cosine

Results with DistilBert

Results with Albert

[BramVanroy]

Hi. Many people are reporting unstable results or just unexpected results. You can search for issues in this library, and even in other ones (e.g. deepset-ai/FARM#202 (comment)). It seems that ALBERT is very sensitive to hyperparameters and even then... For now there seems to be no solution. It is probably best to stick to another model. I'd recommend RoBERTa but it depends on your use-case.

[chikubee]

@BramVanroy I tried with roberta-base as well, the token level similarity is coming out very bad.
Smoking is getting matched with software.

[BramVanroy]

Can you share a repo to your full code?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.