-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Difference in answers predictions using the same model in FarmReader and TransformersReader #248
Comments
Hey thanks for reporting the issue! So it is possible that FARMReader and TransformerReader give different predictions. But not to the degree that we saw in our attempt to replicate your issue. We are working on it and will let you know when we find out more |
Hey, thank you for the support. I have also committed my notebook if it can help to replicate my issue. You can find it here: https://github.com/antoniolanza1996/Haystack_TMP/blob/master/Haystack.ipynb |
Hey @antoniolanza1996 so we dug deeper and actually found that there quite a few differences between FARM and transformer Readers that would contribute to this kind of divergence. Bottom line is that we think both Readers are currently working as they are designed, albeit differently from each other. What I suspect causes the biggest difference between the two models are the following two points:
There are also some more minor differences:
Not sure what level of familiarity you have with QA systems but if you want a primer on sliding windows, logit calculations and aggreation, I'd point you to this! (https://medium.com/deepset-ai/modern-question-answering-systems-explained-4d0913744097) I hope this helps and if you'd like me to clarify anything that I mentioned above, just let me know. I'd be more than happy to try and explain. |
@brandenchan, now it makes sense. Thank you for the support. I am going to close the issue. |
On performance test found FARMReader is better (in speed) compared to TransformerReader. |
Hey @su2twtbridge, |
Question
Why does the behaviour change if I use a model (e.g. deepset/roberta-base-squad2) wrapped in TransformersReader instead of FARMReader?
Additional context
If I run this code:
I expected that
reader1
andreader2
would give the same predictions but it didn't happen. Is there a difference between FARMReader and TransformersReader that I am not taking into account?The text was updated successfully, but these errors were encountered: