Proper Probabilities for Answers

[PiffPaffM]

At the moment, we have pseudo probabilities in Haystack. Proper probabilities would be helpful to decide if it makes sense to show an answer to the user. If an answer has nothing to do with the question, the users are confused or even doubt that the technology is working.

[Timoeller]

Totally, we need better and more reliable scores or even probabilities.

@julian-risch and me will be working on model confidence, that then results in better scores, the coming weeks. I think we will tackle this in FARM, since it is a QA Modelling related task.

[lalitpagaria]

will be working on model confidence, that then results in better scores,

Hey @Timoeller I would like to learn this so I am happy to contribute if given direction.

[Timoeller]

Hey Lalit, thanks for the kind offer. This is actually a bigger topic that we started to plan and will tackle in the coming weeks.

We could definitely use your support and feedback on related the code we will write.
Once we have laid out our detailed plan in the linked FARM issue we can also discuss if there are smaller tasks for you to work on.
We can also have a call where we discuss the solution in general, I know you are very interested in the NLP aspects of haystack and the model confidence would be a good way to show you QA modelling related details.
What do you say?

[lalitpagaria]

Sure make sense. Ping me when you start working on this topic.

Excited to work with @julian-risch, number of publications in his PhD are beyond my imagination. Specially topic related to abusive language detection.

[Timoeller]

We merged in the new FARM version that has better confidence scores and assigned them to the answer field "probability".