Add switch to QA pred head for ranking by confidence scores #836
Conversation
julian-risch
changed the title
farm/modeling/prediction_head.py
Outdated
| @@ -963,6 +964,8 @@ def __init__(self, layer_dims=[768,2], | |||
| :type duplicate_filtering: int | |||
| :param temperature_for_confidence: The divisor that is used to scale logits to calibrate confidence scores | |||
| :type temperature_for_confidence: float | |||
| :param use_confidence_scores_for_ranking: Whether to sort answers by confidence score (normalized between 0 and 1) or by standard score (unbounded) | |||
There was a problem hiding this comment.
Can we document here somewhere a bit clearer what these different scores exactly are and how they are calculated? If I remember correctly we have these three cases:
a) score = start + end logit (unbounded)
b) confidence (default) = logits scaled to 0-1 and incorporating no_answer
c) confidence (calibrated) - same a b) but we multiply it with a learned scaling parameter
I am sure I will forget about it in a couple of weeks and would be helpful to have it documented for others. Probably it's best to do that in the general prediction head doc string (+ Haystack's FARMReader).
There was a problem hiding this comment.
Good point. I added an explanation of the three kinds of scores to the doc string of the QAPredictionHead.
QACandidatescontainscoreandconfidencefields but onlyscorewas used to rankQACandidatesso far.This PR makes ranking
QACandidatesby score the default but also allows to setuse_confidence_scores_for_rankinginQAPredictionHead, which activates rankingQACandidatesbyconfidence.I think, it's better to keep both fields,
scoreandconfidenceinQACandidatesinstead of dropping one of them entirely. Thereby, we are more flexible and allow backwards compatibility.A new test case checks that results are ranked as expected with and without setting
use_confidence_scores_for_ranking.closes #808