Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: AnswerExactMatchEvaluator input should be one answer per query instead of list of answers #7459

Closed
julian-risch opened this issue Apr 3, 2024 · 2 comments · Fixed by #7536
Assignees
Labels
P1 High priority, add to the next sprint topic:eval
Milestone

Comments

@julian-risch
Copy link
Member

julian-risch commented Apr 3, 2024

We should change the inputs of the AnswerExactMatchEvaluator added in #7050 from
questions: List[str], ground_truth_answers: List[List[str]], predicted_answers: List[List[str]]
to
ground_truth_answers: List[str], predicted_answers: List[str]

This change will make all metrics in Haystack core and in integrations, statistical and model-based, consistent in the inputs they expect. For answers always List[str], for queries always List[str] and for documents (contexts) always List[List[str]]. It also simplifies the implementation of the new metrics and will allow us to move faster.

Describe alternatives you've considered
Keeping inputs as is would mean they are inconsistent with the model based metrics and the integrations of evaluation frameworks. However, the behavior would be the same as the exact match metric in Haystack 1.x and more flexible for datasets with multiple ground truth answers such as SQuAD 2.0 and multiple predicted answers like our Reader's output.

@julian-risch julian-risch added topic:eval P1 High priority, add to the next sprint labels Apr 3, 2024
@shadeMe
Copy link
Collaborator

shadeMe commented Apr 4, 2024

Mentioning this here so that we don't forget - We should also remove the questions input parameter as it's unused.

@masci masci self-assigned this Apr 7, 2024
@julian-risch julian-risch added this to the 2.1.0 milestone Apr 9, 2024
@masci
Copy link
Contributor

masci commented Apr 11, 2024

Mentioning this here so that we don't forget - We should also remove the questions input parameter as it's unused.

Removed in #7466

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 High priority, add to the next sprint topic:eval
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants