-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Recall with DocumentRecallEvaluator is not calculated correctly #7867
Comments
Hello! From the docs:
So, in this case (single hit), the result of 1.0 is correct. I am closing the issue. Feel free to reopen it if anything is unclear... |
Damn, you are right 😮💨 Thanks |
Describe the bug
I might have gotten this wrong, but I feel the recall is not calculated correctly
Given this code
I get
{'score': 1.0, 'individual_scores': [1.0]}
If I now change the first Document in retrieved_documents to be for example
Document(content="1st century")
the result is still{'score': 1.0, 'individual_scores': [1.0]}
Expected behavior
As only one of the documents in the two ground_truths docs is found, I would expect that the score is 0.5, not 1.0
Additional context
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.
To Reproduce
Steps to reproduce the behavior
FAQ Check
System:
The text was updated successfully, but these errors were encountered: