Reader returns the same answer twice in isolated eval mode (if there are two labels from the same document)

[tstadel]

Describe the bug
Starting with 1.13.0, BaseReader.run does not deduplicate documents in isolated node eval.

Previously we used

relevant_documents = {label.document.id: label.document for label in labels.labels}.values()

which deduplicates documents in case there are labels for the same documents (but different span).

Now we don't do this anymore:

haystack/haystack/nodes/reader/base.py

Lines 120 to 122 in a2c160e

	relevant_documents = [label.document for label in labels.labels]
	# Filter out empty documents
	relevant_documents = [d for d in relevant_documents if d.content.strip() != ""]

This results in duplicate predictions as the Reader treats the same documents as different ones.

Error message
None, but duplicate predictions.

Expected behavior
No duplicate predictions.

Additional context

To Reproduce

1. Create two labels from the same document but with different answer spans.
2. Run pipeline.eval on a single-noded Reader pipeline in isolated mode with add_isolated_node_eval=True.
3. Get the same predictions twice.

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: any
• GPU/CPU: any
• Haystack version (commit or version number): 1.13.1
• DocumentStore: any
• Reader: FARMReader (but should be the same for TransformersReader)
• Retriever: any