Reader evaluation: KeyError: 'top_n_recall' (refer tutorial 5)

[usaraj]

Describe the bug
Got Key Error from reader (farm.py line 332) while displaying eval results after evaluating a fine tuned (trained) roberta-base-squad2 model on a Squad input file. (Refer. Tutorial 5). The evaluation step completed 100%, the error is thrown while printing the eval results.

Error message
KeyError: 'top_n_recall'. (Trace back below in additional context section)

Expected behavior
The eval result keys Top recall, F1Score, Exact match scores should be displayed.

Additional context
Output and Error traceback after calling reader.eval_on_file(...)

Preprocessing Dataset /content/drive/My Drive/data/test_custom_data.json: 100%
Evaluating: 100%

KeyError Traceback (most recent call last) in ()
Evaluation of Reader can also be done directly on a SQuAD-formatted file without passing the data to Elasticsearch
----> 6 reader_eval_results = new_reader.eval_on_file("/content/drive/My Drive/data", "test_custom_data.json", device=device)

/content/haystack/haystack/reader/farm.py in eval_on_file(self, data_dir, test_filename, device)
330 "EM": eval_results[0]["EM"],
331 "f1": eval_results[0]["f1"],
--> 332 "top_n_recall": eval_results[0]["top_n_recall"]
333 }
334 return results

KeyError: 'top_n_recall'

To Reproduce
Use default deepset roberta base squad2 model as reader
fine tune reader with custom squal2 datasets
save custom reader in path
Try to evaluate the reader with the following command:
reader_eval_results = reader.eval_on_file("local_path", "TestCustomData.json", device=device)
print("Reader Top-N-Recall:", reader_eval_results["top_n_recall"])

System:

• OS: Colab
• GPU/CPU: GPU
• Haystack version (commit or version number): latest git stable master
• DocumentStore: Squad file
• Reader: Farm Reader (fine tuned roberta-squad2 model)
• Retriever: - Not used in this step.

[brandenchan]

Hi, this is now fixed in the latest version of tutorial5 in colab. Could you please let me know if it works for you now?

[usaraj]

Yes this is fixed now, thank you very much