Farm top_k reader error

[aadil-srivastava01]

When using finder.get_answers() with tfidf-retriever and farm reader, top_k_reader parameter has no effect on retrieving the top K answers. It will always return the top 3 answers as FarmReader top_k_per_candidate is set to 3 by default. So in order to get n number of answers, we will have to initialize FarmReader's top_k_per_candidate by n.

[tholor]

Hey @aadil-srivastava01,

We did some changes on these parameters last week. Can you please check how many candidate docs you retriever returns? If you have e.g. only 1 candidate here, it would explain the observed behavior. Otherwise, I am happy to dig in deeper.

To give you some background: There are a couple of different aggregation steps involved. How it should work:

• top_k_per_sample: the model creates this many predictions for one text passage that it can process at once (bounded by max_sequence_length of the model, e.g. 256 tokens)
• top_k_per_candidate: the predictions per sample are aggregated to this many answers per "candidate" (i.e. for one document that the retriever send to the reader, which, depending on the length, consists of 1-x samples )
• top_k_reader: the predictions per candidate are aggregated again to yield the final answers that are returned by the Finder

This means you can only return min(top_k_per_candidate* candidates, top_k_reader) answers.

In any case, we should try to make this more user-friendly in the future. Either by an info message or some automated parameter adjustments for these special cases.

[aadil-srivastava01]

Please refer to the highlighted section in the attached images. I guess the problem is still there. Only when you initialize farm reader with the top_k_reader then will give you the number of specified answers else it will give the default number of answers which is 3, which is the value of its default initialization, making finder.get_answers having no effect on the generated output.