-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Similarity based Answer evaluation #1516
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Following up on research done in deepset-ai/FARM#757
Target Approach
Implementing a new registered metric using the sentencetransformers model.
Also implement a default metric for FARM with default model. Look for ways to pass comparison model to FARM.
Implementation details
Input to our metric functions, preds and labels, are in number format.
Preds are of type QACandidate, but missing text answer information.
Labels are tuples of start and end indices.
Our evaluator.eval pipeline is not meant for dealing with plaintext predictions or labels because we dismiss our "baskets" with all text information. Also using the tokenizer to detokenize input_ids (token ids) will not work since we do not collect input_ids in the evaluator. Therefore we should start an initial implementation based on Inferencer output and not create a default FARM metric.
The text was updated successfully, but these errors were encountered: