Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix error with custom QE COMET #77

Merged
merged 3 commits into from
May 5, 2022

Conversation

Joao-Maria-Janeiro
Copy link
Contributor

When a custom QE metric is trained, the scorer.py gives the error:
parser.error("{} requires -r/--references or -d/--sacrebleu_dataset.".format(cfg.model))

This happens because it is currently comparing the given model name with _REFLESS_MODELS = ["comet-qe"] only, not allowing to use custom QE metrics.

@ricardorei
Copy link
Collaborator

alright, I was thinking about this and I believe that the best option is to add a method to all COMET models that check if the model is reference-free. Then instead of checking the model class, you could call that method to check if the model requires references.

@ricardorei
Copy link
Collaborator

That should be a abstract method in the base class and then implemented on all child classes:

@Joao-Maria-Janeiro
Copy link
Contributor Author

Makes sense!! Submitted!

@ricardorei ricardorei merged commit d3a2f00 into Unbabel:master May 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants