-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TruthfulQA] update rouge-score version or add a way to suppress tokenizer logging #1692
Comments
Does running with |
I am also finding this logging output cumbersome! |
Adding a fix for this in #2090 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When running truthfulqa evaluation i get a lot of unnecessary info/debug prints from the rouge scorer (usually hundreds if not thousands of lines of logging) each looking like this:
Eval harness has a dependency to the package rouge-score > 0.0.4
In the latest version on PyPi the Initialization of the scorer prints an info log each time it is called with the default tokenizer.
Truthfulqa uses a loop to re-initialize the scorer each time it wants to score something which leads to a lot of prints that are supposed to be debug prints but are still stuck at info level due to the outdated version of rouge-scorer (taken from utils.py).
That behaviour was changed 8 months ago with commit 6e232b36782739b234922e2a65e92d7c2651758b. (see history of https://github.com/google-research/google-research/blob/master/rouge/rouge_scorer.py).
The latest package of the rouge-scorer on Pypi however is from july 2022 and doesn't have these changes yet.
Is there a way to get a more updated version of rouge-scorer into the dependencies or a different way to suppress those prints or reduce the amount of them happening (like moving the initialization of the rouge scorer so it isn't called with each ietration of the loop)?
The text was updated successfully, but these errors were encountered: