Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Organize / Cleanup Logging + Levels #1192

Open
haileyschoelkopf opened this issue Dec 21, 2023 · 1 comment
Open

Organize / Cleanup Logging + Levels #1192

haileyschoelkopf opened this issue Dec 21, 2023 · 1 comment
Labels
documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet.

Comments

@haileyschoelkopf
Copy link
Contributor

haileyschoelkopf commented Dec 21, 2023

We should do a pass through the codebase and unify log printing styles + re-check log levels.

Ideally a novice user should only see WARNING levels if they are doing something unintended / footgun-y and INFO levels should (theoretically) be ignorable if you're not a power user. DEBUG should refer to logs that are useful when trying to add a new task and not useful to see every run otherwise. We should try not to print too many logs in the default setting, except where we think it helps the average user (there should be an easily-accessible "almost-silent" mode.)

@haileyschoelkopf haileyschoelkopf added documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet. labels Dec 21, 2023
@haileyschoelkopf
Copy link
Contributor Author

haileyschoelkopf commented Jan 5, 2024

Listing some changes that pop out if I run into them.

  • lm_eval/api/task#L701 should get a more informative log message?
  • Phase out --write_out in favor of just logging some sample instances instead when using DEBUG log level?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet.
Projects
Status: Backlog
Development

No branches or pull requests

1 participant