Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow --include_path to import an externally-defined LM subclass #1457

Open
haileyschoelkopf opened this issue Feb 22, 2024 · 1 comment
Open
Labels
feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.

Comments

@haileyschoelkopf
Copy link
Contributor

As requested by @LSinev ,

it might be helpful for some users to be able to run lm_eval or lm_eval/__main__.py without editing the file, but while using an LM subclass one has defined outside the library. This is already possible via either adding an import to __main__.py of the module containing the register_model() decorated class, or via passing an already-initialized LM class to simple_evaluate(), but that isn't as convenient in all cases.

We could thus allow, similar to the use of external task folders using --include_path, a way to import a desired python module at runtime, enabling users to import models / metrics defined outside the library into its registries.

@haileyschoelkopf haileyschoelkopf added help wanted Contributors and extra help welcome. feature request A feature that isn't implemented yet. good first issue Good for newcomers labels Feb 22, 2024
@LSinev
Copy link
Contributor

LSinev commented May 15, 2024

Are there any pitfalls to avoid (based on recent experience with merging PRs with new models)? Or anything to remember? In case of starting the PR development to close this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
Projects
Status: Backlog
Development

No branches or pull requests

2 participants