Allow --include_path
to import an externally-defined LM subclass
#1457
Labels
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
As requested by @LSinev ,
it might be helpful for some users to be able to run
lm_eval
orlm_eval/__main__.py
without editing the file, but while using anLM
subclass one has defined outside the library. This is already possible via either adding an import to__main__.py
of the module containing theregister_model()
decorated class, or via passing an already-initialized LM class tosimple_evaluate()
, but that isn't as convenient in all cases.We could thus allow, similar to the use of external task folders using
--include_path
, a way to import a desired python module at runtime, enabling users to import models / metrics defined outside the library into its registries.The text was updated successfully, but these errors were encountered: