-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a way to instantiate from HF.AutoModel #521
Comments
In general, yes this sounds like a great contribution. I'm a little unclear on what the blocker is though... what is your usecase where it's important to pre-instantiate the model? That sounds like you have a hosted inference you're looking to query, and we currently support the OpenAI API and if you're using something custom you can add support for your thing based on that. |
We're looking to use lm-eval as a backend for a benchmarks component, which is part of a larger framework. The idea is that a user can load a model once, and use it in different components of the framework without having to reload it. Being able to initialize a HFLM instance using the loaded model solves that problem for our benchmarks module. Perhaps there's a different solution to the problem, but to me it also seems like a convenient option for lm-eval to include. |
This seems like a great contribution! I see 2 potential paths forward:
|
To me, option 1 seems like the clearest way to provide this. I don't think adding a layer of abstraction for this necessarily helps understanding the code, unless there's some benefits that I'm missing. |
Agreed, think that makes sense! option 2 would be for supporting models that may not be HF models (to interface easier with other libs for evals during training) but it probably makes sense to just do 1. |
I agree this makes sense, though for clarity a “—hosted” flag or something like that might be a good idea? Happy to have this added to the library though. |
I was conceptualizing this more as a feature for those who are looking to use the HFLM class as an import, so I haven't clearly thought of any command line arguments for that usecase. Which usecase do you see for such a flag? Perhaps providing a path to a serialized model object or sth? |
I have personally never typed |
I did the CLI part in #550, it casts the base class to one of the children using the model config, should be easy to also accept an initialized |
Thanks @gakada I'll have a look. I'm not sure if either I've managed to find some time to work on this, and I expect to push something the coming days. |
I mean |
I think I understand what you mean, I've added the logic. It seems I'm not authorized to push the branch, if someone could authorize me, I'll create a PR for review. |
You should be able to make a fork and open a PR, but I’ll send you an invite to the EAI org. |
Hi Everyone, I created a PR last week, I'll link it here too for convenience: |
Thanks for the PR!! Will review it this morning :) |
Hi everyone,
I'm looking to use some of the tasks in my own code, which already instantiate a transformers.AutoModelForxyz object. I know that there are wrappers available for hugging face models, such as the HFLM class, however, it seems I can only pass a namestring into the constructor, after which the object instantiates a hugging face model under the hood. It would be great if there was a way to instantiate an HFLM object (or something equivalent which I can pass to simple_evaluate) using an existing hf instance.
My question is therefore twofold:
Let me know what you think, and of course, thanks a lot for sharing this awesome project with the community.
Kind regards,
Sven
The text was updated successfully, but these errors were encountered: