Adding LLaVa support #1832
Adding LLaVa support #1832
Conversation
ashvinnihalani
requested review from
haileyschoelkopf and
lintangsutawika
as code owners
|
|
|
As new release will be soon, can you please consider resolving conflicts with the |
![accesslint[bot]](https://avatars.githubusercontent.com/in/1502?s=60&v=4){kind=small}
Updating APIs for MM support Adding MLLM dependencies Rebase off mainline
Done, just a heads up this is probably not the first CR for LLaVa support adding some more tasks and supporting the latest LLaVa 1.6 is still needed. |
|
Hi @ashvinnihalani , giving you a heads up we're looking at this PR and trying out some multimodal design ideas based on it! See |
|
Cool we are already using this + some other commits to test our MM models in multiple formats (HF LLaVA, LLaVA, VLLM pending) across multiple benchmarks (VQA, MMMU, custom MM benchmarks, text tasks as well, etc). Let me know what I can do to help mainline our changes/get this functionality upstreamed. Happy to continue working on this PR/move to another PR. |
|
Closing this PR as it uses code written by LMMs-eval authors, but as described in #2014 we're working on design for VLM (text-out) backend and will share in that issue for feedback as we update! |
Adding M-LLM support by adding the MMLU task and torch-LLaVa. Based on https://github.com/EvolvingLMMs-Lab/lmms-eval but simplified greatly unify support
Tested with liuhaotian/llava-v1.5-7b and got the following results: https://gist.github.com/ashvinnihalani/563c13b510fb72a57387056e331977c5.
22.3 accuracy which is lower than the reported metric. https://camo.githubusercontent.com/a2210d9fd179342ce7c2a1afa7588c51859e1a627264254e95e4e6c068284b8d/68747470733a2f2f692e706f7374696d672e63632f6a64773439374e532f575832303234303330372d3136323532362d32782e706e67
Addressing #1155