Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
14 Open 61 Closed
14 Open 61 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Multi-GPU inference issue
#224 by ruifengma was closed Jun 19, 2024
11
IndexError: index 1 is out of bounds for dimension 0 with size 1
#150 by lucasjinreal was closed Apr 21, 2024
8
A major problem with the multiple-choice evaluation
#141 by YongLD was closed Apr 9, 2024
7
Questions regarding the metrics for SEED bench
#20 by zjysteven was closed Dec 23, 2023
7
about vlmeval/evaluate/multiple_choice.py
#228 by mary-0830 was closed Jun 20, 2024
6
Model name in demo
#199 by chandrabhuma was closed May 11, 2024
6
有关模型和数据集下载的问题
#213 by xioatian1 was closed May 29, 2024
5
[疑问] 实验分数对不上，使用llava_v1.5_7b在MMMU_DEV_VAL测试集上的分数对不上
#95 by mary-0830 was closed Feb 27, 2024
5
Error when evaluating mmbench_dev_en：The correct answer according to the 'answer' field in the table should be D, but the log says it is A.
#73 by jdy18 was closed Feb 1, 2024
5
Question about LlavaBench Evaluation
#194 by justinphan3110 was closed May 9, 2024
4
Error Encountered in Multi-Node Evaluation Using Distributed Arguments
#142 by jdy18 was closed Apr 21, 2024
4
[Feature Request] To evaluate MMMU test set, you need to transfer the xlsx output to a json file
#124 by StarCycle was closed Apr 9, 2024
4
The scores calculated by VLMEvalKit differ from the score calculated on the MMBench website
#121 by jdy18 was closed Mar 20, 2024
4
MathVista测试问题
#112 by mactavish91 was closed Mar 19, 2024
4
The md5 hashtest of the dataset DocVQA_VAL hasn't passed.
#103 by EmberYS was closed Mar 16, 2024
4
Get a 0 score in Evaluation of qwen chat in ScienceQA
#61 by YongLD was closed Jan 24, 2024
4
Error when evaluating MMBench_TEST_EN
#57 by KosumosuL was closed Jan 21, 2024
4
mmmu should only use mmmu val not MMMU_DEV_VAL
#198 by max-yue was closed May 24, 2024
3
怎么测试LLAVA+LLAMA3的性能？
#181 by xmu-xiaoma666 was closed May 6, 2024
3
Error ‘assert len(chunk_encode) == 2’ when eval MMMU_DEV_VAL
#173 by wutaiqiang was closed Apr 25, 2024
3
ModuleNotFoundError: No module named 'xtuner.parallel'
#139 by starlitsky2010 was closed Apr 11, 2024
3
Detailed results of ScienceQA-IMG
#101 by thecharm was closed Mar 17, 2024
3
Unknown error when loading LLaVA model
#92 by cyj95 was closed Mar 16, 2024
3
where is the vlmeval/utils/data_util.py
#83 by TousenKaname was closed Jan 31, 2024
3
About MMBench data source: Is there any difference from huggingface lmms-lab/MMBench?
#192 by Coobiw was closed May 6, 2024
2
ProTip! Mix and match filters to narrow down what you’re looking for.