Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
14 Open 61 Closed
14 Open 61 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

TypeError in parallel API calling
#39 by moiSusu was closed Jan 3, 2024 updated Jan 3, 2024
1
Cannot reproduce llava v1.5 7b SEEDBench_IMG results
#42 by MagicGeek2 was closed Jan 5, 2024 updated Jan 5, 2024
How do you compute MME(normalized)?
#43 by simplelifetime was closed Jan 8, 2024 updated Jan 8, 2024
1
Numpy compliation issue during installing
#34 by findalexli was closed Jan 10, 2024 updated Jan 10, 2024
2
word_count_mean of COCO Caption
#46 by moiSusu was closed Jan 20, 2024 updated Jan 20, 2024
1
What is the estiamted runtime across benchmarks, and OpenAI api cost?
#33 by findalexli was closed Jan 20, 2024 updated Jan 20, 2024
1
Error when evaluating MMBench_TEST_EN
#57 by KosumosuL was closed Jan 21, 2024 updated Jan 21, 2024
4
upload failed on mmbench website
#59 by mactavish91 was closed Jan 23, 2024 updated Jan 23, 2024
2
Get a 0 score in Evaluation of qwen chat in ScienceQA
#61 by YongLD was closed Jan 24, 2024 updated Jan 24, 2024
4
MMMU test set Feature Request
#65 by mactavish91 was closed Jan 27, 2024 updated Jan 28, 2024
2
How to calculate the average rank?
#71 by liuheng0111 was closed Jan 30, 2024 updated Jan 30, 2024
1
These two download links are no longer accessible...
#78 by zhimin-z was closed Jan 30, 2024 updated Jan 30, 2024
1
where is the vlmeval/utils/data_util.py
#83 by TousenKaname was closed Jan 31, 2024 updated Jan 31, 2024
3
LLava GPT Eval无法对齐
#84 by BlueBlueFF was closed Feb 4, 2024 updated Feb 4, 2024
1
Error when evaluating mmbench_dev_en：The correct answer according to the 'answer' field in the table should be D, but the log says it is A.
#73 by jdy18 was closed Feb 1, 2024 updated Feb 10, 2024
5
[疑问] 实验分数对不上，使用llava_v1.5_7b在MMMU_DEV_VAL测试集上的分数对不上
#95 by mary-0830 was closed Feb 27, 2024 updated Feb 27, 2024
5
If i want to use model of LLaVa in VLMEval, which version should i installed for these?
#100 by EmberYS was closed Mar 7, 2024 updated Mar 7, 2024
1
Enhancing Multi-Choice Question Handling with Case-Sensitive Matching
#99 by jdy18 was closed Mar 12, 2024 updated Mar 12, 2024
1
OPENAI_API_KEY is not set properly, will use exact matching for evaluation
#109 by StarCycle was closed Mar 15, 2024 updated Mar 15, 2024
1
Unknown error when loading LLaVA model
#92 by cyj95 was closed Mar 16, 2024 updated Mar 16, 2024
3
The md5 hashtest of the dataset DocVQA_VAL hasn't passed.
#103 by EmberYS was closed Mar 16, 2024 updated Mar 16, 2024
4
HallusionBench的最终结果如何计算
#113 by echo840 was closed Mar 16, 2024 updated Mar 16, 2024
1
Detailed results of ScienceQA-IMG
#101 by thecharm was closed Mar 17, 2024 updated Mar 17, 2024
3
MathVista测试问题
#112 by mactavish91 was closed Mar 19, 2024 updated Mar 19, 2024
4
The scores calculated by VLMEvalKit differ from the score calculated on the MMBench website
#121 by jdy18 was closed Mar 20, 2024 updated Mar 20, 2024
4
ProTip! Type g i on any issue or pull request to go back to the issue listing page.