Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
16 Open 80 Closed
16 Open 80 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Cannot reproduce llava v1.5 7b SEEDBench_IMG results
#42 by MagicGeek2 was closed Jan 5, 2024
How to calculate the avg score?
#127 by wanxinzzz was closed Mar 29, 2024
Pos_tokens.reshape error while testing mmmu on internlm-xcomposer2-7b
#136 by littleSunlxy was closed Apr 2, 2024
ModuleNotFoundError: No module named 'llava.model.builder'
#166 by TousenKaname was closed Apr 23, 2024
How to calculate Avg Score mentioned in OpenVLM Leaderboard？
#176 by la1n33 was closed Apr 26, 2024
What is the estiamted runtime across benchmarks, and OpenAI api cost?
#33 by findalexli was closed Jan 20, 2024
1
TypeError in parallel API calling
#39 by moiSusu was closed Jan 3, 2024
1
How do you compute MME(normalized)?
#43 by simplelifetime was closed Jan 8, 2024
1
word_count_mean of COCO Caption
#46 by moiSusu was closed Jan 20, 2024
1
How to calculate the average rank?
#71 by liuheng0111 was closed Jan 30, 2024
1
These two download links are no longer accessible...
#78 by zhimin-z was closed Jan 30, 2024
1
LLava GPT Eval无法对齐
#84 by BlueBlueFF was closed Feb 4, 2024
1
Enhancing Multi-Choice Question Handling with Case-Sensitive Matching
#99 by jdy18 was closed Mar 12, 2024
1
If i want to use model of LLaVa in VLMEval, which version should i installed for these?
#100 by EmberYS was closed Mar 7, 2024
1
OPENAI_API_KEY is not set properly, will use exact matching for evaluation
#109 by StarCycle was closed Mar 15, 2024
1
The code for evaluating open question of MMMU is completely wrong
#110 by xwwu2015 was closed Jun 17, 2024
1
HallusionBench的最终结果如何计算
#113 by echo840 was closed Mar 16, 2024
1
How to calculate overall score for HallusionBench
#133 by WizardMx was closed Apr 10, 2024
1
MBench_TEST_CN和MMBench_TEST_EN的tsv里没有answer列
#148 by Jiadwu2 was closed Apr 15, 2024
1
支持LMDeploy作为推理后端
#179 by slchenchn was closed Jul 19, 2024
1
How to calculate avg score after the leaderboard update to 8 multimodal benchmarks
#206 by wanxinzzz was closed May 21, 2024
1
支持GLM4v-9b-chat评测吗？
#220 by yinggoga was closed Jun 13, 2024
1
AesBench support
#222 by yipoh was closed Jun 27, 2024
1
about vlmeval/evaluate/multiple_choice.py
#227 by mary-0830 was closed Jun 20, 2024
1
evaluate TextVQA_VAL error
#237 by LRHstudy was closed Jun 27, 2024
1
ProTip! Follow long discussions with comments:>50.