Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
19 Open 80 Closed
19 Open 80 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

(feature request) can we add load_dotenv() as a small quality of life improvement?
#107 by kdu4108 was closed Mar 20, 2024 updated Mar 20, 2024
2
How to calculate the avg score?
#127 by wanxinzzz was closed Mar 29, 2024 updated Mar 29, 2024
Pos_tokens.reshape error while testing mmmu on internlm-xcomposer2-7b
#136 by littleSunlxy was closed Apr 2, 2024 updated Apr 2, 2024
A major problem with the multiple-choice evaluation
#141 by YongLD was closed Apr 9, 2024 updated Apr 9, 2024
7
Can not reach the info_vqa dataset
#146 by Ezra-Yu was closed Apr 9, 2024 updated Apr 9, 2024
2
Consider integrating MMStar?
#135 by iamlockelightning was closed Apr 9, 2024 updated Apr 9, 2024
2
[Feature Request] To evaluate MMMU test set, you need to transfer the xlsx output to a json file
#124 by StarCycle was closed Apr 9, 2024 updated Apr 9, 2024
4
How to calculate overall score for HallusionBench
#133 by WizardMx was closed Apr 10, 2024 updated Apr 10, 2024
1
ModuleNotFoundError: No module named 'xtuner.parallel'
#139 by starlitsky2010 was closed Apr 11, 2024 updated Apr 11, 2024
3
llava_v1.5_7b wrong results on Seedbench_IMG
#128 by BonitoW was closed Apr 15, 2024 updated Apr 15, 2024
2
MBench_TEST_CN和MMBench_TEST_EN的tsv里没有answer列
#148 by Jiadwu2 was closed Apr 15, 2024 updated Apr 15, 2024
1
[Feature Request] RealWorldQA Benchmark
#149 by StarCycle was closed Apr 16, 2024 updated Apr 16, 2024
2
opencompass多模态榜单上的分数是exact_matching还是GPT辅助计算的分数
#154 by jiezhangGt was closed Apr 17, 2024 updated Apr 17, 2024
2
IndexError: index 1 is out of bounds for dimension 0 with size 1
#150 by lucasjinreal was closed Apr 21, 2024 updated Apr 21, 2024
8
Error Encountered in Multi-Node Evaluation Using Distributed Arguments
#142 by jdy18 was closed Apr 21, 2024 updated Apr 21, 2024
4
ModuleNotFoundError: No module named 'llava.model.builder'
#166 by TousenKaname was closed Apr 23, 2024 updated Apr 23, 2024
A problem with version conflicts
#165 by JesseZZZZZ was closed Apr 23, 2024 updated Apr 23, 2024
2
OSError: Incorrect path_or_model_id: 'xtuner/llava-internlm2-20b/projector'. Please provide either the path to a local folder or the repo_id of a model on the Hub.
#138 by starlitsky2010 was closed Apr 23, 2024 updated Apr 23, 2024
2
Error ‘assert len(chunk_encode) == 2’ when eval MMMU_DEV_VAL
#173 by wutaiqiang was closed Apr 25, 2024 updated Apr 25, 2024
3
llava 34B 评测时CUDA out of memory
#156 by jiezhangGt was closed Apr 17, 2024 updated Apr 25, 2024
2
How to calculate Avg Score mentioned in OpenVLM Leaderboard？
#176 by la1n33 was closed Apr 26, 2024 updated Apr 26, 2024
POPE evaluation
#172 by phquang was closed Apr 29, 2024 updated Apr 29, 2024
2
There are some known issues with VQA tasks like OCRVQA, TextVQA, ChartQA, etc. We will fix them asap.
#168 by BlueBlueFF was closed Apr 27, 2024 updated May 3, 2024
2
怎么测试LLAVA+LLAMA3的性能？
#181 by xmu-xiaoma666 was closed May 6, 2024 updated May 6, 2024
3
About MMBench data source: Is there any difference from huggingface lmms-lab/MMBench?
#192 by Coobiw was closed May 6, 2024 updated May 6, 2024
2
ProTip! What’s not been updated in a month: updated:<2024-06-22.