Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
18 Open 80 Closed
18 Open 80 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Pos_tokens.reshape error while testing mmmu on internlm-xcomposer2-7b
#136 by littleSunlxy was closed Apr 2, 2024
(feature request) can we add load_dotenv() as a small quality of life improvement?
#107 by kdu4108 was closed Mar 20, 2024
2
Evaluation of custom models and datasets.
#108 by juxingyiwan was closed Jul 19, 2024
2
OPENAI_API_KEY is not set properly, will use exact matching for evaluation
#109 by StarCycle was closed Mar 15, 2024
1
The code for evaluating open question of MMMU is completely wrong
#110 by xwwu2015 was closed Jun 17, 2024
1
MathVista测试问题
#112 by mactavish91 was closed Mar 19, 2024
4
HallusionBench的最终结果如何计算
#113 by echo840 was closed Mar 16, 2024
1
The scores calculated by VLMEvalKit differ from the score calculated on the MMBench website
#121 by jdy18 was closed Mar 20, 2024
4
Error Encountered in Multi-Node Evaluation Using Distributed Arguments
#142 by jdy18 was closed Apr 21, 2024
4
How to calculate the avg score?
#127 by wanxinzzz was closed Mar 29, 2024
llava_v1.5_7b wrong results on Seedbench_IMG
#128 by BonitoW was closed Apr 15, 2024
2
How to calculate overall score for HallusionBench
#133 by WizardMx was closed Apr 10, 2024
1
Consider integrating MMStar?
#135 by iamlockelightning was closed Apr 9, 2024
2
Get a 0 score in Evaluation of qwen chat in ScienceQA
#61 by YongLD was closed Jan 24, 2024
4
OSError: Incorrect path_or_model_id: 'xtuner/llava-internlm2-20b/projector'. Please provide either the path to a local folder or the repo_id of a model on the Hub.
#138 by starlitsky2010 was closed Apr 23, 2024
2
ModuleNotFoundError: No module named 'xtuner.parallel'
#139 by starlitsky2010 was closed Apr 11, 2024
3
A major problem with the multiple-choice evaluation
#141 by YongLD was closed Apr 9, 2024
7
[Feature Request] To evaluate MMMU test set, you need to transfer the xlsx output to a json file
#124 by StarCycle was closed Apr 9, 2024
4
Numpy compliation issue during installing
#34 by findalexli was closed Jan 10, 2024
2
TypeError in parallel API calling
#39 by moiSusu was closed Jan 3, 2024
1
Cannot reproduce llava v1.5 7b SEEDBench_IMG results
#42 by MagicGeek2 was closed Jan 5, 2024
How do you compute MME(normalized)?
#43 by simplelifetime was closed Jan 8, 2024
1
word_count_mean of COCO Caption
#46 by moiSusu was closed Jan 20, 2024
1
Error when evaluating MMBench_TEST_EN
#57 by KosumosuL was closed Jan 21, 2024
4
upload failed on mmbench website
#59 by mactavish91 was closed Jan 23, 2024
2
ProTip! Exclude everything labeled bug with -label:bug.