Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
42 Open 520 Closed
42 Open 520 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

The results of instructblip 13b
#187 opened Apr 29, 2024 by YongLD
[Support] multiple process parallel inference large model on multi-gpu In Progress New feature is under working
#298 opened Jul 19, 2024 by junming-yang Loading…
[Benchmark] mmniah
#434 opened Sep 1, 2024 by ttguoguo3 Loading…
Input length of input_ids is 0, but max_length is set to -2009.
#455 opened Sep 10, 2024 by wangli68
How to eval with multigpu when using device_map='auto'
#528 opened Oct 18, 2024 by qianwangn
评测InternVL2-8B出现OOM
#533 opened Oct 21, 2024 by AACengineer
[Benchmark] Add MLLMJudgeDataset
#534 opened Oct 21, 2024 by Misaki-Wang Loading…
LLaVA v1.5 7b very low performance on TextVQA
#554 opened Oct 28, 2024 by wufeim
[Fix] Fix #546
#562 opened Oct 31, 2024 by kennymckormick Loading…
Audio modality evaluation help wanted Extra attention is needed
#313 opened Jul 25, 2024 by Simplesss
1
关于幻觉benchmark的支持 help wanted Extra attention is needed
#384 opened Aug 15, 2024 by shufangxun
1
【提问】关于语音模型的评测 help wanted Extra attention is needed
#473 opened Sep 19, 2024 by xx0412
1
textonly benchmarks
#497 opened Sep 30, 2024 by JcWang20
@PhoenixZ810
1
Different evaluate result between different VLMEvalKit version
#523 opened Oct 16, 2024 by terry-for-github
1
dataset_config.py在哪里
#545 opened Oct 24, 2024 by hello-bluedog
1
[Bug] RuntimeError: world_size (8) is not equal to tensor_model_parallel_size (1) x pipeline_model_parallel_size (1)
#561 opened Oct 31, 2024 by Wiselnn570
1
best to have docker image help wanted Extra attention is needed
#190 opened May 3, 2024 by max-yue
2
[Help Wanted] Supporting the chat_inner API for existing VLMs. help wanted Extra attention is needed
#323 opened Jul 27, 2024 by kennymckormick
2
Inconsistent max_new_tokens
#356 opened Aug 3, 2024 by amitbcp
2
Adding new dataset: worldmedQA-V to VLMEvalKit
#525 opened Oct 16, 2024 by shan23chen
2
qwen2-vl-7b设置device_map为auto时报错
#546 opened Oct 24, 2024 by wwwbq
2
评测MVBench时报错
#551 opened Oct 25, 2024 by helloworld01001
2
The result of gpt4o on MathVista is 61.6, which is lower than the 62.7 on the list.
#553 opened Oct 28, 2024 by kkk123698745
2
grounding数据支持
#219 opened Jun 12, 2024 by sxlyiyiyi
3
[Request] Consider integrating the following list? help wanted Extra attention is needed
#257 opened Jul 5, 2024 by sarapieri
3
ProTip! Mix and match filters to narrow down what you’re looking for.