Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
26 Open 87 Closed
26 Open 87 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Does vlmeval support multi card inference and batch size > 1? Feature Request
#32 opened Dec 28, 2023 by John-Ge updated Jan 27, 2024
7
There is a long gap between the validation accuracy of the dataset of vlmevalkit and the model paper
#94 opened Feb 23, 2024 by YongLD updated Mar 13, 2024
6
How are models that use in-context examples handled? Feature Request
#90 opened Feb 7, 2024 by sachit-menon updated Apr 15, 2024
4
The results of instructblip 13b
#187 opened Apr 29, 2024 by YongLD updated Apr 29, 2024
best to have docker image
#190 opened May 3, 2024 by max-yue updated May 7, 2024
1
requirements.txt 不全
#210 opened May 25, 2024 by max-yue updated May 28, 2024
1
加载本地训练模型
#252 opened Jul 4, 2024 by 2499048212 updated Jul 5, 2024
5
[Request] Consider integrating the following list?
#257 opened Jul 5, 2024 by sarapieri updated Jul 5, 2024
grounding数据支持
#219 opened Jun 12, 2024 by sxlyiyiyi updated Jul 8, 2024
1
error when eval MiniCPM_Llama3_V2.5 on MMMU_TEST
#250 opened Jul 3, 2024 by ruifengma updated Jul 9, 2024
1
Does the generating params changed?
#269 opened Jul 10, 2024 by lucasjinreal updated Jul 11, 2024
4
Difference between MMBench_DEV and MMBench_Test
#274 opened Jul 11, 2024 by royzhang12 updated Jul 11, 2024
1
【提问】关于其他测试集的支持 Feature Request
#97 opened Feb 28, 2024 by mary-0830 updated Jul 17, 2024
4
[rank0]: KeyError: 'answer'
#268 opened Jul 10, 2024 by lucasjinreal updated Jul 18, 2024
7
[Question] Dataset not officially supported ?
#299 opened Jul 19, 2024 by amitbcp updated Jul 20, 2024
1
Suspected error?
#308 opened Jul 23, 2024 by Shijinghang updated Jul 24, 2024
1
评测76B模型在设置device map为auto后依然有报错
#314 opened Jul 25, 2024 by ZhiyuYUE updated Jul 26, 2024
2
MuirBench Integration
#321 opened Jul 27, 2024 by amitbcp updated Jul 27, 2024
1
Audio modality evaluation
#313 opened Jul 25, 2024 by Simplesss updated Jul 27, 2024
1
[Help Wanted] Supporting the chat_inner API for existing VLMs. help wanted Extra attention is needed
#323 opened Jul 27, 2024 by kennymckormick updated Jul 29, 2024
2
InternVL2 Error
#289 opened Jul 17, 2024 by amitbcp updated Jul 30, 2024
5
RuntimeError: cutlassF: no kernel found to launch!
#330 opened Jul 30, 2024 by liuchunming033 updated Jul 30, 2024
1
Regarding the addition of other video datasets
#334 opened Jul 30, 2024 by mary-0830 updated Jul 31, 2024
2
Custom dataset is not considered MCQ by models
#336 opened Jul 31, 2024 by zodiacg updated Aug 1, 2024
可以加入Mantis模型的评测吗 相关模型如下
#342 opened Aug 1, 2024 by BrenchCC updated Aug 1, 2024
ProTip! Type g i on any issue or pull request to go back to the issue listing page.