Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
26 Open 140 Closed
26 Open 140 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

MME-RealWorld-CN的评测集适配问题
#491 opened Sep 27, 2024 by ManiiXu updated Sep 27, 2024
1
Qwen2-VL-2B-Instruct与榜单结果对不齐
#492 opened Sep 27, 2024 by helloworld01001 updated Sep 27, 2024
7B模型在4*A100 80GB上发生OOM
#382 opened Aug 13, 2024 by Rocky77JHxu updated Sep 25, 2024
4
how to run on multi-gpu with device_map='auto'
#482 opened Sep 21, 2024 by qianwangn updated Sep 23, 2024
1
Reproducing QWen2VL Results on Video Benchmarks with VLMEvalKit
#484 opened Sep 23, 2024 by aniki-ly updated Sep 23, 2024
@FangXinyu-0913
liuhaotian/llava-v1.6-vicuna-7b 评测时报错
#480 opened Sep 20, 2024 by Cooperx521 updated Sep 23, 2024
1
[Request] Consider integrating the following list? help wanted Extra attention is needed
#257 opened Jul 5, 2024 by sarapieri updated Sep 23, 2024
2
【提问】关于语音模型的评测 help wanted Extra attention is needed
#473 opened Sep 19, 2024 by xx0412 updated Sep 20, 2024
1
grounding数据支持
#219 opened Jun 12, 2024 by sxlyiyiyi updated Sep 20, 2024
2
About Tablevqabench score
#442 opened Sep 4, 2024 by mary-0830 updated Sep 16, 2024
@FangXinyu-0913
1
best to have docker image help wanted Extra attention is needed
#190 opened May 3, 2024 by max-yue updated Sep 16, 2024
2
ChatAPI - ERROR - HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/chat/completions (Caused by ProxyError('Unable to connect to proxy', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f481233f640>: Failed to establish a new connection: [Errno 111] Connection refused')))
#406 opened Aug 25, 2024 by yansuoyuli updated Sep 14, 2024
3
评测时PermissionError: [Errno 13] Permission denied
#454 opened Sep 10, 2024 by 40459447 updated Sep 13, 2024
1
Input length of input_ids is 0, but max_length is set to -2009.
#455 opened Sep 10, 2024 by wangli68 updated Sep 10, 2024
关于llava-v1.5-7b复现
#391 opened Aug 19, 2024 by helloworld01001 updated Sep 4, 2024
3
Audio modality evaluation help wanted Extra attention is needed
#313 opened Jul 25, 2024 by Simplesss updated Aug 29, 2024
1
KeyError: 'answer'
#378 opened Aug 11, 2024 by lucasjinreal updated Aug 28, 2024
19
关于幻觉benchmark的支持 help wanted Extra attention is needed
#384 opened Aug 15, 2024 by shufangxun updated Aug 28, 2024
1
CMMMU数据集支持 help wanted Extra attention is needed
#398 opened Aug 21, 2024 by DietDietDiet updated Aug 28, 2024
3
Inconsistent max_new_tokens
#356 opened Aug 3, 2024 by amitbcp updated Aug 10, 2024
2
[Help Wanted] Supporting the chat_inner API for existing VLMs. help wanted Extra attention is needed
#323 opened Jul 27, 2024 by kennymckormick updated Jul 29, 2024
2
【提问】关于其他测试集的支持 Feature Request
#97 opened Feb 28, 2024 by mary-0830 updated Jul 17, 2024
4
The results of instructblip 13b
#187 opened Apr 29, 2024 by YongLD updated Apr 29, 2024
How are models that use in-context examples handled? Feature Request
#90 opened Feb 7, 2024 by sachit-menon updated Apr 15, 2024
4
There is a long gap between the validation accuracy of the dataset of vlmevalkit and the model paper
#94 opened Feb 23, 2024 by YongLD updated Mar 13, 2024
6
ProTip! no:milestone will show everything without a milestone.