1 Release published by 1 person

• v0.2rc1

  published Jun 29, 2024

41 Pull requests merged by 13 people

• [Doc] Update readme

  #297 merged Jul 19, 2024

• [Model] Support Llava-Next-Interleave

  #294 merged Jul 19, 2024

• [Update] meta_data

  #295 merged Jul 19, 2024

• [Benchmark] Support MATH-Vision

  #292 merged Jul 19, 2024

• [Fix] fix minor typo

  #291 merged Jul 18, 2024

• Added blink

  #281 merged Jul 18, 2024

• Update bunnyllama3.py

  #287 merged Jul 17, 2024

• Customize prompt for Bunny

  #286 merged Jul 16, 2024

• [Update] update dataset init class

  #285 merged Jul 16, 2024

• Update leaderboard

  #284 merged Jul 15, 2024

• Fix dataset dist init & minor typo

  #283 merged Jul 15, 2024

• Change the displaying of the Dataset Name

  #279 merged Jul 14, 2024

• Support MMLongBench_DOC

  #276 merged Jul 13, 2024

• Refactor branch

  #278 merged Jul 13, 2024

• docs: update README.md

  #277 merged Jul 12, 2024

• include VCR to VLMEvalKit

  #258 merged Jul 12, 2024

• fix api model inference

  #275 merged Jul 12, 2024

• [Fix] Fix #272

  #273 merged Jul 11, 2024

• Update result link

  #267 merged Jul 10, 2024

• [Fix] Update MMBench-Video Prompt for no-pack setting

  #266 merged Jul 9, 2024

• [Improvement] Refactor Dataset

  #265 merged Jul 9, 2024

• Update README

  #262 merged Jul 8, 2024

• add internlm-xcomposer2.5

  #261 merged Jul 8, 2024

• Update Readme

  #260 merged Jul 8, 2024

• Support InternVL2 series

  #259 merged Jul 8, 2024

• [API] Support SenseChat-5-Vision

  #255 merged Jul 5, 2024

• Update transformers version & dataset type

  #254 merged Jul 5, 2024

• update prompts for cogvlm & GLM4v-9B

  #251 merged Jul 4, 2024

• add Aesbench test

  #246 merged Jul 1, 2024

• [Benchmark] Support AesBench_TEST

  #248 merged Jul 1, 2024

• Add AesBench VAL

  #240 merged Jun 27, 2024

• Add cambrian model

  #243 merged Jun 27, 2024

• [Dataset] Add config for MLLMGuard_DS

  #242 merged Jun 27, 2024

• Update video dataset import

  #241 merged Jun 27, 2024

• [Dataset] Support MMBench-Video

  #229 merged Jun 26, 2024

• support internvl_v1.5 for Interleave image input

  #238 merged Jun 25, 2024

• [Fix] Switch to GPT-3.5-Turbo-0125 for Choice Extraction

  #234 merged Jun 22, 2024

• Add chinese quickstart

  #233 merged Jun 22, 2024

• docs: add README in Japanese

  #232 merged Jun 22, 2024

• update tools

  #231 merged Jun 21, 2024

• Add CN development guide

  #230 merged Jun 21, 2024

4 Pull requests opened by 3 people

• support MMDU benchmark

  #249 opened Jul 1, 2024

• Add Support for Video-MME

  #270 opened Jul 10, 2024

• [Update] update development doc

  #296 opened Jul 19, 2024

• [Support] multiple process parallel inference large model on multi-gpu

  #298 opened Jul 19, 2024

21 Issues closed by 7 people

• Error about vlmeval/dataset/utils/multiple_choice.py

  #290 closed Jul 19, 2024

• Evaluation of custom models and datasets.

  #108 closed Jul 19, 2024

• 支持LMDeploy作为推理后端

  #179 closed Jul 19, 2024

• ChartQA augmented & CMMMU

  #69 closed Jul 19, 2024

• Support Blink Dataset

  #171 closed Jul 19, 2024

• How to create OpenVLM.json

  #288 closed Jul 18, 2024

• Missing `dataset_config.py`

  #282 closed Jul 15, 2024

• [BUG] local variable 'structs' referenced before assignment

  #271 closed Jul 12, 2024

• [bug] The same sample was run twice in mp_utils.py

  #272 closed Jul 11, 2024

• 如何多卡并行测评？How to evaluate on multiple cards in parallel?

  #263 closed Jul 10, 2024

• 如何获得 opencompass 上评测模型的最新细分项结果？

  #264 closed Jul 10, 2024

• [Request]Consider integrating MMT-Bench and CONTEXTUAL?

  #196 closed Jul 9, 2024

• Math_Vista result is not summarized

  #253 closed Jul 8, 2024

• SEEDBench2 and SEEDBench2_Plus give the same result

  #247 closed Jul 1, 2024

• Image embeddings

  #245 closed Jun 30, 2024

• AesBench support

  #222 closed Jun 27, 2024

• evaluate TextVQA_VAL error

  #237 closed Jun 27, 2024

• Judge model seems not working

  #236 closed Jun 26, 2024

• MMBench_EN_V11 404 not found after evaluation

  #239 closed Jun 26, 2024

• about vlmeval/evaluate/multiple_choice.py

  #227 closed Jun 20, 2024

• about vlmeval/evaluate/multiple_choice.py

  #228 closed Jun 20, 2024

8 Issues opened by 6 people

• Request Support for VILA

  #293 opened Jul 18, 2024

• InternVL2 Error

  #289 opened Jul 17, 2024

• Difference between MMBench_DEV and MMBench_Test

  #274 opened Jul 11, 2024

• Does the generating params changed?

  #269 opened Jul 10, 2024

• [rank0]: KeyError: 'answer'

  #268 opened Jul 10, 2024

• [Request] Consider integrating the following list?

  #257 opened Jul 5, 2024

• 加载本地训练模型

  #252 opened Jul 4, 2024

• error when eval MiniCPM_Llama3_V2.5 on MMMU_TEST

  #250 opened Jul 3, 2024

2 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• 【提问】关于其他测试集的支持

  #97 commented on Jul 17, 2024 • 0 new comments

• grounding数据支持

  #219 commented on Jul 8, 2024 • 0 new comments