Releases: open-compass/VLMEvalKit
Releases · open-compass/VLMEvalKit
v0.2rc1: [Dataset] Add AesBench VAL (#240)
* Add files via upload * update aesbench * update init * update dataset config * update md5 * update --------- Co-authored-by: kennymckormick <[email protected]>
VLMEvalKit v0.1
What's Changed
- [Feature] Support multi_generate by @kennymckormick in #1
- [Tool] Minor updates 1205 by @kennymckormick in #3
- TranCore-M 20231208 by @PCIResearch in #8
- [Result] update TransCore Results by @kennymckormick in #9
- COREMM Evaluation Benchmark by @youngfly11 in #6
- Adding MMVet by @llllIlllll in #7
- [Doc] README Update 12.11 by @kennymckormick in #10
- Updating mmvet_eval by @llllIlllll in #11
- [Feature] Add run.py and simplify the evaluation. by @kennymckormick in #12
- [Doc] Refine README by @kennymckormick in #13
- [Fix] Fix README by @kennymckormick in #14
- add dataset md5 check integrity by @FangXinyu-0913 in #15
- [Feature] Support two Vision APIs by @kennymckormick in #19
- Add COCO dataset by @FangXinyu-0913 in #16
- [Fix] Fix 1221 by @kennymckormick in #21
- [Feature] More Robust API Evaluation by @kennymckormick in #22
- [Refactor] Refactor Custom Prompt & Fix mPLUG-Owl2 acc by @kennymckormick in #23
- [Dataset] VQA Datasets by @kennymckormick in #25
- [Fix] Bug Fix by @kennymckormick in #26
- Add MMMU dataset by @llllIlllll in #18
- Add QwenVLPlus API by @llllIlllll in #27
- [Result] Update MMMU Acc by @kennymckormick in #30
- [Feature] Support
LLaVA_XTuner
models by @LZHgrla in #17 - [Result] Update XTuner Performance by @kennymckormick in #31
- [Result] Update COCO Caption Results by @kennymckormick in #35
- add dataset ChartQA by @FangXinyu-0913 in #28
- [Feature]: Add ScienceQA by @YuanLiuuuuuu in #24
- [Dataset] MathVista dataset by @llllIlllll in #29
- [Dataset] HallusionBench by @kennymckormick in #38
- Add sharedcaptioner and cogvlm by @fitzpchao in #37
- [Fix] Fix GPT error with parallel calling by @kennymckormick in #40
- [Eval] multiple_choice.py: E->Z by @kennymckormick in #41
- [Eval] Use exact matching for Y/N and multi-choice when OPENAI_API_KEY not set by @kennymckormick in #44
- add VLM: monkey by @ShuoZhang2003 in #45
- [Fix] Fix multiple choice evaluation when OPENAI_API_KEY missing by @kennymckormick in #48
- [Improvement] Support non-contiguous choices by @kennymckormick in #49
- Add Emu2 and Emu2_chat by @llllIlllll in #47
- [Dataset] Add DocVQA by @llllIlllll in #50
- [Benchmark] support AI2D by @kennymckormick in #51
- add monkey-chat by @ShuoZhang2003 in #54
- Add
LLaVA-InternLM2
by @LZHgrla in #53 - [Dataset] Support LLaVABench by @kennymckormick in #55
- Support torchrun for emu2&emu2_chat and fix bug by @llllIlllll in #52
- support sharegpt4v-13b by @xiaoachen98 in #56
- Fix bug in file.py by @Ezra-Yu in #58
- [Result] Update Evaluation Results by @kennymckormick in #60
New Contributors
- @kennymckormick made their first contribution in #1
- @PCIResearch made their first contribution in #8
- @youngfly11 made their first contribution in #6
- @llllIlllll made their first contribution in #7
- @FangXinyu-0913 made their first contribution in #15
- @LZHgrla made their first contribution in #17
- @YuanLiuuuuuu made their first contribution in #24
- @fitzpchao made their first contribution in #37
- @ShuoZhang2003 made their first contribution in #45
- @xiaoachen98 made their first contribution in #56
- @Ezra-Yu made their first contribution in #58
Full Changelog: https://github.com/open-compass/VLMEvalKit/commits/v0.1