Error when evaluating MMBench_TEST_EN

[KosumosuL]

Looks like the annotation in MMBench_TEST_EN does not contains the key "answer"? Yet the prefetch_acc has to access this key, as shown in

VLMEvalKit/vlmeval/inference.py

Line 156 in 7f099c1

if matched == item['answer']:

[YongLD]

Because The test_file have no key "answer", I think they do not add the answer of the choise in the file MMBENCH_TEST_EN.tsv.

[kennymckormick]

Hi, @KosumosuL ,
The label of testing samples is confidential, to obtain the evaluation accuracy of MMBench test split, please upload it to the evaluation service of send it to [email protected]

[KosumosuL]

Hi, @KosumosuL , The label of testing samples is confidential, to obtain the evaluation accuracy of MMBench test split, please upload it to the evaluation service of send it to [email protected]

I can understand that. My point is, however, the code should consider this situation for avoiding this error, or the readme should point this out.

[kennymckormick]

Hi, @KosumosuL , The label of testing samples is confidential, to obtain the evaluation accuracy of MMBench test split, please upload it to the evaluation service of send it to [email protected]

I can understand that. My point is, however, the code should consider this situation for avoiding this error, or the readme should point this out.

Hi, @KosumosuL , that's a good point, I have fixed the problem in this commit: e992046, the evaluation skip for MMBench_TEST will be skipped on any non-official servers. You can try again on the latest main branch.