IndexError: index 1 is out of bounds for dimension 0 with size 1

[lucasjinreal]

cur_image_features = image_features[cur_image_idx]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: index 1 is out of bounds for dimension 0 with size 1

Is there any reason why previous version can evaluate normally, but updated from git got such error?

And this error happens only during evaluation at about 6%

Is there zero images samples in eval MMBench?

[kennymckormick]

Hi, @lucasjinreal ,
Which VLM are you evaluating? I think MMBench does not have zero-image samples. As an ablation, you can try to evaluate some other VLMs like qwen_chat on MMBench to localize the bug.
BTW, VLMEvalKit has recently updated its generate interface (check #140). I'm not sure if this error is due to the refactoring.

[lucasjinreal]

Am not sure as well, but just got this, I used llava model with qwen as LLM. The task is MMBenchCN

I used a try ... catch to generate_inner, not it can run, but looks like would effect the metric

[kennymckormick]

@lucasjinreal
Please provide a command so that I can try to reproduce this problem.

[lucasjinreal]

Sorry, the exception got is on MMMU, not mmbench.

[icrto]

I am also experiencing the same problem when trying to run on MMMU. (I tried it with llava_v1.5_7b.)

[lucasjinreal]

@kennymckormick Can u take a look, actually didn't get this in previous version, some models actually didn't support more than 2 images in a single chat like llava.

[kennymckormick]

@kennymckormick Can u take a look, actually didn't get this in previous version, some models actually didn't support more than 2 images in a single chat like llava.

OK, I will give a try for the llava_v1.5_7b x MMMU configuration.

[kennymckormick]

Hi, @icrto @lucasjinreal
After investigation I find this is a bug due to the refactor of interface. We have fixed it now. Please have a retry with the latest main branch.