llava_v1.5_7b wrong results on Seedbench_IMG

[BonitoW]

Hi,

I have checked the saved result of llava7b on the seedbench_image benchmark and found that: for some problems, the llava7b can give the right prediction but in the evaluation framework it shows that it will return wrong answers.

For example, for index = 1198,4307

Which object is likely found in the boy's hand? A: A book B: A soccer ball C: A calculator D: A pencil

Where is the priest located in the image? A: In front of the stained glass window B: To the right of the bride and groom C: To the left of the bride and groom D: Behind the bride and groom

Can you help explain this? Thanks for your help!

[kennymckormick]

Hi, @BonitoW ,
Honestly, I'm not sure about the reason. Do you mean you got the right prediction when directly infer llava_v1.5_7b with those two questions? If so, I'll give a try.

[kennymckormick]

Hi, @BonitoW ,
any updates?