New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot reproduce llava v1.5 7b SEEDBench_IMG results #42

Closed

MagicGeek2 opened this issue Jan 4, 2024 · 0 comments

MagicGeek2 commented Jan 4, 2024

When i use default setting to initiate llava v1.5 7b model evaluation on SEEDBench_IMG dataset , I got results like this :

I checked intermediate results , but the model seems to generate options correctly :

The default generation config should be

do_sample=True
temperature=0.2
max_new_tokens=512
top_p=None
num_beams=1

And the officially reported results should be

It's really weird . I don't know why there is a huge gap here. Hope to get help. Thank you in advance!

MagicGeek2 closed this as not planned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment