We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When i use default setting to initiate llava v1.5 7b model evaluation on SEEDBench_IMG dataset , I got results like this :
I checked intermediate results , but the model seems to generate options correctly :
The default generation config should be
And the officially reported results should be
It's really weird . I don't know why there is a huge gap here. Hope to get help. Thank you in advance!
The text was updated successfully, but these errors were encountered:
No branches or pull requests
When i use default setting to initiate llava v1.5 7b model evaluation on SEEDBench_IMG dataset , I got results like this :
![屏幕截图 2024-01-04 211703](https://private-user-images.githubusercontent.com/47048568/294190501-cea21a88-da42-4f79-bac2-6ab70f5eb38c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MDY2OTgsIm5iZiI6MTcyNDUwNjM5OCwicGF0aCI6Ii80NzA0ODU2OC8yOTQxOTA1MDEtY2VhMjFhODgtZGE0Mi00Zjc5LWJhYzItNmFiNzBmNWViMzhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDEzMzMxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTYwMjg3MzExNTE4Mjk0NDhjM2E5MzE3MDM4MjIzYjUzMjQ2ZTE1OTY1N2EzNjkyZWQzMmE5MzdiOGZhY2IwZGUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.epE9JvkgeF4hqO6mTThliDFjPloS96fTbKwx1GZGjqo)
I checked intermediate results , but the model seems to generate options correctly :
![屏幕截图 2024-01-04 212152](https://private-user-images.githubusercontent.com/47048568/294191216-195b9900-92fa-43d2-8476-0ab354257488.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MDY2OTgsIm5iZiI6MTcyNDUwNjM5OCwicGF0aCI6Ii80NzA0ODU2OC8yOTQxOTEyMTYtMTk1Yjk5MDAtOTJmYS00M2QyLTg0NzYtMGFiMzU0MjU3NDg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDEzMzMxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyNTkyNGQ2YmU4YmE0NWY0YTAzM2NmNzkxOTIwMTMyZmQyNWIwZTA0NzM5OGVhNDg5ZWQ4ZGVjM2UxZGZkODAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0._Y9LmvxAC2nY9VpQD8htKioH11b1LuMvcfLFyyQqFwI)
The default generation config should be
And the officially reported results should be
![屏幕截图 2024-01-04 212602](https://private-user-images.githubusercontent.com/47048568/294192225-fef77989-4b92-40b8-8aa8-3145bb1e7f33.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1MDY2OTgsIm5iZiI6MTcyNDUwNjM5OCwicGF0aCI6Ii80NzA0ODU2OC8yOTQxOTIyMjUtZmVmNzc5ODktNGI5Mi00MGI4LThhYTgtMzE0NWJiMWU3ZjMzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDEzMzMxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQxY2Q0NzY3NGNiZjE1YTYzMGY2YTNmOTUwMzliMDRmMDhiNzNiMGQ3OWYwNTNjNmZlYjVmNWM2YzE5MjdmNTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.J_fe90-dzHT6pyDloFSY9GHJYijrIBY7ubTyyfgm2jk)
It's really weird . I don't know why there is a huge gap here. Hope to get help. Thank you in advance!
The text was updated successfully, but these errors were encountered: