-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect preprocessing for ImageNet-C evaluation #59
Comments
The suggestion of fixing |
Sorry, took us a long time. Should be fixed with this PR: #85. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I see that the ImageNet-C evaluation uses the preprocessing:
Resize(256)+CenterCrop(224)+ToTensor()
.robustbench/robustbench/data.py
Lines 146 to 154 in 61ce9e9
This causes discrepancies with the scores reported in the original papers (DeepAugment, AugMix, Standard RN-50). The ImageNet-C dataset already contains 224x224 images and hence only
ToTensor()
should be used for consistency.Fixing
prepr='none'
inload_imagenetc
should solve the issue (assuming all the models are capable of handling 224x224 images as input).The text was updated successfully, but these errors were encountered: