-
Notifications
You must be signed in to change notification settings - Fork 282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VICRegL error when grid size is too big #1179
Comments
Hi, our VICRegL implementation makes the assumption that the features passed to the loss have the same size as the grid. So if you are using a resnet18 with 224px input images you get 7x7xdim output features as resnet downsamples inputs by a factor 32. This is why the grid size in the transform is set to 7. Similarly, for the local views we have 96px inputs and grid size 3 as 96/32 = 3. What we probably should do is add a check in the forward pass of the loss that verifies that the number of features and grid size are equal. |
Oh, I see! Thank you! Let's say we have resnet18 and a 224x96 global view instead of 224x224; in that case, the grid should be 7x3, right? |
Yes :) Although grid size is currently fixed to a single number (same width as height), so 7x3 won't be possible. |
Certainly, I had to override it for 3D anyway, so not an issue 😁 thank you so much! Would it be useful to catch the error thoiguh? Simply checking if the max value of |
I believe we should test in vicregloss forward that the features have the same dimension as the grid: lightly/lightly/loss/vicregl_loss.py Line 97 in 00461d1
PR would be very much appreciated! |
Reproduce
Run https://docs.lightly.ai/self-supervised-learning/examples/vicregl.html with
transform = VICRegLTransform(n_local_views=0, global_grid_size=9)
(default isglobal_grid_size=7
).If you run it on GPU, you'll get a very ugly CUDA error that doesn't say much. If you try it on CPU, you can backtrace it.
Description
Calculating
nearest_neighbors
inVICRegLLoss
's_nearest_neighbors_on_l2
and_nearest_neighbors_on_grid
causes the above error. What happens is that the calculated indices inmin_indices
are sometimes out of bound for the given input:lightly/lightly/models/utils.py
Line 518 in dda9e84
That's as far as I've gotten, I haven't looked further into it and do not understand what exactly is going on. I believe that your implementation is correct, but it would be useful to catch the error when the specified grid size is too big and provide a useful error message to the user.
The text was updated successfully, but these errors were encountered: