-
Notifications
You must be signed in to change notification settings - Fork 615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BLIP Image Captioning GradCAM? #155
Comments
Hi, you can look at our code in LAVIS, which provides gradcam computation function for BLIP image-text matching model |
Does it mean, only image-text matching model can perform gradcam? If it only supports image-text matching model, do I need to make another image-text matching model for gradcam? |
You can adapt the gradcam code to work with an image captioning model. |
Thank you I will try it. |
Hi, I am also working on the visualization that goes beyond the image-text matching model, and I've encountered some difficulties when calling 'attn_gradients' and 'attention_map'. Have you had any success with this and if so can you share the code or provide some guidance? Thank you very much! |
Sure if I solve it, I will let you know. |
Did you manage to solve this? |
Hi, I used BlipForConditionalGeneration from transformers for image captioning.
I want to visualize the reason of generated caption (word by word) like GradCAM.
I found a code from Albef (https://github.com/salesforce/ALBEF/blob/main/visualization.ipynb), but it used an image-text matching model, not image captioning model.
Can you give me any hints or simple codes for this?
The text was updated successfully, but these errors were encountered: