-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wondering about attention #1
Comments
Hi @yanivnik Thank you for pointing this out. This was not intentional on my part, and I appreciate you bringing it up. I did not thoroughly compare the placement for the self-attention layer, so am not sure if my version results in decreased sample quality. If you find that it does, I would be happy to modify the code to better reflect what is done in the original work. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
First of all - Thanks for the great implementation, which is very readable and simple to use and reach quick results with.
I noticed you're using the self attention in a different manner than the one written by Ho et al - they used the self attention in the 16x16 resolution layers in the network (both for 32x32 and 256x256 inputs) while you used it in the 8x8 resolution layers in the U-Net. Did you just happen to use it there / was there some guiding logic behind it?
The text was updated successfully, but these errors were encountered: