-
Notifications
You must be signed in to change notification settings - Fork 981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
block-sparse flash attention support #851
Labels
Comments
Hello, |
@natek-1 Welcome! Thank you for your contribution. |
Hey @natek-1 , do you have any updates on this? It's totally alright if you haven't gotten a chance to look at it. Would it be alright if we assigned it to someone else? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I saw flash attention was recently merged.
This approximate attention would be cool to have as well for training very large sequence lengths. https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/flash_blocksparse_attention.py
The text was updated successfully, but these errors were encountered: