Can we get sparse attention working with A100s / CUDA 11?

[sdtblck]

This one will probably be a tonne of work, and i have no idea where to start, but it seems deepspeed's sparse attention only works with cuda 10.x, and only on specific gpu architectures.

It would be great to have a sparse attention implementation that works with our setup, or to fix deepspeed's.

according to the blocksparse dev (pretty sure deepspeed's sparse attn is based on this) we can try using triton.ops.blocksparse (ptillet/torch-blocksparse#38) ?

[sdtblck]

looks like deepspeed is updating their triton support, so maybe integrating this will fix it microsoft/DeepSpeed#902

[sdtblck]

Should work now after installing this Deeperspeed commit EleutherAI/DeeperSpeed@04a52ad