Any plan to support cascading feature of flashinfer?

[wellhowtosay]

No description provided.

[Qubitium]

For context this is what @wellhowtosay is referring to:

https://docs.flashinfer.ai/api/python/cascade.html

[wellhowtosay]

For context this is what @wellhowtosay is referring to:

https://docs.flashinfer.ai/api/python/cascade.html

I tried to implement cascading in sglang on my own, but I just found it's beyond my capability, especially to maintain a shared prefix kv-cache and a unique kv data in a batch.

[Ying1123]

Yes, it is in our plan. However, it is not the highest priority because, in most cases, the bottleneck of long shared-prefix originates from the prefill stage, which can be optimized through automatic prefix sharing with warmup if not sharing across batches. That being said, if you have a batch of requests sharing a long prefix, you can process one request first, and then the radix attention will skip the computation for all the subsequent requests.
However, cascade inference can offer substantial advantages in some scenarios (long shared prefix, short decode, and the bottleneck still being on decode); therefore, we are working on it. I would say the implementation is not trivial, as it requires scheduling multiple kernel launches according to the tree structure of prefix sharing. I will let you know when I have an initial draft, so that you can review and help improve it.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.