Is it possible to define the prompts for KV caching up-front?

[timothylimyl]

For a lot of use cases, there is already a pre-defined system + base prompt that is used.

Can we define the KV cache for these prompts up front manually? For example, if we are extracting information out of a provided context, the provided context prompt changes but the system + base prompt stays the same. Caching the context will make no sense as it is guaranteed to change on the next inference.

[timothylimyl]

Add-on question is how are you currently deciding on how to persist the cache?

I think this should be user defined. For example, typically I can guaranteed that my application calls the LLM in XX numbers of ways (whether is it agentic workflow, RAG, etc). For the XX numbers of ways, it will be best to KV cache all of these different system and base prompts.

If i can define this up-front, then I can also gauge the memory usage.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.