How to cancel a response stream 'mid-stream'? #17
Comments
|
This is definitely a use case for a CancellationToken. Implementing this to exit the stream is easy to do, however I am not sure if the Ollama API will just continue to stream then. Going to take a look. |
|
Amazing - I just did the research and "dusted off" the little bit of Go I know - seems you're several steps ahead of me putting in a PR! 🙏 🚀 (FWIW - I had questions about whether the HTTP context 'transfer' would actually trigger the SIGINT) |
|
I just pushed the new version 1.1.0 that allows an optional |
|
I asked around and got the impression that the HTTP context will "handle itself" which always makes me pull back a bit. Someone originally suggested just closing and re-opening the connection and that felt too ham-handed but I'd be willing to do it as an interim step. This feels much more harmonious to the original intent. Thanks again! |
|
I don't think you'll have to do more than pulling the trigger on a cancellation token. I mean you could simply use a |
|
Oh yeah - I totally get it now - I was just replaying my hand-wringing when I was first looking at this without having "pierced the veil" of looking at the Go code and making a few educated guesses. 😺 |
|
Really nice work - passing CT to the Ollama instance running on the server is at the end of this little walk-thorugh. There's a bit of lag in reporting out the server GPU but it's definitely doing what's expected. Screen.Recording.2024-01-13.at.11.56.48.AM.movThanks again! |
I've been looking at the code for Ollama and working with its console for a bit and trying to get my head wrapped around how to perform the .NET equivalent of
Ctrl+Cin the API. Is there such a command?The goal that I'm seeking is to have the ability to "stop" the current stream of text for a given response so that it can then be ready to take in another request from the API.
This is that use case where the user is like "oh wait this is the wrong way to go" and wants to shut off the response so a new message can be sent. The optimal scenario would be a cancellation token or a specific command that can be sent to shut down the current response. I have thought about simply shutting down the current connection and re-connecting but that seems pretty inelegant.
I like the stream response. I most definitely want to use it. Any thoughts on "shunting" or "stopping" a stream? Thanks!
The text was updated successfully, but these errors were encountered: