Modified streaming setup to use QueryConfig #214
Conversation
…feat/streaming-response
|
reviewing this |
| - To use this, instantiate App with a `InitConfig` instance passing `stream_response=True`. The following example iterates through the chunks and prints them as they appear | ||
| ```python | ||
| app = App(InitConfig(stream_response=True)) | ||
| resp = naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?") |
There was a problem hiding this comment.
app instance is called app not naval_chat_bot (common mistake)
|
|
||
| - To use this, instantiate App with a `InitConfig` instance passing `stream_response=True`. The following example iterates through the chunks and prints them as they appear | ||
| ```python | ||
| app = App(InitConfig(stream_response=True)) |
There was a problem hiding this comment.
can we keep the parameter name as "stream"
stream is self explanatory and implies that response will be streamed.
There was a problem hiding this comment.
fine with me. I wish there was a name that implied a boolean action, maybe is_stream?
README.md
Outdated
|
|
||
| ```python | ||
| app = App() | ||
| query_config = QueryConfig(stream_response = True) |
There was a problem hiding this comment.
lets call the arg as "stream"
| query_config = QueryConfig(stream_response = True) | ||
| resp = naval_chat_bot.query("What unique capacity does Naval argue humans possess when it comes to understanding explanations or concepts?", query_config) | ||
|
|
||
| for chunk in resp: |
There was a problem hiding this comment.
should we give a downstream handler? assuming that a lot of users will use streaming response, so why not there is a function which when called gives the streaming response output.
@cachho @aaishikdutta
There was a problem hiding this comment.
not everyone's going to want to just print it. I haven't looked into how I can get route the stream through flask, but that's my usecase for instance, so no, I'm fine with it how it is.
Streaming is for people that know a little. Not for private users that want to play around. If you want to use it professionally you should be able to copy two lines from the readme.
There was a problem hiding this comment.
Okay, then for now lets leave this.
Removed streaming_option from InitConfig and added it at a Query level.
Added some internal refactors