StreamingResponse

[franzwilding]

In order to have a good LLM chat UX, we need to streame the response to the client. Langserve is doing this with an dedicated endpoint, hayhooks could do the same (pseudocode):

async def pipeline_stream(pipeline_run_req: PipelineRunRequest) -> StreamingResponse:
        buffer = ...
        result = pipe.run(data=pipeline_run_req.dict())

        return StreamingResponse(buffer_generator)

app.add_api_route(
        path=f"/{pipeline_def.name}/stream",
        endpoint=pipeline_stream,
        methods=["POST"],
        name=pipeline_def.name,
        response_model=PipelineRunResponse,
    )

Additionally haystack should provide a special streaming_callback that will write the chunk content to a buffer, that will be available to hayhooks. Maybe the Pipeline could add this logic and provides an pipe.stream method that will return a generator or simething like this.

[vblagoje]

Yes @franzwilding we have this item on our roadmap, thanks for raising this issue and voicing your preferred solution.

[Phlasse]

@vblagoje any idea yet, when this feature will become available? We are using haystack in quite some projects now and want to know if it is worth putting more energy in our work around solution or if we can expect proper streaming out of a pipeline soon :) ?

[vblagoje]

Yes, I understand totally! The support is currently being worked on 😎