Yet Another V2 Assistant Streaming Implemenetation #748
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements Assistant Streaming for the OpenAI API V2 (BETA).
https://platform.openai.com/docs/api-reference/assistants-streaming
Key Features
This PR is built on the excellent work of #737. The main departure from @coolbaluk's work is to improve the streaming response type
AssistantStreamEvent
to handle the polymorphic nature of the stream events better.Work in progress. Would love to seek feedback & suggestions from the community.
Tagging:
@coolbaluk #737
@tanzyy96
@CallOrRet #731
@sashabaranov
Stream Events as Scanner
The StreamerV2 struct
Scan
stream events in a loop as they are received from the server.Unlike the current implemenetation of the
CompletionStreaming
, he events in the stream are polymorphic and are handled naturally through type switching.The
Scan
API mimics thebufio.Scanner
class:Unwrappers for Polymporphic Events
Type switching on stream events allows precise and type-safe access. However, it can be tedious to write user-level code for simpler cases. To address this, unwrapper helpers "cast" an event to a specific type and return the value if the cast is successful:
This is similar to the Bytes and Text methods in bufio.Scanner, which provide access to the current item in different forms:
Libraries designed to handle polymorphic data often provide similar unwrapper helpers. For example, the [gjson]https://github.com/tidwall/gjson?tab=readme-ov-file#result-type) library, offer these unwrapper methods for the polymorphic
Result
type:Stream Events as io.Reader
The most common use case is to just stream the text, and it would be nice to
have a familiar & obvious API for this.
The
StreamerV2
struct implements theio.Reader
interface, which wraps theNext/Scan
API, to provide a simple way read the text deltas from thethread.message.delta
events.TODO Items
Stream
into other methods likeCreateThreadAndStream
.Type()