[serve] Add experimental support for `StreamingResponse` using `RayObjectRefGenerator` #35720

edoakes · 2023-05-24T16:30:03Z

Why are these changes needed?

Adds experimental support for using StreamingResponses to stream intermediate results back to the client. This is currently gated behind a feature flag (must set RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1.

This is implemented by using the Ray ObjectRefStreamingGenerator interface. When the feature flag is on, the HTTP proxy will use num_returns="streaming" for all calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy.

Known limitations & follow-ups (to be addressed before a non-experimental release):

Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on [serve] Support for streaming responses #35468). Most of this should be able to be optimized away before turning this on by default.
Streaming is not yet possible using the ServeHandle interface: [serve][streaming] Support streaming w/ model composition (using ServeHandle) #35777
max_concurrent_queries is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: [serve][streaming] Respect max_concurrent_queries when using streaming calls #35778
The timeout set in the HTTP proxy does not apply to streaming responses: [serve][streaming] Implement timeout logic when using streaming interface #35779

Related issue number

#34266

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: SangBin Cho <[email protected]>

This reverts commit 122b705. Signed-off-by: SangBin Cho <[email protected]>

This reverts commit 05f468a. Signed-off-by: SangBin Cho <[email protected]>

Signed-off-by: SangBin Cho <[email protected]>

python/ray/serve/_private/replica.py

angelinalg · 2023-05-25T22:20:42Z

python/ray/serve/_private/replica.py

- never_set_event = asyncio.Event()
- await never_set_event.wait()
+ If the result is already a Response type, it will be sent directly. Else it
+ will be converted to our custom Response type that handles serialization for


Suggested change

will be converted to our custom Response type that handles serialization for

is converted to a custom Response type that handles serialization for

python/ray/serve/_private/replica.py

python/ray/serve/_private/router.py

Signed-off-by: Edward Oakes <[email protected]>

python/ray/serve/api.py

angelinalg · 2023-05-25T22:24:35Z

Tip: Use the Add suggestion to batch to combine multiple suggestions into a single commit.

angelinalg

LGTM
Made some minor suggestions.

Signed-off-by: Edward Oakes <[email protected]>

edoakes · 2023-05-25T22:25:31Z

Tip: Use the Add suggestion to batch to combine multiple suggestions into a single commit.

Oh wow that is a revelation 😂

…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779

…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779 Signed-off-by: Edward Oakes <[email protected]>

…jectRefGenerator` (#35720) (#35811) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on #35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: #35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: #35778 - The timeout set in the HTTP proxy does not apply to streaming responses: #35779 Signed-off-by: Edward Oakes <[email protected]>

Extends #35720 to also add support for websockets (as well as detecting HTTP disconnects).

…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779

…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779 Signed-off-by: e428265 <[email protected]>

Extends ray-project#35720 to also add support for websockets (as well as detecting HTTP disconnects). Signed-off-by: e428265 <[email protected]>

…jectRefGenerator` (ray-project#35720) (ray-project#35811) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779 Signed-off-by: Edward Oakes <[email protected]>

rkooo567 added 30 commits May 12, 2023 06:19

initial version

452ed1f

Signed-off-by: SangBin Cho <[email protected]>

in progress.

3ebe327

Signed-off-by: SangBin Cho <[email protected]>

finished basics.

c140a5c

Signed-off-by: SangBin Cho <[email protected]>

fix cpp error

b83af80

Signed-off-by: SangBin Cho <[email protected]>

working now.

509b311

Signed-off-by: SangBin Cho <[email protected]>

Merge branch 'master' into streaming-generator-1

d0795e5

fix a bug

f8a90f6

Signed-off-by: SangBin Cho <[email protected]>

Basic version finished.

0a9169d

Signed-off-by: SangBin Cho <[email protected]>

[Please Revert] Work e2e.

05f468a

Signed-off-by: SangBin Cho <[email protected]>

[Revert Please] Support core worker APIs and a generator.

122b705

Signed-off-by: SangBin Cho <[email protected]>

fix a bug

7a8fe2c

Signed-off-by: SangBin Cho <[email protected]>

Revert "[Revert Please] Support core worker APIs and a generator."

d880763

This reverts commit 122b705. Signed-off-by: SangBin Cho <[email protected]>

Revert "[Please Revert] Work e2e."

f501c22

This reverts commit 05f468a. Signed-off-by: SangBin Cho <[email protected]>

Merge branch 'master' into streaming-generator-1

1942394

Fix failing tests.

3e0212e

Signed-off-by: SangBin Cho <[email protected]>

Merge branch 'master' into streaming-generator-2

c9a932e

Merge branch 'streaming-generator-1' into streaming-generator-2

ffe20fd

Merge branch 'master' into streaming-generator-3

0e89ad7

Merge branch 'streaming-generator-2' into streaming-generator-3

d520e47

Fix

7610474

Signed-off-by: SangBin Cho <[email protected]>

Fix a broken test.

aaa0582

Signed-off-by: SangBin Cho <[email protected]>

Merge branch 'streaming-generator-1' into streaming-generator-2

a52f74b

Merge branch 'master' into streaming-generator-1

37c3bdd

Merge branch 'streaming-generator-1' into streaming-generator-2

fd83edd

Merge branch 'streaming-generator-2' into streaming-generator-3

ef08b64

Finished async actor.

74a2e31

Signed-off-by: SangBin Cho <[email protected]>

Add a unit test.

8b9ba39

Signed-off-by: SangBin Cho <[email protected]>

done

a4b62ac

Signed-off-by: SangBin Cho <[email protected]>

Merge branch 'master' into streaming-generator-1

d350b5d

Addressed code review.

9ed05d9

Signed-off-by: SangBin Cho <[email protected]>