[Serve] add new sets of performance test for max_ongoing_requests=100 and doc change

[GeneDer]

Why are these changes needed?

Re: #45943 changed the default max_ongoing_requests down to 5 which caused the overall rps lower as will. This PR did two things:

1. Added a set of performance tests for max_ongoing_requests =100
2. Added related doc change to explicitly calling out setting up a higher value for max_ongoing_requests for lightweight requests

Related issue number

Closes metrics dropping at https://b534fd88.us1a.app.preset.io/explore/?form_data_key=YWenQf8xEN9DHx7wmFVo3cRXYD-xoqObnK7HEC8J2Ho8FMnPCYM-d_as4EO7CG4b&dashboard_page_id=DCipe40Dxo&slice_id=1380

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

@GeneDer how much was the decrease? If it's substantial, we should reconsider the change, as we don't want people to benchmark the defaults and see a big decrease...

When I previously did benchmarks I remember the perf. being very similar at 5 and 10 max concurrent requests

[GeneDer]

@GeneDer how much was the decrease? If it's substantial, we should reconsider the change, as we don't want people to benchmark the defaults and see a big decrease...

When I previously did benchmarks I remember the perf. being very similar at 5 and 10 max concurrent requests

HTTP went from 430ish to 300ish, ~30% drop
Handle went from 900ish to 540ish, ~40% drop

[edoakes]

@GeneDer how much was the decrease? If it's substantial, we should reconsider the change, as we don't want people to benchmark the defaults and see a big decrease...
When I previously did benchmarks I remember the perf. being very similar at 5 and 10 max concurrent requests

HTTP went from 430ish to 300ish, ~30% drop Handle went from 900ish to 540ish, ~40% drop

This is not good. Can we run it w/ the default changed to 10 from 5?

[GeneDer]

Also just paste screenshots, here to keep them in the history for easy reference in the future
before #45943 was merged

after the merge

[GeneDer]

@GeneDer how much was the decrease? If it's substantial, we should reconsider the change, as we don't want people to benchmark the defaults and see a big decrease...
When I previously did benchmarks I remember the perf. being very similar at 5 and 10 max concurrent requests

HTTP went from 430ish to 300ish, ~30% drop Handle went from 900ish to 540ish, ~40% drop

This is not good. Can we run it w/ the default changed to 10 from 5?

sg, let me get that running

[zcin]

From https://buildkite.com/ray-project/release/builds/18857#01907a5e-c295-4eef-b039-1bf6d79eff28 seems like setting max ongoing requests to 10 gives:
Handle throughput: 783.39 (~12% drop)
HTTP throughput: 371.66 (~13% drop)

[aslonnie]

CI running here:
https://buildkite.com/ray-project/premerge/builds/27550

but not sure it will properly post the status or not

[GeneDer]

@edoakes and I talked offline about the approach. We decided to add another set of tests to show the performance numbers of previous config when max_ongoing_requests is at 100. And make an explicit call out for setting this in the doc for lightweight requests.

The release test ran on this branch shows the performance numbers at the previous and current max_ongoing_requests settings. @edoakes can you PTAL