Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Release] Add 2.9.1 release logs #42372

Merged

Conversation

architkulkarni
Copy link
Contributor

Why are these changes needed?

Adds performance logs for cfbf98c.

python compare_perf_metrics 2.9.0 2.9.1
REGRESSION 10.41%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 8.429852592930626 to 7.552591739788697 (10.41%) in 2.9.1/microbenchmark.json
REGRESSION 7.08%: placement_group_create/removal (THROUGHPUT) regresses from 845.7511547073977 to 785.8508797030252 (7.08%) in 2.9.1/microbenchmark.json
REGRESSION 7.03%: single_client_put_gigabytes (THROUGHPUT) regresses from 20.6372354079233 to 19.186501513119794 (7.03%) in 2.9.1/microbenchmark.json
REGRESSION 5.45%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 13.112230033151658 to 12.397164753383363 (5.45%) in 2.9.1/microbenchmark.json
REGRESSION 5.10%: multi_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 12988.08350923366 to 12325.324692399277 (5.10%) in 2.9.1/microbenchmark.json
REGRESSION 4.72%: 1_1_async_actor_calls_async (THROUGHPUT) regresses from 3320.575179316923 to 3163.777669072775 (4.72%) in 2.9.1/microbenchmark.json
REGRESSION 4.45%: 1_1_async_actor_calls_with_args_async (THROUGHPUT) regresses from 2415.0979141352886 to 2307.5841596274245 (4.45%) in 2.9.1/microbenchmark.json
REGRESSION 3.93%: single_client_wait_1k_refs (THROUGHPUT) regresses from 5.424892169925369 to 5.211488336217317 (3.93%) in 2.9.1/microbenchmark.json
REGRESSION 3.41%: n_n_async_actor_calls_async (THROUGHPUT) regresses from 23089.526825423094 to 22301.861041866883 (3.41%) in 2.9.1/microbenchmark.json
REGRESSION 2.70%: 1_1_actor_calls_concurrent (THROUGHPUT) regresses from 5354.497412488227 to 5209.763216908848 (2.70%) in 2.9.1/microbenchmark.json
REGRESSION 2.05%: multi_client_tasks_async (THROUGHPUT) regresses from 24316.337428119852 to 23817.030993400018 (2.05%) in 2.9.1/microbenchmark.json
REGRESSION 1.99%: client__tasks_and_put_batch (THROUGHPUT) regresses from 11472.04637188305 to 11244.160608396189 (1.99%) in 2.9.1/microbenchmark.json
REGRESSION 1.83%: tasks_per_second (THROUGHPUT) regresses from 586.1652936197164 to 575.425069261052 (1.83%) in 2.9.1/benchmarks/many_tasks.json
REGRESSION 1.44%: single_client_tasks_sync (THROUGHPUT) regresses from 1009.4349525282154 to 994.8959357892109 (1.44%) in 2.9.1/microbenchmark.json
REGRESSION 1.24%: single_client_get_calls_Plasma_Store (THROUGHPUT) regresses from 10676.942537676008 to 10544.047488771861 (1.24%) in 2.9.1/microbenchmark.json
REGRESSION 0.84%: 1_1_actor_calls_async (THROUGHPUT) regresses from 8802.650097299196 to 8728.36563834848 (0.84%) in 2.9.1/microbenchmark.json
REGRESSION 0.55%: n_n_actor_calls_async (THROUGHPUT) regresses from 26694.138600078164 to 26547.749741220032 (0.55%) in 2.9.1/microbenchmark.json
REGRESSION 0.55%: tasks_per_second (THROUGHPUT) regresses from 328.5366336052011 to 326.73502105628876 (0.55%) in 2.9.1/benchmarks/many_nodes.json
REGRESSION 0.45%: client__1_1_actor_calls_sync (THROUGHPUT) regresses from 530.5597986550025 to 528.1938300742655 (0.45%) in 2.9.1/microbenchmark.json
REGRESSION 0.29%: client__1_1_actor_calls_async (THROUGHPUT) regresses from 1012.4837493368098 to 1009.5892523668448 (0.29%) in 2.9.1/microbenchmark.json
REGRESSION 0.07%: 1_1_actor_calls_sync (THROUGHPUT) regresses from 2075.2443816745968 to 2073.7767760812935 (0.07%) in 2.9.1/microbenchmark.json
REGRESSION 45.67%: dashboard_p50_latency_ms (LATENCY) regresses from 5.978 to 8.708 (45.67%) in 2.9.1/benchmarks/many_tasks.json
REGRESSION 34.93%: dashboard_p95_latency_ms (LATENCY) regresses from 2569.856 to 3467.574 (34.93%) in 2.9.1/benchmarks/many_actors.json
REGRESSION 19.26%: stage_0_time (LATENCY) regresses from 13.148497581481934 to 15.681537866592407 (19.26%) in 2.9.1/stress_tests/stress_test_many_tasks.json
REGRESSION 5.97%: 3000_returns_time (LATENCY) regresses from 6.094248331000003 to 6.457862126000009 (5.97%) in 2.9.1/scalability/single_node.json
REGRESSION 5.54%: dashboard_p50_latency_ms (LATENCY) regresses from 23.754 to 25.07 (5.54%) in 2.9.1/benchmarks/many_actors.json
REGRESSION 4.40%: avg_iteration_time (LATENCY) regresses from 1.4471865177154541 to 1.5108911991119385 (4.40%) in 2.9.1/stress_tests/stress_test_dead_actors.json
REGRESSION 0.40%: avg_pg_create_time_ms (LATENCY) regresses from 0.9053212447438167 to 0.9089756411408867 (0.40%) in 2.9.1/stress_tests/stress_test_placement_group.json
REGRESSION 0.25%: dashboard_p99_latency_ms (LATENCY) regresses from 10057.867 to 10083.034 (0.25%) in 2.9.1/benchmarks/many_tasks.json

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Archit Kulkarni <[email protected]>
@architkulkarni architkulkarni added release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order labels Jan 12, 2024
@jjyao
Copy link
Collaborator

jjyao commented Jan 12, 2024

REGRESSION 10.41%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 8.429852592930626 to 7.552591739788697 (10.41%) in 2.9.1/microbenchmark.json

Noise

REGRESSION 7.08%: placement_group_create/removal (THROUGHPUT) regresses from 845.7511547073977 to 785.8508797030252 (7.08%) in 2.9.1/microbenchmark.json

Noise

REGRESSION 7.03%: single_client_put_gigabytes (THROUGHPUT) regresses from 20.6372354079233 to 19.186501513119794 (7.03%) in 2.9.1/microbenchmark.json

Noise

REGRESSION 19.26%: stage_0_time (LATENCY) regresses from 13.148497581481934 to 15.681537866592407 (19.26%) in 2.9.1/stress_tests/stress_test_many_tasks.json

Noise.

@architkulkarni architkulkarni merged commit 5233896 into ray-project:master Jan 12, 2024
9 checks passed
vickytsang pushed a commit to ROCm/ray that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants