Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release Performance Regression 2.6.0/2.5.0 #37084

Merged
merged 3 commits into from
Jul 26, 2023
Merged

Release Performance Regression 2.6.0/2.5.0 #37084

merged 3 commits into from
Jul 26, 2023

Conversation

can-anyscale
Copy link
Collaborator

@can-anyscale can-anyscale commented Jul 5, 2023 •
edited
Loading 

REGRESSION 75.75%: dashboard_p95_latency_ms (LATENCY) regresses from 1862.916 to 3274.133 (75.75%) in 2.6.0/benchmarks/many_actors.json
REGRESSION 46.14%: dashboard_p99_latency_ms (LATENCY) regresses from 123.276 to 180.161 (46.14%) in 2.6.0/benchmarks/many_nodes.json
REGRESSION 31.85%: dashboard_p99_latency_ms (LATENCY) regresses from 3114.033 to 4105.733 (31.85%) in 2.6.0/benchmarks/many_actors.json
REGRESSION 28.88%: stage_4_spread (LATENCY) regresses from 0.684079429712877 to 0.8816352064600278 (28.88%) in 2.6.0/stress_tests/stress_test_many_tasks.json
REGRESSION 17.52%: stage_3_creation_time (LATENCY) regresses from 3.7931106090545654 to 4.457586765289307 (17.52%) in 2.6.0/stress_tests/stress_test_many_tasks.json
REGRESSION 13.84%: avg_iteration_time (LATENCY) regresses from 2.237249433994293 to 2.5467939925193788 (13.84%) in 2.6.0/stress_tests/stress_test_dead_actors.json
REGRESSION 13.59%: client__1_1_actor_calls_concurrent (THROUGHPUT) regresses from 1035.9899229285618 to 895.1567380742168 (13.59%) in 2.6.0/microbenchmark.json
REGRESSION 11.12%: client__1_1_actor_calls_async (THROUGHPUT) regresses from 1060.3714461863751 to 942.4465276461463 (11.12%) in 2.6.0/microbenchmark.json
REGRESSION 11.05%: tasks_per_second (THROUGHPUT) regresses from 244.83827951756928 to 217.7813530884885 (11.05%) in 2.6.0/benchmarks/many_nodes.json
REGRESSION 9.69%: placement_group_create/removal (THROUGHPUT) regresses from 1088.4611425326907 to 982.967019188035 (9.69%) in 2.6.0/microbenchmark.json
REGRESSION 9.55%: pgs_per_second (THROUGHPUT) regresses from 18.873242531928092 to 17.07053129158529 (9.55%) in 2.6.0/benchmarks/many_pgs.json
REGRESSION 9.47%: client__get_calls (THROUGHPUT) regresses from 1252.00038300497 to 1133.482124423204 (9.47%) in 2.6.0/microbenchmark.json
REGRESSION 8.68%: 10000_get_time (LATENCY) regresses from 24.125256378000003 to 26.219343377 (8.68%) in 2.6.0/scalability/single_node.json
REGRESSION 8.57%: client__put_calls (THROUGHPUT) regresses from 942.4493801825888 to 861.6587451405634 (8.57%) in 2.6.0/microbenchmark.json
REGRESSION 8.42%: multi_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 13370.618694124361 to 12244.367470814326 (8.42%) in 2.6.0/microbenchmark.json
REGRESSION 8.36%: client__tasks_and_put_batch (THROUGHPUT) regresses from 12556.637086768344 to 11507.323524478936 (8.36%) in 2.6.0/microbenchmark.json
REGRESSION 7.89%: single_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 6364.146068125718 to 5862.27284701274 (7.89%) in 2.6.0/microbenchmark.json
REGRESSION 6.69%: avg_pg_create_time_ms (LATENCY) regresses from 0.8453118138153105 to 0.9018382912905657 (6.69%) in 2.6.0/stress_tests/stress_test_placement_group.json
REGRESSION 5.94%: single_client_get_calls_Plasma_Store (THROUGHPUT) regresses from 5979.727237220568 to 5624.473054749469 (5.94%) in 2.6.0/microbenchmark.json
REGRESSION 4.68%: client__1_1_actor_calls_sync (THROUGHPUT) regresses from 541.0125409991156 to 515.7013650453373 (4.68%) in 2.6.0/microbenchmark.json
REGRESSION 3.34%: 107374182400_large_object_time (LATENCY) regresses from 33.075734592 to 34.17966618600002 (3.34%) in 2.6.0/scalability/single_node.json
REGRESSION 2.13%: single_client_tasks_async (THROUGHPUT) regresses from 11527.454226565262 to 11282.169570159464 (2.13%) in 2.6.0/microbenchmark.json
REGRESSION 1.75%: stage_3_time (LATENCY) regresses from 2559.148703098297 to 2603.871742248535 (1.75%) in 2.6.0/stress_tests/stress_test_many_tasks.json
REGRESSION 1.45%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 11.473556691500509 to 11.307161890413912 (1.45%) in 2.6.0/microbenchmark.json
REGRESSION 1.31%: 1_n_async_actor_calls_async (THROUGHPUT) regresses from 9742.443873422853 to 9614.762907183765 (1.31%) in 2.6.0/microbenchmark.json
REGRESSION 1.23%: 1_n_actor_calls_async (THROUGHPUT) regresses from 10843.297610215956 to 10709.585735591145 (1.23%) in 2.6.0/microbenchmark.json
REGRESSION 0.94%: 1_1_actor_calls_async (THROUGHPUT) regresses from 8177.943178490346 to 8100.897980576367 (0.94%) in 2.6.0/microbenchmark.json
REGRESSION 0.86%: 1_1_async_actor_calls_with_args_async (THROUGHPUT) regresses from 1972.550378341843 to 1955.5773371568473 (0.86%) in 2.6.0/microbenchmark.json
REGRESSION 0.06%: n_n_actor_calls_async (THROUGHPUT) regresses from 32450.558373744876 to 32432.330323783368 (0.06%) in 2.6.0/microbenchmark.json
REGRESSION 0.04%: n_n_async_actor_calls_async (THROUGHPUT) regresses from 25264.429245205785 to 25254.550868265 (0.04%) in 2.6.0/microbenchmark.json

@can-anyscale can-anyscale added the do-not-merge Do not merge this PR! label Jul 5, 2023
@can-anyscale can-anyscale requested a review from a team July 5, 2023 04:05
This was referenced Jul 7, 2023
Closed
Closed
Closed
Closed
@scv119 scv119 added the release-blocker P0 Issue that blocks the release label Jul 17, 2023
@jjyao
Copy link
Collaborator

jjyao commented Jul 17, 2023

@can-anyscale could you update the perf diff in the PR description since I saw you just updated the PR.

@can-anyscale
2.6.0 perf
b7dc94e 
Signed-off-by: can <[email protected]>
@can-anyscale
Copy link
Collaborator Author

@jjyao updated, thanks

@can-anyscale
Update
4ddc921 
Signed-off-by: can <[email protected]>
@scv119
Copy link
Contributor

scv119 commented Jul 18, 2023

all the regression has been identified as non release blocker.

@can-anyscale
Copy link
Collaborator Author

Awesome!

@can-anyscale can-anyscale removed the release-blocker P0 Issue that blocks the release label Jul 26, 2023
@can-anyscale
Update
77590a7 
Signed-off-by: can <[email protected]>
@can-anyscale can-anyscale removed the do-not-merge Do not merge this PR! label Jul 26, 2023
@can-anyscale can-anyscale merged commit 8a4cb0c into master Jul 26, 2023
2 checks passed
@can-anyscale can-anyscale deleted the can-260-perf branch July 26, 2023 17:53
NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023
@can-anyscale @NripeshN
Release Performance Regression 2.6.0/2.5.0 (ray-project#37084)
eb2c9b1 
2.6.0 performance data

Signed-off-by: can <[email protected]>
Signed-off-by: NripeshN <[email protected]>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@can-anyscale @harborn
Release Performance Regression 2.6.0/2.5.0 (ray-project#37084)
8728e0a 
2.6.0 performance data

Signed-off-by: can <[email protected]>
Signed-off-by: harborn <[email protected]>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@can-anyscale @harborn
Release Performance Regression 2.6.0/2.5.0 (ray-project#37084)
ce9f286 
2.6.0 performance data

Signed-off-by: can <[email protected]>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
@can-anyscale @arvind-chandra
Release Performance Regression 2.6.0/2.5.0 (ray-project#37084)
85cca6e 
2.6.0 performance data

Signed-off-by: can <[email protected]>
Signed-off-by: e428265 <[email protected]>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
@can-anyscale
Release Performance Regression 2.6.0/2.5.0 (ray-project#37084)
5fea5b2 
2.6.0 performance data

Signed-off-by: can <[email protected]>
Signed-off-by: Victor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scv119 scv119 scv119 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@can-anyscale @jjyao @scv119