Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Serve] Add Multiplex metrics into dashboard #37722

Merged
merged 16 commits into from
Aug 4, 2023
Merged

[Serve] Add Multiplex metrics into dashboard #37722

merged 16 commits into from
Aug 4, 2023

Conversation

sihanwang41
Copy link
Contributor

@sihanwang41 sihanwang41 commented Jul 24, 2023 •
edited
Loading

Why are these changes needed?

Number of models per replica:
image
Number of times models loaded
image
Number of times models unloaded
image
Model load latency p99
image
Model unloaded latency p99
image
Registered model
image
Cache hit rate
image

Related issue number

Closes: #37517

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@sihanwang41
[Serve][WIP] Add multiplexed metrics into dashboard
b63182f 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Fix
afd8dba 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
34e7298 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
7cba9f7 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Fix
ffbbe3f 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
97933b9 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
6ba887e 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
9076035 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
7ae93e3 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
b799480 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
49f62ba 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
b92e819 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
ff21be6 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Update
2b5402f 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41 sihanwang41 marked this pull request as ready for review July 31, 2023 16:05
@sihanwang41 sihanwang41 changed the title [Serve][WIP] Add Multiplex metrics into dashboard [Serve] Add Multiplex metrics into dashboard Jul 31, 2023
@edoakes edoakes requested a review from ericl August 1, 2023 19:39
edoakes
edoakes approved these changes Aug 1, 2023
View reviewed changes
Copy link
Contributor

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a great start!

dashboard/modules/metrics/dashboards/serve_deployment_dashboard_panels.py
@@ -149,6 +149,108 @@
stack=False,
grid_pos=GridPos(16, 2, 8, 8),
),
Panel(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for most users these panels will be useless.

Is it possible to put them behind a separate grafana dashboard? or maybe a collapsed-by-default grouping?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edoakes I talked to @alanwguo, row is not supported in our dashboard, i think I can take it as a follow up to support it.
issue: #38024

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

@sihanwang41
Update doc
372a00a 
Signed-off-by: Sihan Wang <[email protected]>
alanwguo
alanwguo reviewed Aug 3, 2023
View reviewed changes
Copy link
Contributor

@alanwguo alanwguo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly copy changes to make graph titles seem more like titles

dashboard/modules/metrics/dashboards/serve_deployment_dashboard_panels.py Outdated Show resolved Hide resolved
dashboard/modules/metrics/dashboards/serve_deployment_dashboard_panels.py Outdated Show resolved Hide resolved
dashboard/modules/metrics/dashboards/serve_deployment_dashboard_panels.py Outdated Show resolved Hide resolved
dashboard/modules/metrics/dashboards/serve_deployment_dashboard_panels.py Outdated Show resolved Hide resolved
dashboard/modules/metrics/dashboards/serve_deployment_dashboard_panels.py Outdated Show resolved Hide resolved
@sihanwang41
Update titles
fdd9d91 
Signed-off-by: Sihan Wang <[email protected]>
@sihanwang41
Copy link
Contributor Author

@edoakes ping for merge ^^

@edoakes edoakes merged commit 73311c4 into ray-project:master Aug 4, 2023
39 of 47 checks passed
NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023
@sihanwang41 @NripeshN
[Serve] Add Multiplex metrics into dashboard (ray-project#37722)
eb581af 
Number of models per replica:
<img width="1639" alt="image" src="https://github.com/ray-project/ray/assets/6515354/6f45f047-f02e-453c-914a-96098739c3a3">
Number of times models loaded
<img width="1630" alt="image" src="https://github.com/ray-project/ray/assets/6515354/e5bb0f9a-6958-4e73-9a6a-294aa02891cc">
Number of times models unloaded
<img width="1596" alt="image" src="https://github.com/ray-project/ray/assets/6515354/717a2f45-29a0-4204-8dcf-b3caf3f53035">
Model load latency p99
<img width="1638" alt="image" src="https://github.com/ray-project/ray/assets/6515354/69729506-6f37-4ee7-93c9-979a09c7c234">
Model unloaded latency p99
<img width="1670" alt="image" src="https://github.com/ray-project/ray/assets/6515354/f8ac2a83-42c3-4c49-95b4-2089ca94ff5c">
Registered model
<img width="1661" alt="image" src="https://github.com/ray-project/ray/assets/6515354/d1cd94e1-46f4-4431-a78c-b8177291d948">
Cache hit rate
<img width="1653" alt="image" src="https://github.com/ray-project/ray/assets/6515354/b38472f4-57e7-45d7-aebb-375a0f96cbd1">

Signed-off-by: NripeshN <[email protected]>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@sihanwang41 @harborn
[Serve] Add Multiplex metrics into dashboard (ray-project#37722)
ff3d6dd 
Number of models per replica:
<img width="1639" alt="image" src="https://github.com/ray-project/ray/assets/6515354/6f45f047-f02e-453c-914a-96098739c3a3">
Number of times models loaded
<img width="1630" alt="image" src="https://github.com/ray-project/ray/assets/6515354/e5bb0f9a-6958-4e73-9a6a-294aa02891cc">
Number of times models unloaded
<img width="1596" alt="image" src="https://github.com/ray-project/ray/assets/6515354/717a2f45-29a0-4204-8dcf-b3caf3f53035">
Model load latency p99
<img width="1638" alt="image" src="https://github.com/ray-project/ray/assets/6515354/69729506-6f37-4ee7-93c9-979a09c7c234">
Model unloaded latency p99
<img width="1670" alt="image" src="https://github.com/ray-project/ray/assets/6515354/f8ac2a83-42c3-4c49-95b4-2089ca94ff5c">
Registered model
<img width="1661" alt="image" src="https://github.com/ray-project/ray/assets/6515354/d1cd94e1-46f4-4431-a78c-b8177291d948">
Cache hit rate
<img width="1653" alt="image" src="https://github.com/ray-project/ray/assets/6515354/b38472f4-57e7-45d7-aebb-375a0f96cbd1">

Signed-off-by: harborn <[email protected]>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@sihanwang41 @harborn
[Serve] Add Multiplex metrics into dashboard (ray-project#37722)
b2caaba 
Number of models per replica:
<img width="1639" alt="image" src="https://github.com/ray-project/ray/assets/6515354/6f45f047-f02e-453c-914a-96098739c3a3">
Number of times models loaded
<img width="1630" alt="image" src="https://github.com/ray-project/ray/assets/6515354/e5bb0f9a-6958-4e73-9a6a-294aa02891cc">
Number of times models unloaded
<img width="1596" alt="image" src="https://github.com/ray-project/ray/assets/6515354/717a2f45-29a0-4204-8dcf-b3caf3f53035">
Model load latency p99
<img width="1638" alt="image" src="https://github.com/ray-project/ray/assets/6515354/69729506-6f37-4ee7-93c9-979a09c7c234">
Model unloaded latency p99
<img width="1670" alt="image" src="https://github.com/ray-project/ray/assets/6515354/f8ac2a83-42c3-4c49-95b4-2089ca94ff5c">
Registered model
<img width="1661" alt="image" src="https://github.com/ray-project/ray/assets/6515354/d1cd94e1-46f4-4431-a78c-b8177291d948">
Cache hit rate
<img width="1653" alt="image" src="https://github.com/ray-project/ray/assets/6515354/b38472f4-57e7-45d7-aebb-375a0f96cbd1">
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
@sihanwang41 @arvind-chandra
[Serve] Add Multiplex metrics into dashboard (ray-project#37722)
3126d69 
Number of models per replica:
<img width="1639" alt="image" src="https://github.com/ray-project/ray/assets/6515354/6f45f047-f02e-453c-914a-96098739c3a3">
Number of times models loaded
<img width="1630" alt="image" src="https://github.com/ray-project/ray/assets/6515354/e5bb0f9a-6958-4e73-9a6a-294aa02891cc">
Number of times models unloaded
<img width="1596" alt="image" src="https://github.com/ray-project/ray/assets/6515354/717a2f45-29a0-4204-8dcf-b3caf3f53035">
Model load latency p99
<img width="1638" alt="image" src="https://github.com/ray-project/ray/assets/6515354/69729506-6f37-4ee7-93c9-979a09c7c234">
Model unloaded latency p99
<img width="1670" alt="image" src="https://github.com/ray-project/ray/assets/6515354/f8ac2a83-42c3-4c49-95b4-2089ca94ff5c">
Registered model
<img width="1661" alt="image" src="https://github.com/ray-project/ray/assets/6515354/d1cd94e1-46f4-4431-a78c-b8177291d948">
Cache hit rate
<img width="1653" alt="image" src="https://github.com/ray-project/ray/assets/6515354/b38472f4-57e7-45d7-aebb-375a0f96cbd1">

Signed-off-by: e428265 <[email protected]>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
@sihanwang41
[Serve] Add Multiplex metrics into dashboard (ray-project#37722)
b7aa6d8 
Number of models per replica:
<img width="1639" alt="image" src="https://github.com/ray-project/ray/assets/6515354/6f45f047-f02e-453c-914a-96098739c3a3">
Number of times models loaded
<img width="1630" alt="image" src="https://github.com/ray-project/ray/assets/6515354/e5bb0f9a-6958-4e73-9a6a-294aa02891cc">
Number of times models unloaded
<img width="1596" alt="image" src="https://github.com/ray-project/ray/assets/6515354/717a2f45-29a0-4204-8dcf-b3caf3f53035">
Model load latency p99
<img width="1638" alt="image" src="https://github.com/ray-project/ray/assets/6515354/69729506-6f37-4ee7-93c9-979a09c7c234">
Model unloaded latency p99
<img width="1670" alt="image" src="https://github.com/ray-project/ray/assets/6515354/f8ac2a83-42c3-4c49-95b4-2089ca94ff5c">
Registered model
<img width="1661" alt="image" src="https://github.com/ray-project/ray/assets/6515354/d1cd94e1-46f4-4431-a78c-b8177291d948">
Cache hit rate
<img width="1653" alt="image" src="https://github.com/ray-project/ray/assets/6515354/b38472f4-57e7-45d7-aebb-375a0f96cbd1">

Signed-off-by: Victor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edoakes edoakes edoakes approved these changes

@alanwguo alanwguo alanwguo approved these changes

@ericl ericl Awaiting requested review from ericl

@shrekris-anyscale shrekris-anyscale Awaiting requested review from shrekris-anyscale shrekris-anyscale is a code owner

@zcin zcin Awaiting requested review from zcin zcin is a code owner

@architkulkarni architkulkarni Awaiting requested review from architkulkarni

Assignees

@edoakes edoakes

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Serve] Multiplex observability improvement
3 participants
@sihanwang41 @alanwguo @edoakes