Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[serve] Add telemetry for lightweight config updates #34039

Merged
merged 3 commits into from
Apr 4, 2023
Merged

[serve] Add telemetry for lightweight config updates #34039

merged 3 commits into from
Apr 4, 2023

Conversation

zcin
Copy link
Contributor

@zcin zcin commented Apr 4, 2023

Why are these changes needed?

This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:

  • SERVE_NUM_REPLICAS_UPDATED: whether num_replicas was updated (without any heavyweight option updated)
  • SERVE_USER_CONFIG_UPDATED: whether user_config was updated (without any heavyweight option updated)
  • SERVE_AUTOSCALING_CONFIG_UPDATED: whether autoscaling_config was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@zcin zcin marked this pull request as ready for review April 4, 2023 14:20
edoakes
edoakes approved these changes Apr 4, 2023
View reviewed changes
Copy link
Contributor

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, otherwise LGTM, do we need approval from anyone else on this?

src/ray/protobuf/usage.proto Outdated
Comment on lines 63 to 68
// Whether num_replicas changed as a lightweight config update
SERVE_NUM_REPLICAS_UPDATED = 15;
// Whether user_config changed as a lightweight config update
SERVE_USER_CONFIG_UPDATED = 16;
// Whether autoscaling_config changed as a lightweight config update
SERVE_AUTOSCALING_CONFIG_UPDATED = 17;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Whether num_replicas changed as a lightweight config update
SERVE_NUM_REPLICAS_UPDATED = 15;
// Whether user_config changed as a lightweight config update
SERVE_USER_CONFIG_UPDATED = 16;
// Whether autoscaling_config changed as a lightweight config update
SERVE_AUTOSCALING_CONFIG_UPDATED = 17;
// Whether num_replicas changed as a lightweight config update
SERVE_NUM_REPLICAS_LIGHTWEIGHT_UPDATED = 15;
// Whether user_config changed as a lightweight config update
SERVE_USER_CONFIG_LIGHTWEIGHT_UPDATED = 16;
// Whether autoscaling_config changed as a lightweight config update
SERVE_AUTOSCALING_CONFIG_LIGHTWEIGHT_UPDATED = 17;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we need approval from either @pcmoritz or @thomasdesr.

@sihanwang41 sihanwang41 mentioned this pull request Apr 4, 2023
Merged
8 tasks
thomasdesr
thomasdesr approved these changes Apr 4, 2023
View reviewed changes
Copy link
Contributor

@thomasdesr thomasdesr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Telemetry Boop

@zcin
Copy link
Contributor Author

zcin commented Apr 4, 2023

Failures unrelated. @edoakes ready for merge

zhe-thoughts
zhe-thoughts approved these changes Apr 4, 2023
View reviewed changes
Copy link
Collaborator

@zhe-thoughts zhe-thoughts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for cherry picking

@edoakes edoakes merged commit b4938b2 into ray-project:master Apr 4, 2023
zcin added a commit to zcin/ray that referenced this pull request Apr 4, 2023
@zcin
[serve] Add telemetry for lightweight config updates (ray-project#34039)
7dbbe35 
This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:
- `SERVE_NUM_REPLICAS_UPDATED`: whether `num_replicas` was updated (without any heavyweight option updated)
- `SERVE_USER_CONFIG_UPDATED`: whether `user_config` was updated (without any heavyweight option updated)
- `SERVE_AUTOSCALING_CONFIG_UPDATED`: whether `autoscaling_config` was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.
@zcin zcin mentioned this pull request Apr 4, 2023
Merged
8 tasks
@zcin zcin deleted the lightweight-telemetry branch April 5, 2023 04:33
zcin added a commit to zcin/ray that referenced this pull request Apr 5, 2023
@zcin
[serve] Add telemetry for lightweight config updates (ray-project#34039)
1c01ee4 
This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:
- `SERVE_NUM_REPLICAS_UPDATED`: whether `num_replicas` was updated (without any heavyweight option updated)
- `SERVE_USER_CONFIG_UPDATED`: whether `user_config` was updated (without any heavyweight option updated)
- `SERVE_AUTOSCALING_CONFIG_UPDATED`: whether `autoscaling_config` was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.

Signed-off-by: Cindy Zhang <[email protected]>
clarng pushed a commit that referenced this pull request Apr 5, 2023
@zcin
[serve] Add telemetry for lightweight config updates (#34039) (#34070)
44d4e58 
This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:
- `SERVE_NUM_REPLICAS_UPDATED`: whether `num_replicas` was updated (without any heavyweight option updated)
- `SERVE_USER_CONFIG_UPDATED`: whether `user_config` was updated (without any heavyweight option updated)
- `SERVE_AUTOSCALING_CONFIG_UPDATED`: whether `autoscaling_config` was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.

Signed-off-by: Cindy Zhang <[email protected]>
ArturNiederfahrenhorst pushed a commit to ArturNiederfahrenhorst/ray that referenced this pull request Apr 10, 2023
@zcin @ArturNiederfahrenhorst
[serve] Add telemetry for lightweight config updates (ray-project#34039)
dcb7132 
This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:
- `SERVE_NUM_REPLICAS_UPDATED`: whether `num_replicas` was updated (without any heavyweight option updated)
- `SERVE_USER_CONFIG_UPDATED`: whether `user_config` was updated (without any heavyweight option updated)
- `SERVE_AUTOSCALING_CONFIG_UPDATED`: whether `autoscaling_config` was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@zcin @elliottower
[serve] Add telemetry for lightweight config updates (ray-project#34039)
b5a2062 
This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:
- `SERVE_NUM_REPLICAS_UPDATED`: whether `num_replicas` was updated (without any heavyweight option updated)
- `SERVE_USER_CONFIG_UPDATED`: whether `user_config` was updated (without any heavyweight option updated)
- `SERVE_AUTOSCALING_CONFIG_UPDATED`: whether `autoscaling_config` was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.

Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@zcin @ProjectsByJackHe
[serve] Add telemetry for lightweight config updates (ray-project#34039)
9342ec6 
This change adds new telemetry to get more insight into if and how users are using the lightweight config updates feature of Serve. It adds the telemetry tags:
- `SERVE_NUM_REPLICAS_UPDATED`: whether `num_replicas` was updated (without any heavyweight option updated)
- `SERVE_USER_CONFIG_UPDATED`: whether `user_config` was updated (without any heavyweight option updated)
- `SERVE_AUTOSCALING_CONFIG_UPDATED`: whether `autoscaling_config` was updated (without any heavyweight option updated)

These values are set to True as long as the user uses the feature at least once for any deployment in any application on the cluster.

Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thomasdesr thomasdesr thomasdesr approved these changes

@sihanwang41 sihanwang41 sihanwang41 approved these changes

@edoakes edoakes edoakes approved these changes

@zhe-thoughts zhe-thoughts zhe-thoughts approved these changes

@pcmoritz pcmoritz Awaiting requested review from pcmoritz pcmoritz is a code owner

@shrekris-anyscale shrekris-anyscale Awaiting requested review from shrekris-anyscale

@clarng clarng Awaiting requested review from clarng

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@zcin @thomasdesr @sihanwang41 @edoakes @zhe-thoughts