[Metrics] Add easy-install script for Prometheus #42359

architkulkarni · 2024-01-12T01:16:00Z

Why are these changes needed?

This PR simplifies some of the manual steps currently required for first-time users setting up Prometheus with Ray.

This PR adds an easy install script that automatically downloads and runs Prometheus with the appropriate configuration to scrape from Ray. It's for first-time users to run a quick demo of metrics and is not intended for production use.

TODO before merge:

Figure out the best place to host the install script. Telling the user to copy from site-packages/ray/... is a bit weird. [Solution: Add it to the Ray CLI as ray metrics launch-prometheus]
Figure out why prometheus metrics query doesn't work after ray stop. I would expect this to just work. [Solution: It does work, you just have to view "Graph", or view "Table" after choosing a timestamp from when Ray was up.]
Testing strategy

Possible followups:

Verify checksum
Detect if prometheus is already running and gracefully handle it

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni · 2024-01-12T01:18:52Z

@alanwguo This PR is ready for some early feedback. The remaining work is listed in the PR description.

Signed-off-by: Archit Kulkarni <[email protected]>

…lkarni/ray into easy-install-prometheus Signed-off-by: Archit Kulkarni <[email protected]>

alanwguo · 2024-01-22T20:19:46Z

dashboard/modules/metrics/install_and_start_prometheus.py

+ response = request_method(url, stream=True)
+ response.raise_for_status()
+
+ total_size_in_bytes = int(response.headers.get("content-length", 0))
+ total_size_in_mb = total_size_in_bytes / (1024 * 1024)
+
+ downloaded_size_in_mb = 0
+ block_size = DOWNLOAD_BLOCK_SIZE
+
+ with open(filename, "wb") as file:
+ for chunk in response.iter_content(chunk_size=block_size):
+ file.write(chunk)
+ downloaded_size_in_mb += len(chunk) / (1024 * 1024)
+ print(
+ f"Downloaded: {downloaded_size_in_mb:.2f} MB / "
+ f"{total_size_in_mb:.2f} MB",
+ end="\r",
+ )
+
+ print("\nDownload completed.")


is there a higher level library for downloading byte data into files?
This feels more manual than necessary.

I would think somethign like library.download_file(url, out_path) would already exist in one of the libraries we already have in our environment.

There is one used by Ray Data, but it doesn't provide a percent loading indicator, and I'd like to avoid the Ray Data dependency. I think the percent indicator is useful otherwise it looks like the script is hanging (50mb takes seconds or 1-2 minutes depending on the internet speed). I couldn't find a less manual way of getting a basic progress indicator.

alanwguo · 2024-01-22T20:21:09Z

dashboard/modules/metrics/install_and_start_prometheus.py

+ config_file = Path(PROMETHEUS_CONFIG_INPUT_PATH)
+
+ if not config_file.exists():
+ raise FileNotFoundError(f"Prometheus config file not found: {config_file}")


This configuration only gets created once ray starts. It might make sense to see if we can move the config generation logic to this file instead since this runs before Ray starts.

Discussed offline:

Currently in Ray, "creating the configuration" just consists of copying a hardcoded config from the site-packages Ray directory into some location in /tmp/ray. In this PR, we pull the config file from the first location, so we don't actually have to run Ray first.

However, if in the future we make a change to Ray that makes updates to the hardcoded config upon ray start before copying it to /tmp/ray, then install_and_start_prometheus.py won't have the updated config. So we should at the very least add code comments in the install script and in the "Ray creating the config" step to make sure future maintainers take this into account. I'll update the PR with these comments.

alanwguo · 2024-01-22T20:22:19Z

dashboard/modules/metrics/install_and_start_prometheus.py

+ logging.error("Installation failed.")
+ sys.exit(1)
+
+ # TODO: Add a check to see if Prometheus is already running


let's do this TODO. Maybe just a simple check of the prometheus port?

Will file a followup PR for this. I remember Prometheus prints a reasonable error if the port is already in use, so it should be okay.

angelinalg

Great writing!

Signed-off-by: Archit Kulkarni <[email protected]>

…lkarni/ray into easy-install-prometheus Signed-off-by: Archit Kulkarni <[email protected]>

This PR simplifies some of the manual steps currently required for first-time users setting up Prometheus with Ray. This PR adds an easy install script that automatically downloads and runs Prometheus with the appropriate configuration to scrape from Ray. It's for first-time users to run a quick demo of metrics and is not intended for production use. --------- Signed-off-by: Archit Kulkarni <[email protected]> Signed-off-by: khluu <[email protected]>

architkulkarni added 3 commits January 11, 2024 16:20

Add script to install and start prometheus

5205a26

Signed-off-by: Archit Kulkarni <[email protected]>

Add doc

9513b40

Signed-off-by: Archit Kulkarni <[email protected]>

up

7e8ff75

Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni assigned alanwguo Jan 12, 2024

Lint

be86fbd

Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni marked this pull request as ready for review January 12, 2024 01:18

architkulkarni requested review from maxpumperla, pcmoritz, kevin85421 and a team as code owners January 12, 2024 01:18

architkulkarni added 11 commits January 16, 2024 11:16

Add test

8eb57c6

Signed-off-by: Archit Kulkarni <[email protected]>

Set env var to use HEAD instead of GET

8365afd

Signed-off-by: Archit Kulkarni <[email protected]>

Lint

fe84cfb

Signed-off-by: Archit Kulkarni <[email protected]>

Lint

71813e9

Signed-off-by: Archit Kulkarni <[email protected]>

Fix test

278987f

Signed-off-by: Archit Kulkarni <[email protected]>

Fix test

8bcf6f5

Signed-off-by: Archit Kulkarni <[email protected]>

Add command ray metrics launch-prometheus

eeddffb

Signed-off-by: Archit Kulkarni <[email protected]>

lint

e6be641

Signed-off-by: Archit Kulkarni <[email protected]>

Merge branch 'master' into easy-install-prometheus

aea6c71

Use better command that doesn't require running ray.init()

de64e44

Signed-off-by: Archit Kulkarni <[email protected]>

Merge branch 'easy-install-prometheus' of https://github.com/architku…

824535b

…lkarni/ray into easy-install-prometheus Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni linked an issue Jan 18, 2024 that may be closed by this pull request

Easy metrics setup: install prometheus #41901

Closed

alanwguo reviewed Jan 22, 2024

View reviewed changes

architkulkarni added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jan 23, 2024

architkulkarni assigned angelinalg Jan 23, 2024

angelinalg approved these changes Jan 25, 2024

View reviewed changes

architkulkarni added 3 commits January 25, 2024 11:18

Add comments for future maintainers about config updates

c7311f7

Signed-off-by: Archit Kulkarni <[email protected]>

Merge branch 'master' into easy-install-prometheus

1024b2a

Revert accidental changes

243a8fb

Signed-off-by: Archit Kulkarni <[email protected]>

Merge branch 'easy-install-prometheus' of https://github.com/architku…

551ddf7

…lkarni/ray into easy-install-prometheus Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni merged commit 44db192 into ray-project:master Jan 25, 2024
9 checks passed

architkulkarni mentioned this pull request Jan 25, 2024

[Metrics] Gracefully handle already running Prometheus for easy-install script #42716

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metrics] Add easy-install script for Prometheus #42359

[Metrics] Add easy-install script for Prometheus #42359

architkulkarni commented Jan 12, 2024 •

edited

Loading

architkulkarni commented Jan 12, 2024

alanwguo Jan 22, 2024

architkulkarni Jan 25, 2024

alanwguo Jan 22, 2024

architkulkarni Jan 23, 2024

alanwguo Jan 22, 2024

architkulkarni Jan 25, 2024

angelinalg left a comment

[Metrics] Add easy-install script for Prometheus #42359

[Metrics] Add easy-install script for Prometheus #42359

Conversation

architkulkarni commented Jan 12, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

architkulkarni commented Jan 12, 2024

alanwguo Jan 22, 2024

Choose a reason for hiding this comment

architkulkarni Jan 25, 2024

Choose a reason for hiding this comment

alanwguo Jan 22, 2024

Choose a reason for hiding this comment

architkulkarni Jan 23, 2024

Choose a reason for hiding this comment

alanwguo Jan 22, 2024

Choose a reason for hiding this comment

architkulkarni Jan 25, 2024

Choose a reason for hiding this comment

angelinalg left a comment

Choose a reason for hiding this comment

architkulkarni commented Jan 12, 2024 •

edited

Loading