Skip to content

Commit

Permalink
Add docs for setting up metrics for homebrew installations (ray-proje…
Browse files Browse the repository at this point in the history
  • Loading branch information
alanwguo committed May 11, 2023
1 parent 977cac2 commit 04e414a
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 13 deletions.
9 changes: 1 addition & 8 deletions dashboard/client/src/pages/job/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,7 @@ const columns = [
helpInfo: (
<Typography>
The progress of the all submitted tasks per job. Tasks that are not yet
submitted will not show up in the progress bar.
<br />
<br />
Note: This column requires that prometheus is running. See{" "}
<a href="https://docs.ray.io/en/latest/ray-observability/ray-metrics.html#exporting-metrics">
here
</a>{" "}
for instructions.
submitted do not show up in the progress bar.
</Typography>
),
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ The service discovery file is generated on the :ref:`head node <cluster-head-nod
Ray will periodically update this file with the addresses of all metrics agents in the cluster.

Ray automatically produces a Prometheus config which scrapes the file for service discovery found at `/tmp/ray/session_latest/metrics/prometheus/prometheus.yml`.
You can choose to use this config or modify your own to enable this behavior. The details of the config can be seen below and full documentation can be found at `here <https://prometheus.io/docs/prometheus/latest/configuration/configuration/>`.
You can choose to use this config or modify your own to enable this behavior. The details of the config can be seen below and full documentation can be found `here <https://prometheus.io/docs/prometheus/latest/configuration/configuration/>`_.

With this config, Prometheus will automatically update the addresses that it scrapes based on the contents of Ray's service discovery file.

Expand Down
24 changes: 20 additions & 4 deletions doc/source/ray-observability/ray-metrics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -278,9 +278,21 @@ to `RAY_GRAFANA_HOST=http:https://55.66.77.88:3000`.
Troubleshooting
---------------

Getting Prometheus and Grafana to use the Ray configurations when installed via homebrew on macOS X
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With homebrew, Prometheus and Grafana are installed as services that are automatically launched for you.
Therefore, to configure these services, you cannot simply pass in the config files as command line arguments.

Instead, follow these instructions:
1. Change the --config-file line in `/usr/local/etc/prometheus.args` to read `--config.file /tmp/ray/session_latest/metrics/prometheus/prometheus.yml`.
2. Update `/usr/local/etc/grafana/grafana.ini` file so that it matches the contents of `/tmp/ray/session_latest/metrics/grafana/grafana.ini`.

You can then start or restart the services with `brew services start grafana` and `brew services start prometheus`.

.. _unverified-developer:

Mac does not trust the developer when installing prometheus or grafana
MacOS does not trust the developer to install Prometheus or Grafana
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You may have received an error that looks like this:
Expand All @@ -295,6 +307,10 @@ See `these instructions <https://support.apple.com/guide/mac-help/open-a-mac-app

Grafana dashboards are not embedded in the Ray dashboard
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you're getting error that `RAY_GRAFANA_HOST` is not setup despite you've set it up, please check:
That you've included protocol in the URL (e.g. `http:https://your-grafana-url.com` instead of `your-grafana-url.com`).
Also, make sure that url doesn't have trailing slash (e.g. `http:https://your-grafana-url.com` instead of `http:https://your-grafana-url.com/`).
If you're getting an error that says `RAY_GRAFANA_HOST` is not setup despite having set it up, check that:
You've included the protocol in the URL (e.g., `http:https://your-grafana-url.com` instead of `your-grafana-url.com`).
The URL doesn't have a trailing slash (e.g., `http:https://your-grafana-url.com` instead of `http:https://your-grafana-url.com/`).

Certificate Authority (CA error)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You may see a CA error if your Grafana instance is hosted behind HTTPS. Contact the Grafana service owner to properly enable HTTPS traffic.

0 comments on commit 04e414a

Please sign in to comment.