[Serve] Add logging config doc (ray-project#41708)

Signed-off-by: Sihan Wang <[email protected]>

doc/source/serve/api/index.md

@@ -363,7 +363,8 @@ Content-Type: application/json
  schema.ReplicaDetails
 ```
 
-## Metrics API
+## Observability
+
 ```{eval-rst}
 .. autosummary::
  :nosignatures:
@@ -372,4 +373,5 @@ Content-Type: application/json
  metrics.Counter
  metrics.Histogram
  metrics.Gauge
+ schema.LoggingConfig
 ```

doc/source/serve/configure-serve-deployment.md

@@ -22,6 +22,7 @@ You can also refer to the [API reference](../serve/api/doc/ray.serve.deployment_
 - `health_check_timeout_s` - Duration in seconds, that replicas wait for a health check method to return before considering it as failed. Defaults to 30s.
 - `graceful_shutdown_wait_loop_s` - Duration that replicas wait until there is no more work to be done before shutting down. Defaults to 2s.
 - `graceful_shutdown_timeout_s` - Duration to wait for a replica to gracefully shut down before being forcefully killed. Defaults to 20s.
+- `logging_config` - Logging Config for the deployment (e.g. log level, log directory, JSON log format and so on). See [LoggingConfig](../serve/api/doc/ray.serve.schema.LoggingConfig.rst) for details.
 
 ## How to specify parameters
 

doc/source/serve/doc_code/monitoring/logging_config.py

@@ -0,0 +1,92 @@
+import requests
+import logging
+from ray import serve
+from ray.serve.schema import LoggingConfig
+
+# __json_start__
+
+
+@serve.deployment(logging_config=LoggingConfig(encoding="JSON"))
+class Model:
+ def __call__(self) -> int:
+ return "hello world"
+
+
+# __json_end__
+
+serve.run(Model.bind())
+
+resp = requests.get("http:https://localhost:8000/")
+
+
+# __level_start__
+
+
+@serve.deployment(logging_config=LoggingConfig(log_level="DEBUG"))
+class Model:
+ def __call__(self) -> int:
+ logger = logging.getLogger("ray.serve")
+ logger.debug("This debug message is from the router.")
+ return "hello world"
+
+
+# __level_end__
+
+serve.run(Model.bind())
+
+resp = requests.get("http:https://localhost:8000/")
+
+
+# __logs_dir_start__
+@serve.deployment(logging_config=LoggingConfig(logs_dir="/my_dirs"))
+class Model:
+ def __call__(self) -> int:
+ return "hello world"
+
+
+# __logs_dir_end__
+
+
+# __enable_access_log_start__
+@serve.deployment(logging_config={"enable_access_log": False})
+class Model:
+ def __call__(self) -> int:
+ logger = logging.getLogger("ray.serve")
+ logger.info("hello world")
+ return
+
+
+# __enable_access_log_end__
+
+serve.run(Model.bind())
+
+resp = requests.get("http:https://localhost:8000/")
+
+
+# __application_and_deployment_start__
+@serve.deployment
+class Router:
+ def __init__(self, handle):
+ self.handle = handle
+
+ async def __call__(self):
+ logger = logging.getLogger("ray.serve")
+ logger.debug("This debug message is from the router.")
+ return await self.handle.remote()
+
+
+@serve.deployment
+class Model:
+ def __call__(self) -> int:
+ logger = logging.getLogger("ray.serve")
+ logger.debug("This debug message is from the model.")
+ return "hello world"
+
+
+serve.run(Router.bind(Model.bind()), logging_config={"log_level": "DEBUG"})
+resp = requests.get("http:https://localhost:8000/")
+# __application_and_deployment_end__
+
+# __configure_serve_component_start__
+serve.start(logging_config={"log_level": "DEBUG"})
+# __configure_serve_component_end__

doc/source/serve/monitoring.md

@@ -233,15 +233,77 @@ In addition to the standard Python logger, Serve supports custom logging. Custom
 
 For a detailed overview of logging in Ray, see [Ray Logging](configure-logging).
 
-### JSON logging format
-You can enable JSON-formatted logging in the Serve log file by setting the environment variable `RAY_SERVE_ENABLE_JSON_LOGGING=1`. After setting this environment variable, the logs have the following format:
-```json
-{"levelname": "INFO", "asctime": "2023-07-17 10:34:25,425", "deployment": "default_api", "replica": "default_api#bFDOnw", "request_id": "OGIVJJJPRb", "route": "/app1", "application": "default", "message": "replica.py:664 - Started executing request OGIVJJJPRb"}
-{"levelname": "INFO", "asctime": "2023-07-17 10:34:25,425", "deployment": "default_api", "replica": "default_api#bFDOnw", "request_id": "OGIVJJJPRb", "route": "/app1", "application": "default", "message": "replica.py:691 - __CALL__ OK 0.1ms"}
-{"levelname": "INFO", "asctime": "2023-07-17 10:34:25,433", "deployment": "default_api", "replica": "default_api#bFDOnw", "request_id": "BULmoMIYRD", "route": "/app1", "application": "default", "message": "replica.py:664 - Started executing request BULmoMIYRD"}
-{"levelname": "INFO", "asctime": "2023-07-17 10:34:25,433", "deployment": "default_api", "replica": "default_api#bFDOnw", "request_id": "BULmoMIYRD", "route": "/app1", "application": "default", "message": "replica.py:691 - __CALL__ OK 0.2ms"}
-{"levelname": "INFO", "asctime": "2023-07-17 10:34:25,440", "deployment": "default_api", "replica": "default_api#bFDOnw", "request_id": "jLTczxOqme", "route": "/app1", "application": "default", "message": "replica.py:664 - Started executing request jLTczxOqme"}
-{"levelname": "INFO", "asctime": "2023-07-17 10:34:25,441", "deployment": "default_api", "replica": "default_api#bFDOnw", "request_id": "jLTczxOqme", "route": "/app1", "application": "default", "message": "replica.py:691 - __CALL__ OK 0.1ms"}
+### Configure Serve logging
+From ray 2.9, the logging_config API configures logging for Ray Serve. You can configure logging for Ray Serve. Pass a dictionary of [LOGGING_CONFIG](../serve/api/doc/ray.serve.schema.LoggingConfig.rst) or object of [LOGGING_CONFIG](../serve/api/doc/ray.serve.schema.LoggingConfig.rst) to the `logging_config` argument of `serve.run` or `@serve.deployment`.
+
+#### Configure logging format
+You can configure the JSON logging format by passing `encoding=JSON` to `logging_config` argument of `@serve.deployment`. For example:
+
+```{literalinclude} doc_code/monitoring/logging_config.py
+:start-after: __json_start__
+:end-before: __json_end__
+:language: python
+```
+In the `Model` log file, you should see the following:
+
+```
+{"levelname": "INFO", "asctime": "2023-12-07 12:59:45,271", "deployment": "Model", "replica": "default#Model#PUGBSJ", "request_id": "8d316c3b-4c9f-4769-8080-b6867ca46d1e", "route": "/", "application": "default", "message": "replica.py:719 - Started executing request 8d316c3b-4c9f-4769-8080-b6867ca46d1e"}
+{"levelname": "INFO", "asctime": "2023-12-07 12:59:45,271", "deployment": "Model", "replica": "default#Model#PUGBSJ", "request_id": "8d316c3b-4c9f-4769-8080-b6867ca46d1e", "route": "/", "application": "default", "message": "replica.py:745 - __CALL__ OK 0.1ms"}
+```
+
+#### Disable access log
+
+:::{note}
+Access log is Ray Serve traffic log, it is printed to proxy log files and replica log files per request. Sometimes it is useful for debugging, but it can also be noisy.
+:::
+
+You can also disable the access log by passing `disable_access_log=True` to `logging_config` argument of `@serve.deployment`. For example:
+
+```{literalinclude} doc_code/monitoring/logging_config.py
+:start-after: __enable_access_log_start__
+:end-before: __enable_access_log_end__
+:language: python
+```
+
+The `Model` replica log file doesn't include the Serve traffic log, you should only see the application log in the log file.
+
+```
+INFO 2023-12-07 13:15:07,979 Model default#Model#LhNrRV 8c5fcdb2-87a0-46ed-b2f8-9336280c9c3d / default logging_config.py:50 - hello world
+```
+
+#### Configure logging in different deployments and applications
+You can also configure logging at the application level by passing `logging_config` to `serve.run`. For example:
+
+```{literalinclude} doc_code/monitoring/logging_config.py
+:start-after: __application_and_deployment_start__
+:end-before: __application_and_deployment_end__
+:language: python
+```
+
+In the Router log file, you should see the following:
+
+```
+INFO 2023-12-07 14:18:18,811 Router default#Router#iNdKWL 6fe398fa-17d6-4abe-8264-d69c12aa9884 / default replica.py:719 - Started executing request 6fe398fa-17d6-4abe-8264-d69c12aa9884
+DEBUG 2023-12-07 14:18:18,811 Router default#Router#iNdKWL 6fe398fa-17d6-4abe-8264-d69c12aa9884 / default logging_config.py:68 - This debug message is from the router.
+```
+
+In the Model log file, you should see the following:
+
+```
+INFO 2023-12-07 14:18:18,821 Model default#Model#AHYvjY 6fe398fa-17d6-4abe-8264-d69c12aa9884 / default replica.py:719 - Started executing request 6fe398fa-17d6-4abe-8264-d69c12aa9884
+DEBUG 2023-12-07 14:18:18,821 Model default#Model#AHYvjY 6fe398fa-17d6-4abe-8264-d69c12aa9884 / default logging_config.py:75 - This debug message is from the model.
+INFO 2023-12-07 14:18:18,821 Model default#Model#AHYvjY 6fe398fa-17d6-4abe-8264-d69c12aa9884 / default replica.py:745 - __CALL__ OK 0.1ms
+```
+
+When you set `logging_config` at the application level, Ray Serve applies to all deployments in the application. When you set `logging_config` at the deployment level at the same time, the deployment level configuration will overrides the application level configuration.
+
+#### Configure logging for serve components
+You can also update logging configuration similar above to the Serve controller and proxies by passing `logging_config` to `serve.start`.
+
+```{literalinclude} doc_code/monitoring/logging_config.py
+:start-after: __configure_serve_component_start__
+:end-before: __configure_serve_component_end__
+:language: python
 ```
 
 ### Set Request ID

doc/source/serve/production-guide/config.md

@@ -29,6 +29,12 @@ grpc_options:
  port: ...
  grpc_servicer_functions: ...
 
+logging_config:
+ log_level: ...
+ logs_dir: ...
+ encoding: ...
+ enable_access_log: ...
+
 applications:
 - name: ...
  route_prefix: ...
@@ -42,7 +48,7 @@ applications:
  ...
 ```
 
-The file contains `proxy_location`, `http_options`, `grpc_options`, and `applications`.
+The file contains `proxy_location`, `http_options`, `grpc_options`, `logging_config` and `applications`.
 
 The `proxy_location` field configures where to run proxies to handle traffic to the cluster. You can set `proxy_location` to the following values:
 - EveryNode (default): Run a proxy on every node in the cluster that has at least one replica actor.
@@ -61,6 +67,8 @@ The `grpc_options` are as follows. Note that the gRPC config is global to your R
  set to `9000`.
 - **`grpc_servicer_functions`**: List of import paths for gRPC `add_servicer_to_server` functions to add to Serve's gRPC proxy. The servicer functions need to be importable from the context of where Serve is running. This defaults to an empty list, which means the gRPC server isn't started.
 
+The `logging_config` is global config, you can configure controller & proxy & replica logs. Note that you can also set application and deployment level logging config, which will take precedence over the global config. See logging config API [here](../../serve/api/doc/ray.serve.schema.LoggingConfig.rst) for more details.
+
 These are the fields per application:
 
 - **`name`**: The names for each application that are auto-generated by `serve build`. The name of each application must be unique. 
@@ -131,6 +139,12 @@ grpc_options:
  port: 9000
  grpc_servicer_functions: []
 
+logging_config:
+ encoding: TEXT
+ log_level: INFO
+ logs_dir: null
+ enable_access_log: true
+
 applications:
 - name: default
  route_prefix: /

python/ray/serve/schema.py

@@ -85,7 +85,25 @@ class EncodingType(str, Enum):
 
 @PublicAPI(stability="alpha")
 class LoggingConfig(BaseModel):
- """Logging config schema for configuring serve components logs."""
+ """Logging config schema for configuring serve components logs.
+
+ Example:
+
+ .. code-block:: python
+
+ from ray import serve
+ from ray.serve.schema import LoggingConfig
+ # Set log level for the deployment.
+ @serve.deployment(LoggingConfig(log_level="DEBUG")
+ class MyDeployment:
+ def __call__(self) -> str:
+ return "Hello world!"
+ # Set log directory for the deployment.
+ @serve.deployment(LoggingConfig(logs_dir="/my_dir")
+ class MyDeployment:
+ def __call__(self) -> str:
+ return "Hello world!"
+ """
 
  class Config:
  extra = Extra.forbid