Skip to content

Latest commit

 

History

History
 
 

monitoring

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 

Service Monitoring

Components

Promtail : is a collector which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.

Loki : is a search engine for logs inspired by Prometheus.

OTEL Collector : is vendor-agnostic way to receive, process and export telemetry data.

Tempo : is a distributed tracing backend.

Grafana : is a query frontend to output data.

How to run

cd monitoring

# start containers
docker-compose up -d

# FIXME: maybe we can remove manual setups with help of a config or automation?

Set up logs monitor

  1. Go to page of grafana: https://127.0.0.1:3000/
  2. Use login and password "admin"
  3. Add Loki data source: https://loki:3100
  4. "Save & Exit" should give "Data source connected and labels found." if everything is okay.
  5. Go to "Explore" tab and make a query "{job="varlogs"} |= ``".

Set up logs monitor along with tracing

  1. Navigate to Grafana
  2. Enter "admin" for both username and password [skip if it asks for updating the password]
  3. Add data source (Tempo)
    1. select Tempo
    2. set URL to https://tempo:3200
    3. save
  4. Add data source (Loki)
    1. select Loki
    2. make it default
    3. set URL to https://loki:3100
    4. add Derived fields-
    5. set Name to trace_id
    6. set Regex to trace_id":"(.*?)"(?=,|}|$)
    7. set URL to ${__value.raw}
    8. set URL label to Tempo
    9. enable Internal link and select Tempo
    10. save
  5. Navigate to Explore
  6. Add query [example, job=router]

Notes:

  • Use trace_id in logs to jump to Tempo view to visualize the tracing.
  • Searching through recent trace ids is also possible by selecting appropriate Service Name in Tempo view under Search tab. The UI also provides other filter options.

Helpful commands

https://127.0.0.1:3100/ready : To get status of Loki, it should give "Ready".

docker container ls -as : List running containers.

docker exec -it monitoring_promtail_1 bash : Look inside of promtail container.

docker-compose down : Stop containers