There is a goldmine of information about each of our microservices that is avialable at endpoints provided to us by Spring Boot Actuator. With the use of Micrometer, we can create custom metrics as well, and format all of the data so that it can be read by Prometheus. Prometheus will read this data an display it in a UI.

1. Inside the 3 microservices cards, accounts, and cards add the following dependencies:

   • Micrometer - Micrometer is an open-source project and provides a metric facade that exposes metric data in a vendor-neutral format that a monitoring system can understand.
   • Prometheus - Will feed on the data from Micrometer and allow us to query/visualize this data.
     • Eventually we will use Grafana to build a rich UI dashboard of this data
   • Make sure you have AOP dependency within accounts as well. This is because we will deliver customm metrics from accounts in the form of an Aspect.

2. In accounts AccountsApplication class add the following @Bean. Paste the following:

    /**
     * We want to expose custom metrics related to how 
     * long it takes this service to process a request.
     * MeterRegistry comes from micrometer dependency.
    */
	@Bean
	public TimedAspect timedAspect(MeterRegistry registry) {
	    return new TimedAspect(registry);
	}

3. Now go to your AccountsController and add a @Timed annotation above your getAccountDetails() method like so:

@PostMapping("/myAccount")
// Add it here - We have names, and described the metric to help us understand how
// long it takes to receive a request and return a response.  This configures & exposes a new endpoint.
@Timed(value = "getAccountDetails.time", description = "Time taken to return Account Details")
public Accounts getAccountDetails(@RequestBody Customer customer) {

    Accounts accounts = accountsRepository.findByCustomerId(customer.getCustomerId());
    if (accounts != null) {
        return accounts;
    } else {
        return null;
    }
}

3. If you run configserver > erurekaserver > and then accounts, navigate to localhost:8080/actuator/metrics where you will see all types of metrics regarding this endpoint like /system.cpu.count or process.uptime for example. This is a cumnbersome process to monitor each and every one with just actuator....

This is why we have prepared the endpoints with Micrometer into the format that Prometheus will expect. (Go to localhost:8080/actuator/prometheus). This is the end point that will be requested by Prometheus. This is why we added the dependency name in the URL.

1. In accounts/docker-compose, generate a new folder called monitoring. Make two files in here:
   • docker-compose.yml
   • prometheus.yml

It is not the responsibility of your microservices to push the data into Prometheus. It is prometheus' job to PULL the data. So a Prometheus server must be running at the same time that your services are.

2. Add this into your prometheus.yml file - this tells Prometheus how often to scrape new metrics:

global:
  scrape_interval:     5s # Scrape new data every 5 seconds from your microservices
  evaluation_interval: 5s # Evaluate that data every 5 seconds.
scrape_configs:
  # identify microservices to monitor
  - job_name: 'accounts'
    # endpoint exposed by the service with data on health 
    metrics_path: '/actuator/prometheus'
    static_configs:
    # define host + port number
    # why accounts instead of localhost? The host name is Docker service name when calling another container
    - targets: ['accounts:8080']
  - job_name: 'loans'
    metrics_path: '/actuator/prometheus'
    static_configs:
    - targets: ['loans:8090']
  - job_name: 'cards'
    metrics_path: '/actuator/prometheus'
    static_configs:
    - targets: ['cards:9000']

3. Add this into your docker-compose.yml file:

version: "3.8"

services:

  prometheus:
  # Official image name from Dockerhub
   image: prom/prometheus:latest
   # Default ports that prometheus runs on
   ports:
      - "9090:9090"
   # Docker provides volumes as a mechanism to persist data generated by a container
   # within an environemnt network (like bank).
   volumes:
    # Prometheus expects a configuration file
    # We will copy our promethius.yml in this directory, into the container to configure Prometheus 
    - ./prometheus.yml:/etc/prometheus/prometheus.yml
   networks:
    - bank
   
  zipkin:
    image: openzipkin/zipkin
    mem_limit: 700m
    ports:
      - "9411:9411"
    networks:
     - bank

  configserver:
    image: sophiagavrila/configserver:phase8
    mem_limit: 700m
    ports:
      - "8071:8071"
    networks:
     - bank
    depends_on:
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: default
      SPRING_ZIPKIN_BASEURL: http:https://zipkin:9411/
      
  eurekaserver:
    image: sophiagavrila/eurekaserver:phase8
    mem_limit: 700m
    ports:
      - "8070:8070"
    networks:
     - bank
    depends_on:
      - configserver
    deploy:
      restart_policy:
        condition: on-failure
        delay: 15s
        max_attempts: 3
        window: 120s
    environment:
      SPRING_PROFILES_ACTIVE: default
      SPRING_CONFIG_IMPORT: configserver:http:https://configserver:8071/
      SPRING_ZIPKIN_BASEURL: http:https://zipkin:9411/

  accounts:
    image: sophiagavrila/accounts:phase8
    mem_limit: 700m
    ports:
      - "8080:8080"
    networks:
      - bank
    depends_on:
      - configserver
      - eurekaserver
    deploy:
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
        window: 120s
    environment:
      SPRING_PROFILES_ACTIVE: default
      SPRING_CONFIG_IMPORT: configserver:http:https://configserver:8071/
      EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http:https://eurekaserver:8070/eureka/
      SPRING_ZIPKIN_BASEURL: http:https://zipkin:9411/
  
  loans:
    image: sophiagavrila/loans:phase8
    mem_limit: 700m
    ports:
      - "8090:8090"
    networks:
      - bank
    depends_on:
      - configserver
      - eurekaserver
    deploy:
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
        window: 120s
    environment:
      SPRING_PROFILES_ACTIVE: default
      SPRING_CONFIG_IMPORT: configserver:http:https://configserver:8071/
      EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http:https://eurekaserver:8070/eureka/
      SPRING_ZIPKIN_BASEURL: http:https://zipkin:9411/
    
  cards:
    image: sophiagavrila/cards:phase8
    mem_limit: 700m
    ports:
      - "9000:9000"
    networks:
      - bank
    depends_on:
      - configserver
      - eurekaserver
    deploy:
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
        window: 120s
    environment:
      SPRING_PROFILES_ACTIVE: default
      SPRING_CONFIG_IMPORT: configserver:http:https://configserver:8071/
      EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http:https://eurekaserver:8070/eureka/
      SPRING_ZIPKIN_BASEURL: http:https://zipkin:9411/
      
  gatewayserver:
    image: sophiagavrila/gatewayserver:phase8
    mem_limit: 700m
    ports:
      - "8072:8072"
    networks:
      - bank
    depends_on:
      - configserver
      - eurekaserver
      - cards
      - loans
      - accounts
    deploy:
      restart_policy:
        condition: on-failure
        delay: 45s
        max_attempts: 3
        window: 180s
    environment:
      SPRING_PROFILES_ACTIVE: default
      SPRING_CONFIG_IMPORT: configserver:http:https://configserver:8071/
      EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http:https://eurekaserver:8070/eureka/
      SPRING_ZIPKIN_BASEURL: http:https://zipkin:9411/

networks:
  bank:

4. Now open a terminal within accounts/docker-compose/monitoring and run docker compose up

5. In Postman make a GET call to localhost:8080/myAccount > then check your actuator URL at localhost:8080/actuator/prometheus > ctrl + f to find: "getAccountDetails" and you will find the time that it took to process that request.

6. Now navaigate to localhost:9090/targets which will expose the Prometheus UI which shows you all instances and the prometheus endpoint URLs. You can also look at "Graph" to find out things like system_cpu_usage which easily shows all of the data scraped. If you try to send multiple requests, you'll see the cpu increase.

You can also set up alerts, and status.

7. In a separate terminal, you can kill a container like your cards service, return to Prometheus UI and check its status.

We will now integrate Grafana to expand on this dashboard.

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. We will use Prometheus as our data-source. Enterprises use Grafana to monitor their infrastructure and log analytics, predominantly to improve their operational efficiency. Dashboards make tracking users and events easy because it automates the collection, management and viewing of data.

1. In order to display all of the data captured by Promethesus, we just need to add the following service to the top of our service queue within accounts/docker-compose/monitor/docker-compose.yml:

  grafana:
    image: "grafana/grafana:latest"
    ports:
      - "3000:3000"
    # WHen we open the UI, we will be asked to login with these credentials
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=password
    networks:
     - bank
    depends_on:
      - prometheus  

2. Within the accounts/docker-compose/monitor directory, run docker compose up.

3. After all the containers have started, in your browser navigate to localhost:8080/actuator/prometheus. Here you will see the endpoints generated by actuator and formatted by micrometer to be consumed by prometheus.

4. Next, navigate to localhost:9090/target which is a UI provided by Prometheus. Here you're able to see the services that Prometheus is scraping data from.

You can click on Graphs and enter system_cpu_usage to show you the cpu usage of each individual service.
We can trigger our custom metric which is getAccountDetails_time_seconds_count by entering that into the search bar

5. All of this data is consumed by Grafana which produces a much nicer UI. Navigate to localhost:3000 > login with admin & password.

6. Click on Data Sources > add Prometheus. For the URL add http:https://prometheus:9090 (it will pick up the docker prometheus service) > Scroll down and click save.

7. Create a Dashboard: There are many Grafana templates that you can use like this. In Grafana, go to localhost:3000/dashboard/import and paste in the URL of the dashboard of your choice i.e https://grafana.com/grafana/dashboards/11378 > click load > scroll down, add prometheus as your data source and you're good to go!

You can learn about building custom Grafana Dashboards here.