As microservices scale out, it becomes increasingly important to trace transactions. This is helpful when we need to debug so that we can discover what went wrong, adn where. Additionally, if we are utilizing logging in our microservices, looking at all of the logs generated by all services can become a nightmare - that's where log aggregation comes in to help us organize and group our logs generated by separate services.

Distributed Tracing and Log Aggregation answers these three questions:

We must implement distributed tracing to trace one or more transactions across multiple services, physical machines, and different data stores to try and find exactly where a bug is occuring. There is a way to append the trace id with the help of a distributed tracing tool, rather than manually logging it each time, which is cumbersome.

We must combine all the logs from multiple services into a central location where they can be indexed, searched, filtered, and grouped to find bugs that are contributing to a problem.

We must understand the specific path that a service call took as it travelled within our microservices network, and the time it took as each microservice. This helps us understand why a requess it taking extra long.

• Spring Cloud Sleuth provides Spring Boot auto-configuration for distributed tracing.
• It adds trace and span ids to all the logs, so you can just extract from a given trace or span in a log aggregator.
• It does this by adding filters and interacting with other Spring components to pass the generated correlation ids through all system calls.

Spring Cloud Sleuth will add three pieces of information to all the logs written by a microservice.
[<App Name>, <Trace ID>, <Span ID>
<App Name>: The application name where the log entry is generated. This comes from the spring.application.name property.
<Trace ID>: Trace ID is the equivelent term for the correlation ID. It's a unique number that represents the entire transaction.
<Span ID>: Span ID is a unique ID tha trepresents part of the overall transaction, Each service participating within the transaction will have its own span ID. Span ID's are particularly relevant when you integrat with Zipkin to visualize your transactions.

• Zipkin is an open-source data-visualization tool that helps aggregate all the logs and gather timing data needed to troubleshoot latency problems in microservices architecture.
• It allows us to break a transaction down into into components and visually indentify where there might be performance hotspots, thus reducing triage time by contextualizing errors and delays.

1. Ineach of your microservices, add the Spring Cloud Sleuth starter dependency. It should look like this in your pom.xml's:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

2. We have a few log.info() statements inside of accounts, cards, and loans. So when we send a POST request to the myCustomerDetails() method in the accounts services, we should generate a trace id that travels through all 3 services.

3. Start config > eureka > accounts > cards > loans > gateway. Clear all of the consoles. (Alternatively to save your machine and some time, make sure all of your images are pushed and run docker compose up -d within accounts/docker-compose/default) Then send a POST request to accounts with { "customerId" : 1 } as the Request Body at the location https://localhost:8072/bank/accounts/myCustomerDetails

4. After you've sent the request, go the the console of any of your services, like cards for example. (If you're using docker, run docker ps to find the contianer id of your cards servce > run docker logs -f <cards-container id> to follow the logs within the cards container. This is the first place that receives the request.

The logs, with the help of Spring Cloud Sleuth, will look like this:

2021-10-14 12:10:41.501  INFO [cards,10790a3bbef07309,70dfeef6153bfee7] 1 --- [nio-9000-exec-1] c.r.cards.controller.CardsController     : getCardDetails() method started
2021-10-14 12:10:42.046  INFO [cards,10790a3bbef07309,70dfeef6153bfee7] 1 --- [nio-9000-exec-1] c.r.cards.controller.CardsController     : getCardDetails() method ended

1. Set up Zipkin Server with docker on port 9411. Run: docker run -d -p 9411:9411 openzipkin/zipkin

2. Visit the UI provided by Zipkin server by going to https://localhost:9411/zipkin/ in your browser. There are no traces featured in the console because we have to configure our services! Let's do that.

3. Add the Zipkin dependency to each of your services' pom.xml. It should look like this:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>

1. Now we have to configure the zipkin endpoint for each service within each service's respective application.proeprties file. In each one, add the following:

# By default, for performance issues, Sleuth will send only 10% of the logs to Zipkin
# Here we set it to 100% (if you want 50% you write 0.5)
spring.sleuth.sampler.percentage=1
# This is the endpoint to send the logs 
spring.zipkin.baseUrl=https://localhost:9411/

2. Open your docker-compose.yml file within accounts/docker-compose/default. Paste the following code above the config service so that Zipkin is the first service to be started in the service queue.

  zipkin:
    image: openzipkin/zipkin
    mem_limit: 700m
    ports:
      - "9411:9411"
    networks:
     - bank

2. Under each service, you will need to add: SPRING_ZIPKIN_BASEURL: https://zipkin:9411/ as the last value under environment: (check the docker-compose file here if you need to check what it looks like).

3. You can either run all of your services locally, or you can generate the updated Docker images, push them to Docker Hub, and run docker compose up in a terminal within your accounts/docker-compose/default directory. (Reccomended you use Docker).

4. Send the same POST request to https://localhost:8072/bank/accounts/myCustomerDetails again.

5. Navigate to https://localhost:9411/zipkin/, click the red plus button and click serviceName, after the = sign type accounts > click Run Query

This is the end of this phase. If you're interested in persisting these logs to a data store or stream-analyzation service like Kakfa, you would use RabbitMQ to serve as an asynchronous message service to manage the logs sent from Spring Cloud Sleuth -> to Zipkin -> to any Ddata store of your choice or to Splunk, Kafka, etc.

Warning, the below steps are very heavy on your system because you would be running RabbitMQ on your server as well.

RabbitMQ is an open source message broker software. It accepts messages from producers, and delivers them to consumers. It acts like a middleman which can be used to reduce loads and delivery times taken by web application servers. RabbitMQ is an implementation of Java Messaging Service. It is a place where we can push all of the logs generated by Spring Cloud Sleuth asynchronously.

Splunk, ELK, and Kafka are other products we can use to aggrigate our logs. From the queue, you would use those services to aggregate the logs.

1. Add the Spring for RabbitMQ dependency to each service.

2. We can start RabbitMQ with the following docker command:

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

RabbitMQ has 2 different applications that are running, UI, and a server that is tracking and monitoring the environment for incoming logs.

3. Navigate to localhost: 15672 > enter "guest" for username and password.

4. Set up a new queue: Goe to "Queues" > Click "Add a new queue" > give it the name "zipkin" > click "Add Queue"

5. Clcik on the zipkin queue > scroll down and click "Get Message(s)" > you'll notice that there are no messages in the queue. There are also no connections.

6. Go back to your microservices where you have added the Rabbit dependencies > in each service's application.properties file, add these RabbitMQ related properties:

spring.zipkin.sender.type=rabbit
spring.zipkin.rabbitmq.queue=zipkin
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest

7. This is the addition you would push to the top of your service queue in your docker-compose.yml file.

version: "3.2"
services:
  rabbitmq:
    image: rabbitmq:3-management-alpine
    container_name: 'rabbitmq'
    ports:
        - 5672:5672
        - 15672:15672
    networks:
        - bank

8. You would also make sure that zipkin relies on RabbitMQ. Additionally modify your Zipkin dependency like so:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>

If you're interested in RabbitMQ, you can find specific setup guides online like this one