Demo setup for Spring Boot apps with Prometheus, Grafana, Loki, Tempo, Eureka, and Spring Boot Admin to demonstrate Observability use-cases.
docker compose up
rm -rf logs
If you want to use an in-memory DB (H2):
./gradlew bootRun
If you want to use a real DB (MySQL):
./gradlew bootRun -Pprofiles=mysql
You need a real DB if you want to inject latency on the network (see ToxiProxy).
See SteepTeaSimulation.java
for duration, request rate, and traffic patterns.
./gradlew :load-gen:gatlingRun
docker compose down
docker compose down --volumes
- Tea UI: http:https://localhost:8090/steep
- Tea Service: http:https://localhost:8090
- Tealeaf Service: http:https://localhost:8091
- Water Service: http:https://localhost:8092
- Spring Boot Admin: http:https://localhost:8080
- Eureka: http:https://localhost:8761
- Prometheus: http:https://localhost:9090
- Loki, Grafana, Tempo: http:https://localhost:3000
- ToxiProxy UI (failure injection): http:https://localhost:8484
- MailDev (emails for alerts): http:https://localhost:3001
- Adminer (DB Admin UI): http:https://localhost:8888 (credentials:
root:password
)
When start the apps for the first time, english breakfast
is missing from the DB but you can make requests through the UI using english breakfast
and the load generator also sends requests containing it. Those calls will end-up with HTTP 500; approximately 10% of the requests should fail: ~0.5 rq/sec error- and ~4.5 rq/sec success rate (~5 rq/sec total throughput, see SteepTeaSimulation.java
).
You should see these errors on the throughput panel of the Tea API dashboard and Grafana also alerts on them (see the emails in MailDev).
If you want to fix these errors, you need to create a record in the DB for english breakfast
. The easiest way is sending an HTTP POST request to /tealeaves
to create the resource (you can also log into the DB and insert the record for example using Adminer). The Makefile
contains a goal for this to make it simple for you, you can run this to fix errors (httpie and jq needed):
make errors-fixed
If you want the errors back again, you need to remove the record from the DB, the Makefile
contains a goal for this too, so you can run this to inject errors:
make errors
If you start the apps with the mysql
profile, the apps are not connected to the DB directly but through ToxiProxy so that you can inject failures (i.e.: latency) on the network. You can do this in multiple ways (e.g.: using the ToxiProxy UI or the ToxiProxy CLI). The Makefile
contains a goal for this to make it simple for you, you can run this to inject latency:
make chaos
And this to eliminate the extra latency:
make order