Storm docker cluster using docker-compose. Including a Prometheus/Graphite container for Storms Metrics Reporting API v2 and Grafana for metrics visualization. Docker-compose/stack also includes Kafka service as example testdata source.
Based on offical Apache Storm docker image , Graphite docker image / Prometheus docker image, Grafana docker image, wurstmeister Apache Kafka image and cAdvisor.
- For swarm deploy set up docker swarm and set labels / edit placement labels accordingly in docker-stack.yml
- Adding configs to storm.yaml file overwrites the default values (for version 2.2.0).
- configure the used metric exporter and potential metric filter regex expressions according to Prometheus/Graphite usage in storm.yaml
- to collect host metrics with the included Prometheus service install and start Prometheus Node exporter on hosts
- If graphite is used set cAdvisor enviroment variable to use statsd storage driver of the graphite service:
environment:
- storage_driver=statsd
- storage_driver_host=graphite:8125
$ docker-compose -f docker-stack.yml build
$ docker stack deploy --compose-file docker-stack.yml storm-cluster
or utilize scripts in /buildscripts
$ ./nimbus.sh
$ storm jar /topology.jar MainTopology Mytopology
-
configure enviroment variables (TOPOLOGY_JAR, TOPOLOGY_MAIN, TOPOLOGY_Name) in Dockerfile
-
on nimbus shell:
/./deploy_topology.sh
- get shell on kafka:
./kafka.sh
- execute kafka generator jar
java -jar kafka-generator-1.0-SNAPSHOT-jar-with-dependencies.jar
- the example messages generated by this kafka-generator are compatible to the provided example topology
When using swarm deployment, services are available via all host IPs. (see: ingress)
Storm UI: https://host:8080
Prometheus Dashboard: https://host:9090
Grafana Dashboard: https://host:3000 (userdetails: admin/admin)