Storm docker cluster using docker-compose. Including a Prometheus/Graphite container for Storms Metrics Reporting API v2 and Grafana for metrics visualization. Docker-compose/stack also includes Kafka service as example testdata source.

Based on offical Apache Storm docker image , Graphite docker image / Prometheus docker image, Grafana docker image, wurstmeister Apache Kafka image and cAdvisor.

• For swarm deploy set up docker swarm and set labels / edit placement labels accordingly in docker-stack.yml
• Adding configs to storm.yaml file overwrites the default values (for version 2.2.0).
• configure the used metric exporter and potential metric filter regex expressions according to Prometheus/Graphite usage in storm.yaml
• to collect host metrics with the included Prometheus service install and start Prometheus Node exporter on hosts
• If graphite is used set cAdvisor enviroment variable to use statsd storage driver of the graphite service:

environment: 
- storage_driver=statsd
- storage_driver_host=graphite:8125

$ docker-compose -f docker-stack.yml build

$ docker stack deploy --compose-file docker-stack.yml storm-cluster

or utilize scripts in /buildscripts

$ ./nimbus.sh

$ storm jar /topology.jar MainTopology Mytopology

• configure enviroment variables (TOPOLOGY_JAR, TOPOLOGY_MAIN, TOPOLOGY_Name) in Dockerfile

• on nimbus shell:

  /./deploy_topology.sh

• get shell on kafka:

  ./kafka.sh

• execute kafka generator jar

  java -jar kafka-generator-1.0-SNAPSHOT-jar-with-dependencies.jar

• the example messages generated by this kafka-generator are compatible to the provided example topology

When using swarm deployment, services are available via all host IPs. (see: ingress)

Storm UI: https://host:8080

Prometheus Dashboard: https://host:9090

Grafana Dashboard: https://host:3000 (userdetails: admin/admin)