Some containers does not become the running state in Zenko Swarm Stack.

[InseobKim]

Hi,
I deployed Zenko swarm stack but some containers does not become the running state in Zenko swarm stack. Those are zenko-prod_spark-master, zenko-prod_spark-master and zenko-prod_livy.

# docker service ls
ID                  NAME                           MODE                REPLICAS            IMAGE                                    PORTS
h08dy8pzgw2w        zenko-prod_backbeat-consumer   replicated          1/1                 zenko/backbeat:pensieve-3
yvnrai069v97        zenko-prod_backbeat-producer   replicated          1/1                 zenko/backbeat:pensieve-3
w0mlb7v3jslp        zenko-prod_cache               replicated          1/1                 redis:alpine                             *:30014->6379/tcp
27h85v71f30p        zenko-prod_graphite            replicated          1/1                 scality/clueso-grafana-graphite:latest   *:3000->3000/tcp,*:8005->80/tcp,*:8081->81/tcp
9zg4k9k1oeed        zenko-prod_lb                  global              6/6                 zenko/loadbalancer:latest                *:80->80/tcp
xbtm8jgtvwgu        zenko-prod_livy                replicated          0/1                 scality/clueso-livy:pensieve             *:4040->4040/tcp,*:4041->4041/tcp,*:4042->4042/tcp,*:4043->4043/tcp,*:4044->4044/tcp,*:4045->4045/tcp,*:4046->4046/tcp,*:4047->4047/tcp,*:4048->4048/tcp,*:4049->4049/tcp,*:8998->8998/tcp
5op1u5jlvqqc        zenko-prod_queue               replicated          1/1                 wurstmeister/kafka:1.0.0                 *:30016->9092/tcp
e81x55nmmzdf        zenko-prod_quorum              replicated          1/1                 zookeeper:3.4.11                         *:30017->2181/tcp
5fph6v5y5pnx        zenko-prod_s3-data             replicated          1/1                 zenko/cloudserver:pensieve-1             *:30012->9992/tcp
051pizoatpdy        zenko-prod_s3-front            replicated          1/1                 zenko/cloudserver:pensieve-1             *:30015->8001/tcp
4lu9m8ikgp3p        zenko-prod_s3-metadata         replicated          1/1                 zenko/cloudserver:pensieve-1             *:30013->9993/tcp
0u382awvwprg        zenko-prod_spark-master        replicated          0/1                 scality/clueso-spark-master:pensieve     *:4050->4050/tcp,*:4051->4051/tcp,*:4052->4052/tcp,*:4053->4053/tcp,*:8080->8080/tcp
pxpahj5xz8y5        zenko-prod_spark-worker        replicated          0/1                 scality/clueso-spark-worker:pensieve

zenko-prod_livy becomes up and down repeatly so I checked following logs in zenko-prod_livy.

# docker service logs zenko-prod_livy
zenko-prod_livy.1.rlc2jexkus5r@ip-172-31-4-21    | Waiting for spark master
zenko-prod_livy.1.whbqhss6asan@ip-172-31-4-21    | Waiting for spark master
zenko-prod_livy.1.whbqhss6asan@ip-172-31-4-21    | Cannot contact spark master on 7077
zenko-prod_livy.1.xfco4fq4pcr2@ip-172-31-31-120    | Waiting for spark master
zenko-prod_livy.1.zf0m90z8q016@ip-172-31-31-120    | Waiting for spark master
zenko-prod_livy.1.zf0m90z8q016@ip-172-31-31-120    | Cannot contact spark master on 7077
zenko-prod_livy.1.xfco4fq4pcr2@ip-172-31-31-120    | Cannot contact spark master on 7077
zenko-prod_livy.1.ykj1nwuuft7q@ip-172-31-29-49    | Waiting for spark master
zenko-prod_livy.1.ykj1nwuuft7q@ip-172-31-29-49    | Cannot contact spark master on 7077

How can I fixed this problems?

[LaureVergeron]

Hey @InseobKim ,

livy has been known to be a bit flaky lately. If you want metadata search, you can simply delete your stack and try to redeploy it - usually it fixes itself that way.
If you don't need metadata search, you can simply edit the docker-stack.yml to remove livy, spark master, spark worker, and graphite, and run again.

Let me know if that helps.