Mesos framework termination - not able to run the cells on mesos

[sharmalalit]

What docker image you are using?

pyspark-notebook

What complete docker command do you run to launch the container (omitting sensitive values)?

docker run -it --rm --net=host --pid=host -e TINI_SUBREAPER=true jupyter/pyspark-notebook

What steps do you take once the container is running to reproduce the issue?

I am creating a new notebook with code as given in docs to connect to Mesos master to run my pyspark code.

What do you expect to happen?

What actually happens?
Mesos framework registration is getting failed. pyspark task is not run.
Mesos logs :
I0118 11:40:41.720917 190119936 master.cpp:2231] Received SUBSCRIBE call for framework 'pyspark-shell' at [email protected]:43715
I0118 11:40:41.720995 190119936 master.cpp:2302] Subscribing framework pyspark-shell with checkpointing disabled and capabilities [ ]
I0118 11:40:41.721012 190119936 master.cpp:2312] Framework 43cefdcf-41be-42f2-9007-86bdfcb81df0-0000 (pyspark-shell) at [email protected]:43715 already subscribed, resending acknowledgement
E0118 11:40:41.721050 191193088 process.cpp:1958] Failed to shutdown socket with fd 15: Socket is not connected
I0118 11:40:44.429878 190656512 master.cpp:2231] Received SUBSCRIBE call for framework 'pyspark-shell' at [email protected]:43715
I0118 11:40:44.429954 190656512 master.cpp:2302] Subscribing framework pyspark-shell with checkpointing disabled and capabilities [ ]
I0118 11:40:44.429965 190656512 master.cpp:2312] Framework 43cefdcf-41be-42f2-9007-86bdfcb81df0-0001 (pyspark-shell) at [email protected]:43715 already subscribed, resending acknowledgement

[sharmalalit]

Further logs :
I0118 14:49:12.352545 228540416 master.cpp:1212] Framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134 disconnected
I0118 14:49:12.352654 228540416 master.cpp:2527] Disconnecting framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134
I0118 14:49:12.352692 228540416 master.cpp:2551] Deactivating framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134
I0118 14:49:12.352813 230150144 hierarchical.cpp:375] Deactivated framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000
W0118 14:49:12.352898 228540416 master.hpp:1764] Master attempted to send message to disconnected framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134
I0118 14:49:12.353112 228540416 master.cpp:1236] Giving framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134 0ns to failover
I0118 14:49:12.353237 229076992 master.cpp:5176] Framework failover timeout, removing framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134
I0118 14:49:12.353299 229076992 master.cpp:5909] Removing framework 4d988e7e-9367-4030-b5c4-fc1273ef7197-0000 (pyspark-shell) at [email protected]:42134

[sharmalalit]

I think this is not working due to docker/for-mac#68 . Let us document this.

[parente]

Thanks for the report and pointer to possible underlying cause. We can add a note to the relevant READMEs.

[parente]

A long time in coming, but the note is now in the README. Thanks @sharmalalit .