mosquitto_loop_stop hangs when stopping client run in alpine linux based docker container

[BlackZork]

I cannot shutdown my Modbus to MQTT gateway when run on Apline Linux in docker container.
It works on if container is Debian (glibc) based.

It hangs waiting for mosquitto_loop_stop(ctx, false) in:

#0  __cp_end () at src/thread/x86_64/syscall_cp.s:29
#1  0x000073ef579eb4fc in __syscall_cp_c (nr=202, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=<optimized out>, y=<optimized out>, z=0) at src/thread/pthread_cancel.c:33
#2  0x000073ef579eaa2e in __futex4_cp (to=<optimized out>, val=2, op=128, addr=0x73ef56d3cb70) at src/thread/__timedwait.c:24
#3  __timedwait_cp (addr=addr@entry=0x73ef56d3cb70, val=2, clk=clk@entry=0, at=at@entry=0x0, priv=128, priv@entry=1) at src/thread/__timedwait.c:52
#4  0x000073ef579ecc2d in __pthread_timedjoin_np (t=0x73ef56d3cb38, res=0x0, at=0x0) at src/thread/pthread_join.c:18
#5  0x000073ef576f59ad in mosquitto_loop_stop () from /usr/lib/libmosquitto.so.1

Steps to reproduce:

1. Run mosquitto server on host, reachable from docker container without authorization
2. Build container with service and run shell:

[zork@archdevel modbusmqttgw]$ git clone --branch v1.4.0 https://github.com/BlackZork/mqmgateway.git
[zork@archdevel modbusmqttgw]$ cd mqmgateway
[zork@archdevel modbusmqttgw]$ cp <from_issue>/Dockerfile ./
[zork@archdevel modbusmqttgw]$ cp <from_issue>/config.yaml modmqttd/
[zork@archdevel modbusmqttgw]$ docker build .
[zork@archdevel modbusmqttgw]$ docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --rm -ti --entrypoint /bin/sh -v "$PWD/modmqttd/config.yaml:/etc/modmqttd/config.yaml" --workdir=/etc/modmqttd --name mqm <image_id>

Dockerfile and config.yaml prepared for this bug:
alpine_loop_stop_bug.tar.gz

You may need to change IP address in config.yaml for non std docker settings.

2. Run service in background:

/etc/modmqttd # modmqttd &                                                                                                                                                                                                                                             
/etc/modmqttd # 2024-Jan-19 16:27:17.233694: [INFO]     modmqttd is starting      

3. It will output some logs, important lines are that it connects successfully to configured broker:

2024-Jan-19 16:27:17.234918: [INFO]     Connecting to 172.17.0.1:1883
2024-Jan-19 16:27:17.235200: [INFO]     Connection established
2024-Jan-19 16:27:17.235262: [INFO]     Mqtt connected, sending subscriptions…
2024-Jan-19 16:27:17.235294: [INFO]     Mqtt ready to process messages

4. Send SIGHUP:

/etc/modmqttd # pkill modmqttd
/etc/modmqttd # 2024-Jan-19 16:27:23.176159: [INFO]     Got SIGTERM, exiting…
[...]
2024-Jan-19 16:27:23.176603: [INFO]     Stopping mosquitto message loop

Now it hangs in mosquitto_loop_stop.

5. Find PID, connect gdb, switch to mosquitto thread and see where it hangs:

/etc/modmqttd # ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
    7 root      0:00 modmqttd
   11 root      0:00 ps aux
/etc/modmqttd # gdb -p 7
GNU gdb (GDB) 14.1

Attaching to process 7
[New LWP 9]
__cp_end () at src/thread/x86_64/syscall_cp.s:29

warning: 29     src/thread/x86_64/syscall_cp.s: No such file or directory
(gdb) info threads
  Id   Target Id              Frame 
* 1    LWP 7 "modmqttd"       __cp_end () at src/thread/x86_64/syscall_cp.s:29
  2    LWP 9 "mosquitto loop" __cp_end () at src/thread/x86_64/syscall_cp.s:29
(gdb) thread 2
[Switching to thread 2 (LWP 9)]
#0  __cp_end () at src/thread/x86_64/syscall_cp.s:29
29      in src/thread/x86_64/syscall_cp.s
(gdb) bt
#0  __cp_end () at src/thread/x86_64/syscall_cp.s:29
#1  0x000073ef579eb4fc in __syscall_cp_c (nr=202, u=<optimized out>, v=<optimized out>, w=<optimized out>, x=<optimized out>, y=<optimized out>, z=0) at src/thread/pthread_cancel.c:33
#2  0x000073ef579eaa2e in __futex4_cp (to=<optimized out>, val=2, op=128, addr=0x73ef56d3cb70) at src/thread/__timedwait.c:24
#3  __timedwait_cp (addr=addr@entry=0x73ef56d3cb70, val=2, clk=clk@entry=0, at=at@entry=0x0, priv=128, priv@entry=1) at src/thread/__timedwait.c:52
#4  0x000073ef579ecc2d in __pthread_timedjoin_np (t=0x73ef56d3cb38, res=0x0, at=0x0) at src/thread/pthread_join.c:18
#5  0x000073ef576f59ad in mosquitto_loop_stop () from /usr/lib/libmosquitto.so.1
#6  0x000073ef57911479 in modmqttd::Mosquitto::stop (this=0x73ef57a32d50) at /opt/mqmgateway/source/libmodmqttsrv/mosquitto.cpp:149
#7  0x000073ef57912b5b in modmqttd::MqttClient::onDisconnect (this=0x73ef57287bc0) at /opt/mqmgateway/source/libmodmqttsrv/mqttclient.cpp:78
#8  0x000073ef579116ad in modmqttd::Mosquitto::on_disconnect (this=0x73ef57a32d50, rc=0) at /opt/mqmgateway/source/libmodmqttsrv/mosquitto.cpp:174
#9  0x000073ef57910950 in modmqttd::on_disconnect_wrapper (mosq=0x73ef572638c0, userdata=0x73ef57a32d50, rc=0) at /opt/mqmgateway/source/libmodmqttsrv/mosquitto.cpp:26
#10 0x000073ef576eca53 in ?? () from /usr/lib/libmosquitto.so.1
#11 0x000073ef576f1e9a in ?? () from /usr/lib/libmosquitto.so.1
#12 0x000073ef576ee64b in mosquitto_loop_write () from /usr/lib/libmosquitto.so.1
#13 0x000073ef576ee9ec in mosquitto_loop () from /usr/lib/libmosquitto.so.1
#14 0x000073ef576eea92 in mosquitto_loop_forever () from /usr/lib/libmosquitto.so.1
#15 0x000073ef576f58b1 in ?? () from /usr/lib/libmosquitto.so.1
#16 0x000073ef579ec22e in start (p=0x73ef56d3cac8) at src/thread/pthread_create.c:207
#17 0x000073ef579ee82f in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC
(gdb)