crash in libmosquitto loop when trying to disconnect

[shapiroj]

This is using the current develop branch (279bd33). When trying to disconnect the mosq loop there is a crash because there are two threads trying to call net__socket_close on the same SSL object. One from the thread I'm calling disconnect on, and one from the main loop thread that's running.

gdb)  info threads
  Id   Target Id                                         Frame
  1    Thread 0x7ffff6bd89c0 (LWP 288) "mqtt-client" net__socket_close (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/net_mosq.c:230
  2    Thread 0x7ffff6896700 (LWP 289) "mqtt-client" futex_abstimed_wait_cancelable (private=0, abstime=0x7ffff68857d0, clockid=-158836976, expected=0, futex_word=0x7ca9d4) at ../sysdeps/nptl/futex-internal.h:323
* 4    Thread 0x7ffff5894700 (LWP 291) "mqtt-client" 0x00007ffff7cace51 in net__socket_close (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/net_mosq.c:230
(gdb) bt
#0  0x00007ffff7cace51 in net__socket_close (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/net_mosq.c:230
#1  0x00007ffff7ca92e2 in mosquitto__loop_rc_handle (mosq=0x7d9f70, rc=14) at /build/vendor/github.com/eclipse/mosquitto/lib/loop.c:334
#2  0x00007ffff7ca8d4d in mosquitto_loop_write (mosq=0x7d9f70, max_packets=1) at /build/vendor/github.com/eclipse/mosquitto/lib/loop.c:401
#3  0x00007ffff7ca8a60 in mosquitto_loop (mosq=0x7d9f70, timeout=-1, max_packets=1) at /build/vendor/github.com/eclipse/mosquitto/lib/loop.c:169
#4  0x00007ffff7ca8e48 in mosquitto_loop_forever (mosq=0x7d9f70, timeout=-1, max_packets=1) at /build/vendor/github.com/eclipse/mosquitto/lib/loop.c:258
#5  0x00000000005565e7 in mc::mqtt::MosqClient::loop (this=0x7cd8c0) at /build/mqtt_client/src/mqtt.cpp:84
#6  0x000000000055500b in mc::mqtt::ClientService::runDriverLoop (this=0x7d9d10) at /build/mqtt_client/src/mqtt.cpp:725
#7  0x000000000055b4e4 in std::__invoke_impl<int, int (mc::mqtt::ClientService::*)(), std::reference_wrapper<mc::mqtt::ClientService>>(std::__invoke_memfun_ref, int (mc::mqtt::ClientService::*&&)(), std::reference_wrapper<mc::mqtt::ClientService>&&) (__f=@0x7d0820: (int (mc::mqtt::ClientService::*)(mc::mqtt::ClientService * const)) 0x554360 <mc::mqtt::ClientService::runDriverLoop()>, __t=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:66
#8  0x000000000055b3f2 in std::__invoke<int (mc::mqtt::ClientService::*)(), std::reference_wrapper<mc::mqtt::ClientService> > (__fn=@0x7d0820: (int (mc::mqtt::ClientService::*)(mc::mqtt::ClientService * const)) 0x554360 <mc::mqtt::ClientService::runDriverLoop()>, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:95
#9  0x000000000055b3b5 in std::thread::_Invoker<std::tuple<int (mc::mqtt::ClientService::*)(), std::reference_wrapper<mc::mqtt::ClientService> > >::_M_invoke<0ul, 1ul> (this=0x7d0818) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/thread:264
#10 0x000000000055b365 in std::thread::_Invoker<std::tuple<int (mc::mqtt::ClientService::*)(), std::reference_wrapper<mc::mqtt::ClientService> > >::operator() (this=0x7d0818) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/thread:271
#11 0x000000000055b20e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (mc::mqtt::ClientService::*)(), std::reference_wrapper<mc::mqtt::ClientService> > > >::_M_run (this=0x7d0810) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/thread:215
#12 0x00007ffff7a36ed0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff7c7dea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#14 0x00007ffff7731a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff6bd89c0 (LWP 288))]
#0  net__socket_close (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/net_mosq.c:230
230                                     SSL_shutdown(mosq->ssl);
(gdb) bt
#0  net__socket_close (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/net_mosq.c:230
#1  0x00007ffff7ca500d in do_client_disconnect (mosq=0x7d9f70, reason_code=0, properties=0x0) at /build/vendor/github.com/eclipse/mosquitto/lib/connect.c:281
#2  0x00007ffff7cb1ee3 in packet__write (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/packet_mosq.c:327
#3  0x00007ffff7cb1bf1 in packet__queue (mosq=0x7d9f70, packet=0x7cb480) at /build/vendor/github.com/eclipse/mosquitto/lib/packet_mosq.c:217
#4  0x00007ffff7cb6097 in send__disconnect (mosq=0x7d9f70, reason_code=0 '\000', properties=0x0) at /build/vendor/github.com/eclipse/mosquitto/lib/send_disconnect.c:84
#5  0x00007ffff7ca4fcd in mosquitto_disconnect_v5 (mosq=0x7d9f70, reason_code=0, properties=0x0) at /build/vendor/github.com/eclipse/mosquitto/lib/connect.c:273
#6  0x00007ffff7ca4e6b in mosquitto_disconnect (mosq=0x7d9f70) at /build/vendor/github.com/eclipse/mosquitto/lib/connect.c:243
#7  0x00000000005563ed in mc::mqtt::MosqClient::disconnect (this=0x7cd8c0) at /build/mqtt_client/src/mqtt.cpp:58
#8  0x0000000000551db9 in mc::mqtt::ClientService::shutdown (this=0x7d9d10) at /build/mqtt_client/src/mqtt.cpp:334
#9  0x0000000000488fcc in mc::mqtt::ClientService::~ClientService (this=0x7d9d10) at /build/mqtt_client/src/mqtt.h:88
#10 0x0000000000488f2b in std::default_delete<mc::mqtt::ClientService>::operator() (this=0x7cc608, __ptr=0x7d9d10) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:85
@