Segfaults in v2.0.17

[sauvant]

Keep getting segfaults with v2.0.17 running on docker->plesk->debian:
mosquitto[24889]: segfault at 500000000 ip 00007fba45142fc1 sp 00007fff4bc3e338 error 4 in ld-musl-x86_64.so.1[7fba45104000+4c000]

Did not find a reproduce case yet.

Environment:
Debian 10.13
Plesk Obsidian v18.0.54_build1800230824.08 os_Debian 10.0
Docker version 24.0.5, build ced0996
Mosquitto version 2.0.17

[NorbertHeusser]

Hi Keith,
any more information you are able to provide about your setup/usage of Mosquitto. Or even some log files ?
Best would be to get some log files created with log_type all.

If this is not possible any additional kind of information might help, like:

• Any plugins used (e.g. dynamic security) ?
• Persistence on the broker enabled ?
• Clients using persistence session (clean session = false) ?
• Any messages with retain/qos1/qos2 ?

Without any kind of detail it will be hard to pin down the problem.

[tbclark3]

I'm getting a similar segfault on 2.0.17 almost every time I restart Home Assistant:
mosquitto[608511]: segfault at 200000000 ip 00007f0b8cefffc1 sp 00007ffff9d82498 error 4 in ld-musl-x86_64.so.1[7f0b8cec1000+4c000] likely on CPU 20 (core 16, socket 0)

Home Assistant has several hundred mqtt entities, all of which have to update at startup, but I don't think that's an unusual number. I have been experimenting with configs but haven't been able to find anything consistent.

Persistence is enabled, and I am not using any plugins. There are many messages with retain set, from Home Assistant and other clients.

[RoboMagus]

Been seeing the same thing lately.
At first I thought it had something to do with the Saving in-memory database stuff. See #2726.
But after some monitoring of the docker status for the Mosquitto container I've seen 139 exit codes come by. Looking at the log this occurs both on new connections (not just HA, but other clients as well) as well as the DB writes.

[LucidityCrash]

Wish I could add more then a "Me too", not noticing it for connects/disconnects with mqtt explorer on windows but restarting Home Assistant causes a seg fault in Mosquitto 2.1.7, reverting to 2.1.5 (what I was previously running) and issue goes away

[tbclark3]

I can add that the crash happens during the startup phase of Home Assistant (and not during the shutdown phase). It is not instant; the MQTT crash happens several seconds into the startup, but I'm not sure if it's a single transaction that causes it of the cumulative effect of several startup transactions.

[halfgaar]

Probably the same thing I found in #2881. I have a stack trace there. It started in version 2.0.16 and reverted out servers back to 2.0.15.

[ralight]

We've finally managed to replicate this once after running on test.mosquitto.org for a good while, so hopefully should have an update soon.

[ralight]

Could you please try out the master branch if convenient? I intend to make a new release in the next 12 hours.

[halfgaar]

I haven't been able to reproduce it; I only saw it on the live system, so I don't know what to do.

Does it make sense this behavior was introduced since 2.0.15?

This seems to directly revert 18ea97c "Fixes sub_count is not decreased when client ubsubscribe" BTW.

[ralight]

@halfgaar Yes, it does directly revert that commit - it was an incorrect back port from develop where the case is slightly different. In develop we have sub_capacity, which is the size of the subscription array, and sub_count, which is the count of subscriptions we have. In 2.0, we just have sub_count, which refers to the size of the array. sub_count is also accessible to plugins through the mosquitto_client_sub_count() call. In the case of 2.0, this call may produce incorrect values, hence the change in develop.

[tbclark3]

@ralight, thanks for your work on this! You mentioned a new release within 12 hours, but that was several days ago. Do you have a timeframe for a new release?