Mosquitto stops publishing when using persistent connections with QoS2

[yschroeder]

Ok, I have something very hard to reproduce and I do not know how to reproduce it reliably yet:

Sometimes, when a client publishes a retained message to a topic that already has a retained message the new message is published but the old retained message stays retained. Example:

retained message "foo" on topic "test"
A subscribes to "test", receives "foo"
B publishes "bar" to "test" as retained message
A receives "bar"
A receives "foo" (again?!)
C subscribes to "test", receives "foo" (should receive "bar")

I do not know exactly yet what is going on here, it dos not always happen in my setup. I was not able to reproduce it with a minimal example.

B is a persistent client with LWT set.
In my real test the message sent by B is rather large (some kilobytes)
All messages are published and subscribed with QoS=2.
Everything in running on localhost.
Mosquitto is v1.6.3 on Arch Linux.

To me it looks like some race condition that I cannot trigger reliably.

[yschroeder]

Ok, I found a way to reproduce this (after 4 hours of debugging and finding a minimum example):

# restart mosquitto
sudo systemctl restart mosquitto.service

# listen on topic1
mosquitto_sub -t "topic1"

# publish to topic1 without clean session
mosquitto_pub -t "topic1" -q 2 -c --id "foobar" -m "message1"
# message1 on topic1 is received as expected

# publish to topic2 without clean session
# IMPORTANT: no subscription to this topic is present on broker!
mosquitto_pub -t "topic2" -q 2 -c --id "foobar" -m "message2"
# this goes nowhere, as no subscriber present

# publish to topic1 without clean session
mosquitto_pub -t "topic1" -q 2 -c --id "foobar" -m "message3"
# message3 on topic1 IS NOT RECEIVED

# listen on topic2 
mosquitto_sub -t "topic2"

# publish to topic1 without clean session
mosquitto_pub -t "topic1" -q 2 -c --id "foobar" -m "message4"
# message2 on topic2 is received (WHAT THE F**K?)

# publish to topic1 without clean session
mosquitto_pub -t "topic1" -q 2 -c --id "foobar" -m "message5"
# message5 on topic1 is received as expected (message4 was dropped)

Basically, when you use persistent connections -c with QoS2 -q 2 and you publish to a topic without subscriber, shit hits the fan. Big time.

Mosquitto version: 1.6.3

[yschroeder]

@ralight It would be very nice if you could address this in 1.6.4 as it is a breaking bug for our application.

Our workaround currently is to have some client listen to topic #, to ensure that there is a subscriber on every topic.

By the way, this is a proper "Heisenbug": It only happens when you are not trying to figure out what happens (no subscriber present)...

[ralight]

Thank you for the very straightforward path to follow and the effort to create it! It took a bit of figuring out, but I believe this is now fixed in the fixes branch and will be part of 1.6.4 shortly.

I spent a year in Braunschweig a fair few years ago, it's a nice place.

[yschroeder]

Thanks for the fix @ralight!

We use Mosquitto to relay messages between devices in our timekeeping system for Formula Student Germany. We had version 1.6.4 running this year and it worked like a charm!

From next year on our main timekeeping system will completely work with MQTT. We send data from our RFID readers and light barriers to the main system via MQTT, control attached 7 segment displays via MQTT, have a operator interface in a browser that runs on MQTT via websockets and have a bridge to a second Mosquitto in the Internet that is used for live timing for the bigger audience (also overlay for a Youtube livestream). And all of this is powered by a central Mosquitto instance :-)

Here you can see a bit of the action (and some timekeeping equipment if you watch closely): https://www.youtube.com/watch?v=2GbuATnjenI

Thank you for the great project!

I like Braunschweig as well, I am here for 11 years now ;-)