Persistence Issue - Mosquitto 2.0.15 and Paho MQTT C client - MQTT v5

[rywager]

I have a system I am building that relies on MQTT for communication across nodes. I've done everything i can think of to make persistence work but no matter what whenever a client disconnects, and reconnects, with the exact same client ID, mosquitto with debug all treats them as a brand new connection. Will provide details below but I can provide anything else needed to help find if this is a bug or if I am just missing something.

The clients subscribe to topics with QOS1/2 and reply as they should until restart and then come back up and no longer get the messages.

Here's 2 examples of this occurring for one specific client. The clientID's are all unique because the clientID is their serial number. you can also see here that i am c0.

1709940654: New client connected from 192.168.128.123:50418 as B8A44F3205A4 (p5, c0, k20, u'baton').
1709942173: New client connected from 192.168.128.123:50498 as B8A44F3205A4 (p5, c0, k20, u'baton').

Here's the config file builder in C (sharing it but obviously have the file too)

fprintf(config_file, "listener 1883\n");
fprintf(config_file, "log_type all\n");
fprintf(config_file, "persistence true\n");
fprintf(config_file, "persistence_location /var/log/mosquitto\n");
fprintf(config_file, "persistent_client_expiration 1y\n");
fprintf(config_file, "autosave_on_changes true\n");
fprintf(config_file, "autosave_interval 100\n");
fprintf(config_file, "protocol mqtt\n");
fprintf(config_file, "log_dest file /var/log/mosquitto/baton_mosquitto.log\n");
fprintf(config_file, "allow_anonymous false\n");
fprintf(config_file, "max_connections -1\n");
fprintf(config_file, "password_file %s\n", password_file_path);

Here's logfile showing it's saving to the file when i send SIGUSR1 (however i will also note, that's the ONLY time it ever saves to the file even with the autosave above.

1709929768: Saving in-memory database to /var/log/mosquitto/mosquitto.db.

Here's a subscription to a topic being made that works until the client is restarted

1709940559: Received SUBSCRIBE from B8A44F3205A41
1709940559: 	baton/tags/LightWhenSound (QoS 2)
1709940559: B8A44F3205A4 2 baton/tags/LightWhenSound

I should also note that when the clients reconnect, they send out a subscribeMany call via PahoMQTT for a list of their default topics. I am not sure if this would cause this issue or not but am including it though i believe it's nto the case because before this call ever comes in the logs on the broker are never recognizing the client as existing.

[Daedaluz]

what qos does the publishers use?

[rywager]

Qos 1/2 for different scenarios but not 0.

subscriptions are all confirmed to be worked and then client disconnect, reconnect with same ID shown above, nothing from mosquito about it being recognized, no messages from topics received that are not actively resubscribed to upon connect.

[Daedaluz]

mosquitto.conf:

listener 1883
log_type all
persistence true
persistence_location /var/log/mosquitto
persistent_client_expiration 1y
autosave_on_changes true
autosave_interval 100
protocol mqtt
log_dest stdout
allow_anonymous true
max_connections -1

1. run mosquitto: docker run --rm -ti -p 1883:1883 -v ${PWD}:/mosquitto/config/ eclipse-mosquitto:2.0.15
2. create a subscriber: mosquitto_sub -v -t "test/topic/1" -q 2 -i subscriber1 -V mqttv5 -c and disconnect with ctrl+c
3. publish 3 or 4 messages with: mosquitto_pub -t "test/topic/1" -m 'hello world' -q 2 -V mqttv5
4. reconnect subscríber: mosquitto_sub -v -t "test/topic/1" -q 2 -i subscriber1 -V mqttv5 -c

I get equally many messages as i sent from the broker.

I also took the time to make a tiny python script to test that it would receive queued messages without a new subscription, using the same client id;

#!/usr/bin/python3
import paho.mqtt.client as mqtt

def on_connect(client, userdata, flags, reason_code, properties):
    print(f"Connected {reason_code}")

def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.payload))

mqttc = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="subscriber1", protocol=5)
mqttc.on_connect = on_connect
mqttc.on_message = on_message

mqttc.connect("localhost", 1883, 60, clean_start=False)
mqttc.loop_forever()

Intrestingly, this seems to exhibit your issue where messages stop comming.

The only difference is that mosquitto_sub does another subscribe after connect.

Oddly, this post-connection subscription is what you said your client does, just like mosquitto_sub.

Edit:

1. make initial subscription with mosquitto_sub and then close
2. publish a bunch of messages
3. start python test script

this flow gives me the stored messages once between mosquitto sub and python test code.
restarting python test code does not receive any more messages, not even new ones

[rywager]

I'm not sure if you are saying you are seeing my same issue or not but let me use your format to explain mine and see if we can recreate.

1. Standup broker with persistence and config listed above.
2. Connect MQTTv5 client cleansession=false, with clientID 'B8A44F3205A4' to broker and subscribe to topic/test/1 and topic/test/2.
3. Publish messages to both topics, everything works as it should.
4. Reconnect client, same clientID, and this time only send a new subscription request to topic/test/1 (which you should already be subscribed to along with topic/test/2).
5. Logs show broker treats as new Client.
6. Client only receives messages on topic/test/1 and sees nothing on topic/test/2 (persistence should have maintained this topic subscription because of same ClientID, and not a cleansession.

To be clear, my scenario involves

1. Initial connection always conencts to say 5 different topics.
2. Mid session, I connect to lets say 2 more.
3. Disconnect.
4. Reconnect and re-subscribe to initial 5 topics (but do not resubscribe to the other 2).
5. No unsubscribe, always same client ID, persistence db growing, etc.
6. Messages to the extra 2 topics never reach the client after reconnect.

[ckrey]

Your log shows you are using MQTTV5. You need to set Session Expiry Interval explicitlyl in the CONNECT message. The default is 0 (zero) which means the session expires immediately after disconnect. https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901048 Session Expiry Interval is a Property in MQTTV5 CONNECT. mosquitto_sub sets the Session Expiry Interval per default if used with the -c option. (see mosquitto_sub —help)

…

Am 09.03.2024 um 23:10 schrieb Ryan Wager ***@***.***>: I'm not sure if you are saying you are seeing my same issue or not but let me use your format to explain mine and see if we can recreate. Standup broker with persistence and config listed above. Connect MQTTv5 client cleansession=false, with clientID 'B8A44F3205A4' to broker and subscribe to topic/test/1 and topic/test/2. Publish messages to both topics, everything works as it should. Reconnect client, same clientID, and this time only send a new subscription request to topic/test/1 (which you should already be subscribed to along with topic/test/2). Logs show broker treats as new Client. Client only receives messages on topic/test/1 and sees nothing on topic/test/2 (persistence should have maintained this topic subscription because of same ClientID, and not a cleansession. To be clear, my scenario involves Initial connection always conencts to say 5 different topics. Mid session, I connect to lets say 2 more. Disconnect. Reconnect and re-subscribe to initial 5 topics (but do not resubscribe to the other 2). No unsubscribe, always same client ID, persistence db growing, etc. Messages to the extra 2 topics never reach the client after reconnect. — Reply to this email directly, view it on GitHub <#3014 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6XHGMSCS5XM2QGOD63NVLYXOCETAVCNFSM6AAAAABENXXQROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHE4TGNRSHA>. You are receiving this because you are subscribed to this thread.

[Daedaluz]

ah, right.
subtle.

cheers!

[rywager]

Good sir, to say you are a lifesaver would be an understatement! Thank you! I do have to say as well, was the Paho C client trying to keep this a secret? Haha. I had to dive into the client code itself on github and piece together how to get this to work. If they make setting cleanstart true/false so easy, i have no idea why this critical component is not documented or mentioned almost anywhere. Here's the code that made it work for me, to save the next coder a little trouble :). Thank you again Cristoph! MQTTProperties props = MQTTProperties_initializer; MQTTProperty property; property.identifier = MQTTPROPERTY_CODE_SESSION_EXPIRY_INTERVAL; property.value.integer4 = 3000; MQTTProperties_add(&props, &property); conn_opts.connectProperties = &props;

…

On Sun, Mar 10, 2024 at 5:48 AM Tobias Assarsson ***@***.***> wrote: ah, right. subtle. cheers! — Reply to this email directly, view it on GitHub <#3014 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD236V3LZOGCHBWFKJUWFCTYXQ27BAVCNFSM6AAAAABENXXQROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGE4DCNZTGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Ryan

[rywager]

Quick follow up question - For my use case, a controlled network with controlled amounts of nodes, i really want this to NEVER expire (and i understand the risks here). Are there any settings for perpetual or what is the maximum this can be set to? Thank you again!!!

…

On Sun, Mar 10, 2024 at 11:28 AM Ryan Wager ***@***.***> wrote: Good sir, to say you are a lifesaver would be an understatement! Thank you! I do have to say as well, was the Paho C client trying to keep this a secret? Haha. I had to dive into the client code itself on github and piece together how to get this to work. If they make setting cleanstart true/false so easy, i have no idea why this critical component is not documented or mentioned almost anywhere. Here's the code that made it work for me, to save the next coder a little trouble :). Thank you again Cristoph! MQTTProperties props = MQTTProperties_initializer; MQTTProperty property; property.identifier = MQTTPROPERTY_CODE_SESSION_EXPIRY_INTERVAL; property.value.integer4 = 3000; MQTTProperties_add(&props, &property); conn_opts.connectProperties = &props; On Sun, Mar 10, 2024 at 5:48 AM Tobias Assarsson ***@***.***> wrote: > ah, right. > subtle. > > cheers! > > — > Reply to this email directly, view it on GitHub > <#3014 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AD236V3LZOGCHBWFKJUWFCTYXQ27BAVCNFSM6AAAAABENXXQROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGE4DCNZTGU> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- Ryan

-- Ryan

[Daedaluz]

If the Session Expiry Interval is 0xFFFFFFFF (UINT_MAX), the Session does not expire.

[rywager]

Thank you both, so much. Huge steps forward!

…

On Mon, Mar 11, 2024 at 5:12 AM Tobias Assarsson ***@***.***> wrote: If the Session Expiry Interval is 0xFFFFFFFF (UINT_MAX), the Session does not expire. — Reply to this email directly, view it on GitHub <#3014 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD236V5NU3EH33GCAAX6V7TYXV7RDAVCNFSM6AAAAABENXXQROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGA2DSOBRHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Ryan

[ralight]

Thanks everyone, I'm closing this as complete.