Mosquitto persistence problem

[luigicaiffa]

Use case:

mqtt client -> mosquitto -> bridge -> rabbitmq
mosquitto persistence enabled

Issue description:

Under particular conditions the file mosquitto.db starts to increase in size, even though the bridge is working properly.

This behavior is triggered by changing the remote_clientid configuration parameter.

In fact, if the mosquitto.db file has already been created, changing the parameter causes the following behavior:

• even with bridge active, the message is saved in mosquitto.db
• mosquitto.db reaches the saturation of the available space
• the messages directed to rabbit start to acquire a certain latency in sending

I am attaching a mosquitto.db file that has this problem, although it is very simple to reproduce from scratch.

This is the link

We used the mosquitto_db_dump tool to understand what was in the db, using the command:

./mosquitto_db_dump --stats mosquitto.db

obtaining:

DB_CHUNK_CFG:        1
DB_CHUNK_MSG_STORE:  2260
DB_CHUNK_CLIENT_MSG: 2260
DB_CHUNK_RETAIN:     0
DB_CHUNK_SUB:        2
DB_CHUNK_CLIENT:     2

Is there a parameter that correctly set solves this problem?

Enviroment:

• SO linux Ubuntu 18.04.5 LTS
• mosquitto run on container eclipse-mosquitto:1.6.13
• rabbitmq run on container rabbitmq:3.8.16-management

mosquitto.conf

persistence true
autosave_interval 1
autosave_on_changes true
persistence_location /mosquitto/data/
queue_qos0_messages true
max_queued_bytes 1000000
max_queued_messages 2500

log_type all

connection rabbit-bridge
address rabbitmq
remote_clientid mosquitto
remote_username guest
remote_password guest
topic # out 0 toRabbit/ data/ckb23mf3s01ve0h6gkv5wsfnv/
topic # in 0  toDevice/ data/ckb23mf3s01ve0h6gkv5wsfnv/toMachine/toDevice/

bridge_protocol_version mqttv311
bridge_insecure true
try_private false
keepalive_interval 5

listener 1883
allow_anonymous true

Additional attempts

We have tried to use also the new version 2.0.X with the same configuration but:

• the last working version regarding the bridge is 2.0.5
• we were not able to make persistence work for any 2.0.X version

[ralight]

Thanks for the details. Your persistence file has two clients in it, local.MSIMDEV... and local.mosquitto. Both of them are subscribed to toRabbit/#. It looks like you have changed the local client id. Because the bridges have cleansession set to false by default, they have their session and subscriptions saved. When you change the client id, the old stored client id isn't removed and will keep on receiving messages. A simple way to remove the client is to connect with that client id, then quit, but making sure not to have the clean session set to false:

mosquitto_sub -h <broker> -u <username> -P <password> -i local.MSIMDEV... -t test

What problems are you having with bridges and persistence in version 2?

[luigicaiffa]

@ralight thank you very much for reply.

A simple way to remove the client is to connect with that client id, then quit, but making sure not to have the clean session set to false

At the moment I have implemented a different solution.
I check the client_id list in DB_CHUNK_CLIENT and if there are old client_ids, I delete the db.
In our use case, this should not cause big problems, also because from what you write there is no "clean" solution.

What problems are you having with bridges and persistence in version 2?

After finalizing the fix, I went back to look at the issues with version 2.0.X related to bridge and persistence.
I'm using version 2.0.10.

bridge

Regarding the bridge, actually everything seems to be working,
The problem was related to the startup timing between mosquitto and rabbit.
To solve it I wait until the mqtt port of rabbit is available for bridge connection.

persistence

Instead, persistence still doesn't work.
The malfunction is very simple, when the bridge is disconnected the messages are not saved in the mosquitto.db.

The configuration is the same as described above.

steps to reproduce the problem:

• Start rabbitmq service
• Start the mosquitto service: In the logs the message "Saving in-memory database to /mosquitto/data//mosquitto.db" appears just after the connection to rabbit
• Stop the rabbitmq service: Mosquitto, almost immediately, starts making attempts to reconnect: "Connecting bridge rabbit-bridge (rabbitmq:1883)"
• Send a message on the topic toRabbit/test
• Start rabbitmq service and wait for bridge reconnection

I don't see any evidence of the message in the db or on rabbit.

[ralight]

Thank you - it is the queue_qos0_messages option that is making the persistence bug appear. I'll have that fixed shortly.