out of memory error in syslogs

[dineshu07]

Hi All

I'm using 2.0.15
We are currently facing an issue where our system, supporting 100 clients, experiences a crash after a few hours of operation, with Mosquitto going down due to an out-of-memory error. We are uncertain about the root cause and are seeking guidance on how to initiate the debugging process. Your insights would be highly appreciated. Could you please examine the syslogs for further analysis?

[popovec]

Does any of your clients send large or many small persistent messages? (see how big the database of persistence messages is, on a Debian system it will probably be in /var/lib/mosquitto/, i.e. it should be enough to check the size using the command sudo ls -la /var/lib/mosquitto/*)

[dineshu07]

Thanks for the reply. Yes. All our clients sends small messages very frequently. unfortunately I'm not able to see the file in my system . File is not created.
I'm starting my mosquitto using the command: mosquitto -c /etc/mosquitto/conf.d/mosquitto.conf &
Please find my conf file

allow_anonymous false
password_file /etc/mosquitto/p1.txt
listener 1883
protocol mqtt
listener 1884
protocol websockets

connection bridge-01
address 192.168.200.11:1885
topic # both
remote_username test
remote_password test

do i need to specify db location in my conf file to create? Looks like we are not storing anything in the db.

[popovec]

By default, Debian uses the configuration in /etc/mosquitto/mosquitto.conf, where the path to the persistent database is specified:

persistence true
persistence_location /var/lib/mosquitto/

In your case, you do not use the default location for persistence, but this does not mean that mosquito does not have to keep persistent messages in memory.

You can try to connect to your server, and try to read all the data that is stored persistently in memory, for example like this (use your IP number and your login data):

mosquitto_sub -h _IP_ADDRES_  -t '#' -v

This can list a huge amount of data uploaded by clients who use the "retain" flag when publishing.

Another possibility is that clients use qos1 and 2 when subscribing and publishing, and thus also require, under certain conditions, that the server allocates a large amount of memory for them.

Depending on what you want to enable/disable for your clients, consider adding some of the options to the configuration (Read their specification in the mosquitto.conf manual page)(Read their specification in the mosquitto.conf manual page):

max_packet_size
max_queued_bytes
max_queued_messages
message_size_limit
memory_limit

[dineshu07]

Thanks for your prompt response. I will give a try and let you know. Thanks again.

[NorbertHeusser]

You might as well monitor the number of messages and bytes using subscriptions on the Mosquitto broker $SYS topics regarding message store
$SYS/broker/store/messages/+

So starting a mosquitto_sub should periodically give you information about messages being queued in the broker:
mosquitto_sub -h IP_ADDRES -t '$SYS/broker/store/messages/+' -v

(see https://mosquitto.org/man/mosquitto-8.html, Section Broker Status)