Not able to reload mosquitto on Ubuntu 16.04

[JinPingChng]

Hi,

This case happened on yesterday morning (GMT + 08). and it doesn't restart/start the mosquitto service. I get this issue from my syslog.

// This when I do /etc/init.d/mosquitto reload

Sep 25 10:55:47 nodus-rtls-ytl-prod systemd[1]: Reloading LSB: mosquitto MQTT v3.1 message broker.
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7370]: * Reloading network daemon configuration: mosquitto
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7370]: ...done.
Sep 25 10:55:47 nodus-rtls-ytl-prod systemd[1]: Reloaded LSB: mosquitto MQTT v3.1 message broker.
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7348]: Reloading config.
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7348]: Loading config file /etc/mosquitto/conf.d/default.conf
Sep 25 10:55:47 nodus-rtls-ytl-prod kernel: [12620.743395] mosquitto[7348]: segfault at 88 ip 0000000000423f5c sp 00007ffd9d7bb2a0 error 4 in mosquitto[400000+3d000]

As I understand that

1. Logrotate will do a killall -HUP on all mosquitto, so it will hang up the mosquitto and reload back. Apparently, the mosquitto reload is no longer work (before Yesterday, the mosquitto still work like a charm). So, I decided to temporary remove the killall -HUP mosquitto first
2. Every time I tried to do /etc/init.d/mosquitto reload, I will receive the segfault at 88 ip. And I need to restart manually the mosquitto by restarting
3. I have tried reboot the server, but it's seem like didn't do the trick.
4. Tried to look for solutions and google around.

Please help and I wouldn't hesitate to give out what information you need...

[ralight]

Is this a Raspberry Pi? Others have been having problems with Pi 1 and Pi 2 devices.

[JinPingChng]

No, this is a server that runs on Ubuntu 16.04 Server.

[ralight]

What version of mosquitto, and where did you install from?

[JinPingChng]

Mosquitto Version 1.6.6
I install through sudo apt-get.

This is an example of how do I install mosquitto MQTT broker.
https://www.vultr.com/docs/how-to-install-mosquitto-mqtt-broker-server-on-ubuntu-16-04

[JinPingChng]

Hi railight

Thank for respond. Currently, the main issue is that mosquitto service is not able to reload. However, it still can run as usual. Even when I restart the services. Just that it couldn't reload...

[ralight]

I understand. Are all files in /etc/mosquitto accessible by the mosquitto user? It shouldn't crash, but that is a possible cause for problems.

[ralight]

Are you able to debug on this machine? If so, could you apt install mosquitto-dbgsym, then stop mosquitto and run it manually: sudo valgrind --log-file=vglog /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf. Once it is running, try to reload in the same way. If it doesn't crash, then stop the valgrind process (it will be called memcheck). Please then send me the vglog file to [email protected]

If you aren't able to debug on this machine, then could you please share more of your configuration? I haven't been able to reproduce this so far on a 16.04 machine.

[JinPingChng]

Hi ralight,

I will get back on you once I tested on what you have suggested. Thanks!

[JinPingChng]

I understand. Are all files in /etc/mosquitto accessible by the mosquitto user? It shouldn't crash, but that is a possible cause for problems.

the server only run in root user, temporary no other super user

[ralight]

Mosquitto will try to change to the mosquitto user if run as root, unless you explicitly tell it to stay as root. Which is not recommended.

[JinPingChng]

Hi railight,

I failed to debug by using the valgrind, It will stop the services, I will send you the vglog. And I wasn't able to install the mosquitto-dbgsym.

This will be my configuration

/etc/mosquitto/conf.d/default.conf

General Configuration
log_type all
log_dest syslog
log_dest stdout
connection_messages

Default Listener
port 8883
cafile /root/<root_directory>/mqtt_ca.crt
certfile /root/<root_directory>/mqtt_server.crt
keyfile /root/<root_directory>/mqtt_server.key

Listener 8884 (Third Party Customer)
listener 8884
cafile /root/<root_directory>/mqtt_server_ca.crt
certfile /root/<root_directory>/mqtt_client_server.crt
keyfile /root/<root_directory>/mqtt_client_server.key
require_certificate

WebSocket
listener 9001
protocol websockets

Insecure Connection 1883
listener 1883

/etc/mosquitto/mosquitto.conf

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d'

[ralight]

Could you please try updating, this should be fixed.

[JinPingChng]

Hi Ralight

Thank You for your support and help.

I have updated on the servers and yes I am able to manual reload the mosquitto.

For another question is that, will also affect other versions of Mosquitto? As I have some cloud servers that used another version (I believe is 1.4.8).

Once again, thank for your help.

[ralight]

The other versions should be unaffected.

[JinPingChng]

Hi ralight, one question as our end customer come to us and ask me on it.

I would like to know about the issue, could you specify what kind of issue we were faced before?

Once again, thank for your help

With Regards
Jpchng

[ralight]

It was something like if you had a TLS listener and non-TLS listener configured, and had a persistent client that was disconnected at the time the reload occurred, then an incomplete check could cause a null pointer to be derefenced.