Mosquitto unable to start

[jmichaud111]

Was creating items inside my OpenHab environment and all of a sudden everything went offline. Ive tried more then then I'd like to admit to try and resolve this but I ended up just removing all of eclipse and attemping to reinstall with (sudo apt-get purge --remove mosquitto*) but the error didn't change and I'm still dead in the water. Any insight or suggestions would be marvelous and much appreciated.

skynet@skynet:/etc/mosquitto$ sudo systemctl status mosquitto
● mosquitto.service - Mosquitto MQTT Broker
Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mosquitto.service.d
└─override.conf
Active: failed (Result: exit-code) since Thu 2021-02-04 15:36:03 EST; 6min ago
Docs: man:mosquitto.conf(5)
man:mosquitto(8)
Main PID: 14891 (code=exited, status=1/FAILURE)

Feb 04 15:36:03 skynet systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
Feb 04 15:36:03 skynet systemd[1]: Stopped Mosquitto MQTT Broker.
Feb 04 15:36:03 skynet systemd[1]: mosquitto.service: Start request repeated too quickly.
Feb 04 15:36:03 skynet systemd[1]: mosquitto.service: Failed with result 'exit-code'.
Feb 04 15:36:03 skynet systemd[1]: Failed to start Mosquitto MQTT Broker.

[ralight]

Sorry you're having problems. Do you know what version of Mosquitto you have? apt-cache policy mosquitto should tell you.

You might get more information on what is going wrong by starting Mosquitto manually:

sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

[exetico]

Same problem here. Running the latest version in docker (2.0.7). My watchtower upgraded the container like it normally do, but everything was offline this morning.

The container is reporting "Address not available" - but I'm not sure if it's something related to IPv6 (My local setup are IPv4 only).

It's also reporting: Starting in local only mode. Connections will only be possible from clients running on this machine
And: Create a configuration file which defines a listener to allow remote access.

Therefore, something groundbreaking must have changed, for the v.2+? Maybe something should be baked into the container, for better handling of the users running with the :latest tag. Normally theMQTT-container are allowed to be upgraded through watchtower - but maybe I should blacklist it?

I've reverted back to the 1.6.12-tag, and everything worked just fine.

[Esc4pe]

I had the same issue after watchtower upgraded to the latest version tonight.

The only thing in the log that doesn't seem to be caused by my stuff is this:
Error in epoll re-registering to EPOLLOUT: Invalid argument

For every connection I am seeing this in the log:

1612514911: New connection from 192.168.70.72:57441 on port 1883.
1612514911: Sending CONNACK to 192.168.70.72 (0, 5)
1612514911: Client <unknown> disconnected, not authorised.

What I tried:

• downloaded the latest mosquitto.conf from github and filled it out like before, only difference is that "port" is now "listener" --> not working
• reverted back to 1.6.12 like exetico mentioned --> works (even with the new listener instead of port), and does not show the EPOLLOUT error message above

So I think either there is something wrong with the latest version or there is a config thing or deprecation of some sort.

[ScottRoach]

This broke for me when Watchtower upgraded this to 2.0.7 also. Adding the following to the config file and a restart fixed things for me:

listener 1883

[ralight]

Version 2.0 was released at the start of December and it was made clear that it has breaking changes, with some documentation around the breaking changes: https://mosquitto.org/documentation/migrating-to-2-0/

The latest tag was deliberately left on 1.6 to give people chance to test and migrate.

We advertised (email and twitter) a week ago that the latest tag would be moving yesterday, and so anybody on it should make configuration changes to match - by setting the authentication options explicitly in the config file it would work fine with both 1.6 and 2.0.

Inevitably and understandably there will be people who aren't on the mailing list or saw the tweets, and I'm sorry for the disruption the upgrade can cause some people, but I hope you see that we tried hard to give the chance to upgrade without being disrupted.

In general, I would suggest not using the latest tag for any docker image that you don't want to break on you.

[exetico]

@ScottRoach That makes totally sense, now that it's reported: Create a configuration file which defines a listener to allow remote access.

So, the solution must be to add listener 1883 to the config file, and people are good to go, again :-)

@ralight How can we subscribe to the email-information, what you're referring to? Also, good point about the latest-tag. I've locked it before, but swapped back to latest, a good amount of time ago.

[ralight]

@exetico The mailing list is https://accounts.eclipse.org/mailing-list/mosquitto-dev and twitter is https://twitter.com/MosquittoMQTT

So, the solution must be to add listener 1883 to the config file, and people are good to go, again :-)

The solution is to read the migration docs :) The changes are intended to nudge users into having to consider security. It was too easy before to unintentionally run the broker without any authentication.

In brief:

If you run without a config file, or without a listener defined, Mosquitto will only be accessible on the local machine, but will require no authentication.

If you define a listener, you must consider authentication, otherwise no access will be possible. Choices:

• Setting a password file
• Configuring an authentication plugin (like the new dynamic security plugin)
• Setting allow_anonymous true

[pchevallier]

@ralight thank you for providing those information. That's very useful. That's still not working in my case but at least that gives me some directions to look into.
As a feedback on this release, I get that we wanted to force a higher security policy, but the doc is really not clear when it comes to backward compatibility (i.e. migration from 1.6 to 2.0)

Also, how about making the proper default changes in the default_config.conf file

port 1883
persistence false
user root

this file hasn't changed since the last release in the snap library so that at least we don't get this message in our log files:

Using default config from /snap/mosquitto/533/default_config.conf
The 'port' option is now deprecated and will be removed in a future version. Please use 'listener' instead.

[ralight]

@pchevallier Do you mean on https://mosquitto.org/documentation/migrating-to-2-0/ ? I'm keen to improve anything that isn't clear so feel free to ask and I can then modify the document.

Good point about default_config.conf, I'd missed that. I've just pushed a commit with that change in, when it's built I'll push it to the snap stable channel.

[pjsg]

I got bitten by this -- but it was the snap that got updated and took down my installation. The snap documentation (as far as I can see) was not updated to include the extra step of modifying the config files.

[pchevallier]

@pjsg Exactly what also happened to me yesterday. The new version got released in the official Ubuntu repo, and has been automatically applied during a nightly maintenance window by the AWS system manager.

@ralight yes, the current migration documentation doesn't cover the snap package management. It even has even some issues depending on the OS we are in. It behaves differently whether you are on Ubuntu or Centos. it's not Mosquitto related, but I haven't been able to get neither Mosquitto daemon to run using the legacy setup (version 1.6).
I'm still trying to figure this out and will share as soon as I come up with a solution, but if anyone has come up with a working version using snap, please share.

[pjsg]

The solution is sudo snap refresh mosquitto --channel=1.6/stable

[ralight]

@pjsg you're right that the snap documentation wasn't updated to point users to the need to configure authentication, I've now fixed that, thank you.

The default configuration of the 2.0 snap package not authorising any connections is an oversight which I didn't catch because it seems I have only tested with different variations of configuration file, but not without a custom configuration file. It seems as though the same is true for all of the people who have been using the 2.0 snap over the past two months as well.

Sorry for the disruption.

The snap is now fixed to allow unauthenticated access from the local machine only, as per the default configuration. If you want to allow access from other computers you will need to configure a listener and an authentication method, either a password file, authentication plugin, or setting allow_anonmous true if that is what you wish.

[Diddlik]

Hello everyone, I updated my docker image, changed config but the container start stay in status "unknown".
my conf file:

listener 1883
allow_anonymous true
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log

docker run -it --name mosquitto1 -p 1883:1883 -v $(pwd)/conf/mosquitto.conf:/mosquitto/config/mosquitto.conf eclipse-mosquitto

after few seconds I get:

Unable to find image 'eclipse-mosquitto:latest' locally
latest: Pulling from library/eclipse-mosquitto
801bfaa63ef2: Already exists
0613e57f5869: Already exists
99e1cb91ed0c: Already exists
Digest: sha256:688f5cf03b714127e67a6ee6354f11b746b49ebe532740429b9a23429496a3d7
Status: Downloaded newer image for eclipse-mosquitto:latest

after last line is nothing happens

[exetico]

@Diddlik
Try and run docker pull eclipse-mosquitto first :-) Else, try and map it to a specific tag, like: eclipse-mosquitto:2.0.7

If you're willing to try that, just do the following steps:

docker pull eclipse-mosquitto:2.0.7
docker run -it --name mosquitto1 -p 1883:1883 -v $(pwd)/conf/mosquitto.conf:/mosquitto/config/mosquitto.conf eclipse-mosquitto:2.0.7

Evertything looks just fine, in your config - and the output, too :-)

However, I'll recommend you to add some auth to your setup.

I really doubt that it's something related to the local files, though. I must have read your question a bit too fast.

For inspiration, here's my simple config-file:

allow_anonymous false
password_file /mosquitto/config/file_with_mqttpassword
listener 1883

And here's the part of my docker-compose, related to the mosquotto setup:

# https://hub.docker.com/_/eclipse-mosquitto/
  mqtt:
    image: eclipse-mosquitto:2.0.7
    container_name: mqtt
    ports:
      - "1883:1883"
      - "9001:9001"
    volumes:
      - /home/USERNAME/mqtt/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - /home/USERNAME/mqtt/file_with_mqttpassword:/mosquitto/config/file_with_mqttpassword
    restart: unless-stopped

[Diddlik]

Tried it again.
docker pull eclipse-mosquitto:2.0.7
2.0.7: Pulling from library/eclipse-mosquitto
Digest: sha256:688f5cf03b714127e67a6ee6354f11b746b49ebe532740429b9a23429496a3d7
Status: Downloaded newer image for eclipse-mosquitto:2.0.7
docker.io/library/eclipse-mosquitto:2.0.7
ragnar@york:/opt/docker/appdata/mqtt$ docker run -it --name mosquitto1 -p 1883:1883 -v $(pwd)/conf/mosquitto.conf:/mosquitto/config/mosquitto.conf eclipse-mosquitto:2.0.7

the same output.
with local files is a good point, but at the moment no idea what can be wrong...

[exetico]

Well, have you checked the mosqutto-logs? :-) The status for my container looks just fine.

Have you tried to define the full path, instead of using $(pwd)? Try and, just to see...

I can't bring other things to the table. But you can try and see the logs from the container?
docker logs mosquitto1

Maybe there's something usefull.

[Diddlik]

it is nothing in logs, I htinks it is not started propertly

[exetico]

Well, stop it, remove it, create a new one. There must be something with your setup, one way or another :-) Hopefully you'll figure it out, sooner or later.

[alsak0de]

Hiya.. does anyone know whether the listener can be created through a ENV variable on docker run?
I need the anonymous access and the 0.0.0.0:1883 listener. I'm trying to avoid binding storage in a K8s environment just to pass a Mosquitto.conf. THANKS

[ghost]

It would really help if someone would update https://hub.docker.com/_/eclipse-mosquitto to actually mention that you require to include a configuration file, and what the content should be, if you even want to be able to connect to the darn thing running inside the container

[mosterdt]

There is also some discussion in #2040
I think the documentation on hub.docker.com originates from the documentation in this repository. The maintainer might add something like #2086

[stripler675]

well that was a pain in the butt. I did a docker pull for the latest, and little did I know it would go to the 2.0 branch and break everything.
For docker-compose to get it back up and running again, I made the updates below. All my local config is in /opt, I don't have a good reason for that, change as needed. Now all the file logging works, and I don't have streams of errors.

Added volume for the pid file in docker-compose.yaml, to get rid of the "unable to write PID" error. Also no user is specified in the .yaml file for mosquitto.

volumes:
  - /opt/mosquitto/var/run:/var/run

updated /opt/mosquitto/conf.d/default.conf:

allow_anonymous false
password_file /mosquitto/config/credentials

updated /opt/mosquitto/config/mosquitto.conf:

#locals go in /opt/mosquitto/config/conf.d

pid_file /var/run/mosquoitto.pid

persistence true
persistence_location /mqtt/data
user mosquitto
#user root

#port 1883
listener 1883
log_dest file /mosquitto/log/mosquitto.log
log_dest stdout

include_dir /mosquitto/config/conf.d

chown everything to 1883.1883 and make sure read/write perms are safe

[nilss0n]

I've encountered the same issue as everyone else. It's clearly my mistake for not locking the version, but it would be nice if the new version didn't require a configuration file. For example by setting an environment variable in the docker container.

I guess that locking the server to local connections makes sense when running locally, but it's a pretty bad default when running in docker. Perhaps the docker image should include a default config with the old behavior or something like that.

[alexelite]

Can someone please explain what is not working and is there a fix?
I have a new Debian 10 install and followed dockerhub instructions. Docker log is empty, not even a "Mosquitto is starting" or something, as @Diddlik mentioned.
Thank you in advance. :)

[WombatHollow]

It may be that you are trying to start Mosquitto twice, the first time is a leftover from earlier versions that need deletion. The words below are from a post in another thread.

There was a root CRON job that runs launcher.sh which contained sudo mosquitto -d

How did I find it. Well once I found CRON has a @reboot option I looked further and found CRON is by user including root. So I winged it and tried sudo crontab -l hoping the use of sudo finds root cronjobs instead of pi user jobs and there as a another job.

After that it was a tidy up of all the other things I had tweaked and re-enable the systemd job on reboot. sudo systemctl enable mosquitto.service

So now have 2.0.11 working as I like. I suspect that CRON job was a left over from earlier versions of Mosquitto (I was on 1.5 or 1.6) but the update to 2.0.11 didn't delete it when it created the mosquitto.service job.

[bialpio]

I'm running into similar issue. If I do not mount any paths into the container, it starts fine. The moment I start overriding the (in-container) /mosquitto/config/mosquitto.conf file via a -v <path to my config>:/mosquitto/config/mosquitto.conf, the container has its status listed as "Exited (13) seconds ago". I suspect this is because I'm trying to overwrite a path that exists in-container with something that is mounted from the host, and this fails.

[falense]

With config:

listener 1883
persistence true
persistence_location /mosquitto/data/

Runnning with command:
docker run -it --name mosquitto -p 1883:1883 -v <absolute_host_path>/mosquitto:/mosquitto/ eclipse-mosquitto:1.6.12

Does not work.

While the following config works but has undesirable paths:

allow_anonymous true
listener 1883
persistence true
persistence_location /mosquitto/data

Again launching with:

docker run -it --name mosquitto -p 1883:1883 -v <absolute_host_path>/mosquitto:/mosquitto/ eclipse-mosquitto:1.6.12

Gives this output:

1636186667: Config loaded from /mosquitto/config/mosquitto.conf.
1636186667: Opening ipv4 listen socket on port 1883.
1636186667: Opening ipv6 listen socket on port 1883.
1636186667: mosquitto version 1.6.12 running
1636186668: mosquitto version 1.6.12 terminating
1636186668: Saving in-memory database to /mosquitto/datamosquitto.db.

[falense]

With config:

listener 1883
persistence true
persistence_location /mosquitto/data

Starting 2.0.13 as follows:

docker run -it --name mosquitto -p 1883:1883 -v <abs_path_on_host>/mosquitto/config/:/mosquitto/config/ -v <abs_path_on_host>/mosquitto/data:/mosquitto/data/ eclipse-mosquitto

It both loads the config file found in <abs_path_on_host>/mosquitto/config and saves database to <abs_path_on_host>/mosquitto/data

1636187818: mosquitto version 2.0.13 starting
1636187818: Config loaded from /mosquitto/config/mosquitto.conf.
1636187818: Opening ipv4 listen socket on port 1883.
1636187818: Opening ipv6 listen socket on port 1883.
1636187818: mosquitto version 2.0.13 running
1636187821: mosquitto version 2.0.13 terminating
1636187821: Saving in-memory database to /mosquitto/data/mosquitto.db.

This is probably what most people want. My guess is that the path /mosquitto/data might be defined in the docker image not letting it be overridden unless it is done explicitly.

[falense]

Also using

log_dest file /mosquitto/log/mosquitto.log

seems to disable logging to console which in combination with non working mounts make it seem like the docker image is doing nothing.

[cassidea]

Also using

log_dest file /mosquitto/log/mosquitto.log

seems to disable logging to console which in combination with non working mounts make it seem like the docker image is doing nothing.

Thank you. Had it in my config and was wondering why nothing is happening.
I recommend to disable this option for testing.