mosquitto_pub not working with RabbitMQ + MQTT plugin + TLS

[riker09]

I have a RabbitMQ broker that is secured by a Let's Encrypt certificate. I have also enabled the bundled MQTT plugin and enabled MQTTS on port 8883. I can use MQTT.fx to connect to the broker and publish and subscribe topics just fine. However, when I'm using mosquitto_pub I get a connection error that is very unhelpful.

This is the command I'm using:

user@host:~$ mosquitto_pub -d -h my-rabbit-host.net -p 8883 -u user -P 'password' -t test --insecure -m "test"
Client mosqpub|20501-host sending CONNECT
Error: The connection was lost.
user@host:~$

I've also tried to leave the --insecure flag out, same result.

I'm not sure if I'm using the mosquitto_pub cli tool correct here, but judging from the documentation and my poor understanding on the general subject I would assume that it should be able to connect. Feel free to slap me in the face with the obvious missing puzzle piece if there is such a thing. :)

[iarenaza]

See http:https://mosquitto.org/man/mosquitto_pub-1.html, under section "Encrypted Connections". It seems you need to specify either --capath, --cafile, --psk or --psk-identity option (depending on your broker TLS configuration) to trigger TLS connections from mosquitto_pub command.

[riker09]

Sorry if I'm asking dump questions here... I was under the impression that providing one (or any) of --capath, --cafile, --psk or --psk-identity would enable encrypted authentication of the client (mosquitto_pub) with the broker.

I hope I'm not missing out on something basic, but all I want is to encrypt the data that is transferred from my local computer to my remote broker. I authenticate my local client with a username/password combination. This is for my private home automation setup, no need for any fancy stuff. 😅 🙈

[iarenaza]

Using the official RabbitMQ Docker container from https://hub.docker.com/_/rabbitmq, adding the following extra configuration settings (and copying all the CA and server keys and certificates into /etc/rabbitmq/certs inside the container):

# default TLS-enabled port for MQTT connections
mqtt.listeners.ssl.default = 8883

# SSL/TLS configuration
ssl_options.cacertfile = /etc/rabbitmq/certs/ca.crt
ssl_options.certfile   = /etc/rabbitmq/certs/server.crt
ssl_options.keyfile    = /etc/rabbitmq/certs/server.key
ssl_options.versions.1 = tlsv1.2
ssl_options.verify     = verify_none
ssl_options.depth      = 2
ssl_options.fail_if_no_peer_cert  = false

It seems to successfully publish messages with the following mosquito_pub command line (ca.crt file is a copy of the CA certificate file that is used in the RabbitMQ config):

user@host:~$ mosquitto_pub -d -h localhost -p 8883 --tls-version tlsv1.2 -t test -m "test" --cafile ca.crt
Client mosqpub/21863-osgiliath sending CONNECT
Client mosqpub/21863-osgiliath received CONNACK
Client mosqpub/21863-osgiliath sending PUBLISH (d0, q0, r0, m1, 'test', ... (4 bytes))
Client mosqpub/21863-osgiliath sending DISCONNECT
user@host:~$

[riker09]

Not sure if this has any relevance, but I'm using certificates from Let's Encrypt. I'm extracting the certificates from acme.json which is created by my Traefik instance which does the LE stuff for me automagically. The extracted certificate files are then handed to Rabbit via config:

listeners.ssl.default = 5671

default_user = username
default_pass = Pa$$w0rD

# RabbitMQ SSL configuration
ssl_options.cacertfile = /certs/chain.pem
ssl_options.certfile   = /certs/cert.pem
ssl_options.keyfile    = /certs/privkey.pem
ssl_options.verify     = verify_none
ssl_options.depth      = 2 
ssl_options.fail_if_no_peer_cert = false

# Management
management.ssl.port       = 15671
management.ssl.cacertfile = /certs/chain.pem
management.ssl.certfile   = /certs/cert.pem
management.ssl.keyfile    = /certs/privkey.pem


# MQTT
mqtt.listeners.ssl.default = 8883
mqtt.listeners.tcp.default = 1883
# Default user
mqtt.default_user = username
mqtt.default_pass = Pa$$w0rD

The management interface is encrypted just fine, all browsers happily display the lock-icon. So I take it that the certificates are valid and RabbitMQ is configured correct. When I copy the fullchain.pem file to my local computer and use it as value for the --cafile option I can see within the RabbitMQ logs that a connection attempt is made:

2019-02-06 11:56:38.796 [info] <0.619.0> TLS server: In state certify received CLIENT ALERT: Fatal - Unknown CA

mosquitto_pub still fails, of course:

Client mosqpub|32459-MY-HOST sending CONNECT
Error: A TLS error occurred.

---

[EDIT]

I even tried converting the fullchain.pem to ca.crt with this command:

user@host:~$ openssl x509 -outform der -in fullchain.pem -out ca.crt

I get a different error and no connection attempt in the RabbitMQ logs this way. The error is:

Error: Unable to load CA certificates, check cafile "ca.crt".
Unable to connect (A TLS error occurred.).

[ralight]

Aha! despite its name, fullchain.pem doesn't contain the full chain, it is the chain excluding the CA. You could try mosquitto_pub -d -h my-rabbit-host.net -p 8883 --capath /etc/ssl/certs ... (which will then trust every CA installed on your system), or find a copy of the root CA certificate.

[ralight]

Which is probably /etc/ssl/certs/DST_Root_CA_X3.pem

[riker09]

I can confirm that specifying --cafile /etc/ssl/certs/DST_Root_CA_X3.pem works:

user@host:~$ mosquitto_pub -d -h my-domain.com -p 8883 --cafile /etc/ssl/certs/DST_Root_CA_X3.pem -u username -P 'Pa$$w0rD' -t test -m "test"
Client mosqpub|12977-HOST sending CONNECT
Client mosqpub|12977-HOST received CONNACK
Client mosqpub|12977-HOST sending PUBLISH (d0, q0, r0, m1, 'test', ... (4 bytes))
Client mosqpub|12977-HOST sending DISCONNECT
user@host:~$

It was also successful when giving it the whole folder --capath /etc/ssl/certs, no surprises here. 🙂

So the trick is to tell mosquitto_pub where to look for the correct CA file. The question is, do you think this should be better documented? Why does mosquitto_pub not use the system-wide CA store by default?

[ralight]

This is documented as the very first thing after the description of what mosquitto_pub is... https://mosquitto.org/man/mosquitto_pub-1.html

It doesn't use the system store by default so you always have the most control over what CAs to trust.

[riker09]

Okay, let me put more emphasize on better. 😉

I have read the man page and mosquitto_pub --help. Both cover the subject, but very briefly (in my opinion). I think a small example would greatly enhance the user experience (for noobs like me).

Additionally the debug messages were not very hintful in finding the real issue. Without the CA file I got an Error: The connection was lost. I would have expected something like "Unable to verify TLS connection." or "This looks like a encrypted connection but no CA option was specified." (or similar).

---

[EDIT]

I noticed that the documentation online differs from the man page. Is that by choice? My mosquitto_pub version is 1.4.15