Issue Starting Mosquitto with Cloudflare Certs

[mike-sandstrom]

Hi all,

This is my first time trying to use TLS with mosquitto. I plan on using mosquitto with websockets so I'd like to use a browser recognized CA to issue my credentials. In turn, I used cloudflare to generate client certificates. I made my own key & CSR using the openssl commands in the man page for mosquitto TLS and copied the CSR to Cloudflare which created a client certificate. Additionally, I found the cloudflare root CA certificate here (https://support.cloudflare.com/hc/en-us/articles/115000479507-Managing-Cloudflare-Origin-CA-certificates#h_30cc332c-8f6e-42d8-9c59-6c1f06650639) and I added the RSA key to the ca_certificates folder.

In my certs folder, I have the generated private key from openssl (server.key), the signed certificate from cloudflare (server.pem), and the CSR from the openssl command (server.csr).

If I shouldnt be making the client certificates with cloudflare for any reason I'm not aware of, what other CA should I use if I want to use mutual certificates (certs for the broker and for each IoT device). I've been trying to figure this out for a while so any help would be appreciated.

This is my mosquitto.conf:

include_dir /etc/mosquitto/conf.d
cafile /etc/mosquitto/ca_certificates/ca.crt
keyfile /etc/mosquitto/certs/server.key
certfile /etc/mosquitto/certs/server.pem
tls_version tlsv1.2
port 8883

These are the permissions for the files in the cert folder (i replaced my username and the time):

-rw-rw-r-- 1 MYUSER MYUSER TIME server.csr
-rw------- 1 MYUSER MYUSER TIME server.key
-rw-r--r-- 1 root   root   TIME server.pem`

These are my error cods in the mosquitto logs:

Error: Unable to load server key file "/etc/mosquitto/certs/server.key". Check keyfile.
 OpenSSL Error[0]: error:0200100D:system library:fopen:Permission denied
 OpenSSL Error[1]: error:20074002:BIO routines:file_ctrl:system lib
 OpenSSL Error[2]: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib

[ralight]

Depending on where you installed from, it's likely that the broker is run as the mosquitto user, and that user has to have permission to read the certificate and private key files, instead of MYUSER.

[mike-sandstrom]

hmm, ok are you saying by installing mosquitto it creates a mosquitto user. I only have one user on my machine which is the MYUSER. Thank you for your response!

I installed by this "sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa"

[ralight]

You may only have one user user, but you will have many system users. The mosquitto ppa does add a system user mosquitto, which the broker runs as to keep everything separate.

[mike-sandstrom]

Oh ok I see. So I maybe have two options-find out how to give the mosquitto user these permissions over the cert and ca_certificates files OR change the user the broker is run under to MYUSER? Maybe using this command in the config require_certificate true use_identity_as_username true? Somewhat unrelated--do you recommend/is it possible to use letsencrypt for generating certificates for the server and for the devices?

[mike-sandstrom]

is issue basically the same problem? #1972

[ralight]

Yes, exactly.

One is chown mosquitto <path to file>

The other is adding user MYUSER to your mosquitto config file.

[mike-sandstrom]

adding user MYUSER to the config allowed mosquitto to start! Hopefully, I will be able to use the client certificates successfully now.

[mike-sandstrom]

Ok so here is where I am at now.

Like I mentioned above, I added the RSA key for the Cloudflare Origin CA root certificate (https://support.cloudflare.com/hc/en-us/articles/115000479507-Managing-Cloudflare-Origin-CA-certificates#h_30cc332c-8f6e-42d8-9c59-6c1f06650639) to mosquitto's ca_certificates file, which I am now certain is not correct since this is the root certificate that encrypts data between cloudflares network and my server instead of between client>>cloudflare>> server. I commented this out in the configuration.

For the broker and client-- I created the private key and CSR with openssl, then I copied & pasted the CSR to cloudflare which generated a client certificate. So this leaves me with a "keyfile" and a "certfile". I am now confused what I am to use for the cafile.
When generating the CSR, I used the ip address as the common name for the server (my domain name is currently behind cloudflare and causes problems with mqtt) and a generic name for the client device.

I dont have to use cloudlfare to generate certificates, if it presents unique problems. The goal is to use certificates/keys for the devices & the broker and I was told this isn't done with letsencrypt. Help with this is greatly appreciated and if I can make my question more clear feel free to let me know.

My Error:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

TLS Options:
client.tls_set(certfile="client.crt",keyfile="client.key")

Logs:
1615337170: Error: Unable to load server key file "/etc/mosquitto/certs/server.key". Check keyfile.
1615337170: OpenSSL Error[0]: error:0200100D:system library:fopen:Permission denied
1615337170: OpenSSL Error[1]: error:20074002:BIO routines:file_ctrl:system lib
1615337170: OpenSSL Error[2]: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib

[ptjm]

Typically, commercially-issued certificates are signed by an intermediate CA, not by the root. You end up with a chain consisting of the root, one or two intermediates, and then your certificate. You need this chain to be complete for the certificate to be validated. Each of your cert files (whether it's server.pem or client.crt) should include the client cert plus any intermediate certs. You can see if there are multiple certs by looking at the file. Each cert in a PEM file is a bunch of base-64-encoded garbage bracked by "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----".

The openssl tool can show you information about these certs, but so far as I know it only looks at the first cert in a given file if you run it like this:

openssl x509 -subject -issuer -noout -in client.crt

If there are multiple certs in the file, you can do this for all of them using this awk script (or if you don't have awk on your system, you can save each cert to its own file and run the above against each of them):

awk 'BEGIN { cmd = "openssl x509 -subject -issuer -noout" } /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print | cmd } /END CERTIFICATE/ { close(cmd); print "--" }' client.crt

Assuming there's multiple certificates in the file, the output of that will be something like this:

subject=CN = yyy.xxx.com, O = XXX Corp, L = Toronto, ST = Ontario, C = CA
issuer=C = US, ST = Illinois, L = Chicago, O = "Trustwave Holdings, Inc.", CN = "Trustwave Organization Validation SHA256 CA, Level 1", emailAddress = [email protected]

subject=C = US, ST = Illinois, L = Chicago, O = "Trustwave Holdings, Inc.", CN = "Trustwave Organization Validation SHA256 CA, Level 1", emailAddress = [email protected]
issuer=C = US, O = SecureTrust Corporation, CN = SecureTrust CA

The issuer of each cert has to match the subject of the next cert, and the issuer of the final cert should be your trusted root. In my case, I would need to find the "SecureTrust CA".

If you only have your client cert in that file, the CA vendor will probably have given you another file with the intermediate CA chain. You can just add it to the end of cilent.crt. If the chain includes a self-signed cert (meaning the subject and the issuer are the same), then that's the root CA.

[ralight]

1615337170: OpenSSL Error[0]: error:0200100D:system library:fopen:Permission denied - it looks like there is still a permission error at least.

[mike-sandstrom]

If you only have your client cert in that file, the CA vendor will probably have given you another file with the intermediate CA chain. You can just add it to the end of cilent.crt. If the chain includes a self-signed cert (meaning the subject and the issuer are the same), then that's the root CA.

@ptjm Thank you for your reply! I only received a client certificate from cloudflare. They dont provide a chain of trust. I reached out to cloudflare and this was their response "Cloudflare generates a unique CA for each zone, and it is fully-managed by us. It means that you can only generate a signed client certificate via UI or API". I am opting to use self-signed certificates. Your answer was very helpful though, thank you.

[mike-sandstrom]

1615337170: OpenSSL Error[0]: error:0200100D:system library:fopen:Permission denied - it looks like there is still a permission error at least.

@ralight Ok, I added my user to the config file and my user has root privileges. What is the best way to troubleshoot what files needs permission adjustments? Thank you for your response!

[mike-sandstrom]

I did chown MYUSER for mosquitto.db and thought this would fix my problem. I am getting this error and dont know what file to change.
1615820781: Saving in-memory database to /var/lib/mosquitto//mosquitto.db.
1615820781: Error saving in-memory database, unable to open /var/lib/mosquitto//mosquitto.db.new for writing.
1615820781: Error: Permission denied.

permissions for /var/lib/mosquitto/mosquitto.db
rw------- 1 MYUSER mosquitto 58321 Mar 15 15:06 mosquitto.log

[mike-sandstrom]

I ended up removing and re-installing mosquitto & this problem was resolved.