OpenSSL error when using crlfile after reload

[cracksalad]

I recently added the crlfile line to the configuration below. Since then I observe failed connection attempts (see log below) as soon as the broker is reloaded (using systemctl reload mosquitto). Actually all new connections fail, while already opened connections are still alive. Restarting moqsuitto (using systemctl restart mosquitto) solves the issue until the next reload happens.

Note that the same configuration just without the crlfile-line worked perfectly fine. I just had to revoke some client certificates lately.

This issue might have to do with #2597. Although it is a different Mosquitto version and a slightly different behavior.

I made sure, that all certificates and the CRL are (still) valid. The certificates of the clients failing to connect were not revoked. The CA certificate and the CRL files are readable by all users (mode 0644, parent folders 0755).

Log excerpt

2023-10-06T03:39:41: New connection from 172.18.0.13 on port 8883.
2023-10-06T03:39:41: OpenSSL Error[0]: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher
2023-10-06T03:39:41: Socket error on client <unknown>, disconnecting.

Mosquitto version
1.6.15 (newest version in the OS repository)

OS and version
AlmaLinux 8.8

OpenSSL version
1.1.1k

Mosquitto configuration

# General configuration
acl_file /etc/mosquitto/acl
max_inflight_messages 200
max_queued_messages 10000000
persistent_client_expiration 3d
queue_qos0_messages true
upgrade_outgoing_qos true
user mosquitto

# Default listener
listener 8883
max_connections 64
protocol mqtt
cafile /etc/pki/tls/ca/mqtt-ca.crt
certfile /etc/letsencrypt/live/mqtt.mydomain.com/fullchain.pem
keyfile /etc/letsencrypt/live/mqtt.mydomain.com/privkey.pem
crlfile /etc/pki/tls/crl/mqtt-ca.crl                            # < added this line
dhparamfile /etc/mosquitto/dhparam.pem
require_certificate true
use_identity_as_username true

# Persistence
persistence true
persistence_file mosquitto.db
persistence_location /var/db/mosquitto/

# Logging
log_dest file /var/log/mosquitto/mosquitto.log
log_type error
log_type warning
log_type notice
log_type information
log_type subscribe
log_type unsubscribe
connection_messages true

# Security
allow_anonymous false

[cracksalad]

Some new infos:

1. This issue is not reproducible with Mosquitto v2.0 (the following list items might explain why)
2. As soon as I comment the crlfile line, the issue is gone
3. When copying the Let's Encrypt certificates to a folder like /etc/mosquitto/certs and chown mosquitto:mosquitto this folder, the issue is gone as well: even with the crlfile line uncommented!

I noticed that Mosquitto v2.0 drops to unprivileged user earlier than v1.6 (see migration docs) and would not start at all with the certfile and keyfile pathes I configured. So I thought of missing privileges. I also noticed that in the Mosquitto log files there are additional entries at the time of reloading:

2023-10-07T12:00:59: OpenSSL Error[0]: error:0200100D:system library:fopen:Permission denied
2023-10-07T12:00:59: OpenSSL Error[1]: error:20074002:BIO routines:file_ctrl:system lib
2023-10-07T12:00:59: OpenSSL Error[2]: error:140DC002:SSL routines:use_certificate_chain_file:system lib
...
2023-10-07T12:01:03: Client connection from 42.13.37.42 failed: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher.

I still do not understand how the crlfile config affects the certfile and keyfile privileges. I mean these are located in different directories and belong to different PKIs.

I hope this workaround already helps people, but I think this behavior is unintuitive and there should at least be a note in the docs somewhere to help others. Even the log output is not very helpful. I know, that is more like an OpenSSL problem, but no shared cipher is just not the point and fopen:Permission denied is kind of helpful but not very precise with respect to file name.