Feature Request: Support reconnecting bridge when SIGHUP reload signal is received

[ssnover]

Production broker like GCP's IoT Core require authentication with a JSON Web Token (JWT). These tokens come with an expiration time on them which should be relatively short in production settings. Since reloading parameters for a bridge is not supported as noted in mosquitto.conf.5.xml: "Bridges cannot currently be reloaded on reload signal", my team's current solution is to periodically use a cron job to restart mosquitto after overwriting the JWT in the config. This requires us to have two brokers locally on the machine: one for inter-process communication and one as a bridge to IoT Core.

[ssnover]

I'm totally happy to take on implementing this feature, though I may need a little bit of direction after I have time to look around the codebase more. If there's any gotchas lurking I should know about ahead of time I'd appreciate the heads up!

I see there are currently two means of connecting, one of which is branched behind the WITH_ADNS option. Is it possible to split this feature into two steps: the first one implementing reconnect logic based on the branch without the DNS query and the second implementing reconnection for the more complicated WITH_ADNS branch?

[phlundblom]

I actually ran into a similar problem when planning for a IoT setup using Mosquitto and GCP IoT Core. I found it messy to rewrite configuration files with new JWTs and keeping that running, so my approach was to add the JWT generation + reconnection in mosquitto code. I currently have this running on a few clients but I haven't tested it with all different build flags and documentation isn't updated.

Maybe this is an alternative way forward? One hazzle though, I didn't manage to get a connection to GCP IoT core with latest develop branch HEAD, had to revert a few commits (see #2393).

Here's my take on JWT support: https://github.com/Sensenode/mosquitto/commits/feature/gcp-iot-core-jwt-bridge-auth2

[TLS12]

@phlundblom Thanks for sharing your fork and integrating JWT support. Would you mind sharing your mosquitto config file that you used with your fork to connect to gcp iot core?

[phlundblom]

Oh, of course. This is based on the example configuration file with all comments stripped:

autosave_interval 600
persistence true
persistence_file mosquitto.db
persistence_location ./
log_dest stdout
log_type error
log_type warning
log_type notice
log_type information
log_type debug
log_type all
connection_messages true
log_timestamp true
log_timestamp_format %Y-%m-%dT%H:%M:%S

connection iotcore
address mqtt.googleapis.com:8883
topic /devices/<iot_core_device_name>/events/# out 1
topic /devices/<iot_core_device_name>/commands/# in 0
topic /devices/<iot_core_device_name>/config in 1

bridge_attempt_unsubscribe false
bridge_protocol_version mqttv311
cleansession false
keepalive_interval 120
notifications false

remote_clientid projects/<gcp_project_name>/locations/<gcp_region>/registries/<iot_core_registry_name>/devices/<iot_core_device_name>
remote_password unused

remote_jwt_audience <gcp_project_name>
remote_jwt_expiration 28800
remote_jwt_keyfile <path_to_private_key_pem>
remote_username unused

restart_timeout 20
restart_timeout 10 30
restart_timeout 5 30
try_private false

bridge_cafile ./roots.pem

The added configuration parameters all start with remote_jwt_.

My plan was to try to finish of this by adding documentation and maybe some tests but the time hasn't really found its way to me...

[abiliojr]

Just as a side note, nowadays mosquitto has support for reloading the bridge configuration, and other nice things like TCP keepalives (google charges per PINGREQ).

There are even 2 ways of reloading: "lazy", where configuration changes only influence a future connection, and immediate, where any active connection gets severed and the new parameters work right away.

But I believe these features are all in the develop branch, waiting for a 2.1 release.

Regarding the case of short lived passwords, if you want to use an unmodified broker, you could just split your bridge config into another file and use an include from the main file. Then just rewrite the bridge config with the new JWT, and trigger reload (maybe from cron?).

Now, my two cents... I believe JWT as a password is very Google centric. I have not heard of any other implementation using it. Hypothetically, adding support for JWT directly in the broker, will then trigger the requests for "why not Amazon, or why not Microsoft, or ... insert name here..." And then you see where it can go: Mosquitto stops being lean just to support bridges to companies that like to do it their own way. Hence is my opinion that JWT creation should not belong in the code of the broker.

Maybe, to cover these cases, it would be better to have a plugin API that can implement each case individually, and outside of the main code.

[phlundblom]

I actually agree 100% with everything you write. When I started out this wasn't possible and I deemed the reloading approach a bit messy. If I remember correctly, the auth-related parameters wasn't reloaded when triggering a reload so a code update of what parameters are reloaded was necessary anyway. I haven't verified if this still is the case or not in the develop branch.

Another side note, connecting to GCP IoT Core broke with commit 799cdea (#2393). This was without any modifications, just setting the calculated JWT as password in the configuration file. Last I tried, this is still the case :(

[TLS12]

Same here, I agree as well. Having a plugin API exposed to manage (bridge) authentication credentials would be best to address this. Sadly I'm not familiar with the mosquitto plugin system yet.
@abiliojr Can you tell if things are already in place on mosquitto side so we "just" need to write the plugin or would we need to integrate this on mosquitto side first?

For the issue with the commit in the develop branch I will try this in the next days and will report if I can manage to connect to GCP IoT Core. When was your last try @phlundblom?

[phlundblom]

As I see it, there is no plugin system for authentication yet.

I will check GCP compatibility on develop HEAD later today.

[phlundblom]

I tried latest commit on develop (02d08b98) and the GCP connect problem is gone. Also, now it is possible to update remote_password using reload.

This means an unmodified Mosquitto can be used to connect to GCP IoT Core and keep the connection with external JWT generation. This is great news, it isn't possible with 2.0. ONly thing that one may would want is a way to force a reconnection using signals instead of waiting for GCP MQTT broker to throw you out and thus use the updated JWT.

[abiliojr]

@phlundblom , reconnection on signal can be achieved by setting bridge_reload_type to immediate.

@TLS12, the plugin interface is just an idea. If it sounds sensible, maybe we can open a new "feature request" ticket for such plugin interface, and we could brainstorm there :)

[abiliojr]

@ssnover or @ralight , I believe the feature requested here is already available in develop. Unless you feel something else is missing, feel free to close this ticket.