Load balancing mosquitto broker

[win5923]

I'm currently deploying three Mosquitto brokers using Kubernetes, with two of them handling external load balancing. I have configured connections in mosquitto.conf to bridge internally. However, when testing retain messages, I noticed that due to load balancing, subscribers only receive half of the messages. The other half is received only when connected to the second external broker. I'd like to know how to ensure that subscribers receive the original quantity of retain messages sent by the publisher.

[Daedaluz]

Generally, i think load-balaning mosquitto brokers this way is a bad habbit.
I'm not sure what you are trying to do is possible to do reliably out of the box with mosquitto.
Short version is that the brokers don't share state.

I haven't tried this myself due to the nature of how client-ids play into queuing.
Since the brokers dont share state of which clients are connected and their subscriptions, after a clean session = false client reconnects it will make a new subscription on the second broker while the first broker builds up a huge queue, waiting for the client to come back for its messages just to end up in a situation where the clients will be required to reconnect every now and then to empty queues and receive duplicate messages it already received ages ago as new ones.

You also have a synchronization issue where you don't know which retained message is the latest one due to potential in-flight messages. Partially sent messages on one broker will not match message ids on the other broker either.

My recommendation is that you pick a broker more suitable for clustering or just roll with one broker.

You could also look into mosquitto pro version that should have support for HA.

[win5923]

Hi @Daedaluz, thanks for your reply.

I've created a simple diagram to illustrate the issue I'm currently facing. If you need the mosquitto.conf and client programs, let me know. The original intention was to distribute traffic.

It seems that Cadelo's HA Mosquitto involves directing traffic from the load balancer to a designated lead node, with other brokers set in follower mode. They synchronize data using their Cluster Management tool. In case the lead node fails, traffic is redirected to another node, addressing downtime but not achieving true load balancing.

Is it currently not possible for Mosquitto to implement Load Balancing functionality? I saw that HiveMQ and EMQX both support Load Balancer, but I have not tested the scenario involving retained messages yet.

[Daedaluz]

Not while retaining proper MQTT functionality at least.
One example is that, you can only have one client with the same client-id connected at the same time.
In your case, if you have two subscribers with the same client id, they might end up connecting to different brokers and break this rule.

I don't know what would cause your retained messages to not eventually reach all brokers, but some configs affecting this is retain_available , bridge_outgoing_retain, cleansession, and probably local_cleansession.

one good guess would be to set cleansession to true:

Set the clean session option for this bridge. Setting to false (the default), means that all subscriptions on the remote broker are kept in case of the network connection dropping. If set to true, all subscriptions and messages on the remote broker will be cleaned up if the connection drops. Note that setting to true may cause a large amount of retained messages to be sent each time the bridge reconnects.

however, this probably also has a side-effect that all queued messages with qos > 0 on the other broker will be dropped while down because the persistence broker should remove the subscriptions.

Expect more quirks down the road.

[win5923]

apologize for the misunderstanding. I am using different clients for publishing and subscribing, with qos = 2 and cleanSession = false. Regardless of whether the subscribe client reconnects or not, I always receive only half of the retained messages. I will try adjusting the parameters you provided. Thank you very much!

Edited: 1000 different client for publishing and 1 client for Subscribing