-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
messages are no longer retained with 2.0.16+ (with a lot of reained topics?) #2887
Comments
I tried to repliciate the problem in a test setup including another mosquitto as an mqtt brigde but so far was unable to replicate the problem. Will keep you updated. |
Yes, I did the same after a lot of troubles with no retained messages with 2.0.17 |
@stigvi Good to know we are at least not alone with this problem. Did you find a way to reproduce this problem somehow? |
related to #2785 |
@BubuOT I've been unable to reproduce this so far. Are you able to share any more details about the setup and what sorts of topics you are using? |
I've also seen this. Specifically I'm using mosquitto with Home Assistant and Zigbee2MQTT in this case. Previously, when Home Assistant was restarted, it would read device configurations set as retained messages from Zigbee2MQTT automatically. However on 2.0.17, it would seemingly never be able to receive these retained messages and as a result devices remained unavailable until a restart of Zigbee2MQTT to force it to republish all of the discovery topics at which point they would be successfully received by Home Assistant I confirmed that rolling back to 2.0.15, this began working as expected again. Let me know if there's any other information that would be helpful in reproducing |
@ralight I have so far also been unable to reproduce this outside our production environment :-/ Here's our mosquitto config though:
The bridge config (not sure if relevant.) looks like this:
|
Ok, thank you. I will keep looking, and if you find a way to reproduce it please let me know. |
We have this exact issue when upgrading mosquitto 2.0.15 to 2.0.18 on docker. I can replicate the issue with data "missing" on 2.0.16, 2.0.17 and 2.0.18 but switching back to the 2.0.15 docker image just by editing the docker-compose.yml file and restarting immediately "fixes" everything without needing to republish any data. In case it helps, we only start to see the issue with increasing volumes of data. We have 2 top level topics with a number of nested topics and at ~30k topics published on 2.0.15 or 2.0.18, connecting with MQTT Explorer shows ~1400 topics as the complete data set. Testing with only one top level topic with ~1100 subtopics, we get all of the data sent to the client correctly. It is only when we add a 2nd top level topic with ~28k subtopics that we start to see issues. For reference - our topic structure looks like this
I have a test server that only has dockerized mosquitto on and no personal data that I can let a project dev have root access to if you want to see this happening. |
FWIW, we also have a lot of retained topics (~10k) in production where we saw this issue. This might explain why I've never been able to reproduce this in any test setup 🤔. |
I am having the same issue. All devices come up as "Unavailable" on Home Assistant after an HA reboot. |
Well you can temporarily fix it very easily by reverting to 2.0.15 where retained topics are working as expected |
Not possible. I’m using MQTT HA Addon. No way to go back in version unfortunately. |
I just restored to HA addon version 6.2.1, which still has mosquito 2.0.15 - backups are a life saver. |
I'm happy for you, but unfortunately, I wasn't aware of when the problem started until I had to reboot HA. |
@ZetaWaves: HA does partial backups before every update by default, are you sure you have none? Or have you disabled that? |
I have the same issue with HA Core 2023.10.1, the HA Add-on v6.3.1 (2.0.17) and Zigbee2MQTT. Several devices became Unknown, even though they had a state in Z2M. Restoring my add-on back to the version with 2.0.15 has resolved the issue. I have ~1400 topics. |
That's something you should ask Home Assistant, not Mosqitto. |
Yes, restoring Mosqitto from HA. |
anyone know if this has been fixed in the recent updates or if a fix is coming ? I am still sitting on 2.0.16 for the same reason. |
Yes there is a bugfix for this included. See: |
Aha so this was actually #2893 Perfect, thanks! |
sigh nope... when I log on with MQTT explorer, none of the messages are retained. Back to 2.0.15 :/ |
It is not fixed in 2.0.18. Still broken. Just tested. |
It is not fixed in this release. Actually - it's worse in this release than before. |
Working for me. |
It may be an issue with the MQTT broker addon for HA. I have around 30 clients, each with around 100 entities. And not a single message is retained, despite several reboots etc. Retained messages however work with version 2.0.15 (Addon version 6.21). Anyone know who maintains the addon ? |
Frenk. You can have a chat on discord here: https://discord.gg/jBwTDzJ4sk I'm using the HA addon but don't have anywhere near that many topics. |
From what I can gather, its primarily hitting users with a lot of topics (I guess 5550 topics is high). Its so rarely reported though, I don't even know where to look. Will check out the discord. |
If you find out anything interesting on Discord, a post here would be very much appreciated. |
Tagged French. Will see if he has ny comments or guidance. Might just be time to host the MQTT broker outside HA, which I really was hoping to avoid. |
I don't think it's anything specific to hosting the broker in HA, it's the same container regardless of where you're hosting it. I run mine completely independently and I'm still impacted by this |
2.0.18 doesn't fix this, no. E: Updated the title to reflect this. Also added a hint that this might be related to having a lot of retained topics. |
Just adding that I ran into this problem running 2.0.18 and downgrading to 2.0.15 resolved the issue. Basically the broker just won't retain messages. I even manually try to set a retained message using MQTTExplorer and even that fails to retain. All good on 2.0.15 |
Yeah its likely the same issue. I totally get that this isn't necessarily fixed immediately. But it sure would be nice if it at least could get acknowledged as an actual issue. |
I can also confirm this issue. Rolled back my container tag to 2.0.15 and all is well again with Zigbee2MQTT and HA. |
Why do you ask? What alternatives are there? |
No idea... new 2.0.15 image with vulnerabilities fixed? I mean if you thought that it was bad when devices were not working, I really think it will be way worst than that if someone manages to exploit these vulnerabilities. |
Hey all just to say I was having these issues also with my setup. Tried a lot of things but in the end it was a Home Assistant Core and Operating System update that fixed it for me. I was going to do an add-on restore but luckily didn't need to. |
This is impacting people that don't even run HAOS. |
Can this be solved by increasing max_queued_messages? |
I'll be installing the updated HA Add-on with that change (home-assistant/addons#3615) in a few hours to see. |
Update: installed Result: So far, so good. The previous version with the issue resulted in devices showing as unavailable, whereas everything has come online as expected. |
So far, so good here, too. I think I stay on 2.0.18. Default value for max_queued_messages is 1000 and I have set it to 8192. (0 is unlimited) I have restarted my system several times and there is no problem with retained messages. |
Added the max_queued_messages 8192 to my conf file, upgrade to the latest stable, restarted HA and the issue is still there. Rolled back to 2.0.15 and the issue goes away. |
There is a way to reproduce: a) Setup mosquitto broker with
c) subscribe to the broker to retrieve all the retained message. Make the subscriber "slow" by using debug and json output:
As a result, only part of the messages will be retrieved. d) The cause of the problem is in
This behaviour was introduced by fixing CVE-2024-28366 in 2.0.16 in Not sure if the change was necessary to fix the CVE. Especially because To avoid the problem raise Using QoS1 or QoS2 for retained messages does not help as the |
Since upgrading to mosquitto 2.0.16 (and later 2.0.17) (via docker image) retained messages received by the broker are no longer handled as retained and not delivered to new subscribers.
Downgrading to 2.0.15 makes this work as expected again.
(Happy to add more details about our setup if required)
The text was updated successfully, but these errors were encountered: