You broke my lights! AGAIN

[markg85]

Hi,

Version 2.0.11 worked
Version 2.0.14 broke it

The change i had to make this time.
From:
listener 1883 0.0.0.0

to:
listener 1883

I explicitly added 0.0.0.0 in an attempt to be ahead of config security nightmares so that my node is accessible locally and throughout the local network. Normally in other applications you need to set that explicitly.

So i thought i'd be "smart" and do that for mosquitto too. As i certainly didn't want it to kill my lights yet again.
I did find these hints to add 0.0.0.0 in the days of the 1.x to 2.x version migration on a forum to get it to behave like 1.x did in terms of network security.

Now it turns out that i have to remove 0.0.0.0 to get it working locally. Nice brainfuck.

With this is do have to specifically say that with 0.0.0.0 mosquitto does seem to be working locally when using mosquitto_pub on the same machine. But when running a nodejs script with the mqtt library you get a ECONREFUSED. That is all running on the very same machine with the mqtt connection being done like:

const mqtt = require('mqtt')
mqtt.connect('mqtt:https://localhost')

If you're wondering and just to be sure. No, there is no docker involved in this setup at all! It's all running as local system services. So localhost in my nodejs application and localhost in mosquitto_pub are the same.

So, my question..
Are you gonna break my config in the next couple of releases? If so, please warn me in advance as i will then fork this project and nuke the setting because this is getting ludicrous.

I'm really and truly fed up with debugging broken lights just to figure out - yet again - that it's your fault, not mine.

It does make me super eager to change my lights to be mesh based and not need mosquitto at all anymore. But that's a quite an enormous coding challenge that i don't really want to do just yet. But hey, you might just push me to get working on that.

[Daedaluz]

Okay.
I just don't see how "seem to work" and ECONREFUSED happens at the same time.
There is nothing magical about mosquitto_pub that allows it to publish locally and some other library not.
Connection refused is always connection refused regardless of client.

trying to be "smart" about how to be future-proof with some config is most likely going to be futile with any software as the config might be subject to change over versions, bug or no bug.

As for your debugging methods; I'm not sure how much of your stack you upgrade before you test it, but if it works before you upgraded mosquitto and not after, probably there is some change in mosquitto causing the issue (bug or no bug)
This conclusion shouldn't take more than a minute to reach.

• Did you read the change log or the blog (all red flags and warnings should be here)?
• Did you have a good reason to upgrade (any specific change perhaps, maybe security related)?
• Did you try the new version before putting it in "production" to see if it actually works the way you expect it to work?

So, my answer...

Q: Are you gonna break my config in the next couple of releases?
A: Not intentionally.

If so, please warn me in advance as i will then fork this project and nuke the setting because this is getting ludicrous.

Your warnings are in the change log.
Read it.
No one is going to mail/message you, and tell you that your lights might break again because of a change.
You are the maintainer of your stack - you have to make everything compatible and make sure everything works as you intend,
people here know very little how your lights work.

I'm not sure what you mean by "nuke the setting", but I'm 99% confident that forking mosquitto will cause you more work to keep it up to date than just giving the new, pre-compiled, version a spin and adapt to the changes.
You can also simply just stay put with the version you have that you know works and maybe attempt an upgrade when time allows for it and you are prepared for things to break.

There is also the option to roll back if something breaks and you don't have the time / energy to fix it on the spot and maybe do another attempt later when you have a new idea on how to solve it.
Also, consider doing a backup of your mosquitto database, if you have one, before upgrade.
Just in case there is any irreversible changes made to the database layout and you decide to roll back to the previous version.

These are things you always have to consider doing upgrades in production environments, home use or not.

Posting issues about how we broke your lights by you upgrading your software isn't going to get you very far with solving the problems you have.
This is your own fault, not any one of us.
I think the license also has a clause about this.

Speaking of the problems;

Config documentation says listener port [bind address/host/unix socket path] so I'd say that using a listener with listener 1883 0.0.0.0 should create a listener that accepts connections from all interfaces and should work for your case.
Removing it should default to localhost only if I'm not mistaken and not allow your lights to connect.
0.0.0.0 works for me, so maybe there is something else in your config that might make it not work.

Without additional information about your setup eg. your mosquitto configuration file and some logs would be nice,
there is very little I / we can do to help you further.

The fact that

const mqtt = require('mqtt')
mqtt.connect('mqtt:https://localhost')

gets ECONREFUSED.

And mosquitto_pub "works" is very confusing.

[markg85]

Your warnings are in the change log.
Read it.

Would you mind pointing me to the exact line that caused this? Because i can't find it.
I did read the changes for 2.0.12, 2.0.13 and 2.0.14

I'm not sure what you mean by "nuke the setting", but I'm 99% confident that forking mosquitto will cause you more work to keep it up to date than just giving the new, pre-compiled, version a spin and adapt to the changes.

By that i meant forking it without any future changes.
The alternative (which i did now) is forbid mosquitto to be updated. Which i hate doing because i like to stay up to date.
And that is to detect problems early on and spend relatively little time fixing it. Updating once a year (or decade) is bound to give a severe headache of breaking changes.

Also, consider doing a backup of your mosquitto database, if you have one, before upgrade.
Just in case there is any irreversible changes made to the database layout and you decide to roll back to the previous version.

Why would i want to do that?
Mosquitto is a pas through hatch. A central place to connect to where x parties put something something y others listen to whatever is being put there and do something with it.
In other terms: Definitely no database! Then again, i have 0 use for that.

No one is going to mail/message you, and tell you that your lights might break again because of a change. You are the maintainer of your stack - you have to make everything compatible and make sure everything works as you intend, people here know very little how your lights work.

I don't expect anyone to do that :) It was a sarcastic remark.

There is also the option to roll back if something breaks and you don't have the time / energy to fix it on the spot and maybe do another attempt later when you have a new idea on how to solve it.

I'm running ArchLinux ARM. That is not as simple as you make it sound.
Also, i first need to know what broke it to even consider rolling back. This time it was clear. Last time it wasn't.

Posting issues about how we broke your lights by you upgrading your software isn't going to get you very far with solving the problems you have.
This is your own fault, not any one of us.

Ohh no! This one is fully on you! Specifically the dev that caused this. (might or might not be you)
The previous one was on me. I blindly updated and assumed it all went fine. Just to, after a ton of debugging, figure out that mosquitto was the one causing it. But with that one (i opened an issue about it but only to share the story) i said that mosquitto was not to blame. While i don't like the change, you folks did all the right steps to inform users. I just didn't see any of it because of package managers :)

Config documentation says listener port [bind address/host/unix socket path] so I'd say that using a listener with listener 1883 0.0.0.0 should create a listener that accepts connections from all interfaces and should work for your case. Removing it should default to localhost only if I'm not mistaken and not allow your lights to connect. 0.0.0.0 works for me, so maybe there is something else in your config that might make it not work.

Well, that's not how it works.
Apparently.
As the config file is rather huge, here's a paste to it: https://p.sc2.nl/UpdTK
I can save you looking at it. This command: grep "^[^#;]" /etc/mosquitto/mosquitto.conf shows i have these settings:

log_type all
allow_anonymous true
listener 1883

For the purpose of this test, i changed the config to have these properties and values

log_type all
allow_anonymous true
listener 1883 0.0.0.0

As for MQTT connections.
Both of them run onder my local user (not root).
This works (turns the light off):

mosquitto_pub -h localhost -t "office/light2/switch" -m 'off'

For the json code. You need "the program" and package.json. Below is the program. I removed most of it but kept the structure so you get a feeling of how it works:

'use strict'

const express = require('express')
const app = express()
const mqtt = require('mqtt')
const mqttclient  = mqtt.connect('mqtt:https://localhost')

mqttclient.on('connect', function () {
  console.log(`Connected to MQTT`)
})

mqttclient.on('error', function (err) {
  console.log(`MQTT error: ${err}`)
})

mqttclient.on('offline', function () {
  console.log(`MQTT is offline! Your lights won't work... ouch!`)
})

app.listen(3000, () => {
  console.log('MiLight proxy hub is listening.')
})

and package.json:

{
  "name": "milightnodeproxy",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies": {
    "body-parser": "^1.19.1",
    "express": "^4.17.2",
    "mqtt": "^4.3.5"
  },
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Put them both in a folder and type npm install
Run with node index.js and it shows you:

MiLight proxy hub is listening.
MQTT error: Error: connect ECONNREFUSED ::1:1883
MQTT is offline! Your lights won't work... ouch!
MQTT error: Error: connect ECONNREFUSED ::1:1883
MQTT error: Error: connect ECONNREFUSED ::1:1883
MQTT error: Error: connect ECONNREFUSED ::1:1883

So as you can see. I truly do have mosquito_pub that works and nodejs that fails.
If i now change my mosquitto.conf back to:

log_type all
allow_anonymous true
listener 1883

Then both work fine and local network devices can connect to it just fine too.
And this is why i said "tell me if you break it again" because i see this as bugged.

Sorry for the very long message but this was probably much more helpful then either my or your post ;)

[Daedaluz]

I don't even think this is mosquitto related.
I think this is because you are attempting to connect with ipv6 as per ::1:1883.
Try connect with mqtt:https://127.0.0.1 and it will probably work.

You could also try add another listener to actually listen for ipv6 addresses too (i think).

I wouldn't classify this as a config change; maybe a bugfix to not listen for anything else than you explicitly tell it to listen to.

now; if you lead with configs and maybe some logs in the future instead of ranting about how other people break your stack after an upgrade with an issue covered 90% with just air, maybe things will go a bit smoother.

Would you mind pointing me to the exact line that caused this? Because i can't find it.
I did read the changes for 2.0.12, 2.0.13 and 2.0.14

as for which line could be your warning;

2.0.12 - 2021-08-31
--- snip ---
- Using certain listener related configuration options e.g. `cafile`, that
  apply to the default listener without defining any listener would cause a
  remotely accessible listener to be opened that was not confined to the local
  machine but did have anonymous access enabled, contrary to the
  documentation. This has been fixed. Closes #2283.
--- snip ---

This one talks about listener related fixes and about configuration. Not directly to the configuration line "listener" or ipv6, granted, but should be a red flag / warning about potential issues with your current setup, no?

By that i meant forking it without any future changes.
The alternative (which i did now) is forbid mosquitto to be updated. Which i hate doing because i like to stay up to date.
And that is to detect problems early on and spend relatively little time fixing it. Updating once a year (or decade) is bound to > give a severe headache of breaking changes.

That's my point. Having things automatically upgrade / using rolling release is bound to cause sudden issues with your setup.
I use Arch as my main dist. I run into problems constantly - I also like to detect potential issues early, but I'd never ever run rolling release on something I want to keep running for daily use like my own HA (Home automation) setup where i do pin it.
Today i run 2.0.14. without issues.

Pinning a version does not mean you have to stay there forever.
Just keep up with the release announcements and schedule a upgrade window.
This way it doesn't cause unexpected downtime with potential issues always present with upgrading.

Ohh no! This one is fully on you! Specifically the dev that caused this. (might or might not be you)
The previous one was on me. I blindly updated and assumed it all went fine. Just to, after a ton of debugging, figure out that
mosquitto was the one causing it. But with that one (i opened an issue about it but only to share the story) i said that mosquitto > was not to blame. While i don't like the change, you folks did all the right steps to inform users. I just didn't see any of it because of package managers :)

What a load of crap.
The changes introduced are done so for a reason.
Probably reasoning here is that mosquitto shouldn't listen to anything other than what you configured it to listen to.
That includes different versions of IP.

• I'd still say this is completely on you, breaking your lights.

i'll have to look into why removin 0.0.0.0 allows for external connections.

[markg85]

I don't even think this is mosquitto related.

and you end the message with:

i'll have to look into why removin 0.0.0.0 allows for external connections.

Say it however you want. There's a bug, that much is clear.
Working around that bug right now means i have to remove 0.0.0.0 and once this issue is fixed it probably means i have to explicitly add it again for things to work as they work now. That is a config change from my point of view. The config file itself might not change but the effect it has does.

I think this is because you are attempting to connect with ipv6 as per ::1:1883.
Try connect with mqtt:https://127.0.0.1 and it will probably work.

Hmm, you might be right about ipv6 there. That's probably default behavior in the nodejs mqtt package as i didn't change anything. The code i've shown you is exactly what i've executed without any changes on that side in between config changes. I've demonstrated it working and failing in the previous post. 127.0.0.1 might work but i'm just not going to change it from localhost as that should work too.

as for which line could be your warning;

2.0.12 - 2021-08-31
--- snip ---
- Using certain listener related configuration options e.g. `cafile`, that
  apply to the default listener without defining any listener would cause a
  remotely accessible listener to be opened that was not confined to the local
  machine but did have anonymous access enabled, contrary to the
  documentation. This has been fixed. Closes #2283.
--- snip ---

This one talks about listener related fixes and about configuration. Not directly to the configuration line "listener" or ipv6, granted, but should be a red flag / warning about potential issues with your current setup, no?

So here's my thought process when i was debugging it this time.

1. I update my packages and- yet again - figure out my lights don't work
2. Next up, i immediately look at mosquitto output, noticing no messages arrive from my proxy that lives on the same host. But i did see my lights being on the mosquitto
3. That pointed toward a local issue on the pc itself.
4. Thus sending a message with mosquitto_pub from the pc itself. Surprise, it worked!
5. Next i looked at the changelog entries and indeed caught that very block you mentioned. I even looked at the commit to see if there was something obviously wrong. Didn't seem like it from a first glance (a really simple code commit)
6. At this point i was kinda stuck and needed more logging information. As at this point it all still seemed fine yet the proxy (the nodejs thingy) wasn't sending anything to mosquitto. That much was clear.
7. --- now.. granted, the proxy was only doing a happy day case and not logging anything when a connection would fail. This bug made me change that and on top of it made me add a library to send me a push notification when i have no mosquitto connection. I'll be darn sure to detect a failing connection very fast next time this happens ---
8. Next was looking at the config and being amazed why it didn't work.. Then i thought "well, i added 0.0.0.0 just in case.. perhaps it's broken".. and voila, removing that made it magically work and made me really pissed off. To me not needing 0.0.0.0 is totally counter to many other applications out there that explicitly require 0.0.0.0 to essentially tell "the whole network can see this". So yeah, figuring out that i need to remove it wasn't a nice discovery.

Pinning a version does not mean you have to stay there forever.
Just keep up with the release announcements and schedule a upgrade window.

Nice theory but doesn't always work in practice.
Looking at the changelog - in hindsight - does not give me any explicit confidence that something might get broken. It, at the very most, tells me that there might be something risky. But then again, when mosquitto is in the update list i already see it as "ohh damn, there we go again" ;)

To be fair to you folks, the changelog does look nice and well written! You can't mention bugs you don't know about and i might have well discovered an unknown bug with this bug report. So i suppose the changelog for the version that fixes this (aka, requires 0.0.0.0 for the purposes that i use it) should give me a nice warning.

• I'd still say this is completely on you, breaking your lights.

It depends.
If this truly is a bug then it's all on the dev that caused the bug to be introduced.
You can argue that I should have done testing on my own which might or might not have caught this. There's something to be said for that, i'll give you that. But then again, how much testing do you expect someone for a home project to do?

Also, on the testing subject. I'm just an individual with a home project. It can probably be expected that my testing is close to none in terms of automated testing etc... But mosquitto on the other hand, as an Eclipse project and being company backed, should have tests in place for this. Specifically as this could well relate to security. Now i'm not saying you don't have tests (you probably do), but it certainly didn't caught this one. So some evaluation on the testing side might need to be done to catch this in the future.

[Daedaluz]

I don't even think this is mosquitto related.
and you end the message with:
i'll have to look into why removin 0.0.0.0 allows for external connections.

Well... you may have stumbled upon another bug unrelated to your "bug".

I'm under the impression that the mosquitto broker shouldn't allow external connections unless explicitly configured to do so.
So, removing 0.0.0.0 should either behave like no explicit binding (aka only localhost) OR merely having the line
listener 1883 implies that you have made a deliberate choice about what to listen to and therefore listens to everything, including ipv6.

I do not know which one is correct and therefore maybe there is a bug there. Both behaviors are sound.

However, explicitly listening to 0.0.0.0 (ipv4) as you did should not automatically create another listener for ::1 (ipv6) and therefore this mosquittos behavior is correct in this case, and your application buggy / faulty configuration.

Say it however you want. There's a bug, that much is clear.
Working around that bug right now means i have to remove 0.0.0.0 and once this issue is fixed it probably means i have to
explicitly add it again for things to work as they work now. That is a config change from my point of view. The config file itself
might not change but the effect it has does.

It's not so clear.
How node or the mqtt library does dns resolving i have no knowledge of, but it obviously resolves to ipv6 first (at least for localhost).
You don't HAVE to remove 0.0.0.0 to "work around" it, there are, in fact, several options:

1. hard code your application to use a ipv4 address.
2. simply make your application more configurable and let it connect with whatever you configure it to connect with.
3. Removing ::1 line from /etc/hosts file. this way the library can't resolve localhost to ::1
4. Add another listener in your mosquitto conf to listen for ipv6 addresses as well.
5. figure out why your node uses ipv6 to begin with, when mosquitto_pub (and probably most other software) resolves to ipv4.
6. just use the config without the explicit bind. it seems to work for you. (although this implies to remove 0.0.0.0)
   just to name a few.

Hmm, you might be right about ipv6 there. That's probably default behavior in the nodejs mqtt package as i didn't change
anything. The code i've shown you is exactly what i've executed without any changes on that side in between config changes.
I've demonstrated it working and failing in the previous post. 127.0.0.1 might work but i'm just not going to change it from
localhost as that should work too.

oh, wow, ok. Whether you introduce a simple change in your own software such as changing a string to use ipv4 when you obviously configured it to listen to ipv4, or even better - make it configurable, is up to you.
one thing is clear though, i think mosquitto does right not to bind to ::1 when you tell it to bind to 0.0.0.0.

IF it turns out that leaving it blank after listener 1883 is a bug, and you go with the "workaround" to remove 0.0.0.0,
You should expect further "configuration changes" down the road.

about your thought process; this is where i think you are doing it very easy for yourself to fall into these unexpected issues.
My thought process would normally go something like:

Assuming the time is for maintenance.

1. Read the change log and look for potential issues with upgrading (probably can do this step before attempting upgrade for a better plan).
2. fix eventual, obvious config changes and update.
3. at this point, in a perfect world, the software would run smoothly again. no issues.
   However, if problems arise, something obviously didn't work as i expect. and therefore i would turn to the
   mosquitto logs.
   Here i would find my lights connecting just fine, but not my controller software.
4. Since i didn't find the controller software connecting, i would look into the controller logs and find the very lines:
   MQTT error: Error: connect ECONNREFUSED ::1:1883

if this line didn't spell it out for me, the combination of the fact that mosquitto_pub worked and not my controller software should at the very least point towards something wrong with the controller, not mosquitto.
How / why did you jump to the conclusion that it would be mosquittos fault when the only thing not connecting was your own application at your step 2/3/4?

Next was looking at the config and being amazed why it didn't work.. Then i thought "well, i added 0.0.0.0 just in case.. perhaps
it's broken".. and voila, removing that made it magically work and made me really pissed off

well, okay. so you didn't fully understand the implications of binding to 0.0.0.0, and unlucky enough that your own application happened to use the ::1 address instead of 127.0.0.1?

To me not needing 0.0.0.0 is totally counter to many other applications out there that explicitly require 0.0.0.0 to essentially tell
the whole network can see this

Explicitly requiring 0.0.0.0 will definitively break any application attempting to connect on any ipv6 address.
It makes mosquitto (or any other application) only accept connections from any ipv4 address. you are connecting with ipv6. this does not work.
No need to remove 0.0.0.0 to make it work, just don't expect to be able to connect with ipv6.
OR just add a ipv6 listener. whatever float your boat (or lights).

Nice theory but doesn't always work in practice.

It probably works better than whatever you are doing right now.
Maybe you could provide an example where this does not work?

So i suppose the changelog for the version that fixes this (aka, requires 0.0.0.0 for the purposes that i use it) should give me a nice warning.

requiring the use of 0.0.0.0 for the purpose of connecting with ipv6, to me sounds a bit of a stretch.

But then again, how much testing do you expect someone for a home project to do?

I don't expect you to make any tests at all. You can test much as you want, or nothing.
Depends a little bit on the quality of uptime you want and the risk of something unexpected to happen.
less tests = more risk.
more tests = less risk.

Running as a home user does not remove all the risks involved of something failing in a larger system with an upgrade compared with a company.
The difference usually boils down to how many people are affected and the importance of it running without issues.

[markg85]

I like this sarcastic close to rude conversation. We both seem to be quite good at that, my compliments :)
It's enticing enough to write long very detailed replies but not rude enough to downright quit the conversation. Nice!

My reasoning here is that it worked before and only the update of mosquitto made it fail. I say only because the packages the nodejs application uses were not updated at all in this path. Now sure, there could be system packages having an effect that i don't know about yet. I have seen that before with with NFS (filesystem). But those kind of updates where package x breaks something totally unrelated in package y is super uncommon! I'm using Arch for years now and only have seen it like once or twice.

Therefore my gut instinct, which is proven by running the nodejs app in both config scenarios, is to blame mosquitto. Thus far it still looks like that is the right approach.

As for ipv6 and why i don't want to change localhost to 127.0.0.1 and for that matter don't want to change /etc/hosts either is because those options very definitely smell like fixing the symptom, not the cause. And applying that suggestion (specifically the /etc/hosts one) could very well have unintended consequences elsewhere. There is a cause here and i'm 99% certain it's in mosquitto. I could eventually be wrong (there's a 1% chance for that, hehe) and i will apologize without hesitation if i'm proven wrong. But as it stands, mosquitto "looks" to be at fault.

Think about it. mosquitto_pub with localhost works. Nodejs with localhost doesn't. There was no code update being done on the nodejs side (i did update it since though) so that does point to mosquitto.

The question now becomes

1. How does mosquitto_pub resolve localhost
2. How does nodejs resolve localhost

While i didn't update nodejs packages in the begin (again, i did now), pulling in the updates did update nodejs itself so there's a remote (very unlikely) possibility that something changed there affecting it.

As long as i don't know the cause, i'm not going to change a thing config wise as any change would be symptom fixing, not cause fixing. If you want to, i can have a look at the mqtt nodejs package to see if i can figure out why it even attempts ipv6 as i certainly didn't tell it to.

[Daedaluz]

Did adding 0.0.0.0 work for you with 2.0.11?
Sounds like you upgraded the broker and listener config "just in case something breaks" and went ahead with the upgrade.
My guess is that adding 0.0.0.0 in 2.0.11 would also break your application. (does it?)

If it does, maybe there wasn't a change introduced in the broker that broke your application, just the fact that you added 0.0.0.0 and your application happened to resolve localhost to ::1.

This would explain why your application worked before and not after upgrade, and fit the description of not touching your mqtt library version.

there is obviously a difference in how mosquitto resolves localhost to how your node resolves localhost.
In the end i think both will look into the /etc/hosts file and pick an entry they think is suitable.
the one mosquitto_pub picked happened to be the correct one for your broker configuration.

some might pick the first matching one, others maybe prefer ipv6 and others prefer ipv4.
there may be some saying in how this should be done, but there will always be quirks or different opinions and different implementations for endless reasons.

I would just take the easy route and make the hostname configurable and be done with it. This way the program can be made working even on the quirkiest systems + the user can select which one he / she wanna use.

[markg85]

Did adding 0.0.0.0 work for you with 2.0.11?

I literally said:

The change i had to make this time.
From:
listener 1883 0.0.0.0

to:
listener 1883

How much more explicit do you need it to be?
It had 0.0.0.0 in the config of 2.0.11 and that was working. It's the sole reason why I've reported this issue to begin with because now that needed to go in 2.0.14

Sounds like you upgraded the broker and listener config "just in case something breaks" and went ahead with the upgrade. My guess is that adding 0.0.0.0 in 2.0.11 would also break your application. (does it?)

Re-read the first post. You're now making assumptions that i clearly answered.
Your assumption is wrong. It worked in 2.0.11 with 0.0.0.0. I didn't add anything after 2.0.11 as a "just in case" meassure.

Now i did add 0.0.0.0 just in case - but READ CAREFULLY - that was when migrating from 1.x to 2.0.11. I found forum posts advising me to add that just to be sure on one side but also to be explicit. So in that time, for the migration, i did add it just in case and left it there as it was working as intended. But i did no such thing at all between 1.0.11 and 1.0.14. That means the config between 1.0.11 and 1.0.14 stayed the same.

The rest of your message is based on this wrong assumption and on top of it you're going to assume how /etc/hosts is parsed differently in different applications. Sorry but i'm going to ignore that message after the first line. We need a little more concrete evidence, not blunt assumptions.

Thank you for thinking with me though :)

[Daedaluz]

fact: connecting to an ipv6 address with a service binding to 0.0.0.0 will never work. period.
"just to be sure" does not work in the ipv6 world because that very statement assumes you use ipv4. you seem to be using ipv6.
again: this will not work.

There are a few options i can think of:

1. You somehow had a flawed 2.0.11.
   Adding 0.0.0.0 to the config should never have worked with ipv6 to begin with and your unchanged controller that uses ipv6
   happily relied upon this flaw.
   Then stopped working after an upgrade when mosquitto suddenly started to act correctly for whatever reason.
2. something in node / mqtt library (or any dependencies it has, including any c libraries) changed along with the upgrade
   causing it to choose ipv6, therefore broke your application because mosquitto always only listened to ipv4.

Actually running your test on my arch system exhibit the very same problem on both 2.0.11 and 2.0.14.
As i would expect.

You can test this too, by reverting to mosquitto 2.0.11 (only mosquitto if i may) and see for yourself, does it work or not?

Option number 2 is in this case much more likely if you ask me.

Now, how you choose to deal with the issue is up to you.
The solution is very easy: just make them use the same ip version.

The rest of your message is based on this wrong assumption and on top of it you're going to assume how /etc/hosts is parsed
differently in different applications. Sorry but i'm going to ignore that message after the first line. We need a little more concrete
evidence, not blunt assumptions.

The very fact that mosquitto_pub connected with ipv4 (because it worked) and your node connected with ipv6 (because it doesnt work) could be proof that some utilities resolve dns differently.
Since /etc/hosts is most likely the only place where the information translating "localhost" to either 127.0.0.1 or ::1 exist,
I don't see how mosquitto_pub otherwise could work.

My ping for example, also resolves localhost to ::1.
Running ping on my nas resolves to 127.0.0.1.

You can also test this by removing both localhost entries in your /etc/hosts file to see if any of the utilities still work.
My blunt assumption would be: no.

This is my explanation to why your controller does not work, and mosquitto_pub does.
In the end, the differences is easily overcome by being more specific about where to connect, eg:
if i ever wanted mosquitto pub to connect through ipv6:
mosquitto_pub -h ::1 ...

If you like, you can also see that mosquitto_pub is connecting with ipv4 if you look in your mosquitto logs for something like
New client connected from 127.0.0.1:47576 as ....
If mosquito_pub truly used a ipv6 address, the line would look like:
New client connected from ::1:35086 as ....

You can test this too by setting up a broker in the foreground with:

log_type all
allow_anonymous true
listener 1883

Notice that there is no 0.0.0.0.
Because apparently, this way mosquitto actually listens to both ipv6 and ipv4.

Run

$ # for ipv6
$ mosquitto_pub -h ::1 -t "test" -m "test"
$ # for ipv4
$ mosquitto_pub -h 127.0.0.1 -t "test" -m "test"

and look for the log lines with New client.

/etc/hosts is most likely not parsed differently because the information still has to be correct, just that the content is used differently.

There is actually another option you could try, is to disable ipv6 all together. That is if you are not actually using ipv6.

# <editor> /etc/sysctl.d/disable_ipv6

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

but who knows what consequences this has... i guess.
I know some systems that seems to be running flawlessly without it.