Websockets performance is poor with mosquitto 2.x

[yschroeder]

After upgrading from Mosquitto 1.6.12 to 2.0.6, the performance when sending/receiving large messages (multiple megabytes) via websockets is very poor.

I am working with MQTT.js version 4.2.1 and build Mosquitto with libwebsockets version 4.1.6.

Tag v1.6.12 does not show any performance issues, while tag v2.0.0 is very slow. v2.0.6 is very slow as well.

I did a git bisect and it seems to be either commit 58aa41c or b91e783 that causes the issue. However, I cannot pinpoint it as I can't build mosquitto with some commits (some error in websockets.c).

@ralight in #1974 you mentioned that mqttx does some stupid stuff (that triggered that bug though). mqttx also relies on MQTT.js (same version 4.2.1 according to their package.json). So that stupid connection handling might also cause the performance issue I am seeing...

[ralight]

Do you get the same problems with libwebsockets 2.4.2? That's the most reliable version I have found.

[yschroeder]

Ok I spent some time get mosquitto running with libwebsockets 2.4.2 but was not successful. I built libwebsockets 2.4.2 from source, and built and linked mosquitto (tried 1.6.12 and 2.0.6) against that libwebsockets. It runs, but fails to open the websocket.

The following output was generated with log_type all and websockets_log_level 255

yannic@arch-vm ~/gits/mosquitto/src (git)-[tags/v2.0.6] % LD_LIBRARY_PATH=/home/yannic/gits/libwebsockets/build/lib ./mosquitto -c /etc/mosquitto/mosquitto.conf
1612123576: mosquitto version 2.0.6 starting
1612123576: Config loaded from /etc/mosquitto/mosquitto.conf.
1612123576: Opening ipv4 listen socket on port 1883.
1612123576: Opening ipv6 listen socket on port 1883.
1612123576: Opening websockets listen socket on port 9001.
1612123576: libuv support not compiled in
1612123576: Creating Vhost 'default' port 0, 1 protocols, IPv6 off
1612123576: Unable to find interface ҝ��V
1612123576: init server failed
1612123576: Failed to create default vhost
1612123576: Error: Unable to create websockets listener on port 9001.

Curiously, the line Unable to find interface has some different random bytes every time.

So I'm afraid I am unable to test with libwebsockets 2.4.2 :-(

[ralight]

My guess is that you are compiling against a later set of include headers, and because some public lws structs have been rearranged that is where the error is coming in.

Anyway, I suspect that the real problem is that recent versions of libwebsockets disable the external poll support feature that mosquitto uses. You have to compile it yourself with that option turned on to get proper websockets support. Set LWS_WITH_EXTERNAL_POLL to ON and rebuild libwebsockets and you should be fine. Or use 2.4.2.

[yschroeder]

I will try that.

But still, why does mosquitto 1.6.12 work fine with libwebsockets 4.1.6 while the 2.x versions of mosquitto don't? Or is that external poll feature of libwebsockets only used in the 2.x versions of mosquitto?

[ralight]

It's because in 2.0 a lot of optimisations were made so that for example mosquitto doesn't iterate over every client on each run of the network loop, to do various tasks, but is a lot more intelligent in when it does tasks. That means the external poll support (which was in use anyway, but wouldn't always be required for every operation) has to be correct. The optimisations aren't a huge factor at low client counts, but mean it's now possible to connect >1 million clients at once.

[yschroeder]

I tried the docker version of 2.0.6 and that has no performance issues.

I checked the Arch Linux package and they indeed build libwebsockets without LWS_WITH_EXTERNAL_POLL=ON. I changed the Arch package to enable external polling and with that version I have no issues as well.

So LWS_WITH_EXTERNAL_POLL=ON or using version 2.4.2 indeed fixes my issue.

Thanks for the help!

To avoid the same problem for others: Can Mosquitto detect if the used libwebsockets build has external polling enabled? If it detects that it's not enabled it could log a warning about websockets performance.

[ralight]

Good idea, that's done and in 2.0.7.

[yschroeder]

Quick update:
The problem in Arch Linux was fixed with libwebsockets 4.1.6-2.

Bug report in Arch Linux: https://bugs.archlinux.org/task/70321