Websockets slow for large (ish) payloads

[aalibasic]

I am having an issue where sending a 1mb+ payload over mqtt takes 5-10 seconds to complete over localhost.

The same request over tcp takes 32 ms.

I have created a repo that reproduces the issue: https://github.com/Smart-Ag/mqtt_bug_report

To run the repo:

Open the index.html file in your browser
Run mosquitto docker image (docker-compose up -d)
install python dependencies
run python spammer.py
This will send message to the websocket and tcp listener.

I have tested the same clients with the emqx mosquitto broker and it was 10-40 times faster.

[ralight]

The commit I just pushed helps a lot with this. It seems as though I was making libwebsockets unhappy with write sizes and buffer sizes. The commit message marks this as closed, but it doesn't meet the tcp performance.

[aalibasic]

@ralight what kind of performance does it get with your changes?

How can I get this version of mosquitto to test?

[ralight]

I was getting ~170ms per message. You can get this version by getting the fixes branch:

git clone https://github.com/eclipse/mosquitto
cd mosquitto
git checkout fixes

Or get https://github.com/eclipse/mosquitto/archive/fixes.zip

Are you ok compiling yourself or would you like help?

[aalibasic]

@ralight I would love help getting it working in a docker image.

[ralight]

In the root of the source from above, run make localdocker. This should give you an (untagged) image of the current copy of the repo.

[aalibasic]

@ralight last question, when do you guys plan on making a release cut (tagged docker image) with this code? Or at least an expected wait time?

[ralight]

The aim is for roughly monthly bug fix releases, so soon. I'm trying to just figure a couple of other issues first.

[aalibasic]

@ralight I just tested mosquitto docker image: eclipse-mosquitto:1.6.6

1. Over the local network I am seeing reasonable performance (a few 100 ms)
2. Over a web network (Host the mosquitto broker on ec2) and then host client on local machine, then I get latency of 10,000 ms and it keeps growing as the broker gets backed up.

The emqx broker runs in 200 ms on scenario 1 & 2.

Any ideas on why there is such a difference in performance over local and internet?

[ralight]

That's bonkers. I've tested this out on test.mosquitto.org, which is under load but isn't using the docker image. Using pubsub.py I get:

Unencrypted:
Latency (QoS 0): 30.886173ms
Latency (QoS 1): 31.774521ms
Latency (QoS 2): 162.973881ms
With TLS:
Latency (QoS 0): 30.335188ms
Latency (QoS 1): 30.725718ms
Latency (QoS 2): 163.333654ms

Those results are pretty consistent across runs. I've tried with libwebsockets 2.4.2 and 3.1.0 with the same results. I haven't tried with the docker image yet.

If I understand what you are saying, you get 10,000ms latency right from the first message. Could you please try the python code with your instance to see what you get? You'll need the Paho python client paho-mqtt.