Need help with ECONNRESET using mqtt.js and VerneMQ

[wernermorgenstern]

Hi, I am trying to connect with NodeJs 18, and mqtt.js.

We are using VerneMQ on AWS ECS (Docker)

We have 2000 devices, each one of those sending a message every 10 seconds.
We can connect to the VerneMQ Broker fine, and the client can read all those messages for about 4 hours.

It then does not pull any more messages, even though messages are published. I even tried to put in an automatic fail-safe, that if no messages are received for 1 minute, to unsubscribe, and resubscribe. That does not work. And then I get this error.

Error: write ECONNRESET\n    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)\n    at writevGeneric (node:internal/stream_ │ │
│ base_commons:138:26)\n    at Socket._writeGeneric (node:net:929:11)\n    at Socket._writev (node:net:938:8)\n    at doWrite (node:internal/streams/writable:409:12)\n    at clearBuffer (node:internal/streams/writable:564:5)\n    at Writable.uncor │ │
│ k (node:internal/streams/writable:351:7)\n    at uncork (/app/node_modules/mqtt-packet/writeToStream.js:81:10)\n    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

I saw this Bug Report: #2092
It mentions to change the TCP Linger in the vernemq.conf file.

I am not sure if that could fix our issue. I am willing to try it.

How would I be able to change that option with the DOCKER_ Variables? What would the variable name, and also what would the value be?

[ioolkos]

@wernermorgenstern You can try "DOCKER_VERNEMQ_TCP_LISTEN_OPTIONS = [{nodelay, true}, {linger, {true, 6}}, {send_timeout, 30000}, {send_timeout_close, true}]"

In the Docker container, you can always cat /etc/vernemq.conf to check it generates the correct setting.

But you have another issue, a client should obviously not stop consuming messages after 4 hours. What QoS and MQTT protocol version does that session use? do you handle inflight window settings (based on protocol version this is either max_inflight_messages or receive_max_client)

I have also seen client applications/libraries getting stuck on their own internal buffer mechanisms for incoming messages.

if you need more help, contact us at [email protected]

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[wernermorgenstern]

Our QoS is 2 for the subscribe. The publisher uses QoS 0, and retain is set to false.
And we use MQTT 3.1.1
And we use a TLS Connection to the AWS LoadBalancer, connecting to port 8884. The Load Balancer terminates the TLS connection, and then connects to the ECS Task via 1883.

Right now, we want to test 2000 Devices. However, we want to be able to support 50_000 and even up to 100_000 and higher.

Regarding the variables, here are our settings

DOCKER_VERNEMQ_GRAPHITE_ENABLED: on
DOCKER_VERNEMQ_MAX_MESSAGE_RATE: 0
DOCKER_VERNEMQ_MAX_OFFLINE_MESSAGES: 50000
DOCKER_VERNEMQ_MAX_ONLINE_MESSAGES: 50000
DOCKER_VERNEMQ_MAX_INFLIGHT_MESSAGES: 10000
DOCKER_VERNEMQ_SHARED_SUBSCRIPTION_POLICY: prefer_online_before_local
DOCKER_VERNEMQ_UPGRADE_OUTGOING_QOS: off
DOCKER_VERNEMQ_COORDINATE_REGISTRATIONS: on
DOCKER_VERNEMQ_GRAPHITE_HOST: localhost
DOCKER_VERNEMQ_QUEUE_TYPE: lifo
DOCKER_VERNEMQ_TCP_LISTEN_OPTIONS: [{nodelay, true}, {linger, {true, 6}}, {send_timeout, 30000}, {send_timeout_close, true}]

I did run another test last night, after adding the TCP_LISTEN_OPTIONS, and right now all 1804 devices are "present" on the client side. In our grafana dashboard, it shows no dropped messages for the MQTT Broker.
So that is progress.

Also, we do use shared subscriptions, as we want to be able to scale our client (subscription) horizontally, and not receive and process duplicate messages.

However, if you could maybe help me confirm that our other settings are optimized, especially the max_inflight_messages.

What is the receive_max_client setting?

[ioolkos]

receive_max_client is a v5 only setting, similar to max_inflight_messages for v3.1.1.

A max_inflight_messages of 10k is possible, maybe a bit high. Worst case this whole window will be resent.

50k or 100k client connections are not a problem.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[wernermorgenstern]

Ok, thank you very much. So maybe we can reduce max_inflight_messages to 8k or 5k.

I will need to look into what the difference is between v3.1.1 and v5. Not sure if we will be able to go to V5.
We have mobile apps (Android and iOs) connecting to the MQTT Broker as well. So it depends if those client libraries will be able to handle v5.

So it seems that the DOCKER_VERNEMQ_TCP_LISTEN_OPTIONS configuration did the trick. Most like the linger value to be non-zero. I couldn't find any documentation in what that variable/option actually means.

Is there any documentation you know of?

Also, what other DOCKER_VERNEMQ_TCP_LISTEN_OPTIONS configs would help us in optimizing the MQTT and Connection?

[ioolkos]

Closing, since the issue seems resolved.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.