Bridge keeps sending the same QoS 2 messages

[suhrmann]

Error

I am publishing with HiveMQ-CLI 1000 MQTT QoS 2 messages to Mosquitto with Payload "Hallo Welt <n>" with n ∈ [1 ; 1000] - ca. one message per second. Mosquitto is configured to bridge all messages to HiveMQ-CE - both broker in a Docker container in its own VM. For testing I disabled the network of the bridge end-point HiveMQ-CE (via VM config) for about one minute (~60 messages). After enabling the network HiveMQ-CE receives the same messages over and over again (see logged messages at HiveMQ-CE as CSV and mosquitto.log in Pastebin)

Tested with Mosquitto Docker 2.0.9 and 2.0.10

Reproduce

Stacks:

• Virtualbox VM-Mosqutto [192.168.113.131]: Ubuntu + Docker + Docker-Compose + Mosquitto from Docker hub
• Virtualbox VM-HiveMQ [192.168.113.130]: Ubuntu + Docker + Docker-Compose + HiveMQ-CE from Docker hub

1. Start script that publishes one MQTT message per second to VM-Mosqutto - payload: "Hallo Welt <counter>"
2. After ~100th message disable network of VM-HiveMQ
3. After ~200th message enable network again
4. Highest received message / payload is "Hallo Welt 147" -> 147th message

---

For simplicity both container in the same compose - I run service mosquitto in one VM and service hivemq in the other one:

docker-compose.yml:

version: '3.9'
services:
  mosquitto:
    image: eclipse-mosquitto:latest
    container_name: mosquitto
    restart: always
    volumes:
      - ./mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - /mosquitto/data
      - /mosquitto/log
    ports:
      - 1883:1883
      - 9001:9001
  
  hivemq:
    image: hivemq/hivemq-ce
    container_name: hivemq-ce
    restart: always
    ports:
      - 1883:1883

mosquitto.conf:

listener 1883

persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
log_type all

allow_anonymous true

# Bridge
connection wohnung2thd
address 192.168.113.130:1883
topic # out 2
bridge_protocol_version mqttv50
bridge_insecure true

max_queued_messages 10000
cleansession false
start_type automatic
notifications true
try_private false

Bash script to publish 1000 messages to local broker / VM-Mosqutto using MQTT CLI:

NUM_MESSAGES=1000

for (( cnt=1; cnt <= NUM_MESSAGES; cnt++ ))
do
  #echo "### Sending message number #$cnt..."
  mqtt pub --qos 2 --topic "test2" --message "Hallo Welt $cnt"
  echo "$(date +"%m-%d-%Y %H:%M:%S") Sent message number #$cnt"
  sleep 1s
done

[ralight]

Thanks for the detailed instructions, I'll look at this when I get a chance.

[suhrmann]

Update:

I did further investigation: In Mosquitto 2.0.10 Docker container instead of forwarding the same message again and again the MQTT bridge seems to rather have problems to reconnect. After re-enabling the network no messages arrives at HiveMQ (the endpoint of the bridge), but after restarting the Mosquitto the same message arrive at HiveMQ again and again.

With Mosquitto 2.0.10 and the same test-setup as previously: 1000 QoS 2 Messages from Mosquitto --bridge--> HiveMQ-CE.

1. After Message 92 disconnect network of HiveMQ VM
2. reconnect network
3. No message arrive at HiveMQ (bridge endpoint) at all
4. Restart Mosquitto container
5. The unforwarded messages were persisted (not verified), but with the same problem as above: The same message again and again.
6. The bridge keeps continuously sending, but, again, the same messages

Also interesting: I stopped the script at "Sent message number 899" but HiveMQ received (1078-16tests) = 1062 entries

mosquitto.log (restart Mosquitto at line 5100)

message.log

[suhrmann]

I did the same tests with TLS and auth now and can not reproduce this behavior.