localhost mqtt broker with ditto connection

[Gauravmeenu]

Actually, I'm using MQTT broker in localhost with the help of docker image, and along with I'm also running eclipse ditto with docker-compose up as given in the ditto deployment GitHub repo. whenever I'm trying to testconnection with mqtt tcp localhost protocol it gives me an error like as given below .

{
"status": 504,
"error": "connectivity:connection.failed",
"message": "The Connection with ID 'mqtt-connection-source44' failed to connect.",
"description": "Could not establish a connection on 'localhost:1883'. Make sure the endpoint is reachable and that no firewall prevents the connection."
}

But when I'm trying to connect with mosquitto public server test.mosquitto.org it works fine.

#testconnection file given below

{
"targetActorSelection": "/system/sharding/connection",
"headers": {
"aggregate": false
},
"piggybackCommand": {
"type": "connectivity.commands:testConnection",
"connection": {
"id": "mqtt-connection-source44",
"connectionType": "mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp:https://localhost:1883",
"sources": [{
"addresses": ["my.sensors/#"],
"authorizationContext": ["nginx:ditto"],
"qos": 0,
"filters": []
}],
"mappingContext": {
"mappingEngine": "JavaScript",
"options": {
"incomingScript": "function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload)); const jsonData = JSON.parse(jsonString); const thingId = jsonData.thingId.split(':'); const value = {measurements: {properties: {temperature: jsonData.temperature, humidity: jsonData.humidity}}}; return Ditto.buildDittoProtocolMsg(thingId[0], thingId[1], 'things', 'twin', 'commands', 'modify', '/features', headers, value);}",
"outgoingScript": "function mapFromDittoProtocolMsg(namespace, id, group, channel, criterion, action, path, dittoHeaders, value, status, extra) {return null;}",
"loadBytebufferJS": "false",
"loadLongJS": "false"
}
}
}
}
}

[Daedaluz]

could you provide mosquitto config?

[Gauravmeenu]

Yeah sure this is my mosquitto.conf file.

listener 1883
listener 9001
protocol websockets
allow_anonymous true

[Daedaluz]

sorry, yea.. forgot to ask for mosquitto logs too.

[Gauravmeenu]

This is mosquitto logs.

1679896571: mosquitto version 2.0.15 starting
1679896571: Config loaded from /mosquitto/config/mosquitto.conf.
1679896571: Opening ipv4 listen socket on port 1883.
1679896571: Opening ipv6 listen socket on port 1883.
1679896571: Opening websockets listen socket on port 9001.
1679896571: mosquitto version 2.0.15 running

[Daedaluz]

ugh.. coffee hasn't kicked in.
your config looks like it's configured to tcp:https://localhost:1883.

simply put; all containers has their own localhost (unless net is set to host). you need to have a testconnection config to reach the mosquitto container in some appropriate way.

localhost for where you have testconnection != localhost for mosquitto,

[Gauravmeenu]

Okay, so could you please help me to how I can my testconnection confiig to reach the mosquitto dokcer container .

[Daedaluz]

easiest would probably to just throw in the mosquitto service into the deployment file of ditto, then use tcp:https://mosquitto:1883 as url or similar.

[Gauravmeenu]

Sorry @Daedaluz I can't get you .

[gunjan-it-engg]

easiest would probably to just throw in the mosquitto service into the deployment file of ditto, then use tcp:https://mosquitto:1883 as url or similar.

Actually I also got the same issue , can you please elaborate more about this

[Gauravmeenu]

easiest would probably to just throw in the mosquitto service into the deployment file of ditto, then use tcp:https://mosquitto:1883 as url or similar.

I did this but it gives me a null response error.

[Daedaluz]

how does your docker-compose file look like now then?

[Gauravmeenu]

My docker-compose file looks like this:

version: '3'

services:
mongodb:
image: docker.io/mongo:4.4.18
mem_limit: 256m
restart: always
networks:
default:
aliases:
- mongodb
command: mongod --storageEngine wiredTiger --noscripting
user: mongodb
ports:
- 27017:27017
environment:
TZ: Europe/Berlin

policies:
image: docker.io/eclipse/ditto-policies:${DITTO_VERSION:-latest}
mem_limit: 512m
restart: always
networks:
default:
aliases:
- ditto-cluster
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
# Set additional configuration options here appending JAVA_TOOL_OPTIONS: -Dditto.policies...
- JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class=
- MONGO_DB_HOSTNAME=mongodb
# in order to write logs into a file you can enable this by setting the following env variable
# the log file(s) can be found in /var/log/ditto directory on the host machine
# - DITTO_LOGGING_FILE_APPENDER=true
# only needed if DITTO_LOGGING_FILE_APPENDER is set
# volumes:
# - ditto_log_files:/var/log/ditto
healthcheck:
test: curl --fail hostname:8558/alive || exit 1
interval: 30s
timeout: 15s
retries: 4
start_period: 120s

things:
image: docker.io/eclipse/ditto-things:${DITTO_VERSION:-latest}
mem_limit: 512m
restart: always
networks:
default:
aliases:
- ditto-cluster
depends_on:
- policies
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
# Set additional configuration options here appending JAVA_TOOL_OPTIONS: -Dditto.things...
- JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class=
- MONGO_DB_HOSTNAME=mongodb
# in order to write logs into a file you can enable this by setting the following env variable
# the log file(s) can be found in /var/log/ditto directory on the host machine
# - DITTO_LOGGING_FILE_APPENDER=true
# only needed if DITTO_LOGGING_FILE_APPENDER is set
# volumes:
# - ditto_log_files:/var/log/ditto
healthcheck:
test: curl --fail hostname:8558/alive || exit 1
interval: 30s
timeout: 15s
retries: 4
start_period: 120s

things-search:
image: docker.io/eclipse/ditto-things-search:${DITTO_VERSION:-latest}
mem_limit: 512m
restart: always
networks:
default:
aliases:
- ditto-cluster
depends_on:
- policies
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
# Set additional configuration options here appending JAVA_TOOL_OPTIONS: -Dditto.search...
- JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class=
- MONGO_DB_HOSTNAME=mongodb
# in order to write logs into a file you can enable this by setting the following env variable
# the log file(s) can be found in /var/log/ditto directory on the host machine
# - DITTO_LOGGING_FILE_APPENDER=true
# only needed if DITTO_LOGGING_FILE_APPENDER is set
# volumes:
# - ditto_log_files:/var/log/ditto
healthcheck:
test: curl --fail hostname:8558/alive || exit 1
interval: 30s
timeout: 15s
retries: 4
start_period: 120s

connectivity:
image: docker.io/eclipse/ditto-connectivity:${DITTO_VERSION:-latest}
mem_limit: 768m
restart: always
networks:
default:
aliases:
- ditto-cluster
depends_on:
- policies
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
# if connections to rabbitmq broker are used, you might want to disable ExitOnOutOfMemoryError, because the amqp-client has a bug throwing OOM exceptions and causing a restart loop
# Set additional configuration options here appending JAVA_TOOL_OPTIONS: -Dditto.connectivity...
- JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class=
- MONGO_DB_HOSTNAME=mongodb
# in order to write logs into a file you can enable this by setting the following env variable
# the log file(s) can be found in /var/log/ditto directory on the host machine
# - DITTO_LOGGING_FILE_APPENDER=true
# only needed if DITTO_LOGGING_FILE_APPENDER is set
#volumes:
# - ditto_log_files:/var/log/ditto
healthcheck:
test: curl --fail hostname:8558/alive || exit 1
interval: 30s
timeout: 15s
retries: 4
start_period: 120s

gateway:
image: docker.io/eclipse/ditto-gateway:${DITTO_VERSION:-latest}
mem_limit: 512m
restart: always
networks:
default:
aliases:
- ditto-cluster
depends_on:
- policies
ports:
- "8081:8080"
environment:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
- ENABLE_PRE_AUTHENTICATION=true
# Set additional configuration options here appending JAVA_TOOL_OPTIONS: -Dditto.gateway.authentication.devops.password=foobar -Dditto.gateway...
- JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class=
# in order to write logs into a file you can enable this by setting the following env variable
# the log file(s) can be found in /var/log/ditto directory on the host machine
# - DITTO_LOGGING_FILE_APPENDER=true
# You may use the environment for setting the devops password
#- DEVOPS_PASSWORD=foobar
# only needed if DITTO_LOGGING_FILE_APPENDER is set
# volumes:
# - ditto_log_files:/var/log/ditto
healthcheck:
test: curl --fail hostname:8558/alive || exit 1
interval: 30s
timeout: 15s
retries: 4
start_period: 120s

swagger-ui:
image: docker.io/swaggerapi/swagger-ui:v4.14.3
mem_limit: 32m
restart: always
environment:
- QUERY_CONFIG_ENABLED=true
volumes:
- ../../documentation/src/main/resources/openapi:/usr/share/nginx/html/openapi:ro
- ../../documentation/src/main/resources/images:/usr/share/nginx/html/images:ro
- ./swagger3-index.html:/usr/share/nginx/html/index.html:ro
command: nginx -g 'daemon off;'

nginx:
image: docker.io/nginx:1.21-alpine
mem_limit: 32m
restart: always
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx.htpasswd:/etc/nginx/nginx.htpasswd:ro
- ./nginx-cors.conf:/etc/nginx/nginx-cors.conf:ro
- ./mime.types:/etc/nginx/mime.types:ro
- ./index.html:/etc/nginx/html/index.html:ro
- ../../documentation/src/main/resources/images:/etc/nginx/html/images:ro
- ../../documentation/src/main/resources/wot:/etc/nginx/html/wot:ro
- ../../ui:/etc/nginx/html/ui:ro
ports:
- "${DITTO_EXTERNAL_PORT:-8080}:80"
depends_on:
- gateway
- swagger-ui

mosquitto:
image: eclipse-mosquitto:2.0
container_name: mosquitto
restart: always
volumes:
- ./mosquitto.conf:/mosquitto/config/mosquitto.conf
ports:
- "1883:1883"
- "9001:9001"

volumes:
ditto_log_files:
driver: local
driver_opts:
type: none
device: /var/log/ditto
o: bind,uid=1000,gid=1000

[Daedaluz]

what does "null" error response mean?
does this mean it works? (like "no error")

or is the url supposed to be a websocket url, hence the websocket configuration?

[gunjan-it-engg]

no it's not work

[Gauravmeenu]

what does "null" error response mean? does this mean it works? (like "no error")

or is the url supposed to be a websocket url, hence the websocket configuration?

No it didn't work I got this kind of error

{
"type": "devops.responses:errorResponse",
"status": 503,
"serviceName": "connectivity",
"instance": "1f46323d92f2",
"payload": {
"status": 503,
"error": "ask.error",
"message": "Ask timed out on [ActorSelection[Anchor(akka:https://ditto-cluster/), Path(/system/sharding/connection)]] after [10000 ms]. Message of type [org.eclipse.ditto.connectivity.model.signals.commands.modify.TestConnection]. A typical reason for AskTimeoutException is that the recipient actor didn't send a reply.",
"description": "Please retry the performed action in order to improve resiliency."
}
}

[Daedaluz]

websocket url didn't work either?

[Gauravmeenu]

websocket url didn't work either?

I think connection is establish with tcp:https://mosquitto:1833, but I got error with mosquitto-client sub-pub

Unable to connect (Lookup error.).

[gunjan-it-engg]

is there any update regarding this @Daedaluz @Gauravmeenu @thjaeckle