Implemented code would be a nice example of IoT Hub WebAPIs and simulated SmartHome sensors. This project is an assignment.
IoT Hub provides:
-
Virtual IoT sensors.
-
Virtual IoT Gateway that communicate using light-weight MQTT protocol.
-
Sensors data pipeline, Sensor → MQTT Broker → Kafka → InfluxDB.
-
WebAPIs to query metrics.
-
Authentication and authorization is not taking into consideration.
-
kafka-connect-influxdb-sink
could be replaced with open-source Kafka connector. -
WebAPI in
SensorDataQueryAPI
is highly opinionated. Need better API design.
-
kafka-connect-influxdb-sink
: Kafka InfluxDB Sink. -
virtual-device-simulator
: Simulated IoT sensors and Gateway. -
metrics-query-service
: Query on stored InfluxDB metrics.
-
Improve architectural design, with more added code, pattern should appear.
-
Code improvements:
-
Used
Observer
andObservable
are deprecated, need to find alternative. -
MinValueQueryService
is not implemented yet, but on purpose, just have to followMinValueQueryService
. -
Need to use more specific exception class, instead of
RuntimeException
. -
Add end-to-end tests. Cover more unit tests.
-
Improve logging.
-
-
Generate and check OWASP report to find vulnerable component.
-
Write Gatling performance tests.
-
virtual-device-simulator
: Virtual sensors and gateway, produces MQTT message and post to Mosquitto. -
mosquitto
: MQTT Broker. -
kafka-zookeeper
: Kafka zookeeper. -
kafka
: Event / Message bus. -
kafdrop
: UI to administer Kafka. -
simple-kafka-mqtt-connector
: MQTT to Kafka Connector. -
influxdb
: Time series database. -
chronograf
: Time-Series Data Visualization. -
kafka-connect-influxdb-sink
: Kafka to InfluxDB Connector.
$ docker-compose up # Make sure all services are running
$ telnet localhost 8086 # Verify InfluxDB is running.
$ telnet localhost 8086 # Verify InfluxDB is running.
$ java -jar metrics-query-service/target/metrics-query-service.jar
or
$ cd metrics-query-service
$ mvn spring-boot:run
$ telnet localhost 8080 # Verify metrics-query-service is running.
-
Should able to see Kafka topics at
https://docker-ip:9000/
. -
Should able to see InfluxDB data at
https://docker-ip:8888/sources/0/chronograf/data-explorer
.
Sample request:
curl --location --request POST 'localhost:8080/api/devices/metrics/aggregate?aggregateOperation=max&aggregateField=deviceData.temperature&deviceGroup=TemperatureSensor&startTime=1183495454&endTime=2583495454'
example response:
{
"query": {
"operation": "max",
"deviceId": null,
"deviceGroup": "TemperatureSensor",
"startTime": 1183495454,
"endTime": 2583495454,
"additionalParams": {
"aggregateField": "deviceData.temperature"
}
},
"response": {
"max": 44.903558920361355
}
}
To run the unit tests, execute the following commands
mvn clean test-compile test
To run the integration tests, execute the following commands
mvn clean test-compile verify -DskipTests=true
To run the integration tests, execute the following commands
mvn clean test-compile verify
To run the mutation tests, execute the following commands
mvn clean install test-compile test
mvn org.pitest:pitest-maven:mutationCoverage
$ cd kafka-connect-influxdb-sink
$ mvn clean compile package
$ docker build -t mmahmood/kafka-connect-influxdb-sink:latest -f Dockerfile .
$ docker push mmahmood/kafka-connect-influxdb-sink:latest
Licensed under the MIT License, see the LICENSE file for details.