KBookDemo aims to provide a simple way to define real-time streaming data processing from a DAG (directed acyclic graph). End-user submits DAG definition (called KBook, which will be run as Kafka Streams Tasks) to the server and the server is responsible for start/stop/monitor these instances. Since the DAG itself is just a json description of the graph (see sample1.json), it is possible to visually build the DAG using some sort of frontend and provide interactive development experiences. For example, inspect the schema of nodes or see sample data flowing through edges.

Each node in the DAG definition represents a logic processing step, e.g. branch-filtering, aggregation, generic state machine, sanitizing, or other independent business logic units. Behind the sense, the DAG is translated into an instance of Kafka Streams Topology. It is worth mentioned that in spite of this demo leveraging Kafka Streams to provide parallelism and fault-tolerance, the same algorithm/concept can be used in other streaming data processing frameworks (Spark, Flink) as well. You can see how the algorithm is applied in KBook.Scala.

The next step in the roadmap is to to turn on High Availability with Zookeeper.

Mill is the build tool used by KBookDemo.

mill server.compile

mill server.assembly

java -jar KBookDemo.jar

please refer to sample1.json. Make sure the KBook has a uuid, which is typically generated by frontend (the DAG Maker).

curl -d "@sample1.json" -H "Content-Type: application/json" -X POST https://localhost:9999/book/post

curl -X POST https://localhost:9999/book/play?uuid=the_kbook_uuid_submitted

curl -X POST https://localhost:9999/book/stop?uuid=the_kbook_uuid_submitted

curl -X POST https://localhost:9999/book/delete?uuid=the_kbook_uuid_submitted

curl -X POST https://localhost:9999/book/show?uuid=the_kbook_uuid_submitted

curl -X GET https://localhost:9999/node/list

GNU GPLv3