GitHub

Apache ZooKeeper - Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services
- Documentation
Apache Kafka - Distributed Streaming Platform
Installing kafka in Ubuntu 18.04 x64
- Download the binary

   $ tar xvzf kafka_2.12-2.3.0.tgz
   $ mv ~/Downloads/kafka_2.12-2.3.0 ~/Documents/_applications/

Add these settings to .bashrc for personal preference

   export APPLICATIONS_HOME="${HOME}/Documents/_applications"
   export KAFKA_HOME="${APPLICATIONS_HOME}/kafka_2.12-2.3.0"
   export KAFKA_BIN="${KAFKA_HOME}/bin"
   export KAFKA_CONFIG="${KAFKA_HOME}/config"

   alias kafka-restart='sudo systemctl restart confluentKafka'
   alias kafka-start='sudo systemctl start confluentKafka'
   alias kafka-stop='sudo systemctl stop confluentKafka'
   alias kafka-status='sudo systemctl status confluentKafka'
   alias kafka-enable='sudo systemctl enable confluentKafka'
   alias kafka-disable='sudo systemctl disable confluentKafka'

Configure to start kafka automatically with the server

   $ sudo cp kafka.service /etc/systemd/system
   $ sudo cp zookeepr.service /etc/systemd/system

Update the systemd service after copying kafka/zookeeper services

   $ sudo systemctl daemon-reload

Enable to auto-start kafka service

   $ kafka-enable

Start/Re-Start kafka service

   $ kafka-start
   $ kafka-restart

Check kafka version (OR sudo systemctl status kafka)

   $ kafka-status

CMAK - Cluster Manager for Apache Kafka

  $ $HOME/jdk14.sh
  $ $HOME/Documents/_applications/cmak-3.0.0.5/bin/cmak -Dhttp.port=9990 -Dconfig.file=$HOME/Documents/_applications/cmak-3.0.0.5/conf/application.conf -Dcmak.zkhosts="localhost:2181"

change the following in conf/application.conf

kafka-manager.zkhosts="localhost:2181"
basicAuthentication.username="admin"
basicAuthentication.password="password"

Adding new cluster in Kafka manager

:2181,:2181,:2181

Trifecta CLI

  $ . "$HOME/jdk8.sh"
  $ java -jar "$HOME/Documents/_applications/trifecta-bundle-0.18.13.bin.jar" --http-start -Dtrifecta.web.port=9980

Kafka UI Tools

  $ docker run -d --rm -p 9000:9000 \
      -e KAFKA_BROKERCONNECT=localhost:9092 \
      -e JVM_OPTS="-Xms32M -Xmx64M" \
      -e SERVER_SERVLET_CONTEXTPATH="/" \
      obsidiandynamics/kafdrop:latest

lensesIO - fast-data-dev

fast-data-dev
Docker Hub - This includes zookeeper and kafka
LensesIO Kafka Development Environment http:https://localhost:3030
OR Run individual docker images (see next sections)

  $ docker pull landoop/fast-data-dev
  $ docker run --rm --net=host lensesio/fast-data-dev

lensesIO - schema-registry-ui

schema-registry-ui
Docker Hub
Running the docker image with overridden port 8710 e.g. LensesIO schemaRegistryUI in http:https://localhost:8710
- See Docker run publish or expose port
Resolving lensesIO schema-registry-ui common issues
- lensesIO schema-registry-ui
- Confluent Schema Registry Configuration Options

  $ docker pull landoop/schema-registry-ui

  alias confluentZookeeperStart='cd ${CONFLUENT_HOME}; bin/zookeeper-server-start etc/kafka/zookeeper.properties'
  alias confluentKafkaStart='cd ${CONFLUENT_HOME}; bin/kafka-server-start etc/kafka/server.properties'
  alias confluentSchemaStart='cd ${CONFLUENT_HOME}; bin/schema-registry-start etc/schema-registry/schema-registry.properties'
  alias dockerLensesIOSchemaUI='docker run --rm -it -p 127.0.0.1:8710:8000 -e "SCHEMAREGISTRY_URL=http:https://localhost:8081" landoop/schema-registry-ui'

  $ confluentZookeeperStart
  $ confluentKafkaStart
  $ confluentSchemaStart
  $ dockerLensesIOSchemaUI

lensesIO - kafka-topics-ui

kafka-topics-ui
Docker Hub
Running the docker image with overridden port 8720 e.g. LensesIO kafkaTopicsUI in http:https://localhost:8720
- See Docker run publish or expose port
Resolving lensesIO kafka-topics-ui common issues
- lensesIO kafka-topics-ui
- Confluent ReST Proxy Configuration Options

  $ docker pull landoop/kafka-topics-ui

  alias confluentZookeeperStart='cd ${CONFLUENT_HOME}; bin/zookeeper-server-start etc/kafka/zookeeper.properties'
  alias confluentKafkaStart='cd ${CONFLUENT_HOME}; bin/kafka-server-start etc/kafka/server.properties'
  alias confluentRestStart='cd ${CONFLUENT_HOME}; bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties'
  alias dockerLensesIOTopicsUI='docker run --rm -it -p 127.0.0.1:8720:8000 -e "KAFKA_REST_PROXY_URL=http:https://localhost:8082" -e "PROXY=false" landoop/kafka-topics-ui'

  $ confluentZookeeperStart
  $ confluentKafkaStart
  $ confluentRestStart
  $ dockerLensesIOTopicsUI

lensesIO - kafka-connect-ui

kafka-connect-ui
Docker Hub
Running the docker image with overridden port 8730 e.g. LensesIO kafkaConnectUI in http:https://localhost:8730
- See Docker run publish or expose port
Resolving lensesIO kafka-connect-ui common issues
- lensesIO kafka-connect-ui
- Confluent Connect Configuration Options

  $ docker pull landoop/kafka-connect-ui

  alias confluentKafkaStart='cd ${CONFLUENT_HOME}; bin/kafka-server-start etc/kafka/server.properties'
  alias confluentZookeeperStart='cd ${CONFLUENT_HOME}; bin/zookeeper-server-start etc/kafka/zookeeper.properties'
  alias confluentConnectStart='cd ${CONFLUENT_HOME}; bin/connect-distributed etc/kafka/connect-distributed.properties'
  alias dockerLensesIOConnectUI='docker run --rm -it -p 127.0.0.1:8730:8000 -e "CONNECT_URL=http:https://localhost:8083" -e "PROXY=false" landoop/kafka-connect-ui'

  $ confluentZookeeperStart
  $ confluentKafkaStart
  $ confluentConnectStart
  $ dockerLensesIOConnectUI

Kafka sample commands

# create topics
  $ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic sample-input-topic
  $ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic sample-output-topic

# check/describe/delete topics
  $ kafka-topics.sh --list --zookeeper localhost:2181
  $ kafka-topics.sh --list --zookeeper localhost:2181 --topic sample-input-topic
  $ kafka-topics.sh --describe --zookeeper localhost:2181 --topic sample-input-topic
  $ kafka-topics.sh --delete --zookeeper localhost:2181 --topic sample-input-topic

# start kafka producer to manually enter some data
  $ kafka-console-producer.sh --broker-list localhost:9092 --topic sample-input-topic

# enter
  > kafka hello world
  > kafka sample data processing
  > kafka the quick brown fox jumps over the lazy dog
# exit

# start kafka producer to pipe in some file
  $ kafka-console-producer.sh --broker-list localhost:9092 --topic sample-input-topic < sampleTextFile.txt

# start kafka consumer
  $ kafka-console-consumer.sh --topic sample-input-topic --bootstrap-server localhost:9092 --from-beginning

  $ kafka-console-consumer.sh -bootstrap-server localhost:9092 \
      --topic sample-input-topic \
      --from-beginning \
      --formatter kafka.tools.DefaultMessageFormatter \
      --property print.key=true \
      --property print.value=true \
      --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
      --property value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer

Confluent Platform

Confluent Connectors

MongoDB
- Getting Started with the MongoDB Connector for Apache Kafka and MongoDB
Elastic Search
- Kafka Connect Elasticsearch Connector in Action
Redis
MySQL
RabbitMQ (Commercial License)
Confluent Platform Logs are in

/tmp/confluent.0YtCGnLS/

Confluent Control Center and LensesIO UIs

  $ confluentControlCenterStart
  $ dockerLensesIOSchemaUI
  $ dockerLensesIOTopicsUI
  $ dockerLensesIOConnectUI

Connecting Kafka to MongoDB Source

How to connect Kafka to MongoDB Source
- Setting MongoDB Replica Set
- MongoDB Kafka Connect Tutorial | Apache Kafka
Topic name must be in the form logicalName.databaseName.collectionName
- Kafka Debezium Connector for MongoDB
- So in essence, create the topic name with mongoConn.sampleGioDB.books
  - mongodb.name = mongoConn (see connect-mongodb-source.properties)
  - db name is sampleGioDB (in Mongo)
  - collection name is books (in Mongo)
MongoDB Source Connector in http:https://localhost:8183/connectors
- using bin/connect-standalone script with overridden configs
  - MongoDB Source Worker Config - etc/schema-registry/connect-avro-standalone-mongodb-source.properties
  - MongoDB Source Connector Config - etc/kafka/connect-mongodb-source.properties

  $ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mongoConn.sampleGioDB.books
  $ kafka-topics --list --zookeeper localhost:2181
  $ cd $CONFLUENT_HOME
  $ bin/connect-standalone  etc/schema-registry/connect-avro-standalone-mongodb-source.properties  etc/kafka/connect-mongodb-source.properties  ### This worked so use this
  $ bin/connect-distributed etc/schema-registry/connect-avro-distributed-mongodb-source.properties etc/kafka/connect-mongodb-source.properties  ### Doesn't work. Still needs to investigate why??? 
  $ kafka-console-consumer --bootstrap-server localhost:9092 --topic mongoConn.sampleGioDB.books  --from-beginning

Connecting Kafka to Elasticsearch Sink

Confluent Supported Kafka Connectors
Elasticsearch Service Sink Connector for Confluent Platform
Kafka Connect and Elasticsearch
Kafka Connect in Action: Elasticsearch
Kafka Connect Elasticsearch Connector in Action
Building Streaming Data Pipelines with Elasticsearch, Apache Kafka, and KSQL
Simplest Useful Kafka Connect Data Pipeline
- Part 1
- Part 2
- Part 3
Elasticsearch Sink Connector in http:https://localhost:8283/connectors
- using bin/connect-standalone script with overridden configs
  - Elasticsearch Sink Worker Config - etc/schema-registry/connect-avro-standalone-elasticsearch-sink.properties
  - Elasticsearch Sink Connector Config - etc/kafka-connect-elasticsearch/connect-elasticsearch-sink.properties
Elasticsearch results after running connector standalone elasticsearch sink. Note, kafka topic=mongoConn.sampleGioDB.books is converted to lowercase such as elasticsearch index=mongoconn.samplegiodb.books

  $ kafka-topics --list --zookeeper localhost:2181
  $ cd $CONFLUENT_HOME
  $ bin/connect-standalone  etc/schema-registry/connect-avro-standalone-elasticsearch-sink.properties  etc/kafka-connect-elasticsearch/connect-elasticsearch-sink.properties  ### This worked so use this
  $ bin/connect-distributed etc/schema-registry/connect-avro-distributed-elasticsearch-sink.properties etc/kafka-connect-elasticsearch/connect-elasticsearch-sink.properties  ### Doesn't work. Still needs to investigate why??? 
  $ kafka-console-consumer --bootstrap-server localhost:9092 --topic mongoConn.sampleGioDB.books  --from-beginning

Setting up Confluent Kafka Connect Plugins such as Kafka Connector MongoDB Source

Kafka connect plugin install
How to install connector plugins in Kafka Connect
Install the needed Kafka Connectors from Confluent Kafka Connectors Hub. e.g.

   $ confluent-hub install confluentinc/kafka-connect-elasticsearch:5.5.1
   $ confluent-hub install debezium/debezium-connector-mongodb:1.2.1
   $ confluent-hub install debezium/debezium-connector-mysql:1.2.1
   $ confluent-hub install mongodb/kafka-connect-mongodb:1.2.0
   $ confluent-hub install jcustenborder/kafka-connect-redis:0.0.2.11
   $ confluent-hub install hpgrahsl/kafka-connect-mongodb:1.4.0

Create plugins directory under $CONFLUENT_HOME, then create symlink from lib directory where the .jar files are

   $ cd $CONFLUENT_HOME
   $ mkdir plugins
   $ cd plugins
   $ ln -s ../share/confluent-hub-components/confluentinc-kafka-connect-elasticsearch/lib elasticsearch
   $ ln -s ../share/confluent-hub-components/debezium-debezium-connector-mongodb/lib debezium-mongodb
   $ ln -s ../share/confluent-hub-components/debezium-debezium-connector-mysql/lib debezium-mysql
   $ ln -s ../share/confluent-hub-components/mongodb-kafka-connect-mongodb/lib mongodb
   $ ln -s ../share/confluent-hub-components/jcustenborder-kafka-connect-redis/lib redis

Add the plugins path in the following files
- $CONFLUENT_HOME/etc/kafka/connect-distributed.properties
- $CONFLUENT_HOME/etc/kafka/connect-standalone.properties
- $CONFLUENT_HOME/etc/schema-registry/connect-avro-distributed.properties
- $CONFLUENT_HOME/etc/schema-registry/connect-avro-standalone.properties

plugin.path=$HOME/Documents/_applications/confluent-5.5.1/share/java,$HOME/Documents/_applications/confluent-5.5.1/share/confluent-hub-components,$HOME/Documents/_applications/confluent-5.5.1/plugins

The above plugins .jar files should work but in case it didn't get added to classpath then manually add the $CLASSPATH to .bashrc e.g.

   export CLASSPATH="$HOME/Documents/_applications/confluent-5.5.1/share/confluent-hub-components/debezium-debezium-connector-mongodb/*"

To verify that Mongo is connected to Kafka flowing event stream
- use Trifecta UI localhost
- OR use Confluent KSql
  - KSql Quick Reference
  - KSql Create Stream
  - KSql Print Kafka Topic's Content
  - alias confluentKSqlStart='cd ${CONFLUENT_HOME}; bin/ksql-server-start etc/ksqldb/ksql-server.properties'

   $ trifectaStart
   $ confluentKSqlStart
   $ ksql

   ksql> CREATE STREAM sampleGioBooks (id VARCHAR) WITH (kafka_topic='mongoConn.sampleGioDB.books', value_format='JSON');
   ksql> describe extended sampleGioBooks
   ksql> print 'mongoConn.sampleGioDB.books' from beginning;

Confluent Kafka Connect in Standalone/Distributed Mode

Kafka Blogs

Kafka Diagrams

Multiple Integrations
Multiple Integrations with Kafka
Kafka Connect
ksql - Stream SQL Engine for Apache Kafka

  $ confluent local list
  $ confluent local start # to start `all` services
  $ confluent local start ksql-server # to start `ksql-server` service only

OR

  alias confluentZookeeperStart='cd ${CONFLUENT_HOME}; bin/zookeeper-server-start etc/kafka/zookeeper.properties'
  alias confluentKafkaStart='cd ${CONFLUENT_HOME}; bin/kafka-server-start etc/kafka/server.properties'
  alias confluentKSqlStart='cd ${CONFLUENT_HOME}; bin/ksql-server-start etc/ksqldb/ksql-server.properties'

  $ confluentKSqlStart
  $ ksql

Confluent Connect

  $ confluent local list
  $ confluent local start # to start `all` services
  $ confluent local start connect # to start `connect` service only

Name		Name	Last commit message	Last commit date
Latest commit History 35 Commits
config		config
confluent/etc		confluent/etc
images		images
.bashrc		.bashrc
.ghci		.ghci
.gitignore		.gitignore
.npmrc		.npmrc
.profile		.profile
apacheKafka.service		apacheKafka.service
apacheZookeeper.service		apacheZookeeper.service
cmak.sh		cmak.sh
cmak_trifecta.sh		cmak_trifecta.sh
confluentKafka.service		confluentKafka.service
confluentZookeeper.service		confluentZookeeper.service
elasticsearch.service		elasticsearch.service
exploring_kafka_basics.iml		exploring_kafka_basics.iml
install_nvm.sh		install_nvm.sh
jdk14.sh		jdk14.sh
jdk17.sh		jdk17.sh
jdk8.sh		jdk8.sh
kafkatool.sh		kafkatool.sh
kibana.service		kibana.service
logstash.service		logstash.service
mongodb.service		mongodb.service
openjdk14.sh		openjdk14.sh
openjdk17.sh		openjdk17.sh
openjdk8.sh		openjdk8.sh
readme.md		readme.md
redis.service		redis.service
replicated_mongodb1.conf		replicated_mongodb1.conf
replicated_mongodb1.service		replicated_mongodb1.service
replicated_mongodb2.conf		replicated_mongodb2.conf
replicated_mongodb2.service		replicated_mongodb2.service
replicated_mongodb3.conf		replicated_mongodb3.conf
replicated_mongodb3.service		replicated_mongodb3.service
trifecta_kafka.sh		trifecta_kafka.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

CMAK - Cluster Manager for Apache Kafka

Trifecta CLI

Kafka UI Tools

lensesIO - fast-data-dev

lensesIO - schema-registry-ui

lensesIO - kafka-topics-ui

lensesIO - kafka-connect-ui

Kafka sample commands

Confluent Platform

Confluent Connectors

Confluent Control Center and LensesIO UIs

Connecting Kafka to MongoDB Source

Connecting Kafka to Elasticsearch Sink

Setting up Confluent Kafka Connect Plugins such as Kafka Connector MongoDB Source

Confluent Kafka Connect in Standalone/Distributed Mode

Kafka Blogs

Kafka Diagrams

Confluent Connect

Confluent git repo

About

Releases

Packages

Languages

giocosmiano/exploring_kafka

Folders and files

Latest commit

History

Repository files navigation

Kafka UI Tools

Kafka sample commands

Confluent Control Center and LensesIO UIs

Connecting Kafka to MongoDB Source

Connecting Kafka to Elasticsearch Sink

Setting up Confluent Kafka Connect Plugins such as Kafka Connector MongoDB Source

Kafka Blogs

Kafka Diagrams

About

Resources

Stars

Watchers

Forks

Languages