kafkaproxy is a reverse proxy for the wire protocol of Apache Kafka.

Since the Kafka wire protocol publishes the set of available broker endpoints from the brokers to the clients during client bootstrapping, the brokers must be configured to publish endpoints that are reachable from the client side. This can be a cumbersome restriction in several different situations, such as:

• Network topologies preventing direct access to the broker nodes
• Multiple networks from which broker nodes should be reachable
• DNS resolution restrictions when accessing TLS secured broker nodes
• Using the sidecar pattern for TLS termination in Kubernetes

This is where kafkaproxy comes into play and allows for transparent relaying of the Kafka wire protocol by rewriting the relevant parts of the communication where the brokers publish the endpoint names - with user-configurable endpoints where the the proxy instances can be reached.

kafkaproxy is built to be run in a container. The released versions are available at Docker Hub.

The following configuration parameters are mandatory:

• KAFKAPROXY_HOSTNAME: the hostname at which the proxy can be reached from the clients.
• KAFKAPROXY_BASE_PORT: the first port to be used by kafkaproxy.
• KAFKAPROXY_BOOTSTRAP_SERVERS: the comma separated list of initially mapped endpoints. This is usually the list of bootstrap brokers or a load balancer in front of the kafka brokers.

For example:

docker run \
    --net host \
    -e KAFKAPROXY_HOSTNAME=localhost \
    -e KAFKAPROXY_BASE_PORT=4000 \
    -e KAFKAPROXY_BOOTSTRAP_SERVERS=kafka:9092 \
    -d dajudge/kafkaproxy:0.0.18

Note: You will have to make the proxy ports (starting with KAFKAPROXY_BASE_PORT and incrementing from there) available from outside the container with -p PORT:PORT if you're not using --net host.

If you have docker-compose installed you can try out kafkaproxy by using the demonstration setup provided in the example directory. So clone the kafkaproxy repo and run the following commands:

Step 1: Start kafka, zookeeper and kafkaproxy.

docker-compose -f example/docker-compose.yml up -d

Kafka will take a couple of seconds to fully start and become available.

Step 2: Create my-test-topic.

docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-topics --create --topic my-test-topic --bootstrap-server localhost:4000 --partitions 1 --replication-factor 1

Step 3: Publish a message to my-test-topic.

echo "Hello, kafkaproxy" | docker run --rm --net host -i confluentinc/cp-zookeeper:5.2.1 kafka-console-producer --broker-list localhost:4000 --topic my-test-topic

Step 4: Consume to produced message from my-test-topic.

docker run --rm --net host -it confluentinc/cp-zookeeper:5.2.1 kafka-console-consumer --bootstrap-server localhost:4000 --topic my-test-topic --from-beginning --max-messages 1

Cleanup: Stop and remove the demonstration containers.

docker-compose -f example/docker-compose.yml rm -sf

Explanation: The docker-compose.yml file starts up a kafka broker (along with it's required zookeeper) that is only available from within the docker network as kafka1:9092. The kafkaproxy is configured to proxy this kafka instance as localhost:4000 which is also mapped from outside the docker network.

kafkaproxy is configured using mostly environment variables and a broker map file in YAML format. The following section describe the configuration options in detail.

kafkaproxy requires some general information to start.

The client SSL configuration determines how the Kafka clients have to connect to kafkaproxy. Configuration can be provided using the following environment variables:

The Kafka SSL configuration determines how kafkaproxy connects to the Kafka broker instances. Configuration can be provided using the following environment variables:

• A Guide To The Kafka Protocol
• Kafka protocol guide