Ns4Kafka brings namespaces to Apache Kafka and a new deployment model for your Kafka resources with Kafkactl following the best practices from Kubernetes.

• Principles
  • Namespace Isolation
  • Desired State
  • Server Side Validation
  • CLI
• Download
• Install
• Configuration
  • Managed clusters
  • Admin account
• Administration

Ns4Kafka is an API that exposes all the required controllers to list, create and delete Kafka resources such as topics, connectors, schemas, Kafka Connect clusters and so on...

The solution is based on several principles.

Ns4Kafka implements the concept of namespace. Kafka resources are encapsulated in your namespace and you cannot see resources managed by other namespaces. The isolation is provided by granting ownership on names and prefixes to namespaces.

When you deploy a Kafka resource, Ns4Kafka saves it into a dedicated topic and alignes the Kafka cluster with the desired state of the resource.

Ns4Kafka applies customizable validation rules to enforce values on the configuration of your resources.

Ns4Kafka comes with Kafkactl, a CLI that lets you deploy your Kafka resources "as code" within your namespace using YAML descriptors. It can be used in CI/CD.

Ns4Kafka can be downloaded at https://github.com/michelin/ns4kafka/releases and is available as a fat jar.

A Docker image is available at https://hub.docker.com/repository/docker/michelin/ns4kafka.

Ns4Kafka needs a Kafka broker to store data and GitLab to authenticate users.

The project is based on Micronaut and can be configured with a Micronaut configuration file. There is an example of configuration file in src/main/ressource/application.yml.

If needed, properties from default application.yml can be overrided:

java -Dmicronaut.config.file=application.yml -jar ns4kafka.jar

Or

MICRONAUT_CONFIG_FILE=application.yml java -jar api.jar

Managed clusters are the clusters where namespaces take place, and resources are deployed.

This is how to configure your managed clusters:

ns4kafka:
  managed-clusters:
    clusterNameOne:
      manage-users: false
      manage-acls: false
      manage-topics: true
      manage-connect: false
      manage-role-bindings: false
      drop-unsync-acls: false
      config:
        bootstrap.servers: "localhost:9092"
      schema-registry:
        url: "https://localhost:8081"
        basicAuthUsername: "user"
        basicAuthPassword: "password"
      connects:
        connectOne:
          url: "https://localhost:8083"
          basicAuthUsername: "user"
          basicAuthPassword: "password"
        connect2:

• The name for each managed cluster has to be unique. This is this name you have to set in the field metadata.cluster of your namespace descriptors.

This is where you configure the admin user

micronaut:
  security:
    enabled: true
    authentication: bearer
    gitlab:
      enabled: true
      url: https://gitlab.com
ns4kafka:
  security:
    admin-group: test-ns4kafka
    local-users: # Not for production use.
      - username: admin
        # SHA-256 password.
        password: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
        groups:
          - "admin"

The group as to be set up on GitLab. So, if the admin group is "admin", a user will be admin if he belongs to the GitLab group "admin".

It is up to Ns4Kafka administrators to set up namespaces, owner ACLs, role bindings and quotas as these resources defined the context in which project teams will work. To create your first namespace, check the Kafkactl documentation.