Tested on:
- Debian Jessie
- Debian Stretch
- Ubuntu 14.04 LTS
- Ubuntu 16.04 LTS
- CentOS 7
finn-confluent is a puppet module to install, configure and run confluent.io distribution of kafka, zookeeper, schema-registry and rest service, on same or separate nodes in clusters.
The module places a lot of faith in the packaging of confluent, and only deviations from properties in property files are allowed. In addition, SysV init scripts are provided to control the services (Only start-stop scripts are provided by the packages).
You must classify your nodes with puppet, and java must be provided. See below for simple examples, and ways to provide properties for components.
Set up a standalone server like this:
package { 'openjdk-8-jre':
ensure => present,
} ->
class { 'confluent':
kafka_server => true,
zookeeper => true,
kafka_rest => true,
schema_registry => true,
}
This is the simplest form you will be able to use. For multiple servers, include only the components you wish the node to run:
package { 'openjdk-8-jre':
ensure => present,
} ->
class { 'confluent':
kafka_server => true,
}
See confluent/kafka docs for explanations of the different property files used to configure confluent components. The parameter names map directly to files in /etc/kafka, /etc/kafka-rest and /etc/schemaregistry.
To configure components, you can pass in hashes with properties for relevant components. Any key=value pair can be set.
class { 'confluent':
kafka_server => true,
kafka_server_properties => {}, #main props for kafka server, pass in puppet key => value pairs
kafka_log4j_properties => {},
connect_console_sink_properties => {},
connect_console_source_properties => {},
connect_distributed_properties => {},
connect_standalone_properties => {},
connect_file_sink_properties => {},
connect_file_source_properties => {},
connect_log4j_properties => {},
producer_properties => {},
consumer_properties => {},
tools_log4j_properties => {},
}
class { 'confluent':
zookeeper => true,
zookeeper_myid => '1', #Will create a myid file telling zookeeper who she is
zookeeper_properties => {},
}
class { 'confluent':
schema_registry => true,
schema_registry_properties => {},
schema_registry_log4j_properties => {},
connect_avro_distributed_properties => {},
connect_avro_standalone_properties => {},
}
class { 'confluent':
kafka_rest => true,
kafka_rest_properties => {},
kafka_rest_log4j_properties => {},
}