Status | |
---|---|
Stability | beta |
Supported pipeline types | metrics, traces, logs |
Distributions | contrib |
Kafka receiver receives traces, metrics, and logs from Kafka. Message payload encoding is configurable.
Note that metrics and logs only support OTLP.
The following settings are required:
protocol_version
(no default): Kafka protocol version e.g. 2.0.0
The following settings can be optionally configured:
brokers
(default = localhost:9092): The list of kafka brokerstopic
(default = otlp_spans): The name of the kafka topic to read fromencoding
(default = otlp_proto): The encoding of the payload sent to kafka. Available encodings:otlp_proto
: the payload is deserialized toExportTraceServiceRequest
.jaeger_proto
: the payload is deserialized to a single Jaeger protoSpan
.jaeger_json
: the payload is deserialized to a single Jaeger JSON Span usingjsonpb
.zipkin_proto
: the payload is deserialized into a list of Zipkin proto spans.zipkin_json
: the payload is deserialized into a list of Zipkin V2 JSON spans.zipkin_thrift
: the payload is deserialized into a list of Zipkin Thrift spans.
group_id
(default = otel-collector): The consumer group that receiver will be consuming messages fromclient_id
(default = otel-collector): The consumer client ID that receiver will useauth
plain_text
username
: The username to use.password
: The password to use
tls
ca_file
: path to the CA cert. For a client this verifies the server certificate. Should only be used ifinsecure
is set to true.cert_file
: path to the TLS cert to use for TLS required connections. Should only be used ifinsecure
is set to true.key_file
: path to the TLS key to use for TLS required connections. Should only be used ifinsecure
is set to true.insecure
(default = false): Disable verifying the server's certificate chain and host name (InsecureSkipVerify
in the tls config)server_name_override
: ServerName indicates the name of the server requested by the client in order to support virtual hosting.
kerberos
service_name
: Kerberos service namerealm
: Kerberos realmuse_keytab
: Use of keytab instead of password, if this is true, keytab file will be used instead of passwordusername
: The Kerberos username used for authenticate with KDCpassword
: The Kerberos password used for authenticate with KDCconfig_file
: Path to Kerberos configuration. i.e /etc/krb5.confkeytab_file
: Path to keytab file. i.e /etc/security/kafka.keytab
metadata
full
(default = true): Whether to maintain a full set of metadata. When disabled the client does not make the initial request to broker at the startup.retry
max
(default = 3): The number of retries to get metadatabackoff
(default = 250ms): How long to wait between metadata retries
autocommit
enable
: (default = true) Whether or not to auto-commit updated offsets back to the brokerinterval
: (default = 1s) How frequently to commit updated offsets. Ineffective unless auto-commit is enabled
message_marking
:after
: (default = false) If true, the messages are marked after the pipeline executionon_error
: (default = false) If false, only the successfully processed messages are marked Note: this can block the entire partition in case a message processing returns a permanent error
Example:
receivers:
kafka:
protocol_version: 2.0.0