Skip to content

Commit

Permalink
[exporter/datadogexporter] Add config example for kafka metrics (#30752)
Browse files Browse the repository at this point in the history
**Description:** 
This PR adds an example on how to configure the OTel collector to pull
kafka metrics.

**Link to tracking Issue:** <Issue number if applicable>

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>
  • Loading branch information
mackjmr committed Jan 25, 2024
1 parent 9f6aeb6 commit 166dd9f
Show file tree
Hide file tree
Showing 2 changed files with 65 additions and 0 deletions.
61 changes: 61 additions & 0 deletions exporter/datadogexporter/examples/kafka.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
receivers:
otlp:
protocols:
http:
endpoint: "localhost:4318"
grpc:
endpoint: "localhost:4317"
filelog:
include_file_path: true
poll_interval: 500ms
include:
- /var/log/kafka/server.log
operators:
- type: json_parser
kafkametrics:
brokers: "${env:KAFKA_BROKER_ADDRESS}"
protocol_version: 2.0.0
scrapers:
- brokers
- topics
- consumers
jmx:
jar_path: /path/to/opentelemetry-jmx-metrics.jar
endpoint: ${env:KAFKA_BROKER_JMX_ADDRESS}
target_system: kafka,jvm
jmx/consumer:
jar_path: /path/to/opentelemetry-jmx-metrics.jar
endpoint: ${env:KAFKA_CONSUMER_JMX_ADDRESS}
target_system: kafka-consumer
jmx/producer:
jar_path: /path/to/opentelemetry-jmx-metrics.jar
endpoint: ${env:KAFKA_PRODUCER_JMX_ADDRESS}
target_system: kafka-producer

processors:
batch:
send_batch_max_size: 1000
send_batch_size: 100
timeout: 10s
attributes:
actions:
- key: ddtags
value: "source:kafka"
action: insert

exporters:
datadog:
api:
site: ${env:DD_SITE}
key: ${env:DD_API_KEY}

service:
pipelines:
metrics:
receivers: [otlp, kafkametrics, jmx, jmx/consumer, jmx/producer]
processors: [batch]
exporters: [datadog]
logs:
receivers: [otlp, filelog]
processors: [batch, attributes]
exporters: [datadog]
4 changes: 4 additions & 0 deletions exporter/datadogexporter/examples_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ func TestExamples(t *testing.T) {
files, err := os.ReadDir(folder)
require.NoError(t, err)
for _, f := range files {
if f.Name() == "kafka.yaml" {
// skip validation, as it requires jar file.
continue
}
if f.IsDir() {
continue
}
Expand Down

0 comments on commit 166dd9f

Please sign in to comment.