[exporter/datasetexporter]: Initial implementation of logs and traces

[martin-majlis-s1]

Description: This is the follow up PR after #20733. This PR is adding initial implementation for logs and traces.

Link to tracking Issue: #20660

Testing:

There are two ways how it can be tested:

1. You can run unit tests with make do-integration-tests-with-cover and make test

Documentation:

You can use following:

• Dockerfile:

FROM golang:1.19

# install builder
RUN go install go.opentelemetry.io/collector/cmd/[email protected]

# copy datasetexporter to run the latest version
COPY . ./datasetexporter/

# copy build configuration
COPY otelcol-builder.yaml .

# build it
RUN $GOPATH/bin/builder --config=otelcol-builder.yaml

ENTRYPOINT ["otelcol-dataset/otelcol-dataset"]

• otelcol-builder.yaml

# List of all components is here:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/otelcontribcol/builder-config.yaml

dist:
  name: otelcol-dataset
  description: Local OpenTelemetry Collector binary
  output_path: ./otelcol-dataset
  otelcol_version: 0.77.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datasetexporter v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.77.0
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.77.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.77.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.77.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.77.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanmetricsprocessor v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbytraceprocessor v0.77.0

extensions:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.77.0
  - gomod: go.opentelemetry.io/collector/extension/ballastextension v0.77.0
  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.77.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage v0.77.0
    import: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage

replaces:
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datasetexporter v0.77.0 => ./../datasetexporter

• docker-compose.yaml

version: "3"
services:
  otel-collector-dataset:
    build:
      context: ./
      dockerfile: examples/e2e/Dockerfile
    container_name: otel-collector-dataset
    command: ["--config=/etc/otel-config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel-config.yaml
      - /tmp:/tmp

  # Logs generator
  telemetrygen-logs-dataset:
    build:
      context: ../../cmd/telemetrygen/
    container_name: telemetrygen-logs-dataset
    command:
       - logs
       - --otlp-endpoint=otel-collector-dataset:4317
       - --otlp-insecure
       - --otlp-attributes=key1="open-telemetry-collector-LOGS"
       - --rate=10       # generate 10 records per second
       - --duration=10m  # generate it for 10 minutes
       - --workers=2     # run 2 workers
    depends_on:
      - otel-collector-dataset

  # Traces generator
  telemetrygen-traces-dataset:
    build:
      context: ../../cmd/telemetrygen/
    container_name: telemetrygen-traces-dataset
    command:
      - traces
      - --otlp-endpoint=otel-collector-dataset:4317
      - --otlp-insecure
      - --otlp-attributes=key1="open-telemetry-collector-TRACES"
      - --rate=10       # generate 10 records per second
      - --duration=10m  # generate it for 10 minutes
      - --workers=2     # run 2 workers
    depends_on:
      - otel-collector-dataset

• otel-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch/logs:
    send_batch_size: 10000
    timeout: 5s
  batch/traces:
    send_batch_size: 10000
    timeout: 16s

  memory_limiter:
    check_interval: 2s
    limit_mib: 1800
    spike_limit_mib: 500

  # https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor
  resourcedetection/system:
    detectors: [ "system" ]
    system:
      hostname_sources: [ "os" ]
  resourcedetection/docker:
    detectors: [env, docker]
    timeout: 2s
    override: false

exporters:
  dataset/logs:
    api_key: your-api-key
    dataset_url: https://app.eu.scalyr.com
    buffer:
      max_lifetime: 5s
      group_by:
        - resource.attributes.host.id
      retry_initial_interval: 5s
      retry_max_interval: 30s
      retry_max_elapsed_time: 300s
    retry_on_failure:
      max_interval: 30
      max_elapsed_time: 300
    sending_queue:
      storage: file_storage

  dataset/traces:
    api_key: your-api-key
    dataset_url: https://app.eu.scalyr.com
    buffer:
      max_lifetime: 11s
      retry_initial_interval: 5s
      retry_max_interval: 30s
      retry_max_elapsed_time: 300s
    traces:
      aggregate: true
      max_wait: 10s
    retry_on_failure:
      max_interval: 30
      max_elapsed_time: 300
    sending_queue:
      storage: file_storage
  logging:

extensions:
  health_check:
  pprof:
  zpages:
  memory_ballast:
    size_mib: 1000
  # store unprocessed data persistently
  # do not forget to create this directory
  # and make sure that its readable and writeable
  file_storage:
    directory: /tmp/


service:
  extensions: [ pprof, zpages, health_check, memory_ballast, file_storage ]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch/logs, resourcedetection/system, resourcedetection/docker]
      exporters: [dataset/logs]
    traces:
      receivers: [ otlp ]
      processors: [ batch/traces, resourcedetection/system, resourcedetection/docker ]
      exporters: [ dataset/logs ]

And then:

1. Modify otel-config.yaml
   1. Update dataset_url with the URL to your dataset server
   2. Update api_key with your API key
2. Run docker-compose up --build --abort-on-container-exit

To see that logs and traces are visible in dataset.

[atoulme]

This is going to be hard to maintain, any chance we can simplify this example?

[martin-majlis-s1]

What makes you think that? I would not expect too many changes in the configuration and other changes should not matter.

This should be some configuration from which people can remove stuff that they do not need. It's easier then the other way around.

[atoulme]

A maintainer would be able to offer a more decisive take - if you list components and we need to update them as part of the release, it will be an additional maintenance cost.

[martin-majlis-s1]

From my perspective the most annoying thing is that you copy-paste some example from documentation and it does not work. Therefore I have introduced this test.

Datadog is also testing their examples - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples_test.go.

But their example is very minimal. My understanding is, that you would also prefer as minimalistic example as possible.

[mx-psi]

You don't need the builder configuration file to test the example config (the Datadog exporter is only testing this on unit tests, not by running a Collector). What benefit does the builder config file bring to the example?

[atoulme]

LGTM. That said, most of the logic here is now how you map otel data to what dataset expects, and I can't verify that part, to be clear.

[mx-psi]

Looks like you have to run make crosslink

[mx-psi]

This broke CI on main (see #22039)