[connector/spanmetrics] How to use spanmetrics together with grafana's service graph

[cmergenthaler]

Component(s)

connector/spanmetrics

Is your feature request related to a problem? Please describe.

From what I see, the changes introduced in #18760 made the spanmetrics connector incompatible with grafana's service graph.
As stated in grafana tempo docs:

Grafana uses the traces_spanmetrics_calls_total metric to display the name, rate, and error rate columns, and traces_spanmetrics_latency_bucket to display the duration column. These metrics must exist in your Prometheus data source.

Describe the solution you'd like

Allow customizing the name of the duration metric so that it can be used by grafana.

Describe alternatives you've considered

Would be happy to know about other solutions

Additional context

No response

[github-actions]

Pinging code owners:

• connector/spanmetrics: @albertteoh

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @cmergenthaler, can you share the config you're using for this functionality? There are a couple of options, but it depends on how the collector is configured.

[cmergenthaler]

Hey @crobert-1, appreciate your help!
My config looks the following:

receivers:
  otlp:
    protocols:
      grpc:
      http:

connectors:
  servicegraph:
    latency_histogram_buckets: [0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8]
  spanmetrics:
    namespace: traces.spanmetrics
    histogram:
      explicit:
        buckets: [2ms, 4ms, 8ms, 16ms, 32ms, 64ms, 128ms, 256ms, 512ms, 1.02s, 2.05s, 4.10s]

processors:
  batch:

exporters:
  prometheusremotewrite:
    endpoint: my-prom
    target_info:
      enabled: false

service:
  pipelines:
    metrics:
      receivers: [otlp, servicegraph, spanmetrics]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [servicegraph, spanmetrics]

In prometheus I have the following spanmetrics available:

• traces_spanmetrics_calls
• traces_spanmetrics_duration_bucket
• traces_spanmetrics_duration_count
• traces_spanmetrics_duration_sum

As described, Grafana expects slightly different metrics to be present in prometheus. It also seems that the prometheusremotewrite does not add _total suffix to the calls metric - even though add_metric_suffixes is set to true.
I am running otel-collector-contrib 0.84.0

[crobert-1]

Perfect, the extra information helps!

To get this working you need to enable the add_metric_suffixes config option, as well as the pkg.translator.prometheus.NormalizeName feature gate.

PR #26489 was also just merged which will enable this feature gate by default in the next release of the collector. #26488 is the tracking issue for this effort. Issue #26488 is the tracking issue for this change.

Until release v0.85.0 you can enable the feature gate via CLI when running the collector. Once v0.85.0 is released you won't have to do anything for the feature gate anymore.

[cmergenthaler]

Perfect, the extra information helps!

To get this working you need to enable the add_metric_suffixes config option, as well as the pkg.translator.prometheus.NormalizeName feature gate.

PR #26489 was also just merged which will enable this feature gate by default in the next release of the collector. #26488 is the tracking issue for this effort. Issue #26488 is the tracking issue for this change.

Until release v0.85.0 you can enable the feature gate via CLI when running the collector. Once v0.85.0 is released you won't have to do anything for the feature gate anymore.

Thanks for the help! That indeed fixes the calls metric. Unfortunately, the duration metrics is still not the one grafana expects, it should be called traces_spanmetrics_latency_bucket. Do I need to use the metricstransformprocessor for that or is that something that should be supported in the connector?

[crobert-1]

I don't have a good answer here, so I'll have to defer to the code owners.

[github-actions]

Pinging code owners for exporter/prometheusremotewrite: @Aneurysm9 @rapphil. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tesharp]

Have you looked at this doc https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/architecture/opentelemetry/head-sampling/ ? Describes the otel setup for head sampling with metricstransformprocessor

[cmergenthaler]

Have you looked at this doc https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/architecture/opentelemetry/head-sampling/ ? Describes the otel setup for head sampling with metricstransformprocessor

Thanks for the useful example! I think this solves my initial request, though the serice graph table is still not shown. But I think this issue can be closed for now