[prometheusexporter] / [prometheusremotewriteexporter] Support for exemplars

[gillg]

Is your feature request related to a problem? Please describe.
Exemplars in prometheus are a very new feature but also very basic to expose and are completly consistent with the goal of OpenTelemetry.
They are kind a ephemaral metadata attached to a metric point in OpenMetrics format (https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars)

Describe the solution you'd like
I'm not 100% of the strategy here... but I have some ideas.
Create a config like :

exporters:
   prometheus[...]:
     exemplars_from:
        - attributes list...

It makes no sense to create exemplars from resources or low cardinality labels, but it's definitely useful for trace or span IDs.
I'm not sure how to expose them because I didn't start to work with them for now...

# HELP my_otel_metric Duration of something
# TYPE my_otel_metric histogram
my_otel_metric{net_host_name="localhost",another_low_cardinality_label="something",le="0"} 0 1631180707073 # {trace_id="oHg5SJYRHA0"} 9.8 1631180708123

Format explainations :

Exemplar starts just after "#" sign
<metric name>{<metrics labels stored in TSDB, must not be high cardinality>} <value based on OTEL aggregators> <timestamp where value has been exported>  # {<specific labels for a specific data mesure, can be high cardinality>} <direct mesured value, not aggregated> <timestamp where value has be mesured>

If I understand the concept we can't capture all the traces ids... so we have to one sample randomly or chose a pertinent one when we export a value. Any suggestion ?

Additional context
Higly needed to allow traces and metrics correlation on Grafana or any vizualization tool based on prometheus.

[secat]

There is this issue #5177 which proposes to update the spanmetricsprocessor in order to inject the TraceID as an exemplar in the generated metrics by the span metrics processor.

[gillg]

Oh ! Exemplars are something common in otel metric model ? There is no tricks needed based on attributes ? In this case the emitter (if talks otel protocol) can set exemplars and send it in otlp ?

[secat]

Oh ! Exemplars are something common in otel metric model ? There is no tricks needed based on attributes ? In this case the emitter (if talks otel protocol) can set exemplars and send it in otlp ?

From what I understand from the Otel Protocol, Yes

[Frefreak]

I'm also interested in using prometheusexporter as a way to generate metrics with exemplar data from pdata.Metric. After a brief look of the current implementation, it appears prometheusexporter generates metric with NewConstMetric, which currently has no way to append exemplar to it (populateMetric in client_golang has a parameter to specify the exemplar, and will use it if it is counter type, but constMetric just pass nil to it).

I think maybe the first step is something like making constMetric expose a way to specify the exemplar it uses? However I'm not familiar with prometheus's code base so what I said above might not be correct though.

[dashpole]

For the prometheusexporter, the way to convert OpenTelemetry exemplars to prometheus is now described in the specification:

[gillg]

Great! Thanks @dashpole for all your recents clarifications in the doc !

---

Out of current scope
I love especially the concept of target metric I was starting to define internal guidelines for devs to have two metrics "build_infos" with static labels like all resources attributes, and "runtime_infos" with some standard attributes as labels like some application context specific metadata.
Do you know if your official guidelines are super experimental or in a good shape to be adopted and implemented on collector ? Else have you a set of processors configs working to cover the guidelines?

[dashpole]

#8265 tracks using target_info in the collector

[songjiayang]

hi everyone, I use prometheusreceiver to scape a prometheus exporter with exempalrs metrics.

when I use prometheusremotewriter exporter to push them to mimir (tsdb backend), I find the exemplars data had been dropped.

The Prometheus remote wirte has SendExemplars filed to control exemplars send or not, but prometheusremotewriter exporter does't support this configuration yet.

I don't know if there is any other way to solve this problem，thanks.

[dashpole]

What version of the collector are you using? Unfortunately, exemplar support in the prometheus receiver was just reverted, so that will need to be reintroduced before that scenario will work.

[songjiayang]

@dashpole sorry replay late, I used container with tag 0.59.0, confirm not support yet.

[songjiayang]

hi @dashpole

I do a test with OTel Collector ( receiver is prometheus, exporter is prometheusremotewrite) , but I found the Collector don't support exemplars yet.

here is my test code https://github.com/grafanafans/prometheus-exemplar/blob/remote_prometheus/config/otel-collector.yaml#L7

Why I use prometheus exporter rather than OTLP for my application, because I find I can't add an examplar to a meter with Go SDK.

[vincentfree]

that will need to be reintroduced before that scenario will wo

Any idea when this support gets reintroduced? Also In which version does this get removed? I'd like to know if my application setup does work with exemplars.

[dashpole]

This was fixed by #14132. It will be released next release (v0.63.0)