Sampling race condition causes initial metric to be reported with a 0 value

[sergiuiacob1]

Component(s)

exporter/datadog

What happened?

Description

I've set up a counter metric to track "events". The counter is emitted using Prometheus Push Gateway and scraped by an Open Telemetry agent. I've set OTel to export data to Datadog. I've set it to report only the delta for the counters, i.e.

      datadog:
        metrics:
          sums:
            cumulative_monotonic_mode: to_delta
          histogram:
            counters: true
        api:
          site: datadoghq.com
          key: ${env:DATADOG_API_KEY}

I can see OTel scraping the first 2 values for my metric with the new set of label values

OTEL reports first datapoint:
NumberDataPoints #2
Data point attributes:
     -> app: Str(shoreline)
     -> customer_id: Str(cust0)
     -> exported_instance: Str(cust0-proxy-backend-0)
     -> exported_job: Str(ops_backend)
     -> exported_namespace: Str(cust0-backend)
     -> name: Str(a2)
     -> namespace: Str(cust0-backend)
     -> operation: Str(create)
     -> service: Str(cust0-proxy-backend-0)
     -> status: Str(succeeded)
     -> type: Str(action)
StartTimestamp: 2024-03-15 16:35:05.515 +0000 UTC
Timestamp: 2024-03-15 16:35:05.515 +0000 UTC
Value: 1.000000

OTEL reports second datapoint:
NumberDataPoints #2
Data point attributes:
     -> app: Str(shoreline)
     -> customer_id: Str(cust0)
     -> exported_instance: Str(cust0-proxy-backend-0)
     -> exported_job: Str(ops_backend)
     -> exported_namespace: Str(cust0-backend)
     -> name: Str(a2)
     -> namespace: Str(cust0-backend)
     -> operation: Str(create)
     -> service: Str(cust0-proxy-backend-0)
     -> status: Str(succeeded)
     -> type: Str(action)
StartTimestamp: 2024-03-15 16:35:05.515 +0000 UTC
Timestamp: 2024-03-15 16:36:05.515 +0000 UTC
Value: 1.000000

Datadog receives new time series at 18:36:05. Because the value difference between the 2 timestamps is 0, the initial metric value (initial delta) is 0 as well.
The previous Datadog metric timestamp was at 18:35:05.

Steps to Reproduce

1. Create a counter
2. Set up Datadog exporter
3. Emit datapoints through OTel with new label values
4. Eventually, there will be a "miss" and the delta for a new label set will be 0

Expected Result

OTel should always report the delta for an initial set of label values for a counter to be that metric's value – in my case, 1.

Specifically, in my screenshot in Datadog, I should have seen the initial delta, which is 1 for my metric.

Actual Result

Sometimes the initial counter value is the true initial delta (the first value counted), sometimes it's 0.

Collector version

0.96.0

Environment information

Environment

OS: Ubuntu 20.04
OTel version: 0.96.0. Also reproduced on 0.71.0

OpenTelemetry Collector configuration

datadog:
        metrics:
          sums:
            cumulative_monotonic_mode: to_delta
          histogram:
            counters: true
        api:
          site: datadoghq.com
          key: ${env:DATADOG_API_KEY}

### Log output

_No response_

### Additional context

_No response_

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sergiuiacob1]

Easily recreated this and not very sure it's actually a race condition. I took a look again at the timestamps:

First sample at 2024-03-19 14:12:07.844 +0000 UTC

NumberDataPoints #1
Data point attributes:
     -> app: Str(shoreline)
     -> customer_id: Str(cust0)
     -> exported_instance: Str(cust0-proxy-backend-0)
     -> exported_job: Str(ops_backend)
     -> exported_namespace: Str(cust0-backend)
     -> name: Str(sergiu_test_2)
     -> namespace: Str(cust0-backend)
     -> operation: Str(create)
     -> service: Str(cust0-proxy-backend-0)
     -> status: Str(succeeded)
     -> type: Str(action)
StartTimestamp: 2024-03-19 14:12:07.844 +0000 UTC
Timestamp: 2024-03-19 14:12:07.844 +0000 UTC
Value: 1.000000

Second sample at 2024-03-19 14:13:07.844 +0000 UTC:

NumberDataPoints #1
Data point attributes:
     -> app: Str(shoreline)
     -> customer_id: Str(cust0)
     -> exported_instance: Str(cust0-proxy-backend-0)
     -> exported_job: Str(ops_backend)
     -> exported_namespace: Str(cust0-backend)
     -> name: Str(sergiu_test_2)
     -> namespace: Str(cust0-backend)
     -> operation: Str(create)
     -> service: Str(cust0-proxy-backend-0)
     -> status: Str(succeeded)
     -> type: Str(action)
StartTimestamp: 2024-03-19 14:12:07.844 +0000 UTC
Timestamp: 2024-03-19 14:13:07.844 +0000 UTC
Value: 1.000000

Datadog receives first datapoint at 14:13:05 UTC:

In the case above, the first datapoint at 14:13:05 UTC should have had the value 1.

[sergiuiacob1]

And here is an example where things worked as expected:

First sample reported at 2024-03-19 14:11:07.844 +0000 UTC

NumberDataPoints #0
Data point attributes:
     -> app: Str(shoreline)
     -> customer_id: Str(cust0)
     -> exported_instance: Str(cust0-proxy-backend-0)
     -> exported_job: Str(ops_backend)
     -> exported_namespace: Str(cust0-backend)
     -> name: Str(sergiu_test)
     -> namespace: Str(cust0-backend)
     -> operation: Str(create)
     -> service: Str(cust0-proxy-backend-0)
     -> status: Str(succeeded)
     -> type: Str(action)
StartTimestamp: 2024-03-19 14:11:07.844 +0000 UTC
Timestamp: 2024-03-19 14:11:07.844 +0000 UTC
Value: 1.000000

Second sample reported at 2024-03-19 14:12:07.844 +0000 UTC:

NumberDataPoints #0
Data point attributes:
     -> app: Str(shoreline)
     -> customer_id: Str(cust0)
     -> exported_instance: Str(cust0-proxy-backend-0)
     -> exported_job: Str(ops_backend)
     -> exported_namespace: Str(cust0-backend)
     -> name: Str(sergiu_test)
     -> namespace: Str(cust0-backend)
     -> operation: Str(create)
     -> service: Str(cust0-proxy-backend-0)
     -> status: Str(succeeded)
     -> type: Str(action)
StartTimestamp: 2024-03-19 14:11:07.844 +0000 UTC
Timestamp: 2024-03-19 14:12:07.844 +0000 UTC
Value: 1.000000

Datadog received the first metric datapoint correctly, with a value of 1:

[mx-psi]

Hi @sergiuiacob1, can you try using the cumulative to delta processor instead of the metrics::sums::cumulative_monotonic_mode option?

This is what we currently recommend for these setups, and I want to verify if the issue persists when using this component instead of the fallback logic in the exporter.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.