[exporter/prometheus] does not show metrics from otlp receiver

[sterziev88]

Component(s)

No response

What happened?

Description

I use otlp receiver to collect metrics from my applications and I want to use prometheus exporter in order to be able to see them in prometheus but I don't see them.

Steps to Reproduce

delpoy otlp with helm chart 0.85.0

Expected Result

To be able to see metrics in Prometheus

Actual Result

I am not able to see metrics as system_cpu_usage, system_load_average, etc..

Collector version

0.85.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
        otlp:
          protocols:
            grpc:
              endpoint: x.x.x.x:4317
            http:
              endpoint: x.x.x.x:4318
exporters:
        logging: 
          verbosity: detailed
        prometheus:
          endpoint: "0.0.0.0:8889"
          send_timestamps: true
          resource_to_telemetry_conversion:
            enabled: true

service:
          metrics:
            exporters:
              - prometheus
              - logging

Log output

2024-04-19T06:36:44.203Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 7, "data points": 11}
2024-04-19T06:36:44.203Z info ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.23.1
Resource attributes:
-> container.id: Str(e9673229043eab10ccc347a7bc6de2741b78162235dd864efc8f20e1934283cd)
-> host.arch: Str(amd64)
-> host.name: Str(scheduled-services-deployment-5675c8ccc4-qcmql)
-> os.description: Str(Linux 5.10.210-201.852.amzn2.x86_64)
-> os.type: Str(linux)
-> process.command_args: Slice(["/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java","org.springframework.boot.loader.launch.JarLauncher"])
-> process.executable.path: Str(/layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java)
-> process.pid: Int(1)
-> process.runtime.description: Str(BellSoft OpenJDK 64-Bit Server VM 17.0.7+7-LTS)
-> process.runtime.name: Str(OpenJDK Runtime Environment)
-> process.runtime.version: Str(17.0.7+7-LTS)
-> service.name: Str(billing-scheduler)
-> telemetry.distro.name: Str(opentelemetry-spring-boot-starter)
-> telemetry.distro.version: Str(2.2.0-alpha)
-> telemetry.sdk.language: Str(java)
-> telemetry.sdk.name: Str(opentelemetry)
-> telemetry.sdk.version: Str(1.36.0)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope io.opentelemetry.sdk.logs
Metric #0
Descriptor:
-> Name: processedLogs
-> Description: The number of logs processed by the BatchLogRecordProcessor. [dropped=true if they were dropped due to high throughput]
-> Unit: 1
-> DataType: Sum
-> IsMonotonic: true
-> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
-> dropped: Bool(false)
-> processorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 44
Metric #1
Descriptor:
-> Name: queueSize
-> Description: The number of items queued
-> Unit: 1
-> DataType: Gauge
NumberDataPoints #0
Data point attributes:
-> processorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL:
InstrumentationScope io.opentelemetry.exporters.otlp-grpc
Metric #0
Descriptor:
-> Name: otlp.exporter.exported
-> Description:
-> Unit:
-> DataType: Sum
-> IsMonotonic: true
-> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
-> success: Bool(false)
-> type: Str(log)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 11
NumberDataPoints #1
Data point attributes:
-> success: Bool(false)
-> type: Str(span)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 40
NumberDataPoints #2
Data point attributes:
-> success: Bool(true)
-> type: Str(log)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 44
NumberDataPoints #3
Data point attributes:
-> success: Bool(true)
-> type: Str(span)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 395003
Metric #1
Descriptor:
-> Name: otlp.exporter.seen
-> Description:
-> Unit:
-> DataType: Sum
-> IsMonotonic: true
-> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
-> type: Str(log)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 55
NumberDataPoints #1
Data point attributes:
-> type: Str(span)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 395043
ScopeMetrics #2
ScopeMetrics SchemaURL:
InstrumentationScope io.opentelemetry.sdk.trace
Metric #0
Descriptor:
-> Name: processedSpans
-> Description: The number of spans processed by the BatchSpanProcessor. [dropped=true if they were dropped due to high throughput]
-> Unit: 1
-> DataType: Sum
-> IsMonotonic: true
-> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
-> dropped: Bool(false)
-> processorType: Str(BatchSpanProcessor)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 395003
Metric #1
Descriptor:
-> Name: queueSize
-> Description: The number of items queued
-> Unit: 1
-> DataType: Gauge
NumberDataPoints #0
Data point attributes:
-> processorType: Str(BatchSpanProcessor)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Value: 0
ScopeMetrics #3
ScopeMetrics SchemaURL:
InstrumentationScope io.opentelemetry.spring-webmvc-6.0 2.2.0-alpha
Metric #0
Descriptor:
-> Name: http.server.request.duration
-> Description: Duration of HTTP server requests.
-> Unit: s
-> DataType: Histogram
-> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
-> http.request.method: Str(GET)
-> http.response.status_code: Int(302)
-> network.protocol.version: Str(1.1)
-> url.scheme: Str(http)
StartTimestamp: 2024-04-16 13:37:44.07905719 +0000 UTC
Timestamp: 2024-04-19 06:36:44.083167789 +0000 UTC
Count: 12309
Sum: 6.423931
Min: 0.000430
Max: 0.005979
ExplicitBounds #0: 0.005000
ExplicitBounds #1: 0.010000
ExplicitBounds #2: 0.025000
ExplicitBounds #3: 0.050000
ExplicitBounds #4: 0.075000
ExplicitBounds #5: 0.100000
ExplicitBounds #6: 0.250000
ExplicitBounds #7: 0.500000
ExplicitBounds #8: 0.750000
ExplicitBounds #9: 1.000000
ExplicitBounds #10: 2.500000
ExplicitBounds #11: 5.000000
ExplicitBounds #12: 7.500000
ExplicitBounds #13: 10.000000
Buckets #0, Count: 12308
Buckets #1, Count: 1
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0

Additional context

That is my config in Prometheus:
- job_name: 'sample-job-2'
scrape_interval: 10s
static_configs:
- targets: ['opentelemetry-dev-in-cluster.opentelemetry.svc.cluster.local:8889']

I am able to see that that target is successfully added in Prometheus, according the logs I see that opentelemtry collector successfully receive metrics from my application via otlp protocol but I don't see metrics in my prometheus.

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[odev-swe]

@sterziev88 same situation here

[slashrsm]

I am experiencing the same problem. Was anyone able to figure this out?