Bug with Elasticsearch receiver

[FavourDaniel]

Component(s)

Elasticsearch Receiver

What happened?

Description

I tried running the Elasticsearch receiver with the OpenTelemetry Collector but it kept returning errors.

Steps to Reproduce

I followed the exact steps and used the sample for the Elasticsearch receiver in the receiver repo.

Expected Result

To successfully scrape metrics from my running Elasticsearch cluster

Actual Result

Users-MacBook-Pro-2:otelcol-contrib deimos$ ./otelcol-contrib --config ./config.yml
2023-12-17T22:54:15.523+0100    info    [email protected]/telemetry.go:86 Setting up own telemetry...
2023-12-17T22:54:15.525+0100    info    [email protected]/telemetry.go:203        Serving Prometheus metrics      {"address": "localhost:8888", "level": "Basic"}
2023-12-17T22:54:15.527+0100    info    [email protected]/service.go:145  Starting otelcol-contrib...     {"Version": "0.91.0", "NumCPU": 12}
2023-12-17T22:54:15.527+0100    info    extensions/extensions.go:34     Starting extensions...
2023-12-17T22:54:15.536+0100    info    [email protected]/service.go:171  Everything is ready. Begin running and processing data.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4a4b641]

goroutine 63 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver.(*elasticsearchScraper).scrapeNodeMetrics(0xc00284c880, {0x9734dc0, 0xc001f13830}, 0xc002b87cd8?, 0xc002bf7cd8)
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/scraper.go:138 +0x6c1
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver.(*elasticsearchScraper).scrape(0xc00284c880, {0x9734dc0, 0xc001f13830})
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/scraper.go:76 +0x105
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape(...)
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport(0xc002122b40)
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:197 +0x16b
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1()
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:172 +0xd1
created by go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping in goroutine 1
        go.opentelemetry.io/collector/[email protected]

I also used a later version, 0.89 but it was the same thing.

Collector version

v0.91 and v0.89

Environment information

Environment

OS: mac

OpenTelemetry Collector configuration

receivers:
  elasticsearch:
    metrics:
      elasticsearch.node.fs.disk.available:
        enabled: false
    nodes: ["Users-MacBook-Pro-2.local"]
    skip_cluster_metrics: true
    indices: [".geoip_databases"]
    endpoint: http:https://localhost:9200
    username: xxxxxx
    password: xxxxxx
    collection_interval: 10s

processors:
  batch:
    send_batch_size: 1000
    timeout: 10s

exporters:
  otlp:
    endpoint: "ingest.{region}.signoz.cloud:443"
    tls:
      insecure: false
    timeout: 20s
    headers:
      "signoz-access-token": "xxxxx"

service:
  telemetry:
    metrics:
      address: localhost:8888
  pipelines:
    metrics:
      receivers: [elasticsearch]
      processors: [batch]
      exporters: [otlp]

Log output

Users-MacBook-Pro-2:otelcol-contrib deimos$ ./otelcol-contrib --config ./config.yml
2023-12-17T22:54:15.523+0100    info    [email protected]/telemetry.go:86 Setting up own telemetry...
2023-12-17T22:54:15.525+0100    info    [email protected]/telemetry.go:203        Serving Prometheus metrics      {"address": "localhost:8888", "level": "Basic"}
2023-12-17T22:54:15.527+0100    info    [email protected]/service.go:145  Starting otelcol-contrib...     {"Version": "0.91.0", "NumCPU": 12}
2023-12-17T22:54:15.527+0100    info    extensions/extensions.go:34     Starting extensions...
2023-12-17T22:54:15.536+0100    info    [email protected]/service.go:171  Everything is ready. Begin running and processing data.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4a4b641]

goroutine 63 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver.(*elasticsearchScraper).scrapeNodeMetrics(0xc00284c880, {0x9734dc0, 0xc001f13830}, 0xc002b87cd8?, 0xc002bf7cd8)
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/scraper.go:138 +0x6c1
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver.(*elasticsearchScraper).scrape(0xc00284c880, {0x9734dc0, 0xc001f13830})
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/scraper.go:76 +0x105
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape(...)
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport(0xc002122b40)
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:197 +0x16b
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1()
        go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:172 +0xd1
created by go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping in goroutine 1
        go.opentelemetry.io/collector/[email protected]

Additional context

Before changing the node section to this:
nodes: ["Users-MacBook-Pro-2.local"]

I used what was already in the repo.

[github-actions]

Pinging code owners for receiver/elasticsearch: @djaglowski @BinaryFissionGames. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Thanks for reporting this @FavourDaniel.

@BinaryFissionGames, this panic appears to be due to drilling through structs without checking for nil values. e.g. r.mb.RecordElasticsearchNodeDiskIoReadDataPoint(now, info.FS.IOStats.Total.ReadBytes)

I believe we previously assumed that these structs would always be populated, but it looks like IOStats is a pointer and therefore may be nil. Looks like it's the only pointer in the return structs, so I'll just add a nil check around each access.

[BinaryFissionGames]

Thanks for looking into this @djaglowski - IOStats is a special case, since it's only available on Linux clusters. Definitely warrants a nil check for that case.