[receiver/awscontainerinsight] Gather instance metadata parameters from Kubernetes API when EC2 instance metadata is not accessible

[oleksandr-san]

Component(s)

receiver/awscontainerinsight

Is your feature request related to a problem? Please describe.

AWS Container Insights receiver requires access to the EC2 instance metadata endpoint to get instance ID, instance type, and instance IP address.

AWS recommends to block access to EC2 instance metadata when using IRSA.
When the access is restricted, aws-otel-collector pods are refusing to send metrics to CloudWatch:

2024-02-29T15:15:03.248Z	error	host/ec2metadata.go:69	Failed to get ec2 metadata	{"kind": "receiver", "name": "awscontainerinsightreceiver", "data_type": "metrics", "error": "EC2MetadataRequestError: failed to get EC2 instance identity document\ncaused by: EC2MetadataError: failed to make EC2Metadata request\n\n\tstatus code: 401, request id: "}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver/internal/host.(*ec2Metadata).refresh
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/internal/host/ec2metadata.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver/internal/host.RefreshUntil
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/internal/host/utils.go:51
I0229 15:15:06.316423       1 leaderelection.go:260] successfully acquired lease aws-otel-eks/otel-container-insight-clusterleader

2024-02-29T15:14:57.157Z	warn	cadvisor/cadvisor_linux.go:329	Failed to detect cluster name. Drop all metrics	{"kind": "receiver", "name": "awscontainerinsightreceiver", "data_type": "metrics"}
2024-02-29T15:15:00.122Z	info	host/ec2metadata.go:65	Fetch instance id and type from ec2 metadata	{"kind": "receiver", "name": "awscontainerinsightreceiver", "data_type": "metrics"}

Describe the solution you'd like

It is possible to fetch the needed metadata from Kubernetes API by making a single call to describe the node, where the pod from aws-otel-collector deamonset is running (the node name is already provided via an environment variable):

1. instance ID can be extracted from node spec.providerID
2. instance type can be extracted from node label node.kubernetes.io/instance-type
3. instance IP can be extracted from node internal address

Describe alternatives you've considered

The workaround is to increase the network hop limit for IMDS endpoint to 2, which allows IMDS access for all pods in the cluster.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/awscontainerinsight: @Aneurysm9 @pxaws

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/awscontainerinsight: @Aneurysm9 @pxaws

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.