[receiver/awscloudwatchmetricsreceiver] Add new receiver (open-teleme…

…try#19429)

* Feat(receiver): Initial commit of receiver

Introduces the AWS CloudWatch metrics receiver based on issue open-telemetry#15667

* Chore(docs): Add changelog entry for receiver

Add chloggen entry for new metric receiver component

* fix: Add link to changelog entry

* chore: update github issue templates

* Update .github/ISSUE_TEMPLATE/feature_request.yaml

Co-authored-by: Antoine Toulme <[email protected]>

* Chore(deps): Bump receiver deps

* Fix(errors): Improve tooManyMetrics error message

As per GH comment, include the number of metrics you can define in the
error message.

Lowercase debug messages for start and shutdown receiver method

* Chore: run make goporto

* Chore: make gendependabot

* Chore: add module to `versions.yaml`

* chore: dp templates

* Chore: Update based on PR comments

* Chore: update tests based on PR comments

* fix(spelling): Fix function spelling

* Update .github/CODEOWNERS

Co-authored-by: Antoine Toulme <[email protected]>

* Chore: make gendependabot

* Chore: bump deps

* Fix(docs): update aws cli link

* Chore: make gotidy

* Update receiver/awscloudwatchmetricsreceiver/config_test.go

Co-authored-by: Juraci Paixão Kröhling <[email protected]>

* Update receiver/awscloudwatchmetricsreceiver/config_test.go

Co-authored-by: Juraci Paixão Kröhling <[email protected]>

* Update receiver/awscloudwatchmetricsreceiver/config_test.go

* chore: test formatting

* docs: delete pricing sentence

* Chore: update tests with proper examples

* chore: make gendependabot

* chore: make addlicense

---------

Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Juraci Paixão Kröhling <[email protected]>

.chloggen/add-cloudwatchmetrics-receiver.yaml

@@ -0,0 +1,16 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: new_component
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: awscloudwatchmetricsreceiver
+
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Added [AWS CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html) receiver using GetMetricData API call
+
+# One or more tracking issues related to the change
+issues: [15667]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:

.github/CODEOWNERS

@@ -163,6 +163,7 @@ receiver/aerospikereceiver/ @open-telemetry/collect
 receiver/apachereceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
 receiver/apachesparkreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski @Caleb-Hurshman @mrsillydog
 receiver/awscloudwatchreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski @schmikei
+receiver/awscloudwatchmetricsreceiver/ @open-telemetry/collector-contrib-approvers @jpkrohling @kovrus
 receiver/awscontainerinsightreceiver/ @open-telemetry/collector-contrib-approvers @Aneurysm9 @pxaws
 receiver/awsecscontainermetricsreceiver/ @open-telemetry/collector-contrib-approvers @Aneurysm9
 receiver/awsfirehosereceiver/ @open-telemetry/collector-contrib-approvers @Aneurysm9

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -151,6 +151,7 @@ body:
  - receiver/apache
  - receiver/apachespark
  - receiver/awscloudwatch
+ - receiver/awscloudwatchmetrics
  - receiver/awscontainerinsight
  - receiver/awsecscontainermetrics
  - receiver/awsfirehose

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -145,6 +145,7 @@ body:
  - receiver/apache
  - receiver/apachespark
  - receiver/awscloudwatch
+ - receiver/awscloudwatchmetrics
  - receiver/awscontainerinsight
  - receiver/awsecscontainermetrics
  - receiver/awsfirehose

.github/ISSUE_TEMPLATE/other.yaml

@@ -145,6 +145,7 @@ body:
  - receiver/apache
  - receiver/apachespark
  - receiver/awscloudwatch
+ - receiver/awscloudwatchmetrics
  - receiver/awscontainerinsight
  - receiver/awsecscontainermetrics
  - receiver/awsfirehose

.github/dependabot.yml

@@ -702,6 +702,11 @@ updates:
  schedule:
  interval: "weekly"
  day: "wednesday"
+ - package-ecosystem: "gomod"
+ directory: "/receiver/awscloudwatchmetricsreceiver"
+ schedule:
+ interval: "weekly"
+ day: "wednesday"
  - package-ecosystem: "gomod"
  directory: "/receiver/awscloudwatchreceiver"
  schedule:
@@ -1097,8 +1102,3 @@ updates:
  schedule:
  interval: "weekly"
  day: "wednesday"
- - package-ecosystem: "gomod"
- directory: "/receiver/vcenterreceiver"
- schedule:
- interval: "weekly"
- day: "wednesday"

receiver/awscloudwatchmetricsreceiver/Makefile

@@ -0,0 +1 @@
+include ../../Makefile.Common

receiver/awscloudwatchmetricsreceiver/README.md

@@ -0,0 +1,144 @@
+# CloudWatch Metrics Receiver
+
+| Status | |
+| ------------------------ | ------------- |
+| Stability | [development] |
+| Supported pipeline types | metrics |
+| Distributions | [contrib] |
+
+Receives Cloudwatch metrics from [AWS Cloudwatch](https://aws.amazon.com/cloudwatch/) via the [AWS SDK for Cloudwatch Logs](https://docs.aws.amazon.com/sdk-for-go/api/service/cloudwatchlogs/)
+
+## Getting Started
+
+This receiver uses the [AWS SDK](https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/) as mode of authentication, which includes [Profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) and [IMDS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) authentication for EC2 instances.
+
+## Configuration
+
+### Top Level Parameters
+
+| Parameter | Notes | type | Description |
+| --------------- | ---------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `region` | *required* | string | The AWS recognized region string |
+| `profile` | *optional* | string | The AWS profile used to authenticate, if none is specified the default is chosen from the list of profiles |
+| `IMDSEndpoint` | *optional* | string | The IMDS endpoint to authenticate to AWS | 
+| `poll_interval` | `default=1m` | duration | The duration waiting in between requests | 
+| `metrics` | *optional* | `Metrics` | Configuration for metrics ingestion of this receiver |
+
+### Metrics Parameters
+
+
+| Parameter | Notes | type | Description |
+| ------------------------ | ------------ | ---------------------- | ------------------------------------------------------------------------------------------ |
+| `named` | *required* | `See Named Parameters` | Configuration for Named Metrics, by default no metrics are collected |
+
+
+### Named Parameters
+
+| Parameter | Notes | type | Description |
+| ------------------------ | ------------ | ---------------------- | ------------------------------------------------------------------------------------------ |
+| `namespace` | *required* | `string` | AWS Metric namespace, all AWS namespaces are prefixed with `AWS`, eg: `AWS/EC2` for EC2 metrics |
+| `metric_name` | *required* | `string` | AWS metric name |
+| `period` | `default=5m` | duration | Aggregation period |
+| `aws_aggregation` | `default=sum` | string | type of AWS aggregation, eg: sum, min, max, average |
+| `dimensions` | *optional* | `see Dimensions Parameters` | Configuration for metric dimensions |
+
+### Dimension Parameters
+
+| Parameter | Notes | type | Description |
+| ------------------------ | ------------ | ---------------------- | ------------------------------------------------------------------------------------------ |
+| `name` | *required* | `string` | Dimensions name, can't start with a colon |
+| `value` | *required* | `string` | Dimension value |
+
+
+#### Named Example
+
+```yaml
+awscloudwatchmetrics:
+ region: us-east-1
+ poll_interval: 1m
+ metrics:
+ named:
+ - namespace: "AWS/EC2"
+ metric_name: "CPUUtilization"
+ period: "5m"
+ aws_aggregation: "Sum"
+ dimensions:
+ - Name: "InstanceId"
+ Value: "i-1234567890abcdef0"
+ - namespace: "AWS/S3"
+ metric_name: "BucketSizeBytes"
+ period: "5m"
+ aws_aggregation: "p99"
+ dimensions:
+ - Name: "BucketName"
+ Value: "OpenTelemetry"
+ - Name: "StorageType"
+ Value: "StandardStorage"
+```
+
+## Sample Configs
+
+```yaml
+receivers:
+ awscloudwatchmetrics:
+ region: eu-west-1
+ poll_interval: 10m
+ metrics:
+ named:
+ - namespace: "AWS/EC2"
+ metric_name: "CPUUtilization"
+ period: "5m"
+ aws_aggregation: "Sum"
+ dimensions:
+ - Name: "InstanceId"
+ Value: "i-035e091c31292427a"
+
+processors:
+
+exporters:
+ logging:
+ verbosity: detailed
+
+service:
+ pipelines:
+ metrics:
+ receivers: [awscloudwatchmetrics]
+ processors: []
+ exporters: [logging]
+```
+
+## AWS Costs
+
+This receiver uses the [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) API call, this call is *not* in the AWS free tier. Please refer to [Amazon's pricing](https://aws.amazon.com/cloudwatch/pricing/) for further information about expected costs.
+
+
+[alpha]:https://github.com/open-telemetry/opentelemetry-collector#alpha
+[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
+[Issue]:https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/15667
+
+## Troubleshooting / Debugging
+
+## My metrics are intermittent / not receing any metrics
+
+Try a bigger `poll_interval`. CloudWatch returns no data if the period of the metric, by default for AWS supplied metrics, it's 300 seconds (5 minutes). Try out a period of 600 seconds and a poll interval of 600 seconds.
+
+## Help, I'm getting IAM permission denied
+
+Make sure your IAM role/user has the required permissions:
+
+```yaml
+"cloudwatch:GetMetricData",
+"cloudwatch:GetMetricStatistics",
+"cloudwatch:ListMetrics"
+```
+
+The following IAM permissions are required for transit gateways to work:
+
+```
+"ec2:DescribeTags",
+"ec2:DescribeInstances",
+"ec2:DescribeRegions",
+"ec2:DescribeTransitGateway*"
+```
+
+

receiver/awscloudwatchmetricsreceiver/config.go

@@ -0,0 +1,162 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package awscloudwatchmetricsreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchmetricsreceiver"
+
+import (
+ "errors"
+ "fmt"
+ "net/url"
+ "strings"
+ "time"
+
+ "go.uber.org/multierr"
+)
+
+var (
+ defaultPollInterval = 5 * time.Minute
+)
+
+// Config is the overall config structure for the awscloudwatchmetricsreceiver
+type Config struct {
+ Region string `mapstructure:"region"`
+ Profile string `mapstructure:"profile"`
+ IMDSEndpoint string `mapstructure:"imds_endpoint"`
+ PollInterval time.Duration `mapstructure:"poll_interval"`
+ Metrics *MetricsConfig `mapstructure:"metrics"`
+}
+
+// MetricsConfig is the configuration for the metrics part of the receiver
+// added this so we could expand to other inputs such as autodiscover
+type MetricsConfig struct {
+ Names []*NamedConfig `mapstructure:"named"`
+}
+
+// NamesConfig is the configuration for the metric namespace and metric names
+// https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html
+type NamedConfig struct {
+ Namespace string `mapstructure:"namespace"`
+ MetricName string `mapstructure:"metric_name"`
+ Period time.Duration `mapstructure:"period"`
+ AwsAggregation string `mapstructure:"aws_aggregation"`
+ Dimensions []MetricDimensionsConfig `mapstructure:"dimensions"`
+}
+
+// MetricDimensionConfig is the configuration for the metric dimensions
+type MetricDimensionsConfig struct {
+ Name string `mapstructure:"Name"`
+ Value string `mapstructure:"Value"`
+}
+
+var (
+ errNoMetricsConfigured = errors.New("no metrics configured")
+ errNoNamespaceConfigured = errors.New("no metric namespace configured")
+ errNoRegion = errors.New("no region was specified")
+ errInvalidPollInterval = errors.New("poll interval is incorrect, it must be a duration greater than one second")
+
+ // https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-data.html
+ errEmptyDimensions = errors.New("dimensions name and value is empty")
+ errTooManyDimensions = errors.New("you cannot define more than 30 dimensions for a metric")
+ errDimensionColonPrefix = errors.New("dimension name cannot start with a colon")
+
+ errInvalidAwsAggregation = errors.New("invalid AWS aggregation")
+)
+
+func (cfg *Config) Validate() error {
+ if cfg.Region == "" {
+ return errNoRegion
+ }
+
+ if cfg.IMDSEndpoint != "" {
+ _, err := url.ParseRequestURI(cfg.IMDSEndpoint)
+ if err != nil {
+ return fmt.Errorf("unable to parse URI for imds_endpoint: %w", err)
+ }
+ }
+
+ if cfg.PollInterval < time.Second {
+ return errInvalidPollInterval
+ }
+ var errs error
+ errs = multierr.Append(errs, cfg.validateMetricsConfig())
+ return errs
+}
+
+func (cfg *Config) validateMetricsConfig() error {
+ if cfg.Metrics == nil {
+ return errNoMetricsConfigured
+ }
+ return cfg.validateNamedConfig()
+}
+
+func (cfg *Config) validateNamedConfig() error {
+ if cfg.Metrics.Names == nil {
+ return errNoMetricsConfigured
+ }
+ return cfg.validateDimensionsConfig()
+}
+
+func (cfg *Config) validateDimensionsConfig() error {
+ var errs error
+
+ metricsNames := cfg.Metrics.Names
+ for _, name := range metricsNames {
+ if name.Namespace == "" {
+ return errNoNamespaceConfigured
+ }
+ err := validateAwsAggregation(name.AwsAggregation)
+ if err != nil {
+ return err
+ }
+ if name.MetricName == "" {
+ return errNoMetricsConfigured
+ }
+ errs = multierr.Append(errs, validate(name.Dimensions))
+ }
+ return errs
+}
+
+// https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Statistics-definitions.html
+func validateAwsAggregation(agg string) error {
+ switch {
+ case agg == "SampleCount":
+ return nil
+ case agg == "Sum":
+ return nil
+ case agg == "Average":
+ return nil
+ case agg == "Minimum":
+ return nil
+ case agg == "Maximum":
+ return nil
+ case strings.HasPrefix(agg, "p"):
+ return nil
+ case strings.HasPrefix(agg, "TM"):
+ return nil
+ case agg == "IQM":
+ return nil
+ case strings.HasPrefix(agg, "PR"):
+ return nil
+ case strings.HasPrefix(agg, "TC"):
+ return nil
+ case strings.HasPrefix(agg, "TS"):
+ return nil
+ default:
+ return errInvalidAwsAggregation
+ }
+}
+
+func validate(nmd []MetricDimensionsConfig) error {
+ for _, dimensionConfig := range nmd {
+ if dimensionConfig.Name == "" || dimensionConfig.Value == "" {
+ return errEmptyDimensions
+ }
+ if strings.HasPrefix(dimensionConfig.Name, ":") {
+ return errDimensionColonPrefix
+ }
+ }
+ if len(nmd) > 30 {
+ return errTooManyDimensions
+ }
+ return nil
+}