Unable to scrape metrics from Azure monitor

[anoop2503]

Hi,

I could manage to set up the promitor application in kubernetes environment. The application is up and running without any issues. However, it does not scrape metrics from Azure monitor.

Here is the configuration for setting up the promitor scraper agent (without resourceDiscovery).

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promitor-agent-scraper
  namespace: monitoring
  labels:
    app: promitor-agent-scraper
data:
  runtime.yaml: |-
      server:
        httpPort: "5000"
      authentication:
        mode: UserAssignedManagedIdentity
        identityId: metrics-identity
      metricSinks:
        prometheusScrapingEndpoint:
          metricUnavailableValue: "NaN"
          enableMetricTimestamps: "true"
          baseUriPath: "/metrics"
          labels:
            transformation: "None"
      metricsConfiguration:
        absolutePath: /config/metrics-declaration.yaml
      telemetry:
        applicationInsights:
          isEnabled: "false"
        containerLogs:
          isEnabled: "true"
          verbosity: "Debug"
        defaultVerbosity: "Debug"
  metrics-declaration.yaml: |-
    version: v1
    azureMetadata:
      tenantId: xxxx-xxxxx-xxxxx-xxxx-xxxxx
      subscriptionId: xxxx-xxx-xxx-xxx-xxxx
      resourceGroupName: xxxxxxxxx
      cloud: Global
    metricDefaults:
      aggregation:
        interval: 00:02:00
      scraping:
        schedule: "*/2 * * * *"
    metrics:
      - name: azure_load_balancer_traffic_bytes
        description: "Average amount of bytes sent through an Azure Load Balancer"
        resourceType: LoadBalancer
        scraping:
          schedule: "*/2 * * * *"
        azureMetricConfiguration:
          metricName: ByteCount
          aggregation:
            type: Average
            interval: 00:05:00
        resources: # Optional, required when no resource discovery is configured
        - loadBalancerName: bda-azr-dev-wu2-zonal-agent-pool-internal
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: promitor-agent-scraper
  namespace: monitoring
  labels:
    aadpodidbinding: metrics-identity
    app: promitor-agent-scraper
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: promitor-agent-scraper
  template:
    metadata:
      labels:
        app: promitor-agent-scraper
        aadpodidbinding: metrics-identity
    spec:
      containers:
        - name: promitor-agent-scraper
          image: "ghcr.io/tomkerkhove/promitor-agent-scraper:latest"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 5000
              protocol: TCP
          volumeMounts:
          - name: config-volume-promitor-agent-scraper
            mountPath: /config/
          - name: tmp
            mountPath: /tmp/
      imagePullSecrets:
      - name: registry-credentials
      volumes:
        - name: config-volume-promitor-agent-scraper
          configMap:
            name: promitor-agent-scraper
        - emptyDir: {}
          name: tmp

I can see there is some authentication failure logs in the container logs though. Here is the container logs:

██████╗ ██████╗  ██████╗ ███╗   ███╗██╗████████╗ ██████╗ ██████╗ 
██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗
██████╔╝██████╔╝██║   ██║██╔████╔██║██║   ██║   ██║   ██║██████╔╝
██╔═══╝ ██╔══██╗██║   ██║██║╚██╔╝██║██║   ██║   ██║   ██║██╔══██╗
██║     ██║  ██║╚██████╔╝██║ ╚═╝ ██║██║   ██║   ╚██████╔╝██║  ██║
╚═╝     ╚═╝  ╚═╝ ╚═════╝ ╚═╝     ╚═╝╚═╝   ╚═╝    ╚═════╝ ╚═╝  ╚═╝
[14:06:55 INF] Booting up Promitor v2.6.1 running .NET .NET 6.0.3 - Thank you for using Promitor!
[14:06:55 INF] Running .NET 6.0.3 on Linux (alpine.3.14-x64 | Linux 5.10.61-flatcar #1 SMP Tue Aug 31 15:41:58 -00 2021).
[14:06:55 INF] Using configuration folder '/config/'
[14:06:56 INF] Starting validation of Promitor setup
[14:06:56 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                              │ Outcome │ Details                        ┃
┠───────────────────────────────────┼─────────┼────────────────────────────────┨
┃ Metrics Declaration Path          │ Success │ Everything is well-configured. ┃
┃ Azure Authentication              │ Success │ Everything is well-configured. ┃
┃ Metrics Declaration               │ Success │ Everything is well-configured. ┃
┃ Resource Discovery                │ Success │ Everything is well-configured. ┃
┃ StatsD Metric Sink                │ Success │ Everything is well-configured. ┃
┃ Prometheus Scraping Endpoint      │ Success │ Everything is well-configured. ┃
┃ Metric Sink                       │         │                                ┃
┃ Atlassian Statuspage Metric Sink  │ Success │ Everything is well-configured. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                   Validation                                   
[14:06:56 INF] Promitor configuration is valid, we are good to go.
[14:06:56 INF] Here's an overview of what was configured:
╭───────────────────┬───────────────────┬───────────────────┬──────────────────╮
│ Tenant Id         │ Subscription Id   │ Resource Group    │ Azure Cloud      │
│                   │                   │ Name (Default)    │                  │
├───────────────────┼───────────────────┼───────────────────┼──────────────────┤
│ xxxx-xxxx-xxxxxxx │ xxxxxxxx-xxxx-xxx │ xxxxxxxxxxxxxxxxx │ Azure Global     │
│ x-xxxx-xxxxxxxxxx │ x-xxxx-xxxxxxxxxx │ xx                │ Cloud            │
│ xx                │ xx                │                   │                  │
╰───────────────────┴───────────────────┴───────────────────┴──────────────────╯
                                 Azure Metadata                                 
╭─────────────┬─────────────┬────────┬─────────────┬─────────────┬─────────────╮
│ Metric Name │ Resource    │ Labels │ Azure       │ Resources   │ Resource    │
│             │ Type        │        │ Monitor     │ To Scrape   │ Discovery   │
│             │             │        │ Metric      │             │ Groups To   │
│             │             │        │             │             │ Scrape      │
├─────────────┼─────────────┼────────┼─────────────┼─────────────┼─────────────┤
│ azure_load_ │ Load        │ None   │ ByteCount   │ xxx-azr-dev │ None        │
│ balancer_tr │ Balancer    │        │             │ -xxx-zonal- │             │
│ affic_bytes │             │        │             │ agent-pool- │             │
│             │             │        │             │ internal    │             │
╰─────────────┴─────────────┴────────┴─────────────┴─────────────┴─────────────╯
                               Configured Metrics                               
[14:06:57 INF] [Job][f7926029-4df3-43de-b973-baf17deefa69-k8sdevwu2clusterbda-azure_load_balancer_traffic_bytes-xxx-azr-dev-xxx-zonal-agent-pool-internal-fd6a9af8-799f-4136-b69a-8dd648655f61] is running under time zone: UTC
[14:06:57 INF] Scraping Azure Monitor - 04/21/2022 14:06:57 +00:00
[14:06:57 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:06:57 INF] Adding Prometheus sink to expose on /metrics
[14:06:57 INF] Adding Prometheus sink to expose on /metrics
[14:06:57 INF] Now listening on: https://[::]:5000
[14:06:57 INF] Application started. Press Ctrl+C to shut down.
[14:06:57 INF] Hosting environment: Production
[14:06:57 INF] Content root path: /app
[14:08:00 INF] Scraping Azure Monitor - 04/21/2022 14:08:00 +00:00
[14:08:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:10:00 INF] Scraping Azure Monitor - 04/21/2022 14:10:00 +00:00
[14:10:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:12:00 INF] Scraping Azure Monitor - 04/21/2022 14:12:00 +00:00
[14:12:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:14:00 INF] Scraping Azure Monitor - 04/21/2022 14:14:00 +00:00
[14:14:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:16:00 INF] Scraping Azure Monitor - 04/21/2022 14:16:00 +00:00
[14:16:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:18:00 INF] Scraping Azure Monitor - 04/21/2022 14:18:00 +00:00
[14:18:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:20:00 INF] Scraping Azure Monitor - 04/21/2022 14:20:00 +00:00
[14:20:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:22:00 INF] Scraping Azure Monitor - 04/21/2022 14:22:00 +00:00
[14:22:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:24:00 INF] Scraping Azure Monitor - 04/21/2022 14:24:00 +00:00
[14:24:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:26:00 INF] Scraping Azure Monitor - 04/21/2022 14:26:00 +00:00
[14:26:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:28:00 INF] Scraping Azure Monitor - 04/21/2022 14:28:00 +00:00
[14:28:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:30:00 INF] Scraping Azure Monitor - 04/21/2022 14:30:00 +00:00
[14:30:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:32:00 INF] Scraping Azure Monitor - 04/21/2022 14:32:00 +00:00
[14:32:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:34:00 INF] Scraping Azure Monitor - 04/21/2022 14:34:00 +00:00
[14:34:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:36:00 INF] Scraping Azure Monitor - 04/21/2022 14:36:00 +00:00
[14:36:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:38:00 INF] Scraping Azure Monitor - 04/21/2022 14:38:00 +00:00
[14:38:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:40:00 INF] Scraping Azure Monitor - 04/21/2022 14:40:00 +00:00
[14:40:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:42:00 INF] Scraping Azure Monitor - 04/21/2022 14:42:00 +00:00
[14:42:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:44:00 INF] Scraping Azure Monitor - 04/21/2022 14:44:00 +00:00
[14:44:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:46:00 INF] Scraping Azure Monitor - 04/21/2022 14:46:00 +00:00
[14:46:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:48:00 INF] Scraping Azure Monitor - 04/21/2022 14:48:00 +00:00
[14:48:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:48:26 FTL] Failed to scrape resource for metric 'azure_load_balancer_traffic_bytes'
Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSIMaxRetryReachedException: MSI: Failed to acquire tokens after retrying %20 times
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.RetrieveTokenFromIMDSWithRetryAsync(String resource, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetTokenFromIMDSEndpointAsync(String resource, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetAuthenticationHeaderForVirtualMachineAsync(String resource, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetAuthenticationHeaderAsync(CancellationToken cancellationToken)
   at Microsoft.Rest.TokenCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperations.ListWithHttpMessagesAsync(String resourceUri, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperationsExtensions.ListAsync(IMetricDefinitionsOperations operations, String resourceUri, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.Microsoft.Azure.Management.Monitor.Fluent.IMetricDefinitions.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetMetricDefinitionsAsync(String resourceId) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 124
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 80
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 54
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 78

Here are the metrics what promitor is exposing now:

# HELP promitor_runtime_dotnet_collection_count_total GC collection count
# TYPE promitor_runtime_dotnet_collection_count_total counter
promitor_runtime_dotnet_collection_count_total{generation="0"} 2
promitor_runtime_dotnet_collection_count_total{generation="1"} 1
promitor_runtime_dotnet_collection_count_total{generation="2"} 0
# HELP promitor_runtime_dotnet_totalmemory Total known allocated memory
# TYPE promitor_runtime_dotnet_totalmemory gauge
promitor_runtime_dotnet_totalmemory 10389832
# HELP promitor_runtime_process_cpu_seconds_total Total user and system CPU time spent in seconds
# TYPE promitor_runtime_process_cpu_seconds_total counter
promitor_runtime_process_cpu_seconds_total 10.1
# HELP promitor_runtime_process_virtual_bytes Process virtual memory size
# TYPE promitor_runtime_process_virtual_bytes gauge
promitor_runtime_process_virtual_bytes 23826468864
# HELP promitor_runtime_process_working_set Process working set
# TYPE promitor_runtime_process_working_set gauge
promitor_runtime_process_working_set 195641344
# HELP promitor_runtime_process_private_bytes Process private memory size
# TYPE promitor_runtime_process_private_bytes gauge
promitor_runtime_process_private_bytes 174456832
# HELP promitor_runtime_process_num_threads Total number of threads
# TYPE promitor_runtime_process_num_threads gauge
promitor_runtime_process_num_threads 28
# HELP promitor_runtime_process_processid Process ID
# TYPE promitor_runtime_process_processid gauge
promitor_runtime_process_processid 1
# HELP promitor_runtime_process_start_time_seconds Start time of the process since unix epoch in seconds
# TYPE promitor_runtime_process_start_time_seconds gauge
promitor_runtime_process_start_time_seconds 1650550015
# HELP promitor_runtime_http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE promitor_runtime_http_request_duration_seconds histogram
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.005"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.01"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.025"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.05"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.075"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.25"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.75"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="2.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="7.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="10"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="+Inf"} 1
promitor_runtime_http_request_duration_seconds_sum{status_code="200",method="GET",path="/metrics"} 0.040377444
promitor_runtime_http_request_duration_seconds_count{status_code="200",method="GET",path="/metrics"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.005"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.01"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.025"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.05"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.075"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.25"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.75"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="2.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="7.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="10"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="+Inf"} 1
promitor_runtime_http_request_duration_seconds_sum{status_code="404",method="GET",path="/favicon.ico"} 0.030159532
promitor_runtime_http_request_duration_seconds_count{status_code="404",method="GET",path="/favicon.ico"} 1
# HELP promitor_scrape_error Provides an indication that the scraping of the resource has failed
# TYPE promitor_scrape_error gauge
promitor_scrape_error{tenant_id="xxxx-xxxx-xxxx-xxxx-xxxxxx",subscription_id="xxxxxx-xxxx-xxxxx-xxxxx-xxxxxx",resource_type="LoadBalancer",resource_name="xxx-azr-dev-xxx-zonal-agent-pool-internal",resource_group="xxxxxxxxxxxxx",metric_name="azure_load_balancer_traffic_bytes"} 1 1650552506423
# HELP promitor_scrape_success Provides an indication that the scraping of the resource was successful
# TYPE promitor_scrape_success gauge
promitor_scrape_success{tenant_id="xxxx-xxxx-xxxx-xxxx-xxxxxx",subscription_id="xxxxxx-xxxx-xxxxx-xxxxx-xxxxxx",resource_type="LoadBalancer",resource_name="xxx-azr-dev-xxx-zonal-agent-pool-internal",resource_group="xxxxxxxxxxxxx",metric_name="azure_load_balancer_traffic_bytes"} 0 1650552506422

What I understood from the container log is, the UserAssignedManagedIdentity authentication is failing when scraping metrics from azure monitor. But, I tried the same UserAssignedManagedIdentity as aadpodidbinding with azure_metrics_exporter and it returns the metrics without any issues. So, wondering if I have done any mistakes in the configurations side?

For authentication.identityId in runtime.yaml, I am using the same value what I am using for aadpodidbinding in deployment label section. I believe that is the correct configuration.

Please help.

Thanks,

[tomkerkhove]

I might be wrong but I believe that has to be the app id instead, but @Mimetis implemented it so he might be able to refresh my memory

[anoop2503]

Thanks @tomkerkhove for the response.

I tried the same configuration with "Client ID" / "Object (principal) ID" instead of using aadpodidbinding value for authentication.identityId in runtime.yaml. But, still the application is not scraping the metrics from Azure monitor. Since we are using user assigned managed identity, i believe we don't need to create any applications. So, can you please provide some more details about what could be the app id value when we use "user assigned managed identity"?

Thanks,

[anoop2503]

Any suggestion on this please?

[tomkerkhove]

I haven't had time to check yet - Sorry!

[tomkerkhove]

Created #2021 for this