Scraping a virtual machine with promitor helm chart

[deggja]

Hello!

I've recently started using promitor so I might be going about this the wrong way.

I'm using the promitor helm charts for the scraper and resource discovery. I've successfully scraped my SQL instances with this approach.

However, I'm having some issues adding additional resources to be scraped (The resource discovery seems to work).

When I'm installing the helm chart for the agent scraper I'm getting the following error:

[13:19:23 INF] Booting up Promitor v2.7.0 running .NET .NET 6.0.6 - Thank you for using Promitor!
[13:19:23 INF] Running .NET 6.0.6 on Linux (alpine.3.14-x64 | Linux 5.4.0-1085-azure #90~18.04.1-Ubuntu SMP Fri Jun 10 18:32:22 UTC 2022).
[13:19:23 INF] Using configuration folder '/config/'
[13:19:23 FTL] Promitor Scraper Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
System.ArgumentNullException: Value cannot be null. (Parameter 'key')
   at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
   at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.GroupMetricsByScrapingInterval(MetricsDeclaration allMetrics) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 59
   at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.ScheduleMetricScraping(IServiceCollection services) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 46
   at Promitor.Agents.Scraper.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 55
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 35

The only difference here is that the virtual machine I am trying to scrape (as opposed to the SQL database) is in another subscription and resource group entirely. So I suspect this is where the issue is - and I configured something the wrong way.

Here are the values from my scraper and resource discovery charts:

## Metric Declaration YAML
azureMetadata:
  tenantId: redacted
  subscriptionId: redacted
  resourceGroupName: redacted
  cloud: "Global"
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "*/5 * * * *"
metrics:
  # Get cpu percent metric from sql database
  - name: azure_sql_database_cpu_percent_discovery
    description: "The CPU percentage used by an Azure SQL Database."
    resourceType: SqlDatabase
    labels:
      app: promitor
      customer: customer
      resource: sql
    azureMetricConfiguration:
      metricName: cpu_percent
      aggregation:
        type: Average
    resourceDiscoveryGroups:
      - name: sql-databases
  - name: azure_virtual_machine_percentage_cpu
    description: "Average percentage cpu usage on an Azure virtual machine"
    resourceType: VirtualMachine
    #resourceGroupName: redacted
    #subscriptionId: redacted
    labels:
      app: promitor
      customer: customer
      resource: vm
    azureMetricConfiguration:
      metricname: Percentage CPU
      aggregation:
        type: Average
    resourceDiscoveryGroups:
      - name: virtual-machines

# Metric Declaration YAML
azureLandscape:
  cloud: Global
  tenantId: redacted tenant id
  subscriptions:
  - redacted subscription ID 1
  - redacted subscription ID 2

resourceDiscoveryGroups:
- name: sql-databases
  type: SqlDatabase
- name: virtual-machines
  type: VirtualMachine
  criteria:
    include:
      resourceGroups:
      - redacted resource group name
      subscriptions:
      - redacted subscription id
      region:
      - redacted region

Am I missing something? I expect this to be able to continue scraping my SQL instances, in addition to scraping the virtual machines in the other resource group and subscription.

Note: I tried adding the resourceGroupName and subscriptionId directly to the scraper configuration (the two lines I commented out) - but it fails the same. I did also find this thread - but the links to the example linked in there is not working..

Any nudge in the right direction is appreciated.

Thanks.

[tomkerkhove]

Hi,

This exception typically means that the configuration is invalid and there's a bug open for this, but I thin the problem is here:

## Metric Declaration YAML
azureMetadata:
  tenantId: redacted
  subscriptionId: redacted
  resourceGroupName: redacted
  cloud: "Global"
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "*/5 * * * *"
metrics:
  # Get cpu percent metric from sql database
  - name: azure_sql_database_cpu_percent_discovery
    description: "The CPU percentage used by an Azure SQL Database."
    resourceType: SqlDatabase
    labels:
      app: promitor
      customer: customer
      resource: sql
    azureMetricConfiguration:
      metricName: cpu_percent
      aggregation:
        type: Average
    resourceDiscoveryGroups:
      - name: sql-databases
  - name: azure_virtual_machine_percentage_cpu
    description: "Average percentage cpu usage on an Azure virtual machine"
    resourceType: VirtualMachine
-   resourceGroupName: redacted
-   subscriptionId: redacted
    labels:
      app: promitor
      customer: customer
      resource: vm
    azureMetricConfiguration:
      metricname: Percentage CPU
      aggregation:
        type: Average
    resourceDiscoveryGroups:
      - name: virtual-machines

Did you try it running like this?

[tomkerkhove]

Yeah - I saw those samples, but I couldn't find any samples on how to specifiy the additional resourceGroupNames in the agent-scraper helm chart while using resourceDiscoveryGroups.

You can't because the discovery happens by the discovery agent so either you need another group, extend the current group or declare the resources manually that you want to scrape.

I noticed that the criteria specified for the virtual-machines resourceDiscoveryGroup appears within the sql-databases block in the configMap for resource discovery. Not sure if that's a problem (It's specified as criteria for the virtual-machines resouceDiscoveryGroup in the chart)?

Actually it is fine but it's just sorted differently:

   resourceDiscoveryGroups:
    - name: sql-databases
      type: SqlDatabase
    - criteria:
        include:
          region:
          - redacted
          resourceGroups:
          - redacted
          subscriptions:
          - redacted
      name: virtual-machines
      type: VirtualMachine

I still see this though, can you make sure it's not in your Helm config or delete it manually please?

    - azureMetricConfiguration:
        aggregation:
          type: Average
        metricname: Percentage CPU
      description: Average percentage cpu usage on an Azure virtual machine
      labels:
        app: promitor
        customer: redacted
        resource: vm
      name: azure_virtual_machine_percentage_cpu
      resourceDiscoveryGroups:
      - name: virtual-machines
-     resourceGroupName: redacted
      resourceType: VirtualMachine
-     subscriptionId: redacted

Once the scraper restarted it should be fine

[deggja]

Aha, actually that makes sense. Thanks.

So I've made the changes accordingly, but it seems like it's still crashing with the same error message.

Edit: I don't know if I wrote that or not - but I should say that it works perfectly fine with the SqlDatabase and the sql-database resourceDiscoveryGroup, but as soon as I add the below config for the VirtualMachine resource and resourceDiscoveryGroup it ends up in a crashLoop.

Running this config:

 - azureMetricConfiguration:
        aggregation:
          type: Average
        metricname: Percentage CPU
      description: Average percentage cpu usage on an Azure virtual machine
      labels:
        app: promitor
        customer: redacted
        resource: vm
      name: azure_virtual_machine_percentage_cpu
      resourceDiscoveryGroups:
      - name: virtual-machines
      resourceType: VirtualMachine

[tomkerkhove]

Aaah YAML. Can you change azureMetricConfiguration.metricname to azureMetricConfiguration.metricName please?

[deggja]

... Yeah, that was the issue 🤦‍♂️

THANKS - I'm pretty sure I never would have noticed.

[tomkerkhove]

My pleasure!