Promitor Scraper Agent has encountered an unexpected error

[kad-meedel]

Report

We changed the promitor version to 2.0.0., because we had problems monitoring flexible server instance for Postgresql.
But now the promitor service failed to start

Expected Behavior

A running promitor service

Actual Behavior

pod/promitor-784f66c89d-z6m8w 0/1 CrashLoopBackOff 5 (2m39s ago) 5m40s

Steps to Reproduce the Problem

...
deploy a the new promitor image version 2.0.0

Component

Scraper

Version

2.0.0

Configuration

Configuration:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:

• rbac.yaml
• deployment.yaml
• service.yaml

configMapGenerator:

• name: promitor-config
  files:
  • config/runtime.yaml

images:

• name: promitor
  newName: tomkerkhove/promitor-agent-scraper
  newTag: 2.0.0

Logs

2022-04-25T06:39:50.508556579Z ██████╗ ██████╗ ██████╗ ███╗ ███╗██╗████████╗ ██████╗ ██████╗
2022-04-25T06:39:50.508676980Z ██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗
2022-04-25T06:39:50.508683980Z ██████╔╝██████╔╝██║ ██║██╔████╔██║██║ ██║ ██║ ██║██████╔╝
2022-04-25T06:39:50.508688080Z ██╔═══╝ ██╔══██╗██║ ██║██║╚██╔╝██║██║ ██║ ██║ ██║██╔══██╗
2022-04-25T06:39:50.508692180Z ██║ ██║ ██║╚██████╔╝██║ ╚═╝ ██║██║ ██║ ╚██████╔╝██║ ██║
2022-04-25T06:39:50.508695780Z ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝
2022-04-25T06:39:50.599551631Z [06:39:50 INF] Booting up Promitor v2.0.0 - Thank you for using Promitor!
2022-04-25T06:39:50.599588832Z [06:39:50 INF] Using configuration folder '/config/'
2022-04-25T06:39:52.401077199Z [06:39:52 INF] Metrics declaration is using spec version v1
2022-04-25T06:39:52.890857814Z [06:39:52 FTL] Promitor Scraper Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
2022-04-25T06:39:52.890892414Z System.InvalidOperationException: Unable to resolve service for type 'Prometheus.Client.Abstractions.IMetricFactory' while attempting to activate 'Promitor.Integrations.Sinks.Prometheus.RuntimeMetricsCollector'.
2022-04-25T06:39:52.890898914Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.CreateArgumentCallSites(Type serviceType, Type implementationType, CallSiteChain callSiteChain, ParameterInfo[] parameters, Boolean throwIfCallSiteNotFound)
2022-04-25T06:39:52.890907314Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.CreateConstructorCallSite(ResultCache lifetime, Type serviceType, Type implementationType, CallSiteChain callSiteChain)
2022-04-25T06:39:52.890913914Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.TryCreateExact(ServiceDescriptor descriptor, Type serviceType, CallSiteChain callSiteChain, Int32 slot)
2022-04-25T06:39:52.890919214Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.TryCreateExact(Type serviceType, CallSiteChain callSiteChain)
2022-04-25T06:39:52.890922714Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.CreateCallSite(Type serviceType, CallSiteChain callSiteChain)
2022-04-25T06:39:52.890926914Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.<>c__DisplayClass7_0.b__0(Type type)
2022-04-25T06:39:52.890931114Z at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
2022-04-25T06:39:52.890935214Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.GetCallSite(Type serviceType, CallSiteChain callSiteChain)
2022-04-25T06:39:52.890938115Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.CreateServiceAccessor(Type serviceType)
2022-04-25T06:39:52.890941015Z at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
2022-04-25T06:39:52.890944215Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
2022-04-25T06:39:52.890966915Z at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType)
2022-04-25T06:39:52.890972015Z at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
2022-04-25T06:39:52.890976315Z at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
2022-04-25T06:39:52.890981115Z at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.ScheduleMetricScraping(IServiceCollection services) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 34
2022-04-25T06:39:52.890985215Z at Promitor.Agents.Scraper.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 55
2022-04-25T06:39:52.890988315Z at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
2022-04-25T06:39:52.890991315Z at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
2022-04-25T06:39:52.890994415Z at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
2022-04-25T06:39:52.890997315Z at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.g__Startup|0(IServiceCollection serviceCollection)
2022-04-25T06:39:52.891000215Z at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
2022-04-25T06:39:52.891003115Z at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.b__0(IServiceCollection services)
2022-04-25T06:39:52.891006115Z at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
2022-04-25T06:39:52.891010115Z at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.b__0(HostBuilderContext context, IServiceCollection services)
2022-04-25T06:39:52.891013115Z at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
2022-04-25T06:39:52.891016215Z at Microsoft.Extensions.Hosting.HostBuilder.Build()
2022-04-25T06:39:52.891019215Z at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 34

Platform

Microsoft Azure

Contact Details

[email protected]

[tomkerkhove]

Hi, would you mind adding some more information on how you deploy please? Also, since you are upgrading I highly recommend going to v2.6.1.

[kad-meedel]

We followed the information on https://docs.promitor.io/walkthrough/scrape-promitor-with-prometheus-on-azure-kubernetes-service.html
Deploy Promitor and Prometheus
Create a metrics declaration for Promitor

The problem I got in version 1.5.0 was the message:
[08:23:02 FTL] ResourceNotFound: The Resource 'Microsoft.DBforPostgreSQL/servers/pg-dev-test.postgres.database.azure.com' under resource group 'rg-pg-dev-test' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

And this message is correct because we are using flexible postgres servers. The info in azure for this kind of resource is:
"id": "/subscriptions/xxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-pg-dev-test/providers/Microsoft.DBforPostgreSQL/flexibleServers/pg-dev-test",

I implemeted the setting type: Flexible according the info from https://docs.promitor.io/configuration/v2.x/metrics/postgresql, but that doesn't seem to work.

So I tried to have an update of promitor to see if that solved my problem.

The metrics-decleration.yaml looks like this:

version: v1
azureMetadata:
tenantId: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
subscriptionId: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resourceGroupName: rg-pg-dev-test
metricDefaults:
aggregation:
"# azure metrics need 5 minutes aggregation interval (some won't work @1 minute)"
interval: 00:05:00
scraping:
"# Every minute"
schedule: "0 * * ? * *"
"# all metrics: https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported"
metrics:

• name: azure_postgresql_cpu_percent
  description: "The CPU percentage on the server"
  resourceType: PostgreSql
  azureMetricConfiguration:
  metricName: cpu_percent
  aggregation:
  type: Average
  resources:
  • serverName: pg-dev-test
    resourceGroupName: rg-pg-dev-test
    type: Flexible

[kad-meedel]

I have now used version 2.6.1, but get the same error message.

[12:55:34 INF] Booting up Promitor v2.6.1 running .NET .NET 6.0.3 - Thank you for using Promitor!
[12:55:34 INF] Running .NET 6.0.3 on Linux (alpine.3.14-x64 | Linux 5.4.0-1067-azure #70~18.04.1-Ubuntu SMP Thu Jan 13 19:46:01 UTC 2022).
[12:55:34 INF] Using configuration folder '/config/'
[12:55:37 FTL] Promitor Scraper Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
System.InvalidOperationException: Unable to resolve service for type 'Promitor.Core.Metrics.Prometheus.Collectors.Interfaces.IPrometheusMetricsCollector' while attempting to activate 'Promitor.Integrations.Sinks.Prometheus.Collectors.AzureScrapingPrometheusMetricsCollector.

I can stil run Version 1.5.0 but than I get the ResourceNotFound error

[tomkerkhove]

And what Helm configuration are you using?

[kad-meedel]

Hello Tom, we don't use helm for deployment. We use kustomize

[tomkerkhove]

Would you mind sharing your configuration then please?

[kad-meedel]

These are the two config files I use for promitor

runtime.yaml.txt
metrics-declaration.yaml.txt

promitor-pod-config-at-runtime.txt

[tomkerkhove]

Given you are upgrading to a new major version, you should align your configuration with the introduced breaking changes as announced in https://blog.tomkerkhove.be/2021/01/06/announcing-promitor-2-0-bringing-azure-monitor-metrics-where-you-need-them/

This migration guide should be able to help you: https://docs.promitor.io/walkthrough/migrate-from-1.x-to-2.x

The problem you are having (at least) is that you are not defining any metric sinks.

So what I would suggest is:

1. Migrate to latest Promitor version
2. Follow migration guide to align
3. Re-deploy

Feel free to let me know how it goes

[kad-meedel]

Tom, I changed the runtime config and it's all working fine now. Thanks for your support.

For your info: the url link https://promitor.io/configuration/v2.x/runtime/scraper#prometheus-scraping-endpoint give's an 404

[tomkerkhove]

Yes, I've noticed it this morning - Sorry!

Also, if you want to become a listed Promitor end-user feel free to let me know!