Promitor scraper not working for resourceType: ApiManagement #1194

jaisegrg · 2020-08-04T13:52:44Z

jaisegrg
Aug 4, 2020

I have installed promitor scraper agent for "resourceType: ApiManagement" on AKS cluster.

Helm chart used:- https://hub.helm.sh/charts/promitor/promitor-agent-scraper/2.0.0-preview-1
Metric declaration used:- https://promitor.io/configuration/v1.x/metrics/api-management.html

But the pod is getting crashed with the following error,

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
at Promitor.Scraper.Host.Startup..ctor(IConfiguration configuration) in /src/Promitor.Scraper.Host/Startup.cs:line 20
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetServiceOrCreateInstance(IServiceProvider provider, Type type)
at Microsoft.AspNetCore.Hosting.Internal.StartupLoader.LoadMethods(IServiceProvider hostingServiceProvider, Type startupType, String environmentName)
at Microsoft.AspNetCore.Hosting.WebHostBuilderExtensions.<>c__DisplayClass1_0.b__1(IServiceProvider sp)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, ServiceProviderEngineScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(IServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite singletonCallSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.b__0(ServiceProviderEngineScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureStartup()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at Promitor.Scraper.Host.Program.BuildWebHost(String[] args) in /src/Promitor.Scraper.Host/Program.cs:line 31
at Promitor.Scraper.Host.Program.Main(String[] args) in /src/Promitor.Scraper.Host/Program.cs:line 16

Tried configuring "Resourcetype: Azure Cache for Redis" using the metrics declaration https://promitor.io/configuration/v1.x/metrics/redis-cache and got the same error. But able to make it work using the below runtime configuration,

prometheus:
metricUnavailableValue: NaN # Optional. Default: NaN
enableMetricTimestamps: false # Optional. Default: true
scrapeEndpoint:
baseUriPath: /metrics # Optional. Default: /metrics

Tried the same runtime configuration for "ApiManagement", but got the below error,

Unhandled Exception: System.Runtime.Serialization.SerializationException: Unable to deserialize the configured metrics declaration. Please verify that it is a well-formed YAML specification. ---> System.ArgumentException: Requested value 'ApiManagement' was not found.
at System.Enum.TryParseEnum(Type enumType, String value, Boolean ignoreCase, EnumResult& parseResult)
at System.Enum.Parse[TEnum](String value, Boolean ignoreCase)
at Promitor.Core.Scraping.Configuration.Serialization.YamlMappingNodeExtensions.GetEnum[TEnum](YamlMappingNode node, String propertyName) in /src/Promitor.Core.Scraping/Serialization/YamlMappingNodeExtensions.cs:line 37
at Promitor.Core.Scraping.Configuration.Serialization.v1.Core.MetricDefinitionDeserializer.Deserialize(YamlMappingNode node) in /src/Promitor.Core.Scraping/Serialization/v1/Core/MetricDefinitionDeserializer.cs:line 35
at Promitor.Core.Scraping.Configuration.Serialization.Deserializer`1.Deserialize(YamlSequenceNode nodes) in /src/Promitor.Core.Scraping/Serialization/Deserializer.cs:line 34
at Promitor.Core.Scraping.Configuration.Serialization.v1.Core.V1Deserializer.DeserializeMetrics(YamlMappingNode rootNode) in /src/Promitor.Core.Scraping/Serialization/v1/Core/V1Deserializer.cs:line 83
at Promitor.Core.Scraping.Configuration.Serialization.v1.Core.V1Deserializer.Deserialize(YamlMappingNode rootNode) in /src/Promitor.Core.Scraping/Serialization/v1/Core/V1Deserializer.cs:line 27
at Promitor.Core.Scraping.Configuration.Serialization.ConfigurationSerializer.InterpretYamlStream(YamlStream metricsDeclarationYamlStream) in /src/Promitor.Core.Scraping/Serialization/ConfigurationSerializer.cs:line 60
at Promitor.Core.Scraping.Configuration.Serialization.ConfigurationSerializer.Deserialize(String rawMetricsDeclaration) in /src/Promitor.Core.Scraping/Serialization/ConfigurationSerializer.cs:line 39
--- End of inner exception stack trace ---
at Promitor.Core.Scraping.Configuration.Serialization.ConfigurationSerializer.Deserialize(String rawMetricsDeclaration) in /src/Promitor.Core.Scraping/Serialization/ConfigurationSerializer.cs:line 45
at Promitor.Core.Scraping.Configuration.Providers.MetricsDeclarationProvider.Get(Boolean applyDefaults) in /src/Promitor.Core.Scraping/Providers/MetricsDeclarationProvider.cs:line 29
at Promitor.Scraper.Host.Extensions.IServiceCollectionExtensions.ScheduleMetricScraping(IServiceCollection services) in /src/Promitor.Scraper.Host/IServiceCollectionExtensions.cs:line 46
at Promitor.Scraper.Host.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Scraper.Host/Startup.cs:line 49
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at Promitor.Scraper.Host.Program.BuildWebHost(String[] args) in /src/Promitor.Scraper.Host/Program.cs:line 31
at Promitor.Scraper.Host.Program.Main(String[] args) in /src/Promitor.Scraper.Host/Program.cs:line 16

Can you please help on this?

Answered by jaisegrg

Aug 5, 2020

@tomkerkhove

Deployment yaml file below,

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    meta.helm.sh/release-name: promitor-agent-scraper-apim
    meta.helm.sh/release-namespace: aksmonitor
  creationTimestamp: "2020-08-04T13:35:30Z"
  generation: 4
  labels:
    app: promitor-agent-scraper
    app.kubernetes.io/managed-by: Helm
    chart: promitor-agent-scraper-1.6.0
    heritage: Helm
    release: promitor-agent-scraper-apim
  name: promitor-agent-scraper-apim
  namespace: aksmonitor
  resourceVersion: "11816974"
  selfLink: /apis/extensions/v1beta1/namespaces/aksmonitor/deployments/promitor-agent-scraper-apim
  uid:

View full answer

tomkerkhove · 2020-08-04T14:30:36Z

tomkerkhove
Aug 4, 2020
Maintainer

That's not nice.

Would you mind sharing:

Full logs
Server config
Metrics declaration

0 replies

jaisegrg · 2020-08-04T15:19:32Z

jaisegrg
Aug 4, 2020
Author

configmap.txt
logs.txt
metrics_Declaration.txt

@tomkerkhove,

Please find the files attached.

Full logs
Server config (ConfigMaps)
Metrics declaration

1 reply

tomkerkhove Aug 4, 2020
Maintainer

Since you are using Kubernetes, can you share your deployment YAML please?

jaisegrg · 2020-08-05T03:05:07Z

jaisegrg
Aug 5, 2020
Author

@tomkerkhove,

The steps followed as below,

Helm installation command,

helm install promitor-agent-scraper-apim promitor/promitor-agent-scraper --values ./metric-declaration.yaml

metric-declaration yaml file below,

cat metric-declaration.yaml

azureAuthentication:
  appId: "<appId>"
  appKey: "<appKey>"
azureMetadata:
  tenantId: "<tenantId>"
  subscriptionId: "<subscriptionId>"
  resourceGroupName: "<resourceGroupName>"
metricDefaults:
  aggregation:
    interval: 00:01:00
  scraping:
    schedule: "0 * * ? * *"
metrics:
  - name: promitor_test_azureapimanagement_capacity
    description: "The amount of capacity used an Azure API Management instance."
    resourceType: ApiManagement
    azureMetricConfiguration:
     metricName: Capacity
     aggregation:
       type: Average
    resources:
     - instanceName: "<instanceName>"

0 replies

jaisegrg · 2020-08-05T03:08:44Z

jaisegrg
Aug 5, 2020
Author

@tomkerkhove

Deployment yaml file below,

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    meta.helm.sh/release-name: promitor-agent-scraper-apim
    meta.helm.sh/release-namespace: aksmonitor
  creationTimestamp: "2020-08-04T13:35:30Z"
  generation: 4
  labels:
    app: promitor-agent-scraper
    app.kubernetes.io/managed-by: Helm
    chart: promitor-agent-scraper-1.6.0
    heritage: Helm
    release: promitor-agent-scraper-apim
  name: promitor-agent-scraper-apim
  namespace: aksmonitor
  resourceVersion: "11816974"
  selfLink: /apis/extensions/v1beta1/namespaces/aksmonitor/deployments/promitor-agent-scraper-apim
  uid: 94c2a2ed-feed-4fb6-a500-04c4f0de8e83
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: promitor-agent-scraper
      release: promitor-agent-scraper-apim
      type: runtime
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: 59aeec025eb4412a83fe1dff5052efb689b305e34d39c5fc9521fad852f58798
        checksum/secret: 91f05f2cd7568f445af8055aeb528f47e1063daa1eaa1bd93e109ecccbd2e902
        kubectl.kubernetes.io/restartedAt: "2020-08-04T07:52:25-07:00"
      creationTimestamp: null
      labels:
        app: promitor-agent-scraper
        release: promitor-agent-scraper-apim
        type: runtime
    spec:
      containers:
      - env:
        - name: PROMITOR_AUTH_APPID
          valueFrom:
            secretKeyRef:
              key: azure-app-id
              name: promitor-agent-scraper-apim
        - name: PROMITOR_AUTH_APPKEY
          valueFrom:
            secretKeyRef:
              key: azure-app-key
              name: promitor-agent-scraper-apim
        image: tomkerkhove/promitor-agent-scraper:1.0.0
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/v1/health
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: promitor-agent-scraper
        ports:
        - containerPort: 88
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/v1/health
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config/
          name: config-volume-promitor-agent-scraper-apim
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: promitor-agent-scraper-apim
      serviceAccountName: promitor-agent-scraper-apim
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: config-map-promitor-agent-scraper-apim
        name: config-volume-promitor-agent-scraper-apim
status:
  conditions:
  - lastTransitionTime: "2020-08-04T13:35:30Z"
    lastUpdateTime: "2020-08-04T13:35:30Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-08-04T15:01:44Z"
    lastUpdateTime: "2020-08-04T15:01:44Z"
    message: ReplicaSet "promitor-agent-scraper-apim-8667d8f9" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 4
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

11 replies

jaisegrg Aug 5, 2020
Author

@tomkerkhove,

No, i didnt configure it in runtime also. Only thing tried is disabling it using the parameter "resourceDiscovery.enabled: false".

tomkerkhove Aug 5, 2020
Maintainer

Ok, that's a bug. Sorry about that 👎 #1196

Will release a new alpha version asap.

jaisegrg Aug 5, 2020
Author

@tomkerkhove,

Thank you for the help!!
Please let me know once it is fixed.

tomkerkhove Aug 10, 2020
Maintainer

This should be fixed in https://github.com/tomkerkhove/promitor/releases/tag/2.0.0-preview-2

jaisegrg Aug 11, 2020
Author

Thank you @tomkerkhove !!

The reported issue is fixed in the release https://github.com/tomkerkhove/promitor/releases/tag/2.0.0-preview-2 and i am able to scrape API Management metrics (Metrics Name:- Capacity) from Azure monitor to Prometheus.

jaisegrg · 2020-08-11T07:46:53Z

jaisegrg
Aug 11, 2020
Author

@tomkerkhove,

I am able to scrape metrics "Capacity" for ApiManagement, but other metrics are not working. Getting the below errors,

Error for metrics "Requests",

[07:38:33 INF] Metrics declaration is using spec version v1
[07:38:33 FTL] Promitor Scraper Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
System.ArgumentNullException: [scraping] cannot be Null. (Parameter 'scraping')
   at GuardNet.Guard.For[TException](Func`1 predicate, TException exception)
   at GuardNet.Guard.NotNull[TParam,TException](TParam param, TException exception)
   at GuardNet.Guard.NotNull[TParam](TParam param, String paramName, String message)
   at GuardNet.Guard.NotNull[TParam](TParam param, String paramName)
   at Promitor.Core.Scraping.Configuration.Model.Metrics.ScrapeDefinition`1..ctor(AzureMetricConfiguration azureMetricConfiguration, PrometheusMetricDefinition prometheusMetricDefinition, Scraping scraping, TResourceDefinition resource, String subscriptionId, String resourceGroupName) in /src/Promitor.Core.Scraping/Model/Metrics/ScrapeDefinition.cs:line 34
   at Promitor.Core.Scraping.Configuration.Model.Metrics.MetricDefinition.CreateScrapeDefinition(IAzureResourceDefinition resource, AzureMetadata azureMetadata) in /src/Promitor.Core.Scraping/Model/Metrics/MetricDefinition.cs:line 64
   at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.ScheduleResourceScraping(IAzureResourceDefinition resource, AzureMetadata azureMetadata, MetricDefinition metric, AzureMonitorClientFactory azureMonitorClientFactory, MetricSinkWriter metricSinkWriter, IRuntimeMetricsCollector runtimeMetricCollector, IConfiguration configuration, IOptions`1 azureMonitorLoggingConfiguration, ILoggerFactory loggerFactory, ILogger`1 logger, IServiceCollection services) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 69
   at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.ScheduleMetricScraping(IServiceCollection services) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 34
   at Promitor.Agents.Scraper.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 63
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 33

Metrics Declaration as below,

azureAuthentication:
  appId: "appId"
  appKey: "appKey"
azureMetadata:
  tenantId: "tenantId"
  subscriptionId: "subscriptionId"
  resourceGroupName: "resourceGroupName"
metricDefaults:
  aggregation:
    interval: 00:01:00
  scraping:
    schedule: "0 * * ? * *"
metrics:
  - name: Requests
    description: "Requests"
    resourceType: ApiManagement
    azureMetricConfiguration:
     metricName: Requests
     aggregation:
       type: Sum
    resources:
     - instanceName: "instanceName"

Error for metrics "Duration of Backend Requests",

[07:34:32 INF] Scraping Duration of Backend Requests for resource type ApiManagement
[07:34:33 FTL] Failed to scrape resource for metric 'Duration of Backend Requests'
Promitor.Integrations.AzureMonitor.Exceptions.MetricNotFoundException: The metric 'Duration of Backend Requests' was not found
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter)
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 46
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 69
[07:34:34 INF] Metrics declaration is using spec version v1

1 reply

tomkerkhove Aug 11, 2020
Maintainer

Duration of Backend Requests - You'll need to use BackendDuration instead of the display name (learn more here)
Requests - Odd exception and I'm not sure what's up but your aggregation is at least not correct because it has to be Total (learn more here)

jaisegrg · 2020-08-12T04:00:29Z

jaisegrg
Aug 12, 2020
Author

@tomkerkhove,

Thank you!! it worked and i was able to add the metrics.

Do Promitor have support for scraping Application Gateway metrics from Azure Monitor to Prometheus?
I have tried the same. but failed.

1 reply

tomkerkhove Aug 20, 2020
Maintainer

It's not supported yet but it's coming, see #313.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Promitor scraper not working for resourceType: ApiManagement #1194

{{title}}

Replies: 6 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Promitor scraper not working for resourceType: ApiManagement #1194

jaisegrg Aug 4, 2020

Replies: 6 comments · 14 replies

tomkerkhove Aug 4, 2020 Maintainer

jaisegrg Aug 4, 2020 Author

tomkerkhove Aug 4, 2020 Maintainer

jaisegrg Aug 5, 2020 Author

jaisegrg Aug 5, 2020 Author

jaisegrg Aug 5, 2020 Author

tomkerkhove Aug 5, 2020 Maintainer

jaisegrg Aug 5, 2020 Author

tomkerkhove Aug 10, 2020 Maintainer

jaisegrg Aug 11, 2020 Author

jaisegrg Aug 11, 2020 Author

tomkerkhove Aug 11, 2020 Maintainer

jaisegrg Aug 12, 2020 Author

tomkerkhove Aug 20, 2020 Maintainer

jaisegrg
Aug 4, 2020

Replies: 6 comments 14 replies

tomkerkhove
Aug 4, 2020
Maintainer

jaisegrg
Aug 4, 2020
Author

tomkerkhove Aug 4, 2020
Maintainer

jaisegrg
Aug 5, 2020
Author

jaisegrg
Aug 5, 2020
Author

jaisegrg Aug 5, 2020
Author

tomkerkhove Aug 5, 2020
Maintainer

jaisegrg Aug 5, 2020
Author

tomkerkhove Aug 10, 2020
Maintainer

jaisegrg Aug 11, 2020
Author

jaisegrg
Aug 11, 2020
Author

tomkerkhove Aug 11, 2020
Maintainer

jaisegrg
Aug 12, 2020
Author

tomkerkhove Aug 20, 2020
Maintainer