[PromQL] How to use these metrics?

[samuel-begin]

I've managed to find some useful visualizations of our devops data, but it has been hard due to the nature of the format in which this exporter scrapes.

I managed to list average build times per definition

avg(
  azure_devops_build_status{type="jobDuration"}
    + on(buildID) group_left(buildDefinitionID) (0 *azure_devops_build_info)
) by (buildDefinitionID)

And Ive had a hard time finding consistent data for current build count. for exemaple, these two metrics give off different data (probably due to the scrape timing being different)

azure_devops_stats{name="Build"}

rate(azure_devops_agentpool_builds[2m])

Would you consider adding some query samples or even a simple grafana dashboard to get us started exploring your metrics exporter?

[mblaschke]

Currently we're exporting most informations which are available. But Azure DevOps can delete builds/releases so there can be a difference for the count.

azure_devops_stats is counting the last builds from last scraping time which is more reliable.

Will add some examples later :)

PS: Any metrics you need?

[gandhiprachi]

+1 for this request. it will be helpful to have some sample prometheus queries in the docs for this repo.
Here's a few things i'm trying to collect:
-average queuetime per agent
-average build time per build
-plot of success/failure on daily basis per build.
-some visualization that shows which agents are busy over time and which agents are not fully utilized. This would be very helpful in capacity estimation.

The last one would be very useful for us.

[samuel-begin]

Here's some metrics that I think would be useful.
@gandhiprachi had good ones, so ill iterate on them

• average queue time
• build/release count in queue
• current build/release count sorted by agent (the busy meter)
• average build/release time
  • by build definition
  • by agent
• success/failure
  • by build definition
  • by agent

Additionally, do you have any tips on configuration which would help with the accuracy or usability of the exported metrics?

TIA, @mblaschke, for your time!

After our exploration phase, I will share our queries a/o grafana dashboard to contribute

[samuel-begin]

Ive played a bit with promql and have come up with a few useful metrics in this dashboard: https://gist.github.com/samuel-begin/a67ca79c5d46baa94c57f360010172ee

at this point, i have the following metrics:

• current build status (failed or succeeded) with version number
• finished build count grouped by status (which gives an estimate of parallel build count)
• some network metrics of my agent (coming from another datasource)
• build times by definition (only if succeeded)
• build success rate by definition

[mblaschke]

Still working on more statistic metrics

[mblaschke]

With 0.8.0-beta1 i've added:

Metric	Scraper	Description
azure_devops_stats_agentpool_builds	stats	Number of buildsper agentpool, project and result (counter)
azure_devops_stats_agentpool_builds_wait	stats	Build wait time per agentpool, project and result (summary)
azure_devops_stats_agentpool_builds_duration	stats	Build duration per agentpool, project and result (summary)
azure_devops_stats_project_builds	stats	Number of builds per project, definition and result (counter)
azure_devops_stats_project_builds_wait	stats	Build wait time per project, definition and result (summary)
azure_devops_stats_project_builds_duration	stats	Build duration per project, definition and result (summary)

But removed azure_devops_agentpool_builds and azure_devops_agentpool_wait.
Should help a lot for getting better statistics.

[chelnak]

@mblaschke I'm running beta3 and don't see the metrics you mention above.. or am i missing something obvious?

[viniston]

@mblaschke I'm not sure how to use this azure-devops-exporter. The application is running in my container and I'm seeing it is pulling enough details from Azure DevOps.
INFO : collector_base.go:63: collector[AgentPool]: finished metrics collection (duration: 35.954ms) INFO : collector_base.go:54: collector[AgentPool]: starting metrics collection

But, the metrics are not showing in Prometheus (http:https://localhost:9090/graph). Can you please help me how to use this metrics in grafana?

[chelnak]

@viniston Have you a job to your Prometheus config? E.g

  - job_name: azure-devops-exporter
    honor_timestamps: true
    scrape_interval: 1m
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - prometheus-azure-devops-exporter.default.svc:8080

[viniston]

@chelnak thank you very much for your help. I don't have any job like that. I will try to configure it as you mentioned above and will post my status here.

[viniston]

@viniston Have you a job to your Prometheus config? E.g

  - job_name: azure-devops-exporter
    honor_timestamps: true
    scrape_interval: 1m
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - prometheus-azure-devops-exporter.default.svc:8080

Hi @chelnak , I'm seeing below error in Prometheus UI

Promotheus YAML config file:

Please help me to understand where I'm doing wrong.

[chelnak]

Assuming that you have all of this running in k8s you might want to check your target address.. The one in my example is specific to how I named the exporter service and the namespace i am using.

[viniston]

@chelnak I have azure-devops-exporter locally running in docker container where my Prometheus also running. Do we need to see those new metric should display in Prometheus as well?

address="prometheus-azure-devops-exporter.default.svc:8080" Is this my container localhost endpoint?

[viniston]

@chelnak I have found my issue which was wrong exporter URL and not it is fixed. 👍

[viniston]

Assuming that you have all of this running in k8s you might want to check your target address.. The one in my example is specific to how I named the exporter service and the namespace i am using.

Thank you for your valuable reply and you saved my day.

[mblaschke]

I've added some promql queries in the README, hope that helps.
Please open a new issue if more help is needed.

Thanks for all replies and answers here, you're great :)