Incorrect count returned in azure_devops_stats_project_builds

[roshini-cp]

Hi,
I'm using azure_stats_project_builds metric to count the number of succeded builds in a specific project for a specific builddefinitionid, however I'm observing that count returned in this metrics is incorrect for my project, it seems like a pattern is getting followed here.

One of the scenario I have observed is:

azure_devops_stats_project_builds{buildDefinitionID="XXX",projectID:"XXX", result="succeeded"} 2
this is when the exporter is initailly installed for the collection (there were no build running/created in this project during installation)

after 30 mins the same metric counter is increased by 2 (there is no build that is succeeded in the past 30 mins) azure_devops_stats_project_builds{buildDefinitionID="XXX",projectID:"XXX", result="succeeded"} 4

Now a build gets succeeded in the project, the count gets increased to 7
azure_devops_stats_project_builds{buildDefinitionID="XXX",projectID:"XXX", result="succeeded"} 7

and agin after 30 mins metric shows the value as 10 (again there is no build that is succeded or created/queued/cancelled in that project)
azure_devops_stats_project_builds{buildDefinitionID="XXX",projectID:"XXX", result="succeeded"} 10

(similar observations for failed/cancelled builds even with the different projects)

What could be the issue here?

[davejhahn]

@roshini-cp I'm also trying to understand these numbers. From what I can see, it is a perpetual increase. None of the metrics I have looked at make any sense to me.

I just want to be able to see the # of builds, success, failures, etc... over time, be able to identify problems (e.g. an increase in failures), but I look at a short of time, like 1 hour, I see everything, and if you graph it it just increases.

If you look at the raw metrics from the exporter, it just has everything. I tried adjusting the durations for the exporter to a 5m window (--scrape.time=5m) but didn't seem to make a difference.

I tried one of the example queries, to get the current running jobs, and what it shows when graphed is a scale of 1.7G, and the listed jobs are not running. The queries are doing a bunch of joins also, not sure why that would be needed.

I'm guessing it is something I am doing wrong, just no idea what.

[mblaschke]

these metrics are counters, you have to use increase(azure_devops_stats_project_builds[1h]) to get builds per hour similar to all other counters in Prometheus.

https://prometheus.io/docs/concepts/metric_types/#counter

[roshini-cp]

HI @mblaschke even with increase or any such functions we get the incorrect value, due to the count getting increased as I tried to describe it initially when I opened this issue.