feat: Provide support for batch scraping

[hkfgo]

Hey Tom, hope all is well!

I've attached this diagram to illustrate my proposed change.

How to Batch?

Scrape jobs are currently defined per <resource, metric>. I'm taking the simplest approach to just batch resources looking for the same metric.

Max batch size will be a configurable parameter, under runtime configuration.

Batch Job Execution Flow

Changes to Internal Metrics

Internal Promitor metrics will need to be modified under batch scraping mode.
promitor_scrape_success and promitor_scrape_error metrics should be appended with an additional tag batched, to indicate the mode of operation(0/1)
A histogram metric promitor_batch_size should be added to track actual batch sizes
promitor_ratelimit_arm_throttled/promitor_ratelimit_arm metrics will no longer be published, since batch scraping does not go through ARM.

Other considerations

Batch scraping will not be free anymore but comes with other benefits(like no longer being subject to ARM throttling). This should be documented for all users

Implements #2284

[github-actions]

Thank you for your contribution! 🙏 We will review it as soon as possible.

[tomkerkhove]

Thanks! Is it safe to say that the top of the diagram is current and bottom shows these changes?

[tomkerkhove]

Good start, added a few remarks but no concerns!

[hkfgo]

Hey I'm revisiting this PR now since SDK migration is almost across the finish line. Do you know if the batch scraping API is available in sovereign clouds and what URLs we should use? We are particularly interested in the US Gov cloud. I can't find any information online(Azure doesn't really mentioned the existence of sovereign clouds to begin with)

[tomkerkhove]

Do you know if the batch scraping API is available in sovereign clouds and what URLs we should use?

I do not know, sorry! What is the URL for public cloud?

[tomkerkhove]

• US Government cloud : .metrics.monitor.azure.us
  • Token audience: https://metrics.monitor.azure.us
• China cloud : .metrics.monitor.azure.cn
  • Token audience: https://metrics.monitor.azure.cn

[hkfgo]

• https://metrics.monitor.azure.cn

Awesome! How did you find this out?

[tomkerkhove]

I reached out to the team :D

[tomkerkhove]

@hkfgo Do you know if you'll have time to finalize this PR next week? I'd like to cut a release next week.

[hkfgo]

@hkfgo Do you know if you'll have time to finalize this PR next week? I'd like to cut a release next week.

We should plan for batch scraping to be on the release after this coming one. There are still a decent amount of work left and I'm not able to work on this full time.

I do have time to get SDK migration done though. In that case we should have something major-ish to release :)

[tomkerkhove]

That's OK - Thank you!

[tomkerkhove]

@hkfgo Any thoughts on this PR? Are you able to continue working on it?

[hkfgo]

@hkfgo Any thoughts on this PR? Are you able to continue working on it?

Hey I got caught up in some other work and took a few weeks PTO. I'm back working on it now :)

[hkfgo]

Any thoughts on this btw: #2459 (comment)?

[hkfgo]

Alright, I've cleaned up the code and addressed all comments! This PR should be in a mergeable state now, besides that the Scraper CI is still failing. I created an issue to track that. I think the tests are just flaky since you can see them passing on recent runs: https://dev.azure.com/tomkerkhove/Promitor/_build?definitionId=50&_a=summary

[hkfgo]

Oh and I'm working on the public documentation at the moment. PR for that will coming soon.

[tomkerkhove]

LGTM, two more comments

[tomkerkhove]

Great! We're ready to go, let me know when the CI is passing 👌