Azure monitor scraper #18898

altuner · 2023-02-23T21:54:57Z

Description:
The new receiver allows to collect Azure Monitor metrics per a subscription/resource group(s) by using Azure REST API.

Link to tracking Issue: #18899

Example configuration for the component

receivers:
  azuremonitor:
    subscription_id: "${subscription_id}"
    tenant_id: "${tenant_id}"
    client_id: "${client_id}"
    client_secret: "${env:CLIENT_SECRET}"
    resource_groups:
      - "${resource_group1}"
      - "${resource_group2}"
    services:
      - "${service1}"
      - "${service2}"
    collection_interval: 60s

Example of metrics output:

codeboten

Thanks for the contrib @altuner, can you open a new component proposal if one doesn't already exist: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components

I looked for one but couldn't find it.

runforesight · 2023-03-13T07:04:17Z

Foresight Summary

Major Impacts

build-and-test duration(40 minutes 9 seconds) has decreased 6 minutes 25 seconds compared to main branch avg(46 minutes 34 seconds).

View More Details

⭕ build-and-test-windows workflow has finished in 5 seconds (30 minutes 25 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
windows-unittest-matrix	- 🔗	N/A	See Details
windows-unittest	- 🔗	N/A	See Details

✅ telemetrygen workflow has finished in 59 seconds and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
build-dev	- 🔗	N/A	See Details
publish-latest	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details

✅ check-links workflow has finished in 1 minute 26 seconds and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
changed files	- 🔗	N/A	See Details
check-links	- 🔗	N/A	See Details

✅ changelog workflow has finished in 2 minutes 17 seconds and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
changelog	- 🔗	N/A	See Details

✅ prometheus-compliance-tests workflow has finished in 3 minutes 18 seconds (3 minutes less than `main` branch avg.) and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
prometheus-compliance-tests	- 🔗	N/A	See Details

✅ load-tests workflow has finished in 6 minutes 31 seconds (3 minutes 58 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
setup-environment	- 🔗	N/A	See Details
loadtest (TestIdleMode)	- 🔗	N/A	See Details
loadtest (TestBallastMemory\|TestLog10kDPS)	- 🔗	N/A	See Details
loadtest (TestMetric10kDPS\|TestMetricsFromFile)	- 🔗	N/A	See Details
loadtest (TestMetricResourceProcessor\|TestTrace10kSPS)	- 🔗	N/A	See Details
loadtest (TestTraceNoBackend10kSPS\|TestTrace1kSPSWithAttrs)	- 🔗	N/A	See Details
loadtest (TestTraceBallast1kSPSWithAttrs\|TestTraceBallast1kSPSAddAttrs)	- 🔗	N/A	See Details
loadtest (TestTraceAttributesProcessor)	- 🔗	N/A	See Details

✅ e2e-tests workflow has finished in 13 minutes 10 seconds and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
kubernetes-test (v1.25.3)	- 🔗	N/A	See Details
kubernetes-test (v1.24.7)	- 🔗	N/A	See Details
kubernetes-test (v1.26.0)	- 🔗	N/A	See Details
kubernetes-test (v1.23.13)	- 🔗	N/A	See Details

✅ build-and-test workflow has finished in 40 minutes 9 seconds (6 minutes 25 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

Job	Failed Steps	Tests
govulncheck	- 🔗	N/A	See Details
setup-environment	- 🔗	N/A	See Details
check-collector-module-version	- 🔗	N/A	See Details
check-codeowners	- 🔗	N/A	See Details
build-examples	- 🔗	N/A	See Details
checks	- 🔗	N/A	See Details
correctness-metrics	- 🔗	N/A	See Details
correctness-traces	- 🔗	N/A	See Details
integration-tests	- 🔗	N/A	See Details
lint-matrix (receiver-0)	- 🔗	N/A	See Details
lint-matrix (receiver-1)	- 🔗	N/A	See Details
lint-matrix (processor)	- 🔗	N/A	See Details
lint-matrix (exporter)	- 🔗	N/A	See Details
lint-matrix (extension)	- 🔗	N/A	See Details
lint-matrix (connector)	- 🔗	N/A	See Details
lint-matrix (internal)	- 🔗	N/A	See Details
lint-matrix (other)	- 🔗	N/A	See Details
unittest-matrix (1.20, receiver-0)	- 🔗	N/A	See Details
unittest-matrix (1.20, receiver-1)	- 🔗	N/A	See Details
unittest-matrix (1.20, processor)	- 🔗	N/A	See Details
unittest-matrix (1.20, exporter)	- 🔗	N/A	See Details
unittest-matrix (1.20, extension)	- 🔗	N/A	See Details
unittest-matrix (1.20, connector)	- 🔗	N/A	See Details
unittest-matrix (1.20, internal)	- 🔗	N/A	See Details
unittest-matrix (1.20, other)	- 🔗	N/A	See Details
unittest-matrix (1.19, receiver-0)	- 🔗	N/A	See Details
unittest-matrix (1.19, receiver-1)	- 🔗	N/A	See Details
unittest-matrix (1.19, processor)	- 🔗	N/A	See Details
unittest-matrix (1.19, exporter)	- 🔗	N/A	See Details
unittest-matrix (1.19, extension)	- 🔗	N/A	See Details
unittest-matrix (1.19, connector)	- 🔗	N/A	See Details
unittest-matrix (1.19, internal)	- 🔗	N/A	See Details
unittest-matrix (1.19, other)	- 🔗	N/A	See Details
unittest (1.20)	- 🔗	N/A	See Details
unittest (1.19)	- 🔗	N/A	See Details
lint	- 🔗	N/A	See Details
cross-compile (darwin, amd64)	- 🔗	N/A	See Details
cross-compile (darwin, arm64)	- 🔗	N/A	See Details
cross-compile (linux, 386)	- 🔗	N/A	See Details
cross-compile (linux, amd64)	- 🔗	N/A	See Details
cross-compile (linux, arm)	- 🔗	N/A	See Details
cross-compile (linux, arm64)	- 🔗	N/A	See Details
cross-compile (linux, ppc64le)	- 🔗	N/A	See Details
cross-compile (windows, 386)	- 🔗	N/A	See Details
cross-compile (windows, amd64)	- 🔗	N/A	See Details
build-package (deb)	- 🔗	N/A	See Details
build-package (rpm)	- 🔗	N/A	See Details
windows-msi	- 🔗	N/A	See Details
publish-check	- 🔗	N/A	See Details
publish-dev	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details
rotate-milestone	- 🔗	N/A	See Details

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}

receiver/azuremonitorreceiver/go.mod

receiver/azuremonitorreceiver/internal/configazure/configazure.go

receiver/azuremonitorreceiver/metadata.yaml

receiver/azuremonitorreceiver/internal/metadata/generated_metrics.go

receiver/azuremonitorreceiver/internal/configazure/configazure.go

receiver/azuremonitorreceiver/config.go

nslaughter

Just a couple of thoughts on simplifying.

receiver/azuremonitorreceiver/scraper.go

nslaughter

Good work. Thanks for submitting this.

altuner · 2023-04-18T17:05:54Z

@rajim17 I've added the dimensions change

mwear

LGTM. Is it true that all azure metrics map to OTLP as gauges?

cartersocha · 2023-04-19T00:03:33Z

@trask @reyang @lmolkova could a contributor from the Azure team potentially take a look at this?

altuner · 2023-04-19T01:02:47Z

LGTM. Is it true that all azure metrics map to OTLP as gauges?

@mwear good question, technically Azure provides data with different aggregations by default, which suits more "histogram", however since "histogram" in otlp doesn't have "average" option, I've picked "gauge" and I submit every aggregation type as separate value with "_aggregation" suffix. Please let me know if that's not correct approach.

linux-foundation-easycla · 2023-04-19T06:15:42Z

The committers listed above are authorized under a signed CLA.

✅ login: altuner / name: Dmitry Av (2df2008, e790f5e, d76b805, a9d0af9, 97ad16b, 954b16b, 893af3b, 3f437bd, 899ffb2, bc3ba31, 31cbf9c, bec660d)
✅ login: codeboten / name: Alex Boten (bec660d)

receiver/azuremonitorreceiver/internal/metadata/metrics.go

codeboten · 2023-04-12T22:40:16Z

receiver/azuremonitorreceiver/config.go

+ }
+)
+
+// Config defines the configuration for the various elements of the receiver agent.


Suggested change

// Config defines the configuration for the various elements of the receiver agent.

// Config defines the configuration for the various elements of the receiver.

codeboten · 2023-04-12T22:45:56Z

receiver/azuremonitorreceiver/scraper.go

+ }
+}
+
+func (s *azureScraper) processTimeserieData(resourceID string, metric *armmonitor.Metric, timeserieData *armmonitor.MetricValue) {


Suggested change

func (s *azureScraper) processTimeserieData(resourceID string, metric *armmonitor.Metric, timeserieData *armmonitor.MetricValue) {

func (s *azureScraper) processTimeseriesData(resourceID string, metric *armmonitor.Metric, timeseriesData *armmonitor.MetricValue) {

receiver/azuremonitorreceiver/metadata.yaml

receiver/azuremonitorreceiver/internal/metadata/generated_status.go

receiver/azuremonitorreceiver/factory.go

altuner requested a review from a team as a code owner February 23, 2023 21:54

altuner requested a review from mx-psi February 23, 2023 21:54

github-actions bot assigned TylerHelmuth Feb 23, 2023

altuner force-pushed the receiver/azuremonitorreceiver branch from f9aa2c8 to 0177696 Compare February 23, 2023 21:59

altuner marked this pull request as draft February 23, 2023 22:04

codeboten reviewed Feb 23, 2023

View reviewed changes

altuner force-pushed the receiver/azuremonitorreceiver branch from 0177696 to 0195768 Compare March 10, 2023 23:10

cparkins mentioned this pull request Mar 14, 2023

[receiver/azureeventhub] add metric support #19436

Merged

altuner force-pushed the receiver/azuremonitorreceiver branch 2 times, most recently from f877f77 to 1645a96 Compare March 23, 2023 21:15

github-actions bot added cmd/configschema configschema command cmd/otelcontribcol otelcontribcol command labels Mar 23, 2023

github-actions bot requested review from dmitryax and pmcollins March 23, 2023 21:15

altuner changed the title ~~[WIP] Azure monitor scraper~~ Azure monitor scraper Mar 23, 2023

altuner marked this pull request as ready for review March 23, 2023 21:19

github-actions bot assigned bogdandrutu Mar 23, 2023

altuner force-pushed the receiver/azuremonitorreceiver branch from 1645a96 to 5ada9d4 Compare March 23, 2023 21:30

codeboten reviewed Mar 24, 2023

View reviewed changes

altuner force-pushed the receiver/azuremonitorreceiver branch from 5ada9d4 to 2981edc Compare March 27, 2023 22:08

nslaughter suggested changes Mar 29, 2023

View reviewed changes

receiver/azuremonitorreceiver/scraper.go Outdated Show resolved Hide resolved

receiver/azuremonitorreceiver/scraper.go Outdated Show resolved Hide resolved

altuner force-pushed the receiver/azuremonitorreceiver branch 8 times, most recently from 7704820 to dc60054 Compare April 4, 2023 19:06

altuner force-pushed the receiver/azuremonitorreceiver branch from 08f828c to 9345670 Compare April 17, 2023 20:56

nslaughter approved these changes Apr 18, 2023

View reviewed changes

mwear approved these changes Apr 18, 2023

View reviewed changes

altuner force-pushed the receiver/azuremonitorreceiver branch 2 times, most recently from bec660d to 1c7e46d Compare April 19, 2023 23:15

TylerHelmuth reviewed Apr 20, 2023

View reviewed changes

receiver/azuremonitorreceiver/internal/metadata/metrics.go Show resolved Hide resolved

codeboten reviewed Apr 21, 2023

View reviewed changes

codeboten approved these changes Apr 21, 2023

View reviewed changes

altuner and others added 13 commits April 21, 2023 12:15

Azure monitor scraper

cbe950f

Added filter for services

1007a12

addressing comments

0797e5a

add new component into dependabot.yml

8b506c6

added into templates

ee9c80b

addressing comments

5590ecc

updated dependencies

fd5d36d

addressing comments

534a4ab

added golden copy

3d89916

added dimensions

f58faea

comments addressed

cf3497b

Update receiver/azuremonitorreceiver/README.md

20abc7b

update type to azuremonitor

9a737c8

altuner force-pushed the receiver/azuremonitorreceiver branch from d2cccbf to 7685c84 Compare April 21, 2023 19:18

small fixes

a9fa1ba

altuner force-pushed the receiver/azuremonitorreceiver branch from 7685c84 to a9fa1ba Compare April 21, 2023 21:19

codeboten merged commit 97b4d5c into open-telemetry:main Apr 24, 2023

github-actions bot added this to the next release milestone Apr 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure monitor scraper #18898

Azure monitor scraper #18898

altuner commented Feb 23, 2023 •

edited

Loading

codeboten left a comment

runforesight bot commented Mar 13, 2023 •

edited

Loading

`build-and-test` duration(40 minutes 9 seconds) has decreased 6 minutes 25 seconds compared to main branch avg(46 minutes 34 seconds).

⭕ build-and-test-windows workflow has finished in 5 seconds (30 minutes 25 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

✅ telemetrygen workflow has finished in 59 seconds and finished at 14th Apr, 2023.

✅ check-links workflow has finished in 1 minute 26 seconds and finished at 14th Apr, 2023.

✅ changelog workflow has finished in 2 minutes 17 seconds and finished at 14th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 3 minutes 18 seconds (3 minutes less than `main` branch avg.) and finished at 14th Apr, 2023.

✅ load-tests workflow has finished in 6 minutes 31 seconds (3 minutes 58 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

✅ e2e-tests workflow has finished in 13 minutes 10 seconds and finished at 14th Apr, 2023.

✅ build-and-test workflow has finished in 40 minutes 9 seconds (6 minutes 25 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

nslaughter left a comment

nslaughter left a comment

altuner commented Apr 18, 2023

mwear left a comment

cartersocha commented Apr 19, 2023 •

edited

Loading

altuner commented Apr 19, 2023

linux-foundation-easycla bot commented Apr 19, 2023 •

edited

Loading

codeboten Apr 12, 2023

codeboten Apr 12, 2023

	// Config defines the configuration for the various elements of the receiver agent.
	// Config defines the configuration for the various elements of the receiver.

	func (s azureScraper) processTimeserieData(resourceID string, metric armmonitor.Metric, timeserieData *armmonitor.MetricValue) {
	func (s azureScraper) processTimeseriesData(resourceID string, metric armmonitor.Metric, timeseriesData *armmonitor.MetricValue) {

Azure monitor scraper #18898

Azure monitor scraper #18898

Conversation

altuner commented Feb 23, 2023 • edited Loading

Example configuration for the component

codeboten left a comment

Choose a reason for hiding this comment

runforesight bot commented Mar 13, 2023 • edited Loading

Foresight Summary

build-and-test duration(40 minutes 9 seconds) has decreased 6 minutes 25 seconds compared to main branch avg(46 minutes 34 seconds).

⭕ build-and-test-windows workflow has finished in 5 seconds (30 minutes 25 seconds less than main branch avg.) and finished at 14th Apr, 2023.

✅ telemetrygen workflow has finished in 59 seconds and finished at 14th Apr, 2023.

✅ check-links workflow has finished in 1 minute 26 seconds and finished at 14th Apr, 2023.

✅ changelog workflow has finished in 2 minutes 17 seconds and finished at 14th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 3 minutes 18 seconds (3 minutes less than main branch avg.) and finished at 14th Apr, 2023.

✅ load-tests workflow has finished in 6 minutes 31 seconds (3 minutes 58 seconds less than main branch avg.) and finished at 14th Apr, 2023.

✅ e2e-tests workflow has finished in 13 minutes 10 seconds and finished at 14th Apr, 2023.

✅ build-and-test workflow has finished in 40 minutes 9 seconds (6 minutes 25 seconds less than main branch avg.) and finished at 14th Apr, 2023.

nslaughter left a comment

Choose a reason for hiding this comment

nslaughter left a comment

Choose a reason for hiding this comment

altuner commented Apr 18, 2023

mwear left a comment

Choose a reason for hiding this comment

cartersocha commented Apr 19, 2023 • edited Loading

altuner commented Apr 19, 2023

linux-foundation-easycla bot commented Apr 19, 2023 • edited Loading

codeboten Apr 12, 2023

Choose a reason for hiding this comment

codeboten Apr 12, 2023

Choose a reason for hiding this comment

altuner commented Feb 23, 2023 •

edited

Loading

runforesight bot commented Mar 13, 2023 •

edited

Loading

`build-and-test` duration(40 minutes 9 seconds) has decreased 6 minutes 25 seconds compared to main branch avg(46 minutes 34 seconds).

⭕ build-and-test-windows workflow has finished in 5 seconds (30 minutes 25 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 3 minutes 18 seconds (3 minutes less than `main` branch avg.) and finished at 14th Apr, 2023.

✅ load-tests workflow has finished in 6 minutes 31 seconds (3 minutes 58 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

✅ build-and-test workflow has finished in 40 minutes 9 seconds (6 minutes 25 seconds less than `main` branch avg.) and finished at 14th Apr, 2023.

cartersocha commented Apr 19, 2023 •

edited

Loading

linux-foundation-easycla bot commented Apr 19, 2023 •

edited

Loading