Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pkg/pdatatest] Do not ignore timestamps in CompareMetrics, use options #17983

Merged

Conversation

dmitryax
Copy link
Member

It's pretty common to ignore timestamps when using CompareMetrics in tests for scrapers, but it's not apparent behavior for other use cases. We should provide options to ignore timestamps explicitly. That will make the Compare... functions provide a clean comparison with no implicit behavior.

Updates #17865

@runforesight
Copy link

runforesight bot commented Jan 24, 2023

Foresight Summary

    
Major Impacts

TestScrape ❌ failed 1 times in 1 runs (100% fail rate).
TestScrape/Fully_successful_scrape ❌ failed 1 times in 1 runs (100% fail rate).
TestScrape/Scrape_with_errors ❌ failed 1 times in 1 runs (100% fail rate).
build-and-test-windows duration(18 minutes 58 seconds) has decreased 21 minutes 46 seconds compared to main branch avg(40 minutes 44 seconds).
View More Details

✅  telemetrygen workflow has finished in 4 minutes 37 seconds (⚠️ 2 minutes 8 seconds more than main branch avg.) and finished at 24th Jan, 2023.


Job Failed Steps Tests
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details
build-dev -     🔗  N/A See Details

✅  tracegen workflow has finished in 4 minutes 49 seconds (⚠️ 2 minutes 21 seconds more than main branch avg.) and finished at 24th Jan, 2023.


Job Failed Steps Tests
build-dev -     🔗  N/A See Details
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  changelog workflow has finished in 5 minutes 22 seconds (⚠️ 3 minutes 17 seconds more than main branch avg.) and finished at 24th Jan, 2023.


Job Failed Steps Tests
changelog -     🔗  N/A See Details

✅  prometheus-compliance-tests workflow has finished in 6 minutes 53 seconds and finished at 24th Jan, 2023.


Job Failed Steps Tests
prometheus-compliance-tests -     🔗  ✅ 21  ❌ 0  ⏭ 0    🔗 See Details

✅  check-links workflow has finished in 8 minutes 34 seconds (⚠️ 6 minutes 42 seconds more than main branch avg.) and finished at 24th Jan, 2023.


Job Failed Steps Tests
changed files -     🔗  N/A See Details
check-links -     🔗  N/A See Details

 build-and-test workflow has finished in 16 minutes 58 seconds (34 minutes 37 seconds less than main branch avg.) and finished at 24th Jan, 2023.


Job Failed Steps Tests
unittest-matrix (1.18, internal) N/A  ✅ 561  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, internal) N/A  ✅ 561  ❌ 0  ⏭ 0    🔗 See Details
correctness-traces N/A  ✅ 17  ❌ 0  ⏭ 0    🔗 See Details
correctness-metrics N/A  ✅ 2  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, processor) N/A  ✅ 1470  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, extension) N/A  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, extension) N/A  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, processor) N/A  ✅ 1470  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-0) N/A  ✅ 2568  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-0) N/A  ✅ 2568  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, exporter) N/A  ✅ 2498  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, exporter) N/A  ✅ 2498  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, other) N/A  ✅ 4630  ❌ 0  ⏭ 0    🔗 See Details
integration-tests N/A  ✅ 55  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, other) N/A  ✅ 4630  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-1) N/A  ✅ 1895  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-1) N/A  ✅ 1895  ❌ 0  ⏭ 0    🔗 See Details

✅  load-tests workflow has finished in 16 minutes 58 seconds (⚠️ 2 minutes 38 seconds more than main branch avg.) and finished at 24th Jan, 2023.


Job Failed Steps Tests
loadtest (TestTraceAttributesProcessor) -     🔗  ✅ 3  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestIdleMode) -     🔗  ✅ 1  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) -     🔗  ✅ 10  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) -     🔗  ✅ 8  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile) -     🔗  ✅ 6  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) -     🔗  ✅ 12  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestBallastMemory|TestLog10kDPS) -     🔗  ✅ 19  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details

❌  build-and-test-windows workflow has finished in 18 minutes 58 seconds (21 minutes 46 seconds less than main branch avg.) and finished at 24th Jan, 2023. 1 job failed. There are 3 test failures.


Job Failed Steps Tests
windows-unittest-matrix (internal) -     🔗  ✅ 525  ❌ 0  ⏭ 0    🔗 See Details
windows-unittest-matrix (extension) -     🔗  ✅ 515  ❌ 0  ⏭ 0    🔗 See Details
windows-unittest-matrix (receiver-0) Run Unit tests     🔗  ✅ 82  ❌ 3  ⏭ 0    🔗 See Details
windows-unittest-matrix (other) -     🔗  ✅ 81  ❌ 0  ⏭ 0    🔗 See Details
windows-unittest-matrix (exporter) -     🔗  ✅ 692  ❌ 0  ⏭ 0    🔗 See Details
windows-unittest-matrix (receiver-1) -     🔗  ✅ 377  ❌ 0  ⏭ 0    🔗 See Details
windows-unittest-matrix (processor) -     🔗  ✅ 431  ❌ 0  ⏭ 0    🔗 See Details
windows-unittest -     🔗  N/A See Details

🔎 See details on Foresight

*You can configure Foresight comments in your organization settings page.

@dmitryax dmitryax force-pushed the pmetric-test-dont-ignore-timestamps branch 7 times, most recently from f780be6 to de7941d Compare January 24, 2023 07:38
@dmitryax dmitryax force-pushed the pmetric-test-dont-ignore-timestamps branch 3 times, most recently from 0154cf6 to 0feee66 Compare January 24, 2023 17:27
@dmitryax dmitryax added the Run Windows Enable running windows test on a PR label Jan 24, 2023
@dmitryax
Copy link
Member Author

Windows tests are blocked by #17999

It's pretty common to ignore timestamps when using CompareMetrics in tests for scrapers, but it's not obvious behavior for other use cases. We should provide options to explicitly ignore timestamps. That will make the Compare... functions provide clean comparison with not implicit behavior.
@dmitryax dmitryax force-pushed the pmetric-test-dont-ignore-timestamps branch from 0feee66 to 4878248 Compare January 24, 2023 19:28
@bogdandrutu bogdandrutu merged commit 27e1370 into open-telemetry:main Jan 24, 2023
@dmitryax dmitryax deleted the pmetric-test-dont-ignore-timestamps branch January 24, 2023 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants