Generalize testbed #1062

kbrockhoff · 2020-06-01T20:08:40Z

Description: Extracted out TestResultsSummary (in testbed/testbed/results.go), DataProvider (in testbed/testbed/data_provider.go), OtelcolRunner (in testbed/testbed/otelcol_runner.go), TestCaseValidator (in testbed/testbed/validator.go) interfaces with multiple implementations. Added tracing correctness tests in testbed/correctness using the testbed with different implementations of the 5 interfaces listed than what the perf tests use.

Link to tracking Issue: Provides the support to cleanly implement #652, #1022, #1023, #1027, #1031

Testing: All existing testbed-driven tests still pass. Correctness tests run without any panics. Correctness tests are reporting a number of bugs with translations.

Documentation: Godocs on all public methods.

codecov · 2020-06-01T20:14:47Z

Codecov Report

Merging #1062 into master will increase coverage by 0.12%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1062      +/-   ##
==========================================
+ Coverage   86.73%   86.85%   +0.12%     
==========================================
  Files         201      201              
  Lines       14513    14518       +5     
==========================================
+ Hits        12588    12610      +22     
+ Misses       1471     1458      -13     
+ Partials      454      450       -4

Impacted Files	Coverage Δ
service/service.go	`52.06% <100.00%> (+1.54%)`	⬆️
service/builder/exporters_builder.go	`72.85% <0.00%> (+2.85%)`	⬆️
translator/internaldata/resource_to_oc.go	`76.47% <0.00%> (+2.94%)`	⬆️
internal/collector/telemetry/telemetry.go	`87.93% <0.00%> (+3.44%)`	⬆️
exporter/loggingexporter/logging_exporter.go	`83.68% <0.00%> (+3.54%)`	⬆️
service/telemetry.go	`88.88% <0.00%> (+3.70%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f134f50...6f6545e. Read the comment docs.

tigrannajaryan

Thanks for undertaking this.

testbed/testbed/data_providers.go

linux-foundation-easycla · 2020-06-05T20:03:29Z

The committers are authorized under a signed CLA.

✅ Kevin Brockhoff (3a683a6, 9e9613d, 90b8605, 2557a0f, 49440b9, 36bafbc)

testbed/correctness/correctness_test.go

testbed/testbed/child_process.go

testbed/correctness/correctness_test.go

tigrannajaryan · 2020-06-09T15:30:14Z

testbed/correctness/correctness_test.go

+ tc.ValidateData()
+}
+
+func createConfigFile(t *testing.T, sender testbed.DataSender, receiver testbed.DataReceiver, resultDir string,


Is this a copy of createConfigFile from scenarios.go or is different from it? If it is a copy perhaps extract as testbed helper func.

I wonder if it would make things easier to create a configmodels.Config directly instead of building the yaml string then converting that into a configmodels.Config.

This is a different configuration than used by perf tests. Generating string and converting to Config because functionality does not currently exist to write out Config back to a string.

testbed/testbed/child_process.go

tigrannajaryan

This is an excellent and very useful PR. I left some comments to refine it, let's get it merged asap. I believe it is going to help us catch a lot of correctness problems (as you posted it found a bunch, now we need to analyse them).
Can we run the correctness test as part of the build, but ignore the failures for now?

testbed/README.md

testbed/testbed/test_case.go

testbed/testbed/validator.go

kbrockhoff · 2020-06-16T14:27:08Z

Commented out the test assertion for zero span diffs so tests are not failed because of diffs but the report is still created.

Added correctness tests into the Makefile testbed-runtests and testbed-listtests targets.

tigrannajaryan

Thanks for the replies. Only a couple open comments remaining, please address.

testbed/testbed/otelcol_runner.go

testbed/correctness/correctness_test.go

testbed/testbed/otelcol_runner.go

tigrannajaryan

LGTM, only minor output formatting issue left.

tigrannajaryan · 2020-06-17T22:23:14Z

@kbrockhoff Thanks a lot! This is a much wanted contribution.

…telemetry#1062) Extracted out TestResultsSummary (in testbed/testbed/results.go), DataProvider (in testbed/testbed/data_provider.go), OtelcolRunner (in testbed/testbed/otelcol_runner.go), TestCaseValidator (in testbed/testbed/validator.go) interfaces with multiple implementations. Added tracing correctness tests in testbed/correctness using the testbed with different implementations of the 5 interfaces listed than what the perf tests use. **Link to tracking Issue:** Provides the support to cleanly implement open-telemetry#652, open-telemetry#1022, open-telemetry#1023, open-telemetry#1027, open-telemetry#1031 **Testing:** All existing testbed-driven tests still pass. Correctness tests run without any panics. Correctness tests are reporting a number of bugs with translations. **Documentation:** Godocs on all public methods.

* Integration tests for OTel Collector Attributes. * Update CHANGELOG.md * Fix review * Fix review. * Fix test. * Fix test. Co-authored-by: Tyler Yahn <[email protected]>

kbrockhoff added 2 commits May 29, 2020 14:47

refactor out TestResultSummary

3a683a6

refactor out DataProvider

9e9613d

kbrockhoff requested review from bogdandrutu, dmitryax, flands, owais, pjanotti, rghetia, songy23 and tigrannajaryan as code owners June 1, 2020 20:08

tigrannajaryan self-assigned this Jun 2, 2020

tigrannajaryan added this to In progress in Collector via automation Jun 2, 2020

tigrannajaryan reviewed Jun 3, 2020

View reviewed changes

testbed/testbed/data_providers.go Outdated Show resolved Hide resolved

kbrockhoff added 4 commits June 5, 2020 08:30

refactor out OtelcolRunner

90b8605

Merge branch 'master' into generalize-testbed

2557a0f

update to latest upstream

49440b9

initial dev of correctness tests

36bafbc

kbrockhoff added 2 commits June 6, 2020 11:05

Merge branch 'master' into generalize-testbed

55275e4

Refactor NewTestCase to specify DataProvider and OtelcolRunner

9d9ac71

tigrannajaryan mentioned this pull request Jun 9, 2020

Add E2E receiver/export correctness tests #652

Closed

6 tasks

tigrannajaryan reviewed Jun 9, 2020

View reviewed changes

kbrockhoff added 6 commits June 10, 2020 09:31

Golden dataset provider bug fixes

3df9ed2

Extract validators and inprocess runner bug fixes

be5f6e8

Fixed correctness processors config

712b782

Merge branch 'master' into generalize-testbed

d6ded01

Update to latest metrics API

e3b78f8

Added span data validation

6673d25

kbrockhoff changed the title ~~Generalize testbed [WIP]~~ Generalize testbed Jun 12, 2020

kbrockhoff added 4 commits June 13, 2020 09:36

Merge branch 'master' into generalize-testbed

b0bb782

consolidate assertion failures reporting

239c6c5

improve test coverage

004e716

Document test suite creation procedures.

68ec886

kbrockhoff requested review from tigrannajaryan and pmcollins June 13, 2020 18:45

tigrannajaryan reviewed Jun 15, 2020

View reviewed changes

kbrockhoff added 2 commits June 16, 2020 07:54

Merge branch 'master' into generalize-testbed

dffab9d

Fix PR comments and wire in correctness tests in Makefile

0eab46d

Fix more PR comments

a0b4915

kbrockhoff requested a review from tigrannajaryan June 16, 2020 18:44

tigrannajaryan reviewed Jun 16, 2020

View reviewed changes

testbed/testbed/otelcol_runner.go Outdated Show resolved Hide resolved

kbrockhoff added 3 commits June 17, 2020 07:51

Merge branch 'master' into generalize-testbed

36e0026

Remove ResourceSpec because not applicable

6aad53c

Improve OtelcolRunner Config by returning cleanup function

00afa6f

kbrockhoff requested a review from tigrannajaryan June 17, 2020 15:18

tigrannajaryan reviewed Jun 17, 2020

View reviewed changes

testbed/correctness/correctness_test.go Outdated Show resolved Hide resolved

testbed/testbed/otelcol_runner.go Outdated Show resolved Hide resolved

Collector automation moved this from In progress to Reviewer approved Jun 17, 2020

tigrannajaryan approved these changes Jun 17, 2020

View reviewed changes

Shorten test name

6f6545e

tigrannajaryan merged commit e3df9f3 into open-telemetry:master Jun 17, 2020

Collector automation moved this from Reviewer approved to Done Jun 17, 2020

kbrockhoff deleted the generalize-testbed branch June 18, 2020 12:21

This was referenced Jun 18, 2020

Add E2E test for Zipkin format #482

Closed

Don't spawn Collector process for certain E2E tests #1024

Closed

Troels51 pushed a commit to Troels51/opentelemetry-collector that referenced this pull request Jul 5, 2024

Otlp http log example (open-telemetry#1062)

97fbdaa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generalize testbed #1062

Generalize testbed #1062

kbrockhoff commented Jun 1, 2020 •

edited

Loading

codecov bot commented Jun 1, 2020 •

edited

Loading

tigrannajaryan left a comment

linux-foundation-easycla bot commented Jun 5, 2020 •

edited

Loading

tigrannajaryan Jun 9, 2020

pmcollins Jun 10, 2020

kbrockhoff Jun 13, 2020

tigrannajaryan left a comment

kbrockhoff commented Jun 16, 2020

tigrannajaryan left a comment •

edited

Loading

tigrannajaryan left a comment

tigrannajaryan commented Jun 17, 2020

Generalize testbed #1062

Generalize testbed #1062

Conversation

kbrockhoff commented Jun 1, 2020 • edited Loading

codecov bot commented Jun 1, 2020 • edited Loading

Codecov Report

tigrannajaryan left a comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Jun 5, 2020 • edited Loading

tigrannajaryan Jun 9, 2020

Choose a reason for hiding this comment

pmcollins Jun 10, 2020

Choose a reason for hiding this comment

kbrockhoff Jun 13, 2020

Choose a reason for hiding this comment

tigrannajaryan left a comment

Choose a reason for hiding this comment

kbrockhoff commented Jun 16, 2020

tigrannajaryan left a comment • edited Loading

Choose a reason for hiding this comment

tigrannajaryan left a comment

Choose a reason for hiding this comment

tigrannajaryan commented Jun 17, 2020

kbrockhoff commented Jun 1, 2020 •

edited

Loading

codecov bot commented Jun 1, 2020 •

edited

Loading

linux-foundation-easycla bot commented Jun 5, 2020 •

edited

Loading

tigrannajaryan left a comment •

edited

Loading