pkg/trace/stats: support measured spans for trace stats calculation

[jdgumz]

What does this PR do?

Adds support for a _dd.measured flag. This flag can be used to trigger metrics and stats calculations for any span.

Note:
I did not change some of the places where only top-level spans are considered. E.g.,

datadog-agent/pkg/trace/stats/statsraw.go

Line 224 in 1c76b83

if s.TopLevel {

datadog-agent/pkg/trace/stats/statsraw.go

Line 265 in 1c76b83

if s.TopLevel {

I have also left the legacy APM event extractor untouched, so it will continue to only process top-level spans:

datadog-agent/pkg/trace/event/extractor_legacy.go

Line 38 in 1c76b83

if !traceutil.HasTopLevel(s) {

Motivation

To allow more spans than just top-level ones to be included for measurement.

[bits-bot]

All committers have signed the CLA.

[pkalmakis]

The application code changes are simple enough and seem logical. I think it would be better if you changed less of the test code though, as others will have a higher level of confidence that this change hasn't broken existing functionality.

[jdgumz]

@pkalmakis @gbbr

Thank you for the great feedback! I've made the following changes:

• Use Span.Metrics to look for _dd.measured.
• Revert the addition of new test cases to old tests. Now the only changes to old tests are things to help clean up in some small ways (like using countValsEq).
• Added table-driven tests for new tests that evaluate the measured span-related behavior.
• I did not include tests for flushing logic with measured spans. I do not believe the flushing code considers top-level spans specifically. Sublayer calculation does however, which is why I wanted that test coverage.

[gbbr]

Thanks for making the changes Jesse. This is starting to look a bit better but please bear with me while we iron out all the details. I would also like to kindly ask you to abstain from sprinkling notes as comments all over the PR as these are not helping (quite the opposite). They are distracting from the review. I think most reviewers (including myself) prefer a clean page of code to look at. If anything is unclear or feels like it needs explaining, people generally tend to ask (example)...

Generally, when reviewing, I prefer to read the code as it would be read by someone looking at it in the codebase, where no PR comments exist.

[gbbr]

Forgot to say: you'll have to use reno to create the release notes. They need to be prefixed with APM :, something like:

---
features:
    - |
      APM: Add support for computing statistics on user-selected spans.

[gbbr]

Nice work! Thanks for bearing through the iterations.