Add Agentless Datadog Trace Exporter

[ericmustin]

This is probably at a point where it could benefit from review, cc @mx-psi , some things to note:

• There's still some uncertainty around a few trace's span translation choices, as well as how to grab the global config stuff like hostname and env and apply it to the trace payloads (as well as the order of precedence for things like DD_SERVICE as opposed to the service already set by the tracer). but for the most part the implementation is complete, as is the flushing of trace and stats payloads.

• the most of what's in /apm, including tests and the snapshots, i've just grabbed from serverless. I tried to remove everything that wasn't being used by our implementation, but didn't update the serverless tests, so they'll probably fail.

• The big things here are going to be:

  • having a better set of eyes around my very hacky Go code. Anything you see that looks unusual, or could be optimised, there's likely not a good reason i've chosen not to, it's just because i don't know Go especially well.
  • The implementation itself around converting Internal Traces to Datadog Traces largely follows the approach the Zipkin Exporter took. There are a few things zipkin is doing that we don't support and vice versa but the methods and code paths are similar.
  • The conventions of the coercing the internal spans to datadog spans largely follow the python exporter's translation implementation, while the conventions and approach around generating+aggregating+flushing trace+stats payloads is largely based on the serverless implementation. As i mentioned previously, I more or less ported everything in /apm directly from that repo.
  • I've been testing this locally using our otel-collector-e2e-tests package, lmk if you have any trouble with that.
  • How to appropriately manage the config. it's possible my approach has drifted from what we've ended up doing in the metric exporter.
  • Tests are going to be a hassle, If you see any shortcuts here or good similar test setups in other parts of the repo that I can use as a guide, let me know. I will try to start writing tests for this stuff next

[mx-psi]

I reviewed some parts of the code and left a few comments so that you can keep working on this; most importantly I think you need to run make fmt and make to run all the linters and tests that can help catch errors (we will need them to pass to have a PR accepted upstream)

[mx-psi]

Second round; impi (after running make fmt) and misspell still give some errors that need to be addressed

[NathanBaulch]

Well that's annoying, I just stumbled on this branch/PR after successfully migrating the tracing portion of opencensus-go-exporter-datadog to OpenTelemetry here. I admit it did seem weird that this work hadn't been done already (at least not somewhere easily discoverable).

Oh well, mine works but is just a minimal 1-1 (where possible) port, so I look forward to this official one being released. 👍

While I'm here (and sorry if off-topic), I'm curious about the pros and cons of an OTEL exporter vs an OTEL API compatible wrapper as used in dd-trace-go (example)?

Also, does "agentless" mean it'll go direct to datadoghq without dd-agent? Thanks.

[ericmustin]

@NathanBaulch the goal here is we really need to be able to live in the otel-collector. simply put, existing otel users likely aren't going to want to run a datadog agent along with an opentelemetry-collector (maybe they will? we plan to add agent export as well down the road). So, yes agentless means direct to api intake. We're still iterating here, as circumventing the agent and our own tracing clients introduces a whole can of worms around metric computation but we hope to have a PR in with the Otel folks by EOW.

as far as the pros and cons of an OTEL exporter vs an OTEL API compatible wrapper, i think it's different use cases, an OTEL Exporter that lives in the opentelemetry-collector would be language agnostic but require the opentelemetry collector to be running, whereas the OTEL API Compatibility wrapper would leverage existing DD tracing architecture.

Sorry to hear about the duped work (nice work tho, fwiw!), but we can certainly keep you updated here on how this work is going and we'd love any feedback.

[mx-psi]

Looking better! I left some more comments, I think we would benefit from some unit test coverage at this point. One thing I am worried about is that the Agent might spam logs when importing its packages/running its internal functions, have you noticed anything in the end to end tests?

[ericmustin]

@mx-psi I tried to address most of the feedback and add unit tests for the span/trace translation work. Some of the trace connection stuff is still not covered in tests, I think I have to wire up a mock server for some of that, and I wasn't totally sure how to manage merging up the config tags. But punting on those bits for now so I can work on rebasing off the most up to date metrics exporter work so that the config is in a good place. I'd also like to start gameplanning on how to start the upstream PR process if you think this is in a reasonably good place, so let me know if there are other areas here where the code needs to be fixed before that.

[ericmustin]

Added some additional tests, rebasing off master brought in a bunch of changes to metric stuff that probably is now out of date. Going to wait for your metrrics PR and try to rebase off that branch

[ericmustin]

@mx-psi So i think this is pretty much in a good state, i added mock server tests to test the trace connection so i think our code coverage is pretty good at this point. I just need to rebase off master + your metric exporter work

[mx-psi]

Looks good, I have yet to test it on the backend but since you already did I am going to approve. I left a bunch of small typos to fix/TODOs that can be removed too.

[mx-psi]

Should we close this now that it is available upstream?