Add a "l5d-trace" user header which can be used to manipulate the "l5d-ctx-trace" header

[klingerf]

Per the request in #1428, we should consider adding an additional user header to set the trace id, so that users don't have to set the "l5d-ctx-trace" header, which requires trace id encoding. Instead, we could support a plaintext "l5d-trace" header on the request, which will be rewritten into the "l5d-ctx-trace" header on the response. I picture this working a lot like the "l5d-dtab" and "l5d-ctx-dtab" headers, described here.

Editorial note: I'm not sure if it would be better for this header to be called "l5d-trace" and require a root:parent:id tuple, or to instead to be called "l5d-reqid" with just the root id, which would be more similar to the "l5d-reqid" header that's we're already setting on the response.

[amitsaha]

I would be curious to know what would your thoughts be regarding making the same possible for RPC services (i.e. when the incoming request is from a RPC service).?

[klingerf]

@amitsaha Sorry for the delay. By RPC do you mean thrift? We could definitely make something work for TTwitter thrift, but not sure if there's an option for native thrift. For context, the l5d-dtab header is currently only supported for http protocols, so we'd probably start with that.

[codefromthecrypt]

hmm is there an existing issue out to synchronize and/or link the linkerd trace header with another format like X-B3? I might have misspoken about compatibility, which is cool, but better to at least have a tracking issue to refer to.

[klingerf]

Hey @adriancole, good question. As far as I know there aren't any other open issues for adopting additional trace header formats. Since we use the l5d-ctx-* headers to propagate both tracing information and routing information, we think of them as being Linkerd-specific for the most part, and we generally encourage service owners for forward these headers without modification.

Similarly, Linkerd forwards tracing headers from all of the other tracing systems without modification, such that Linkerd's traces are non-overlapping with those systems' traces. This has the drawback that traces generated by Linkerd are not linked to traces generated by the applications themselves, but it also gives us the flexibility to propagate whatever type of context we need.

I think that the l5d-trace header described in this issue could be a good middle ground, allowing applications to inject their own trace IDs if they have them, without wiring through context propagation for every tracing system.

[codefromthecrypt]

Hmm so I suppose I might be oversimplifying, but what is actually preventing a trace join plugin? Like a bijection. For example, what would be the risk of continuing a user-created B3 trace and also B3'ing on the other side (even if l5 headers are also sent). For example, this is likely how folks I've talked to will support switching formats. Ex double-propagating. The cost would be a coordinated plugin, but at least B3 is extremely common and understood in finagle world.

[klingerf]

Yeah, that makes sense. Linkerd could provide a pluggable server-side filter that converts inbound headers to a finagle Trace object, and a pluggable client-side filter that converts the Trace object back to headers on the outbound request. It's a bit more complicated because we'd still need to figure out a way for Linkerd to continue to propagate routing context, such as per-request routing rules, and we'd rather not rely on plugins to carry that forward. But something additive for advancing trace context for other systems would probably work.

[codefromthecrypt]

yeah I think of these contexts as different, for example, a user ID context being different than a routing rule context being different than tracing context (although they are indeed inter-related). Happy to help think this through together, even pow-wow at the next tracing workshop https://docs.google.com/document/d/19SQOio1z7mHqb79MzLQCiKdkq5tdirOW8OCcTQNQdUg/edit#

[klingerf]

Good thinking -- I'd be happy to join the discussion to talk about service mesh tracing as it relates to Linkerd. I indicated my interest in the document. Thanks!

[yllions]

Hello, is there any plan about passing existing trace context through linkerd?

[klingerf]

Hi @xiaoerlyl -- linkerd is already setup to propagate context from any tracing system. For http traffic, context is usually passed via http request headers, and linkerd will forward those headers, regardless of tracing system. This ticket is setup to track linking traces generated externally to those generated by linkerd. This issue describes one such approach, but it's not being actively worked on at the moment.

[yllions]

@klingerf thanks for your reply. I want that linkerd to be a part of my entire end to end trace both for http and other protocol. For http, I just followed the approach from issue #1428 . and i further modified linkerd internal implementation to pass the trace info generated in linkerd to downstream server.

Here is the modification in LinkerdHeaders.scala

 def set(headers: HeaderMap, id: TraceId): Unit = {
        val bytes = TraceId.serialize(id)
        val b64 = Base64.getEncoder.encodeToString(bytes)
        val _ = headers.set(Key, b64)
        val map = headers.set("X-B3-SpanId", id.spanId.toString)
    }

[jmalvarezf]

Hi!

I tried to find this in the Roadmap but didn't. Is it still something that you will have in the medium term? We are planning to use linkerd but not being able to participate in an existing tracing context is a big drawback for us.

Thanks