Zipkin tracing format update

[zsojma]

Issue Type:

• Bug report
• Feature request

What happened:

Hello, when a trace is logged into Zipkin (we actually use Jaeger https://github.com/jaegertracing, but there should be no difference related to this issue - it is backward compatible with zipkin), linkerd logs names of services in following complex format (we are using io.zipkin.http telemeter):

• service mesh route from a pod to linkerd: %/io.l5d.k8s.daemonset/l5d-system/http-incoming/l5d/#/io.l5d.k8s.http/k2-development/http/orchestration
• service mesh route from linkerd to a pod: %/io.l5d.k8s.localnode/k8s-master-1.alz.lcl/#/io.l5d.k8s/k2-development/http/orchestration - differs for each node, which is here logged as k8s-master-1.alz.lcl

We currently have kubernetes cluster with 6 nodes and something about 20 applications deployed (it will be more in future), which results to a state, where we see something around 140 (20 + 6 * 20) different services logged in the tracer:

And this makes it hard to locate all traces for a concrete deployed application, because we need to choose between incomming/outgoing requests and between nodes. A concrete trace looks like the following now:

What you expected to happen:

It would be nice, if linkerd could have an option to log only names of concrete k8s services with no other related information. Something like the following (related to previous picture):

> 0.0.0.0/80
  > k2-development/http/orchestration
    > 0.0.0.0/4340
      > k2-development/grpc/metadatamanagement
        > 0.0.0.0/4341
          > k2-development/grpc/metadatamanagement
    > 0.0.0.0/4140
      > k2-development/http/rightsmanagement
        > 0.0.0.0/4141
          > k2-development/http/rightsmanagement

Where k2-development is name of a k8s namespace, http / grpc are names of ports defined by k8s service and orchestration etc. are names of k8s services. Other related information (like name of a router %/io.l5d.k8s.daemonset/l5d-system/grpc-incoming/l5d/#/io.l5d.k8s.grpc/) would be logged as either span tag or Process tag.

What do you think? Thanks

Environment:

• linkerd/namerd version, config files: 1.3.5
• Platform, version, and config files (Kubernetes, DC/OS, etc): Kubernetes 1.8.3

[klingerf]

Hey @zsojma -- thanks for reporting this. I had previously opened #1132 to track changing the long span names to something more intelligible, but we ultimately decided against it. More context about that here: #1135 (comment)

The underlying issue is that the client span name corresponds to the id used by linkerd to label the clients that it builds dynamically. If we strip out parts of the id, then they become ambiguous.

But I totally agree that in a kubernetes setup, where multiple transformers and namers are in use, the ids become unwieldy. Maybe we should think about a kubernetes-specific fix like you suggested, since the issue is most pronounced in that environment.

[yurishkuro]

The underlying issue is that the client span name corresponds to the id used by linkerd to label the clients that it builds dynamically. If we strip out parts of the id, then they become ambiguous.

@klingerf This sounds like you're deciding on a user-facing feature based on the internal implementation detail. You can still keep the unique IDs internally, and associate them with a normalized structure like

{
    service: "myservice",
    cluster: "k8-development",
    transport: "grpc",
    // etc
}

tracing systems can easily handle high dimension metadata, and can actually provide better / more interesting aggregations when metadata is provided in a normalized form instead of a single denormalized string.

[klingerf]

@yurishkuro That sounds totally reasonable. I haven't looked at Finagle's trace implementation in a while. Do you know if it's possible to set span names to the normalized structure like you suggest? I seem to remember that it required strings for span names.

[yurishkuro]

Haven't looked at finagle in a couple of years, but fairly certain span names are plain strings. I'm not suggesting that they should be structs, rather that the rest of the dimensions can be recorded as span tags (finagle might call them binary annotations, following zipkin notation).