Dark Traffic

[adleong]

linkerd should have some mechanism to fork incoming requests and send them to two separate backends with one of the responses being returned to the caller and the other response being discarded. This can be used for generating real traffic on a pre-production service before making it live. Some open questions:

• How should linkerd determine which requests to fork? Based on service name? Client name? Sample rate?
• What mechanism should be used to specify the forked destination? Config file? API?
• Where in the linkerd stack should this be implemented?
• Should linkerd compare the responses in any way (a la diffy) or simply discard the non-live response?

[gsogol]

Another use case could be used to send to two different data centers, whichever one returns first, send back to the consumer based on latency. Or could that be configured using different load balancing strategies?Thoughts?

[adleong]

@gsogol actually that sounds more like speculative/backup requests: #1069

[gsogol]

It's exactly that.

…

On May 24, 2017, at 6:05 PM, Alex Leong ***@***.***> wrote: @gsogol actually that sounds more like speculative/backup requests: #1069 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nmurthy]

Hey @adleong I'm in the midst of implementing this and would love to give back to the community. Here's how I was thinking about doing it:
Designate a new HTTP header x-dark-route whose value is a dtab entry representing where the response should be multicast to. Our use case definitely involves analyzing the response differences, so I'm also thinking about having a config variable optionally point to a kafka topic/http endpoint where both reqs/responses will be sent. (We could def just wire it up to diffy this way too)

Thoughts?

[adleong]

Cool! I'd love to see it!

[prdoyle]

Can I ask, why do you guys use HTTP headers for things like this? Doesn't that mean that anyone using a REST client can play your application like a fiddle and make it do all kinds of unintended things?

[klingerf]

@prdoyle Good question. Check out the clearContext server option, documented here:

https://linkerd.io/config/1.1.3/linkerd/index.html#server-parameters

[DavidParks8]

Any progress with this issue? It seems this is one of the few capabilities istio has that linkerd is lacking now.

[cpretzer]

@DavidParks8 I don't think anyone has been working on this for Linkerd 1. It sounds like @nmurthy started to think through an implementation.

[DavidParks8]

@cpretzer what about for linkerd 2? My team uses 2.x versions.

[cpretzer]

@DavidParks8 great to hear that you're using Linkerd 2! There is an open issue around shadow/mirrored traffic in Linkerd 2 repository.

We'd love to get your thoughts there.