Allow for compressed dtabs in Consul

[chrismikehogan]

Issue Type:

• Bug report
• Feature request

This is motivated by the same use case describedin #2201. I'll copy/paste that use case here:

Use case

We are trying to use Linkerd's dtab feature to build very large, dynamic routing tables. The goal to use Linkerd (instead of, e.g. Consul) to control which versions of which applications get routed to when a a request for a particular service is received.

This requires us to dynamically build most of our dtab, and requires that the dtab have an explicit dentry for every service in our eco system. E.g.:

...
/svc/cool-thing => /tagged/ver-1.3/cool-thing;
/svc/lame-thing => 0.1 * /tagged/ver-2.0/lame-thing & 0.9 * /tagged/ver-1.8/lame-thing;
...

This means that our dtab will get very large! The example I'm working with at the moment is 1811 lines long, and 143kb.

Problem

We're using Consul as the KV store. Consul enforces a 512kb limit on the base64-encoded values. While my example dtab at 143kb is close to what our current "worst case" would be, we're concerned that this will not scale with time.

Proposed solution

We'd like to have the ability to read/write gzipped (or otherwise compressed) dtabs from Consul (or any other datastore). The dynamic dtabs compress extremely well -- my current example is only 16k. The compression gains are only slightly lost with base64 encoding that Consul demands -- my final encoded, gzipped dtab was 21k.

[adleong]

Thanks for filling this! I think this totally makes sense as a feature and seems pretty well scoped. @chrismikehogan, are you interested in working on this? If so, I can point you in the right direction.

[koiuo]

FWIW, at NCBI we use different approach. Our consul store plugin combines pieces of dtabs stored in different keys into single dtab (originally the idea belongs to @ashald)

That's how it may look in KV with our approach:

kv/namerd/dtabs contains the index, what keys should be read and in which order should be combined

kv/namerd/dtabs/base
kv/namerd/dtabs/cool-thing
kv/namerd/dtabs/lame-thing

Then, for example
kv/namerd/dtabs/base:

/http=>/svc

kv/namerd/dtabs/cool-thing:

/svc/cool-thing => /tagged/ver-1.3/cool-thing;

kv/namerd/dtabs/lame-thing:

/svc/lame-thing => 0.1 * /tagged/ver-2.0/lame-thing & 0.9 * /tagged/ver-1.8/lame-thing;

This approach also lets you having fine-grained ACLs for your dtabs. For example, you may issue a restricted token specifically for lame-thing, and let your deployment pipeline for lame-thing to update only that single key in consul KV without affecting anything else.

[chrismikehogan]

Thanks for filling this! I think this totally makes sense as a feature and seems pretty well scoped. @chrismikehogan, are you interested in working on this? If so, I can point you in the right direction.

I'm interested! And I could definitely use some directions haha

[chrismikehogan]

@edio That sounds really promising as well. How do the separate k/v entries get combined in the final dtab?

Or did you write custom code that knows how to parse these?

[adleong]

@chrismikehogan KvApi is the class that defines how we interact with the Consul KV api. I think you'll want to add a parameter to that class that controls if gzipping should be used. You'll then need to modify the read and write methods in that file to gzip the values on the way in and unzip them on the way out.

You'll then need to add a property to the consul store config object to control the gzip behavior. The value of that property would be passed into KvApi here: https://github.com/linkerd/linkerd/blob/master/namerd/storage/consul/src/main/scala/io/buoyant/namerd/storage/consul/ConsulDtabStoreInitializer.scala#L49

Hope that makes sense! I'm happy to provide more guidance if you run into any problems.

[koiuo]

@chrismikehogan , we have custom plugin for that. We haven't thought about publishing it, but if there's interest, we can do this.

[adleong]

@chrismikehogan please take a look at #2303 and let us know if that satisfies your use case.