Add zstd support as a compression method

[zamazan4ik]

Summary

Add zstd codec support.

Desired Behaviour

CouchDB already supports multiple compression codecs (by the way, which exactly? Cannot find the documentation about it), so we want just to have the option to use zstd as another compression codec.

Additional context

We want to use zstd since it seems like one of the most advanced compression on the market and has a very good CPU/compression ratio.

[nickva]

That's a good idea. We've even discussed it 7 years or so ago :-)

https://lists.apache.org/thread/kvvjodld2ly0t9rrllfd4d27pwf43hy5

My comment from 2016 was:

There has been a surprising resurgence of compression research lately with things like Brotli from Google and zstd from Facebook (http:https://facebook.github.io/zstd/). zstd has an interesting "training" mode where it can do a pass over small documents and learn a common dictionary, and CouchDB already passed over data during compaction, would that be a good time to train a compression dictionary?

At the time it wasn't sure how zstd would do, but it seems to have survived pretty well and even made its way into the Linux kernel.

That being said, if we just do the naive compression of doc bodies like we do now, it might not be worth it as there is a real future backward compatibility cost having to support another compression scheme essentially forever. It's kind of like that with snappy already. Now, one motivating factor about zstd would be if we could apply per-btree or per db compression using a dictionary. CouchDB's compaction provides a nice place to learn the dictionary and update during each compaction cycle.

CouchDB already supports multiple compression codecs (by the way, which exactly?

The current compression methods include snappy and the built-in Erlang term compression. See the docs for more details. The built-in Erlang compression is essentially deflate with 10 configurable levels.