helm chart default install de-facto doesn't work (inconsistent adminHash)

[lbruun]

A default install of the helm chart, like so:

 helm install couchdb couchdb/couchdb                                                             \
    --namespace $namespace                                                                                \
    --set couchdbConfig.couchdb.uuid=1edba19879809c3ae67e2abec4054579  \
    --set persistentVolume.enabled=true

will simply not work. It may seem to work, but in reality it doesn't. Here's why: In Kubernetes everything must expect to be bounced occasionally. Perhaps your PaaS provider for the Kubernetes cluster will restart/re-cycle underlying Kubernetes Nodes on a weekly basis or whatever. This causes Pods to restart. With the above config, the adminHash will be generated on Pod startup. It will inevitably be different on different Pods. For our deployment this has meant that - after some time - you can no longer login to Fauxton. (and when that happens it produces very strange behavior/error message which I doubt anyone except 1-2 couchdb experts would ever figure out to attribute to inconsistent adminHash'es on different nodes ... but that is not the focus of this ticket).

My argument here is that the above default install is indeed a time bomb. Sooner or later it will stop working.

The solution - for us - has been to create the adminHash ourselves and put it into the Secret which we then need produce ourselves too. This is hinted in the helm chart documentation, but as if it is optional, however it is not, you cannot really make CouchDB work (at least if using Fauxton) without force-setting the adminHash.

Creating your own Secret is indeed a very involved process as you then need to figure out a way to auto-create all the other values in the Secret too. (I'm assuming here that everyone is using IaC these days: the goal is to use the helm chart from a deployment pipeline, not as a tool randomly fired from the command line).

In any case, creation of the Secret would look something like this:

 kubectl --namespace "$k8s_namespace" \
     create secret generic "$secretname" \
     --from-literal=adminUsername=admin \
     --from-literal=adminPassword="$adminpwd" \
     --from-literal=cookieAuthSecret="$cookieauthsecret" \
     --from-literal=adminHash="$adminhash"  \
     --from-literal=erlangCookie="$erlangcookie"

cookieAuthSecret, erlangCookie and adminPassword are password-like strings. You need to find a way to auto-generate those in your pipeline.

adminHash is worse. CouchDB uses PBKDF2 for the adminHash but stores it in a custom format. You can find a recipe here for how to create the adminHash. The input to the hash is the value you've created for adminPassword.

Now for the final step, you need to let the helm chart know that you've already created Secret and you want the chart to use that one. This is done like so:

 helm install couchdb couchdb/couchdb                                                             \
    --namespace $namespace                                                                                \
    --set couchdbConfig.couchdb.uuid=1edba19879809c3ae67e2abec4054579  \
    --set createAdminSecret=false                                                                          \
    --set adminHash=loremipsum                                                                           \
    --set persistentVolume.enabled=true

The strange - and as far as I can tell - undocumented trick is that you need to set adminHash as a chart value too (in addition to the fact that you just defined it in the Secret). As a chart value it doesn't matter what value it has, as long as it is set. I set it to lorem ipsum in the example, just to make a point of this fact.

I think this could be either automated in the helm chart or (second best) be documented. In particular, I argue, that having a default recommendation for how the helm chart should be used, which in reality will not work after some time, is not good.

I hope this can be used as input for an improvement process on the helm chart.

Version of CouchDB Helm chart:

Latest at time of this ticket. (v4.5.0)