Evaluate and implement a method for checking the health of a CouchDB installation

[big-r81]

Summary

Starting with this Slack discussion, I would like to capture the thoughts from the discussion and talk about ideas to enhance/implement such a "global health-check-system" (gHCS).

In this particular case, the user performed a cluster setup without using the cluster setup endpoint (API or wizard). A load balancer was then placed in front of it, which led to authentication problems since not every node had the same authentication information.

Desired Behaviour

After installation and configuring CouchDB, a kind of self-test should be carried out in order to discover the most frequently occurring "errors" or "misconfigurations".

Possible Solution

A health check could be triggered after a start, a configuration change, by a user, at recurring intervals or in other cases. The "Cookie-Monster-Check" method could be such a point at the gHCS, as well as verifying the authentication information for each node or the check for correct node names or whatever should be checked in order to get a system that is as "error-free configured" as possible.

[nickva]

We have https://github.com/apache/couchdb/tree/main/src/weatherreport which can diagnose some of the issues, maybe we can enhance to check for more things?