Error 500 when creating a db below quorum

[calonso]

Expected Behavior

On a 4 nodes cluster, having two down, if I try to create a new database the server should either reject the request or accept it returning a 202 status code. Once the down nodes come back, the new db should be replicated to them.

Current Behavior

Currently an error 500 is returned but the db is indeed created (not sure if it replicated when the down nodes came back).

Possible Solution

Return a more friendlier status code as 202 Accepted if the db was actually created or completely reject it with a 412 Precondition Failed instead.

Steps to Reproduce (for bugs)

1. Setup a 4 nodes cluster
2. Bring 2 of them down
3. curl -X PUT "https://xxx.xxx.xxxx.xxx:5984/testdb"
4. An error 500 is returned.
5. curl -X GET "https://xxx.xxx.xxx.xxx:5984/_all_dbs" will return testdb

Context

I'm building a CouchDB automatic administration toolset and I was trying out different scenarios to see it working, in this case I was simulating an operation where on a cluster without quorum (4 nodes, 2 of them down), one tries to create a new database.

Your Environment

It is a Kubernetes managed cluster where the Docker image in use is https://hub.docker.com/r/klaemo/couchdb/

• Version used: 2.0.0
• Link to your project: https://github.com/cabify/couchdb-admin

[wohali]

Hi there, can you please include an excerpt from couch.log showing what happens when the 500 is returned? Also, if possible, can you include the exact message returned to your client along with the 500, if any? Thanks.

[calonso]

Sure!!

$ curl -X PUT "https://admin:[email protected]:5984/newtestdb2"
{"error":"error","reason":"internal_server_error"}

and the log, in debug mode

[debug] 2017-06-20T13:42:58.420676Z [email protected] <0.18754.0> ac122995ef cache miss for admin
[debug] 2017-06-20T13:42:58.421663Z [email protected] <0.18754.0> ac122995ef no record of user admin
[notice] 2017-06-20T13:42:58.564874Z [email protected] <0.18754.0> ac122995ef 127.0.0.1:5984 127.0.0.1 undefined PUT /newtestdb2 500 ok 144

[eiri]

This one is a bit tricky. We are throwing internal_server_error here when the writers quorum not met, but we can only know that after the fact, i.e. after we created db on all available nodes and didn't get enough replies from the writers.

The database is available after that, we can add docs to it and it'll be replicated to the rest of the nodes when they get up, so strictly speaking this is not a complete failure. But at the same time we want to have some kind of warning that the cluster is unstable, after all, if we'll get available nodes down and previously unavailable nodes up at the same time the database will as if suddenly disappear.

I guess we can add a check for the nodes availability on db creation/deletion before starting operations and throw 412 there if we know we wouldn't have the quorum. It wouldn't solve the issue completely, the nodes could go down during db creation, but at least it'll narrow the confusion gap. @davisp what do you think? Is it worth of try to implement?

[davisp]

Assuming the db record gets written on one node then we should return a 202 Accepted as the dbs db will be replicated and at that point we'll have created the database.

[ChiragMoradiya]

I also have observed same behavior for my production cluster. Any idea, when it's going to be fixed? Is it targeted for specific release?

[ChiragMoradiya]

In addition to this, When I was testing behavior with 3 nodes cluster. Whenever 1 of the node is down then I was able to create database and receiving 200 status code.

But, it sends 500 status code when 2 nodes (out of 3) are down & only 1 is up.

[jjrodrig]

I've also noticed this issue during my cluster testing.
I think the problem is here

couchdb/src/fabric/src/fabric_db_create.erl

Line 143 in d16f2db

maybe_stop(W, Counters) ->

Accepted is returned if the number of responses are at least (W div 2 + 1). If you get less than this value an internal server error is returned.

In a three node cluster where the quorum is 2, you need at least 2 responses in order to get accepted as response which is the same number you need for a 200 status code response.

This seems not to be consistent with this description in the documentation "11.2 Theory" which applies to document reads/writes not to db creation but I expected a similar behaviour .

The number of copies of a document with the same revision that have to be read before CouchDB returns with a 200 is equal to a half of total copies of the document plus one. It is the same for the number of nodes that need to save a document before a write is returned with 201. If there are less nodes than that number, then 202 is returned. Both read and write numbers can be specified with a request as r and w parameters accordingly.

W div 2 + 1, seems to be the default quorum required for a 200 status code, not for a 202 status code response

[marcinrozanski]

Same question as @ChiragMoradiya : "When will this fix make it to a Debian package?" Asking here as I was not able to find info about CouchDB release cycle.

[wohali]

@jjrodrig thanks for the fix, closing this issue.

As to when it will be available in a package, the answer is "with the next release of CouchDB." Our intention is to release CouchDB approximately twice a year.

The last release of CouchDB was November 7, 2017.

[dynamite-ready]

I experience the same issue as the OP on a single node, when cluster.n=1 has been set in the config.
I've been trying to post a document to a newly created user db (with the Couch Per User feature).

Typical log output look like this:

[notice] 2018-12-30T00:18:28.028842Z [email protected] <0.8860.0> 4cf3f1a0e6 localhost:5984 127.0.0.1 admin POST /userdb-*** 404 ok 17
[debug] 2018-12-30T00:18:29.031705Z [email protected] <0.8872.0> 8f64a178af no record of user admin
[notice] 2018-12-30T00:18:29.032887Z [email protected] <0.8872.0> 8f64a178af localhost:5984 127.0.0.1 admin POST /userdb-**** 404 ok 2
[debug] 2018-12-30T00:18:31.036855Z [email protected] <0.9002.0> 6745f08438 no record of user admin
[notice] 2018-12-30T00:18:31.038036Z [email protected] <0.9002.0> 6745f08438 localhost:5984 127.0.0.1 admin POST /userdb-**** 404 ok 2
[debug] 2018-12-30T00:18:35.041849Z [email protected] <0.9086.0> c3579d0751 no record of user admin
[notice] 2018-12-30T00:18:35.047631Z [email protected] <0.9086.0> c3579d0751 localhost:5984 127.0.0.1 admin POST /userdb-**** 404 ok 6
[debug] 2018-12-30T00:18:43.054896Z [email protected] <0.9199.0> 6a15101a52 no record of user admin
[notice] 2018-12-30T00:18:43.056778Z [email protected] <0.9199.0> 6a15101a52 localhost:5984 127.0.0.1 admin POST /userdb-**** 404 ok 3

Where posts to /userdb-**** are all the same recently created DB. The 5 calls are made rough 4 seconds apart, in a 20 second interval, and by that time, /userdb-**** had definitely been created.

It happens somewhat inconsistently too, which is surprising, because if cluster.n=1, I shouldn't have to worry about the chance of the POST going to a non existent node, isn't that right?