Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.30813.1> exit with reason normal in context child_terminated

[simpsojo]

Seeing the following messages repeatedly in /var/log/couchdb/couchdb.log:

[error] 2018-11-14T15:54:51.599025Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.4622.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:54:56.606364Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.4775.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:01.614952Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.4886.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:06.622537Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.5031.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:11.631585Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.5141.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:16.638075Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.5284.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:21.645321Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.5396.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:26.652175Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.5537.2> exit with reason normal in context child_terminated
[error] 2018-11-14T15:55:31.658958Z [email protected] <0.419.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.5647.2> exit with reason normal in context child_terminated

Expected Behavior

No error messages.

Current Behavior

Many error messages and beam.smp memory usage gradually rises until it crashes.

Otherwise, no visible issues with CouchDB installation.

Possible Solution

Steps to Reproduce (for bugs)

1. Occurs even whilst CouchDB is idle

Context

Happy to provide more information.

Your Environment

• Version used: 2.2.0
• Browser Name and version:
• Operating System and version (desktop or mobile): Ubuntu 16.04.4 x64 runing at Digital Ocean with 1 GB Memory / 25 GB Disk
• Link to your project:

[Coinbird]

I had this issue as well. CouchDB 2.3.1

Strangely, I was unable to log out as well, when it happens. In Fauxton, I noticed that there was a database that was "unable to load" even though I was logged into Fauxton.

As far as I can tell, it had something to do with per_user databases getting in a weird state; I'm not entirely sure how I got there. I was messing with permissions on the _users DB and that perhaps I caused it (I was creating users from another _users account.) Either a user hadn't been properly deleted (somehow) or some kind of conflict between userdb- and the corresponding username.

My fix was from Fauxton, find the userdb-xyz database that "cannot be loaded" and manually issue a DELETE from curl, Postman, (or tool of your choice.) I used Basic auth for this DELETE command, using my administrator user/pw.
Once that "Cannot be loaded" DB was gone, everything started working again. Luckily this was a local dev DB and I didn't need the contents of the userdb.

UPDATE: Also realized I had some dead nodes from playing with clustering before--this is perhaps why I was seeing the "cannot load DB". http:https://localhost:5984/_membership showed 3 nodes even though I was intending to run just 1. I followed this to axe the extra ones. BE VERY CAREFUL WITH THIS- it can potentially delete data if you axe the wrong cluster, it appears. https://docs.couchdb.org/en/master/cluster/nodes.html

[alessandroingargiola]

This problem happens to me everytime after deleting a peruser user from _users table, whether I delete it from fauxton UI or by CURL command.

couchdb.log files writes a new row like this every 5 seconds:
[error] 2019-08-01T18:02:25.247630Z [email protected] <0.391.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.14430.1> exit with reason normal in context child_terminated

The only way I've found to stop it is to reinstall couchdb.

[baptistedonaux]

I have the same issue. This issue seems appears when few data into _users database have data with wrong scheme. An easy way to reproduce, it's to insert a user with minimum data (you can try to except password field) and constat.

[jimm101]

Same issue. Any workarounds? Is there a way to identify bad _users entries?

[Crispy1975]

I am seeing this with the official Docker image of CouchDB 3.1.0. Exactly the same behaviour as others have described. When a user doc is deleted - along with the database that gets created with it (couch_peruser) - I start to see the following error messages in the CouchDB logs:

couchdb_1         | [error] 2020-05-18T08:30:57.689442Z nonode@nohost <0.395.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.29782.0> exit with reason normal in context child_terminated
couchdb_1         | [error] 2020-05-18T08:31:02.693066Z nonode@nohost <0.395.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.29847.0> exit with reason normal in context child_terminated
couchdb_1         | [error] 2020-05-18T08:31:07.695809Z nonode@nohost <0.395.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.29909.0> exit with reason normal in context child_terminated

I would very much like to make use of the couch_peruser option as it's a very nice feature to keep user data segregated etc. I've tried various options to try and clear the error, like recreating the same deleted user and database combination, restarts of the server process but nothing seems to help.

I should add that after user creation I also store some addition data in each user document, also I am currently running a single node without any kind of replication (at this point).

[janl]

@Crispy1975 can you check if setting the log level to debug gets you some more output? Would be great to get the entire log segment from the recording of the user deletion onwards.

And can you share the response from / _membership?

[Crispy1975]

@janl here is the output from the CouchDB logs with debug turned on:

couchdb_1         | [debug] 2020-05-19T08:32:28.850719Z nonode@nohost <0.1853.70> -------- peruser: handling _users/org.couchdb.user:example1
couchdb_1         | [debug] 2020-05-19T08:32:28.851228Z nonode@nohost <0.1936.70> -------- peruser: handling _users/org.couchdb.user:example2
couchdb_1         | [debug] 2020-05-19T08:32:28.851650Z nonode@nohost <0.31834.69> -------- peruser: starting on node nonode@nohost in pid <0.31834.69>
couchdb_1         | [error] 2020-05-19T08:32:28.851649Z nonode@nohost <0.395.0> -------- Supervisor couch_peruser_sup had child couch_peruser started with couch_peruser:start_link() at <0.1642.70> exit with reason normal in context child_terminated
couchdb_1         | [debug] 2020-05-19T08:32:28.851733Z nonode@nohost <0.31834.69> -------- peruser: enabled on node nonode@nohost
couchdb_1         | [debug] 2020-05-19T08:32:28.851983Z nonode@nohost <0.1961.70> -------- peruser: skipping, cluster unstable shards/40000000-5fffffff/_users.1539158245/org.couchdb.user:example3
couchdb_1         | [debug] 2020-05-19T08:32:28.852002Z nonode@nohost <0.395.0> -------- Supervisor couch_peruser_sup started couch_peruser:start_link() at pid <0.31834.69>```

Currently example3 is the only existing user (in _users) with a related database.

The output from the _membership request is:

{"all_nodes":["nonode@nohost"],"cluster_nodes":["nonode@nohost"]}

[janl]

and this is just during CouchDB startup? Or when you DELETE users example1 and example2? If the latter, please include lines for those DELETE requests.

[Crispy1975]

I will create a fresh install and test the scenarios later on and report back. Also worth adding is that since the restart this morning and turning debug on I see a lot of crashes, not sure if this is related. But the process eventually died completely. I created a gist of the output here: https://gist.github.com/Crispy1975/904e4579108ce4c883a3063ee76787e2

[Crispy1975]

I restarted CouchDB after the above crash and perhaps the startup log might also help, that is here in this gist: https://gist.github.com/Crispy1975/ea936e3a01a4a83ce6286c2b27219c1e

Interestingly it's still referencing the deleted users and having issues with an unstable shard?

[Crispy1975]

An interesting update. The errors seem to have stopped. I thought that perhaps the manual deletion of users and their associated databases were perhaps getting CouchDB into a bad state as some internal process was not running correctly. I turned on the couch_peruser option delete_dbs. I then created all the deleted users again (CouchDB created the associated databases). I then deleted the users via Fauxton. CouchDB did it's thing and removed the associated per user databases, and on checking the logs no more recurring errors.

The only thing left to do is for CouchDB to purge the deleted docs from the _users database (shown in the below grab).

So in summary it seems that this error is caused by deleting users and their associated databases manually. Not sure if that is a server bug, but turning on the delete_dbs option seems to keep CouchDB in a good state. You just have to be careful not to nuke your databases. ;-)

[alcastle]

Just ran into this as well and can confirm
Turning on delete_dbs under the couch_peruser section of my config and restarting the service fixed it.