Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_dbs.couch file is zero bytes and server won't start up #4077

Closed
tjrobinson opened this issue Jun 22, 2022 · 4 comments
Closed

_dbs.couch file is zero bytes and server won't start up #4077

tjrobinson opened this issue Jun 22, 2022 · 4 comments

Comments

@tjrobinson
Copy link

Description

After the following error, the server won't start and the _dbs.couch file is zero bytes.

[notice] 2022-06-21T20:12:22.766823Z [[email protected]](mailto:[email protected]) <0.15742.6857> 6b430e88a3 52.158.43.32:5984 51.142.253.239 undefined GET / 401 ok 0
[notice] 2022-06-21T21:02:57.099484Z [[email protected]](mailto:[email protected]) <0.371.0> -------- couch_replicator_scheduler: Job {"52a8eeedef822cfeb63311b33862f6ea","+create_target"} started as <0.22297.6858>
[notice] 2022-06-21T21:03:01.009956Z [[email protected]](mailto:[email protected]) <0.18144.6858> 2b0a3dab49 couchdev.northeurope.cloudapp.azure.com:5984 52.158.43.32 undefined POST /_session 200 ok 2
[notice] 2022-06-21T21:03:01.026533Z [[email protected]](mailto:[email protected]) <0.18144.6858> d35d41db0e couchdev.northeurope.cloudapp.azure.com:5984 52.158.43.32 admin GET /sessions/ 404 ok 12
[error] 2022-06-21T21:03:01.028105Z [[email protected]](mailto:[email protected]) <0.22297.6858> -------- throw:{db_not_found,<<"could not open http:https://couchdev.northeurope.cloudapp.azure.com:5984/sessions/">>}: Replication 52a8eeedef822cfeb63311b33862f6ea+create_target failed to start http:https://couchdev.northeurope.cloudapp.azure.com:5984/sessions/ -> http:https://couchdev.northeurope.cloudapp.azure.com:5984/sessionIndex/ doc <<"shards/80000000-ffffffff/_replicator.1627909308">>:<<"66bdbf9229bcacfe4155170f87cecd1b">> stack:[{couch_replicator_api_wrap,db_open,3,[{file,"src/couch_replicator_api_wrap.erl"},{line,114}]},{couch_replicator_scheduler_job,init_state,1,[{file,"src/couch_replicator_scheduler_job.erl"},{line,572}]}]
[info] 2022-06-22T01:18:13.344895Z [[email protected]](mailto:[email protected]) <0.239.0> -------- Preflight check: Asserting Admin Account
[info] 2022-06-22T01:18:13.372870Z [[email protected]](mailto:[email protected]) <0.239.0> -------- Apache CouchDB 3.1.1 is starting.
[info] 2022-06-22T01:18:13.373174Z [[email protected]](mailto:[email protected]) <0.240.0> -------- Starting couch_sup
[info] 2022-06-22T01:18:14.036249Z [[email protected]](mailto:[email protected]) <0.239.0> -------- Apache CouchDB has started. Time to relax.
[notice] 2022-06-22T01:18:14.161367Z [[email protected]](mailto:[email protected]) <0.272.0> -------- rexi_server : started servers
[notice] 2022-06-22T01:18:14.188250Z [[email protected]](mailto:[email protected]) <0.276.0> -------- rexi_buffer : started servers
[error] 2022-06-22T01:18:14.478062Z [[email protected]](mailto:[email protected]) <0.286.0> -------- Could not open file /bitnami/couchdb/data/_nodes.couch: permission denied
[info] 2022-06-22T01:18:14.478482Z [[email protected]](mailto:[email protected]) <0.285.0> -------- open_result error {error,eacces} for _nodes

Steps to Reproduce

Unknown.

Expected Behaviour

The server starts as expected and _dbs.couch file is not empty.

Your Environment

  • CouchDB version used: 3.1.1
  • Browser name and version: N/A
  • Operating system and version: Bitnami Azure VM store image (Debian GNU/Linux 10 (buster)) with CouchDB pre-installed

Additional Context

We're unable to use curl to get any additional information as the server won't start up.

The _nodes.couch looks to be intact, and there are .couch files for the databases inside the shards directory.

bitnami@LDC-AZU-COU2:/bitnami/couchdb/data$ ls -l
total 16
-rw-r--r-- 1 root root    0 Apr  4 17:58 _dbs.couch
-rw-r--r-- 1 root root 8389 Apr  4 17:55 _nodes.couch
drwxr-xr-x 3 root root 4096 Apr  4 17:55 shards

Is there any way to rebuild the _dbs.couch file? We don't have backups.

@tjrobinson
Copy link
Author

Output from running couchdb directly:

root@LDC-AZU-COU2:/opt/bitnami/couchdb/etc# couchdb
Configuration Settings:
  [admins] admin="-pbkdf2-8857f71e0b770c567210b14b157ba415350f067c,d9b9f7602fd331a8750bf69714ce5529,10"
  [attachments] compressible_types="text/*, application/javascript, application/json, application/xml"
  [attachments] compression_level="8"
  [chttpd] backlog="512"
  [chttpd] bind_address="0.0.0.0"
  [chttpd] max_db_number_for_dbs_info_req="100"
  [chttpd] port="5984"
  [chttpd] prefer_minimal="Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary"
  [chttpd] require_valid_user="true"
  [chttpd] server_options="[{recbuf, undefined}]"
  [chttpd] socket_options="[{sndbuf, 262144}, {nodelay, true}]"
  [cluster] n="3"
  [cluster] q="2"
  [cors] credentials="true"
  [cors] headers="accept, authorization, content-type, origin, referer"
  [cors] methods="GET, PUT, POST, HEAD, DELETE"
  [cors] origins="*"
  [couch_httpd_auth] allow_persistent_cookies="true"
  [couch_httpd_auth] auth_cache_size="50"
  [couch_httpd_auth] authentication_db="_users"
  [couch_httpd_auth] authentication_redirect="/_utils/session.html"
  [couch_httpd_auth] iterations="10"
  [couch_httpd_auth] require_valid_user="true"
  [couch_httpd_auth] secret="2ddfd16d6682d9e3037cbf234911474e"
  [couch_httpd_auth] timeout="600"
  [couch_peruser] database_prefix="userdb-"
  [couch_peruser] delete_dbs="false"
  [couch_peruser] enable="false"
  [couchdb] attachment_stream_buffer_size="4096"
  [couchdb] changes_doc_ids_optimization_threshold="100"
  [couchdb] database_dir="/bitnami/couchdb/data"
  [couchdb] default_engine="couch"
  [couchdb] default_security="admin_only"
  [couchdb] file_compression="snappy"
  [couchdb] max_dbs_open="500"
  [couchdb] max_document_size="8000000"
  [couchdb] os_process_timeout="5000"
  [couchdb] users_db_security_editable="false"
  [couchdb] uuid="0ed37b0e0f1c26daae00df09e76bff56"
  [couchdb] view_index_dir="/bitnami/couchdb/data"
  [couchdb_engines] couch="couch_bt_engine"
  [csp] enable="true"
  [feature_flags] partitioned||*="true"
  [httpd] WWW-Authenticate="Basic realm=\"administrator\""
  [httpd] allow_jsonp="false"
  [httpd] authentication_handlers="{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}"
  [httpd] bind_address="127.0.0.1"
  [httpd] config_whitelist="[{httpd,config_whitelist}]"
  [httpd] enable_cors="true"
  [httpd] enable_xframe_options="false"
  [httpd] max_http_request_size="4294967296"
  [httpd] port="5986"
  [httpd] secure_rewrites="true"
  [httpd] socket_options="[{sndbuf, 262144}]"
  [indexers] couch_mrview="true"
  [ioq] concurrency="10"
  [ioq] ratio="0.01"
  [ioq.bypass] compaction="false"
  [ioq.bypass] os_process="true"
  [ioq.bypass] read="true"
  [ioq.bypass] shard_sync="false"
  [ioq.bypass] view_update="true"
  [ioq.bypass] write="true"
  [log] file="/opt/bitnami/couchdb/var/log/couch.log"
  [log] level="debug"
  [log] writer="file"
  [query_server_config] os_process_limit="100"
  [query_server_config] reduce_limit="true"
  [replicator] connection_timeout="30000"
  [replicator] http_connections="20"
  [replicator] interval="60000"
  [replicator] max_churn="20"
  [replicator] max_jobs="500"
  [replicator] retries_per_request="5"
  [replicator] socket_options="[{keepalive, true}, {nodelay, false}]"
  [replicator] ssl_certificate_max_depth="3"
  [replicator] startup_jitter="5000"
  [replicator] verify_ssl_certificates="false"
  [replicator] worker_batch_size="500"
  [replicator] worker_processes="4"
  [ssl] port="6984"
  [uuids] algorithm="sequential"
  [uuids] max_count="1000"
  [vendor] name="The Apache Software Foundation"
{"Kernel pid terminated",application_controller,"{application_start_failure,mem3,{{shutdown,{failed_to_start_child,mem3_shards,{{case_clause,{{badmatch,{error,enospc}},[{couch_bt_engine,init,2,[{file,\"src/couch_bt_engine.erl\"},{line,163}]},{couch_db_engine,init,3,[{file,\"src/couch_db_engine.erl\"},{line,775}]},{couch_db_updater,init,1,[{file,\"src/couch_db_updater.erl\"},{line,43}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]}},[{mem3_util,ensure_exists,1,[{file,\"src/mem3_util.erl\"},{line,269}]},{mem3_shards,get_update_seq,0,[{file,\"src/mem3_shards.erl\"},{line,321}]},{mem3_shards,init,1,[{file,\"src/mem3_shards.erl\"},{line,208}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]}}},{mem3_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,mem3,{{shutdown,{failed_to_start_child,mem3_shards,{{case_clause,{{badmatch,{error,enospc}},[{couch_bt_engine,init,2,[{file,"

Crash dump is being written to: erl_crash.dump...done

@big-r81
Copy link
Contributor

big-r81 commented Jun 22, 2022

Please use the latest CouchDB (v3.2.2) with the latest security fixes.

Use our docs to install CouchDB.

I don't know how Bitnami configured CouchDB, but there should a separate couchdb user. So I would guess it is an access rights problem for _dbs.couch, _nodes.couch and shards files/directories.

Additional see this discussion.

@tjrobinson
Copy link
Author

It turned out to be that there was no disk space remaining on the partition containing the data directory. The clue was error,enospc in the logs.

@big-r81
Copy link
Contributor

big-r81 commented Jun 22, 2022

Great, but use the current version and don't run CouchDB as root user!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants