error "Unauthorized" (401) when loading "_security"-doc

[gambolputty]

Hello!

I installed CouchDb 2.1.0 on my Ubuntu 16.04 machine. Somehow I cannot load the content of the _security doc without an error. I came across this issue after realizing that Fauxton is not displaying any "admins" or "members" on the "Permissions" page for a database.

When I do curl http:https://admin:my_password@my_server:5984/my_database/_security I receive the same output when I open the URL in my web browser:

{
  "error": "unauthorized",
  "reason": "You are not authorized to access this db.",
  "admins": {
    "names": [
      "admin"
    ]
  },
  "members": {
    "names": [
      "admin"
    ]
  }
}

• Version used: CouchDB 2.1.0
• Browser Name and version: Fauxton in Chrome 60.0.31 (macOS Sierra 10.12.5)
• Operating System and version: CouchDB on Ubuntu 16.04

local.ini:

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes
;os_process_timeout = 5000

[couch_peruser]
; If enabled, couch_peruser ensures that a private per-user database
; exists for each document in _users. These databases are writable only
; by the corresponding user. Databases are in the following form:
; userdb-{hex encoded username}
;enable = true
; If set to true and a user is deleted, the respective database gets
; deleted as well.
;delete_dbs = true

[chttpd]
;port = 5984
;bind_address = 127.0.0.1
; Options for the MochiWeb HTTP server.
;server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
; For more socket options, consult Erlang's module 'inet' man page.
;socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

[httpd]
; NOTE that this only configures the "backend" node-local port, not the
; "frontend" clustered port. You probably don't want to change anything in
; this section.
; Uncomment next line to trigger basic-auth popup on unauthorized requests.
;WWW-Authenticate = Basic realm="administrator"

; Uncomment next line to set the configuration modification whitelist. Only
; whitelisted values may be changed via the /_config URLs. To allow the admin
; to change this value over HTTP, remember to include {httpd,config_whitelist}
; itself. Excluding it from the list would require editing this file to update
; the whitelist.
;config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]

[query_servers]
;nodejs = /usr/local/bin/couchjs-node /path/to/couchdb/share/server/main.js

[httpd_global_handlers]
;_google = {couch_httpd_proxy, handle_proxy_req, <<"http:https://www.google.com">>}

[couch_httpd_auth]
; If you set this to true, you should also uncomment the WWW-Authenticate line
; above. If you don't configure a WWW-Authenticate header, CouchDB will send
; Basic realm="server" in order to prevent you getting logged out.
; require_valid_user = false

[os_daemons]
; For any commands listed here, CouchDB will attempt to ensure that
; the process remains alive. Daemons should monitor their environment
; to know when to exit. This can most easily be accomplished by exiting
; when stdin is closed.
;foo = /path/to/command -with args

[daemons]
; enable SSL support by uncommenting the following line and supply the PEM's below.
; the default ssl port CouchDB listens on is 6984
; httpsd = {chttpd, start_link, [https]}

[ssl]
;cert_file = /full/path/to/server_cert.pem
;key_file = /full/path/to/server_key.pem
;password = somepassword
; set to true to validate peer certificates
;verify_ssl_certificates = false
; Set to true to fail if the client does not send a certificate. Only used if verify_ssl_certificates is true.
;fail_if_no_peer_cert = false
; Path to file containing PEM encoded CA certificates (trusted
; certificates used for verifying a peer certificate). May be omitted if
; you do not want to verify the peer.
;cacert_file = /full/path/to/cacertf
; The verification fun (optional) if not specified, the default
; verification fun will be used.
;verify_fun = {Module, VerifyFun}
; maximum peer certificate depth
;ssl_certificate_max_depth = 1
;
; Reject renegotiations that do not live up to RFC 5746.
;secure_renegotiate = true
; The cipher suites that should be supported.
; Can be specified in erlang format "{ecdhe_ecdsa,aes_128_cbc,sha256}"
; or in OpenSSL format "ECDHE-ECDSA-AES128-SHA256".
;ciphers = ["ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA"]
; The SSL/TLS versions to support
;tls_versions = [tlsv1, 'tlsv1.1', 'tlsv1.2']

; To enable Virtual Hosts in CouchDB, add a vhost = path directive. All requests to
; the Virual Host will be redirected to the path. In the example below all requests
; to http:https://example.com/ are redirected to /database.
; If you run CouchDB on a specific port, include the port number in the vhost:
; example.com:5984 = /database
[vhosts]
;example.com = /database/

[update_notification]
;unique notifier name=/full/path/to/exe -with "cmd line arg"

; To create an admin account uncomment the '[admins]' section below and add a
; line in the format 'username = password'. When you next start CouchDB, it
; will change the password to a hash (so that your passwords don't linger
; around in plain-text files). You can add more admin accounts with more
; 'username = password' lines. Don't forget to restart CouchDB after
; changing this.
[admins]
;admin = mysecretpassword

[dharders]

I can confirm this too (Debian Jessie). At first I thought it was a Fauxton issue, or the fact I compiled from master source and did something wrong, but standard http/curl with official 2.1.0 release experiencing same bug as OP has stated above.

[wohali]

Can both of you confirm how you installed CouchDB? Was it from the packages, or from source? This may be something specific to the Debian/Ubuntu package configuration, which hasn't had as much testing as the source release. We've not seen this error when installing from source.

[dharders]

I installed from source firstly, then tried package, with both experiencing same issue. I simply followed instructions in INSTALL.Unix.md.

Other install details:

 Single-node
 Binding to 127.0.0.1
 chttpd -> require_valid_user = true    
 couch_httpd_auth -> require_valid_user = true    (also tested with false.... issue still present)
 admin password changed
 Chrome / Firefox / Curl all experienced same issue

Possibly related to an old similar issue to do with cookie/basic auth (and having to workaround by POST both Authorization: Base64(user:pass) AND Content-Type: application/x-www-form-urlencoded first request!). But could also be completely unrelated ;)

[wohali]

@dharders Interesting. If it's not package-related, we may be onto something real (or just a misconfiguration of some sort.)

Some questions:

1. With the require_valid_user settings set to false, does it still happen?
2. Your admin user - is it defined in the local.ini file, in the _users database, or both?
3. Can you post the full set of curl commands you've run to get this behaviour?

Thanks!

[dharders]

@wohali

1. Yes
2. Both, however, admin is defined _users db and in /opt/couchdb/etc/local.d/10-admins.ini instead of local.ini as my current installation used the package installer and interactive terminal put it there instead. Manually copying it to local.ini didn't seem to help either :(

///opt/couchdb/etc/local.d/10-admins.ini

# Package-introduced administrative user
[admins]
admin = -pbkdf2-longstring,10

[couchdb]
uuid = longstring

[chttpd]
bind_address = 127.0.0.1
port = 5984
require_valid_user = false

[cluster]
n = 1

[couch_httpd_auth]
secret = longstring
require_valid_user = false

3. Curl / Steps after install complete

// stop warnings in log if dbs don't exist
curl -u admin:admin -X PUT http:https://127.0.0.1:5984/_users
curl -u admin:admin -X PUT http:https://127.0.0.1:5984/_replicator
curl -u admin:admin -X PUT http:https://127.0.0.1:5984/_global_changes
curl -u admin:admin -X PUT http:https://127.0.0.1:5984/_metadata

//Check install by visiting http:https://127.0.0.1:5984/_utils
//Click verify, all good
//I think I then Clicked Setup, Single Node, admin & password, Bind to 127.0.0.1 Port 5984, even though 
//...I had done it via interactive terminal install ? Can't remember.

// Set require_valid_user to true
curl  -u admin:admin -X PUT 127.0.0.1:5984/_node/couchdb@localhost/_config/couch_httpd_auth/require_valid_user -d '"true"'
curl  -u admin:admin -X PUT 127.0.0.1:5984/_node/couchdb@localhost/_config/chttpd/require_valid_user -d '"true"'

// Set admin as member of db's
curl -u "admin:admin" -X PUT "127.0.0.1:5984/_users/_security" -d '{"admins":{"names":[],"roles":[]},"members":{"names":["admin"],"roles":[]}}'
curl -u "admin:admin" -X PUT "127.0.0.1:5984/_replicator/_security" -d '{"admins":{"names":[],"roles":[]},"members":{"names":["admin"],"roles":[]}}'
curl -u "admin:admin" -X PUT "127.0.0.1:5984/_global_changes/_security" -d '{"admins":{"names":[],"roles":[]},"members":{"names":["admin"],"roles":[]}}'
curl -u "admin:admin" -X PUT "127.0.0.1:5984/_metadata/_security" -d '{"admins":{"names":[],"roles":[]},"members":{"names":["admin"],"roles":[]}}'

// create new user
curl -u admin:admin -H 'Content-Type: application/json' -X PUT -d '{"_id": "org.couchdb.user:newuser", "type": "user", "name": "newuser", "password": "newuser", "roles": [] }' 127.0.0.1:5984/_users/org.couchdb.user:newuser

// create new db for newuser
curl -u "admin:admin" -X PUT "127.0.0.1:5984/newuser"

// set db permissions (access to admin and newuser)
curl -u "admin:admin" -X PUT "127.0.0.1:5984/newuser/_security" -d '{"admins":{"names":[],"roles":[]},"members":{"names":["admin","newuser"],"roles":[]}}'

// manually change admin password through fauxton
// goto any db and click permissions
// no members present
// chrome devtools show 401 Unauthorized fetch to db/_security, looks like no auth info sent in header
// manually type same db/_security url in address bar. 200 OK, but body contents is:
// {"error":"unauthorized","reason":"Authentication required.","members":{"names":["admin","newuser"]}} 

I also noted a few console errors within fauxton related to worker CSP 'unsafe-eval', DOMException all seemingly due to Worker related stuff.

[gambolputty]

Previously I had CouchDB 1.6 installed and it worked perfectly fine. Then I upgraded to 2.1.0, but purged the installation, because I had difficulties migrating my old DBs to the new installation (also replicating failed during the "verify installation"-process in Fauxton).

I installed 2.1.0 via apt-get as described in the docs.

Regarding your questions, @wohali:

1. Same results with require_valid_user set to false.
2. Admins are set in /op/couchdb/etc/local.d/10-admins.ini. Values in /op/couchdb/etc/local.ini seem to be ignored (as I realize by now thx to @dharders)
3. It took me just one CURL command to get this behaviour (the one I posted above)

file: /op/couchdb/etc/local.d/10-admins.ini

# Package-introduced administrative user
[admins]
admin = longstring

[couchdb]
uuid = 6424d76bc3dd3cd9ab9ef6edfad80786

[couch_httpd_auth]
secret = longstring

[httpd]
enable_cors = true
allow_jsonp = false

[cors]
credentials = true
origins = *
headers = accept, authorization, content-type, origin, referer
methods = GET, PUT, POST, HEAD, DELETE

[chttpd]
bind_address = 0.0.0.0
port = 5984
require_valid_user = false

[cluster]
n = 1

[rnewson]

well, this is fun. I think what you've done is written that error into the security object;

curl foo:bar@localhost:15984/db1/_security -XPUT -d ' {"error":"unauthorized","reason":"Authentication required.","members":{"names":["admin","newuser"]}} '
{"ok":true}

curl foo:bar@localhost:15984/db1/_security -s
{"error":"unauthorized","reason":"Authentication required.","members":{"names":["admin","newuser"]}}

can you do curl with the -v so we can see if you're getting, as I suspect, a 200 respose, not a 401.

[gambolputty]

@rnewson I am getting a 200 status code!

curl http:https://admin:password@server:5984/database/_security -v

*   Trying 1.2.3.4...
* TCP_NODELAY set
* Connected to 1.2.3.4 (1.2.3.4) port 5984 (#0)
* Server auth using Basic with user 'admin'
> GET /database/_security HTTP/1.1
> Host: 1.2.3.4:5984
> Authorization: Basic longstring==
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-CouchDB-Body-Time: 0
< X-Couch-Request-ID: 57fcb3932b
< Server: CouchDB/2.1.0 (Erlang OTP/18)
< Date: Thu, 10 Aug 2017 18:42:21 GMT
< Content-Type: application/json
< Content-Length: 137
< Cache-Control: must-revalidate
<
{"error":"unauthorized","reason":"You are not authorized to access this db.","admins":{"names":["admin"]},"members":{"names":["admin"]}}
* Curl_http_done: called premature == 0
* Connection #0 to host 1.2.3.4 left intact

Hm. I didn't execute any CURL commands before, nor did I touch the _security doc. Is it Fauxton-related?

[gambolputty]

Ok. I was able to reproduce this.

Step 1: update the _security-doc:

~/Python/projects $ curl http:https://admin:password@server:5984/database/_security -XPUT -d '{"admins":{"names":["admin"],"roles":[]},"members":{"names":[],"roles":[]}}'
{"ok":true}

Step 2: check the content of _security-doc

~/Python/projects $ curl http:https://admin:password@server:5984/database/_security -v
*   Trying 1.2.3.4...
* TCP_NODELAY set
* Connected to 1.2.3.4 (1.2.3.4) port 5984 (#0)
* Server auth using Basic with user 'admin'
> GET /newscorpus/_security HTTP/1.1
> Host: 1.2.3.4:5984
> Authorization: Basic JdNbLWnpE6WhWE4ddaWUAGYGnlU2ZtFydQRRUFtHjV==
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-CouchDB-Body-Time: 0
< X-Couch-Request-ID: a2d27bbf76
< Server: CouchDB/2.1.0 (Erlang OTP/18)
< Date: Thu, 10 Aug 2017 19:36:57 GMT
< Content-Type: application/json
< Content-Length: 76
< Cache-Control: must-revalidate
<
{"admins":{"names":["admin"],"roles":[]},"members":{"names":[],"roles":[]}}
* Curl_http_done: called premature == 0
* Connection #0 to host 1.2.3.4 left intact

Step 3: Navigate to Fauxton http:https://server:5984/_utils/ & log in
Step 4: Click on the database name & open the "Permissions"-page. "admin" is listed under admins on that page.
Step 5: Adding a new member ("test") works, but after reloading the page, "admin" & "test" listed before are gone and Chrome DevTools reveals 401 Unauthorized-Error (still logged in as admin). Response content:

{"error":"unauthorized","reason":"You are not authorized to access this db."}

Meanwhile member "test" was added to the _security-doc:

{"admins":{"names":["admin"],"roles":[]},"members":{"names":["test"],"roles":[]}}

Step 6: But reloading the "Permissions"-page doesn't reveal any name. I'm still getting a 401-Error in DevTools. And this is when the error message was written to the _security-doc, as @rnewson suggested: In my assumption that my changes made on the Permissions-page have not been saved (because I kept reloading the page and didn't see them), I tried to add the same member ("test") again. After reproducing this right now (adding the same member again), I can see the error message inside the _security-doc now, too, after executing CURL:

~/Python/projects $ curl http:https://admin:password@server:5984/database/_security
{"error":"unauthorized","reason":"You are not authorized to access this db.","members":{"names":["test"]}}

[wohali]

@dharders Signs point to a Fauxton bug. Can you tell me - before you go to Fauxton and change your admin password, right after the curl -u "admin:admin" -X PUT "127.0.0.1:5984/newuser/_security" -d '{"admins":{"names":[],"roles":[]},"members":{"names":["admin","newuser"],"roles":[]}}' command, can you do a curl http:https://admin:password@server:5984/database/_security -v and see if you get the same weird error or not?

[wohali]

@garrensmith you might want to have a look at the above, if confirmed we can move this issue over to the couchdb-fauxton repo if you like.

[dharders]

@wohali Yes your suspicions were correct. It does seem that it is a fauxton bug, as the issue isn't present if I curl from a fresh install BEFORE using fauxton permissions screen

curl -u "admin:admin" http:https://127.0.0.1:5984/newuser/_security -v
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
* Server auth using Basic with user 'admin'
> GET /newuser/_security HTTP/1.1
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.38.0
> Host: 127.0.0.1:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< X-CouchDB-Body-Time: 0
< X-Couch-Request-ID: 31b36ae0fe
* Server CouchDB/2.1.0 (Erlang OTP/18) is not blacklisted
< Server: CouchDB/2.1.0 (Erlang OTP/18)
< Date: Fri, 11 Aug 2017 00:41:18 GMT
< Content-Type: application/json
< Content-Length: 86
< Cache-Control: must-revalidate
<
{"admins":{"names":[],"roles":[]},"members":{"names":["admin","newuser"],"roles":[]}}
* Connection #0 to host 127.0.0.1 left intact

After visiting permissions screen (and seeing no present members) and adding 'admin' and 'newuser' to members then running the same curl command

curl -u "admin:admin" http:https://127.0.0.1:5984/newuser/_security -v
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
* Server auth using Basic with user 'admin'
> GET /newuser/_security HTTP/1.1
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.38.0
> Host: 127.0.0.1:5984
> Accept: */*
>
< HTTP/1.1 200 OK
< X-CouchDB-Body-Time: 0
< X-Couch-Request-ID: 4460928ace
* Server CouchDB/2.1.0 (Erlang OTP/18) is not blacklisted
< Server: CouchDB/2.1.0 (Erlang OTP/18)
< Date: Fri, 11 Aug 2017 00:45:44 GMT
< Content-Type: application/json
< Content-Length: 101
< Cache-Control: must-revalidate
<
{"error":"unauthorized","reason":"Authentication required.","members":{"names":["admin","newuser"]}}
* Connection #0 to host 127.0.0.1 left intact

Definitely a fauxton bug. It's adding the error message to the _security document, which is a weird thing to do.

So in summary, need to:

• Don't write error message in a 200 OK db/_security document GET
• Send auth Header data in Fauxton db/_security fetch on permissions page so can display members

[wohali]

This is a Fauxton bug, so I'm closing it out in apache/couchdb. See the other referenced issues for more info.