Cluster setup API does not support adding nodes via SSL

[dsietz]

When adding a node to a CouchDB 2.1 cluster using the Cluster Setup API, using the default SSL port 6984 with a self-signed cert, returns the following response:
{"error":"bad_request","reason":"retry_later"}

Expected Behavior

Return value
{"ok":true}

Current Behavior

Is clustering over a SSL port is not supported? If not, then it presents a potential security issue.

Steps to Reproduce (for bugs)

1. Setup 2 nodes with SSL (self signed cert) using the default port 6984
2. Follow the setup steps in the Cluster API where the remote port is 6984
   (NOTE: This error also appears when using Fuxton)

Your Environment

• Version used: 2.1
• Browser Name and version: Chrome
• Operating System and version (desktop or mobile): Ubutnu

[wohali]

At the moment, the Cluster Setup API doesn't officially support SSL connections - meaning we haven't tested that configuration. It's really a shortcut to help people who haven't learned the manual process, and isn't intended to cover all cases.

As a workaround, you should be able to join the cluster together by following the manual process for adding nodes as described here:

http:https://docs.couchdb.org/en/2.1.0/cluster/nodes.html

Once all nodes have been added, you'll then need to manually create the system databases (_replicator, _users and _global_changes).

One thing to watch out for is that as you create the admin user on every node independently, you'll need to be sure that the ini file contains the identical hashed password on each node if you expect sessions to work correctly with the entire cluster residing behind a load balancer.

[dsietz]

Thank you wohali for the fast reply and followup information.
However, if I manually add the nodes to the cluster using the call...

curl -X PUT "http:https://xxx.xxx.xxx.xxx:5986/_nodes/[email protected]" -d {}
... how do I specify that the clustered nodes uses the port 6984 (not the default 5984) when "managing / syncing" data? For example, I will come in the add a document to a data base on port 6984 (SSL), but will the cluster automatically "talk" between the nodes (e.g.: replicating their data) using that port? The _membership states the all_nodes and cluster_nodes are (e.g.: [email protected]).

I probably don't understand the cluster architecture well enough, so sorry for any misunderstanding on my part.

[wohali]

CouchDB doesn't sync between nodes via the HTTP interface. Instead, it uses the Erlang distribution mechanism, which is managed via the Erlang port mapper daemon (epmd). This traffic occurs on dynamically allocated ports, the range of which can be altered. We describe the options necessary for constraining the epmd port range here: http:https://docs.couchdb.org/en/2.1.0/cluster/setup.html

To encrypt this traffic as well is beyond the scope of our installation docs. You can read more about the process here: https://www.erlang-solutions.com/blog/erlang-distribution-over-tls.html

[dsietz]

Thank you VERY much for the help. That makes it all clear. I will work through the Erlang documentation to secure the port (e.g.: 9100)

PS: I would never get such faster and specific support from a vendor. Hat's off to your team!

[wohali]

Note you need 1 port open per node in your cluster. Limiting to JUST port 9100 means you can only have a 2-node cluster.

This was wrong, instead SEE BELOW: #783 (comment)

[dsietz]

good point. Thanks for the tip!

[dsietz]

If anyone is interested, I figured out how to get ports 9100 ( through 9200) using SSL.
I added the following lines to the following files

local.ini

[ssl]
port = 6984
cert_file = /etc/couchdb/cert/couchdb.pem
key_file = /etc/couchdb/cert/privkey.pem
ciphers = undefined
tls_versions = undefined
secure_renegotiate = undefined
certfile = /etc/couchdb/cert/couchdb.pem
keyfile = /etc/couchdb/cert/privkey.pem
cacertfile = undefined

vm.args

-kernel inet_dist_listen_min 9100
-kernel inet_dist_listen_max 9200
-pa /opt/couchdb/cert
-proto_dist inet_tls
-ssl_dist_opt server_certfile /etc/couchdb/cert/couchdb.pem server_keyfile /etc/couchdb/cert/privkey.pem

To verify a port is secure I used the following command:

openssl s_client -connect xxx.xxx.xxx.xxx:9100

[dsietz]

Just to add to wohali statement

One thing to watch out for is that as you create the admin user on every node independently, you'll need to be sure that the ini file contains the identical hashed password on each node if you expect sessions to work correctly with the entire cluster residing behind a load balancer.

You also need to make sure that the secret is also the same on all the nodes if you are using cookie authentication.

[couch_httpd_auth]
secret

[wohali]

I want to revise my advice on the inet_dist_listen range, as I was wrong. You should be able to crank this down to a single port, i.e. have _min and _max match. All the restriction does is to say "this local node can use this range (of 1)." So long as you don't try to open a second node with that same port range restriction on the same machine, you're fine.

[janl]

The fix for this would be to allow self-signed certs in the request options for the ibrowse requests sent in src/setup/src/setup.erl.