CouchDB sends A dns request when configured -proto_dist inet6_tcp

[sergey-safarov]

Description

We have an IPv6-only installation. No IPv4 is configured on the server (except loopback). In the vm.agrs configured

-proto_dist inet6_tcp

When I start replication to then CouchDB sends only A dns request and does not try to send AAAA dns requsts.
[NOTE]: # ( Describe the problem you're encountering. )
TIP: # ( Do NOT give us access or passwords to your actual CouchDB! )

Steps to Reproduce

Create request.json file like

{
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http:https://ippbx-1a.la.example.com:5984/mydb",
    "headers": {
      "Authorization": "Basic XXXXXXXXXXXX"
    }
  },
  "target": {
    "url": "http:https://127.0.0.1:5984/mydb",
    "headers": {
      "Authorization": "Basic XXXXXXXXX"
    }
  },
  "create_target": true,
  "continuous": false
}

Then send curl command like

curl \
 -H 'Content-Type: application/json' \
 -H 'Accept: application/json' \
 -d @request.json \
 'http:https://admin:[email protected]:5984/_replicator'

in the logs i can see

Jul 07 09:21:51 ippbx-1b.la.example.com docker[3677369]: [error] 2023-07-07T09:21:51.599467Z [email protected] <0.6095.0> -------- couch_replicator_httpc: auth plugin initialization failed "http:https://ippbx-1a.la.example.com:5984/mydb/" {session_request_failed,"http:https://ippbx-1a.la.example.com:5984/_session","admin",{conn_failed,{error,nxdomain}}}
Jul 07 09:21:51 ippbx-1b.la.example.com docker[3677369]: [error] 2023-07-07T09:21:51.599676Z [email protected] <0.6095.0> -------- throw:{replication_auth_error,{session_request_failed,"http:https://ippbx-1a.la.example.com:5984/_session","admin",{conn_failed,{error,nxdomain}}}}: Replication 69411eb9179c970995746a1ecd781fac+create_target failed to start "http:https://ippbx-1a.la.example.com:5984/mydb/" -> "http:https://127.0.0.1:5984/mydb/" doc <<"shards/00000000-7fffffff/_replicator.1688721035">>:<<"477deb9261a9927a1d20e90f2a001b31">> stack:[{couch_replicator_httpc,setup,1,[{file,"src/couch_replicator_httpc.erl"},{line,62}]},{couch_replicator_api_wrap,db_open,3,[{file,"src/couch_replicator_api_wrap.erl"},{line,68}]}]

Expected Behaviour

CouchDB should send a AAAA dns request to get a remote host IP address. If the AAAA request fails, then try to send an A DNS request.

Your Environment

• CouchDB version used:
  Used apache/couchdb:3.3.2 docker image.

• Browser name and version:
  Used curl to trigger replication.

• Operating system and version:

Additional Context

IPv6 only CouchDB installation.

[rnewson]

the erlang/otp proto_dist parameter controls how the erlang nodes communicate with each other with the distributed erlang protocol, it is not related to the protocols the (http-based) replicator uses.

Support for replicating to ipv6-only hosts was added back in couchdb 2.3 (by upgrading ibrowse).

given you are within an docker image, have you confirmed the environment is capable of ipv6 host resolution and routing? When you said couchdb only does an A lookup and not AAAA lookup is this speculation or do you have a confirming packet capture?

[sergey-safarov]

When you said couchdb only does an A lookup and not AAAA lookup is this speculation or do you have a confirming packet capture?

Yes, I have started replication and in another tab, I have started tcpdump -ni any port 53 and will able to see which DNS requests were issued to the DNS server. All requests for hostname I configured for replication is 'A'.

have you confirmed the environment is capable of ipv6 host resolution and routing

Please check my real result without "example" masking. The same results you will get on the your PC.

[centos@ippbx-1b ~]$ nslookup ippbx-0a.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ippbx-0a.la.nga911.com
Address: 2605:84c0:4a:604::6

[centos@ippbx-1b ~]$ nslookup ippbx-0b.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ippbx-0b.la.nga911.com
Address: 2605:84c0:4a:605::6

[centos@ippbx-1b ~]$ nslookup ippbx-1a.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ippbx-1a.la.nga911.com
Address: 2605:84c0:48:604::6

[centos@ippbx-1b ~]$ nslookup ippbx-1b.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Name:	ippbx-1b.la.nga911.com
Address: 2605:84c0:48:605::6

[sergey-safarov]

routing also works
standalone couchdb server with 3.3.2

[centos@ippbx-1b ~]$ curl http:https://ippbx-1b.la.nga911.com:5984
{"couchdb":"Welcome","version":"3.3.2","git_sha":"11a234070","uuid":"065b865d8615b590a225292060d6c415","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

couchdb cluster with 2.3.1

[centos@ippbx-1b ~]$ curl http:https://ippbx-0a.la.nga911.com:5984
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"e79401873eae0a741ad58421c4521c1e","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
[centos@ippbx-1b ~]$ curl http:https://ippbx-0b.la.nga911.com:5984
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"a581bad23b6f91b65710d9c2481f295a","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
[centos@ippbx-1b ~]$ curl http:https://ippbx-1a.la.nga911.com:5984
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"355ce185adb189bc576b757410c208d3","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

[rnewson]

Thanks. looks like this change in the third party http client we use (ibrowse): apache/couchdb-ibrowse@e6a0c36

try setting;

[replicator]
socket_options = [{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6

You'll need to restart couchdb if you edit the config files, or you can use the /_node/_local/_config endpoint to change the settings live.

[sergey-safarov]

on the server I have created file /opt/couchdb/etc/local.d/replicator.ini with content

[replicator]
; See the `inet` Erlang module's man page for the full list of options.
socket_options = [{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]

; Valid socket options. Options not in this list are ignored. The full list of
; options may be found at https://www.erlang.org/doc/man/inet.html#setopts-2.
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6

Then I mounted this file into the docker container. The CouchDB container started with this file mounted.

[root@ippbx-1b ~]# docker exec couch ls -l /opt/couchdb/etc/local.d/replicator.ini
-rw-r--r--. 1 couchdb couchdb 403 Jul 10 09:53 /opt/couchdb/etc/local.d/replicator.ini

Started replications again and using tcpdump can see A DNS requests

10:00:49.900494 IP 127.0.0.1.42238 > 127.0.0.1.domain: 44128+ A? ippbx-1a.la.nga911.com. (40)
10:00:49.900627 IP 127.0.0.1.domain > 127.0.0.1.42238: 44128* 0/1/0 (140)
10:00:49.900711 IP 127.0.0.1.52185 > 127.0.0.1.domain: 8288+ A? ippbx-1a.la.nga911.com. (40)
10:00:49.900828 IP 127.0.0.1.domain > 127.0.0.1.52185: 8288* 0/1/0 (140)

I also have checked local node config after CouchDB restart. I see replicator settings applied as expected.

[root@ippbx-1b ~]# curl -s -S  http:https://xxxxx:[email protected]:5984/_node/_local/_config | jq
{
  "chttpd": {
    "bind_address": "::",
    "port": "5984"
  },
  "admins": {
    "xxxxxx": "xxxxxxxxxxx"
  },
  "vendor": {
    "name": "The Apache Software Foundation"
  },
  "feature_flags": {
    "partitioned||*": "true"
  },
  "chttpd_auth": {
    "hash_algorithms": "sha256, sha"
  },
  "replicator": {
    "socket_options": "[{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]",
    "valid_socket_options": "buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6"
  },
  "log": {
    "level": "error"
  },
  "indexers": {
    "couch_mrview": "true"
  },
  "prometheus": {
    "additional_port": "false",
    "bind_address": "127.0.0.1",
    "port": "17986"
  },
  "httpd": {
    "bind_address": "::",
    "port": "5986"
  },
  "smoosh": {
    "state_dir": "./data"
  },
  "couch_httpd_auth": {
    "authentication_db": "_users",
    "secret": "c8bf8a8d517074d779a1a31a9dac75bd"
  },
  "couchdb_engines": {
    "couch": "couch_bt_engine"
  },
  "couchdb": {
    "database_dir": "./data",
    "uuid": "065b865d8615b590a225292060d6c415",
    "view_index_dir": "./data"
  }
}

[rnewson]

hi,

I've tried this locally and it doesn't work, sorry for the bad steer.

I see several bugs that prevent you from getting the necessary {prefer_ipv6, true} setting to where it needs to be, we'll fix in the next release.

[rnewson]

we have a fix for the next release, thank you for filing the issue!

[sergey-safarov]

Should I use?

[replicator]
socket_options = [{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6
ibrowse_options = [{prefer_ipv6, true}]

[rnewson]

@sergey-safarov when there's a release with the fix all you'll need to do is;

[replicator]
ibrowse_options = [{prefer_ipv6, true}]

and then ipv6 will be attempted for all replications, which is appropriate for your ipv6-only setup.

In other setups you might just specify {"source":"X", ... "ibrowse_options":"[{prefer_ipv6,true}]}"} on certain replication docs as needed.