The replication job for couchdb is stucked

[zhangjingcat]

Description

[NOTE]: # The replication job for couchdb is stucked.

Steps to Reproduce

source couchdb-A : a version 3.3.3 cluster with 3 nodes on RHEL 9 vms
source couchdb-B : a version 3.3.3 cluster with 3 nodes on RHEL 9 vms

1.Create a database in the source couchdb-A named "penny" and added two documents naming {"_id":"penny-001"} and {"_id":"penny-002"}.

2.Config a one-time replication job in target couchdb-B named "penny" and run the pulling job replicating form couchdb-A, the 2 documents are located into"penny" in couchdb-B very quickly.

3.Delete the previous replication job.

4.Added one more document into "penny" in couchdb-B named {"_id":"penny-003"}.

5.Reverse the replication order, make couchdb-B to be source, make couchdb-A to be target. Config a one-time replication job from source couchdb-B "penny" to target couchdb-A "penny", the data does not locate in couchdb-A "penny" even after waiting a very long time.

Your Environment

I am testing in RHEL 9 vms and very puzzled by this issue for the replication failure. And it seems something might be wrong in source "couchdb-A" or target "couchdb-B". I'm not sure whether we could do the replication from each order or we should only follow one order to sync data, but I think the official document said it should only follows the _changes api to get data so that we should be able to sync as long as we could get the response from _changes.

One discovery from my side (for couchdb-B) :

https://username:[email protected]/penny/_changes?feed=longpoll&filter=CustomFilters/RelCompLicFilter&limit=2000&since=0

{
"error": "not_found",
"reason": "missing"
}

This api is failed so I think this could be the root cause for syncing data from B to A, but I do not know how to fix it. Hope you could give us some valuable suggestions. Thank you very much!

The ini files we used is, does some of the labels affects the replication behavior ?

[couchdb]
max_document_size = 4294967296 ; bytes
uuid = ca6e9bf190c040ec9987fc5512563704
database_dir = ./data
view_index_dir = ./data
file_compression = snappy
max_document_size = 8000000 ; bytes
os_process_timeout = 7000 ; 7 seconds. for view servers.
max_document_size = 8000000 ; bytes
max_dbs_open = 500
attachment_stream_buffer_size = 8192

[purge]
max_document_id_number = 1000
max_revisions_number = 1000
index_lag_warn_seconds = 86400

[cluster]
q=8
n=3
seedlist = xxx,xxx,xxx

[database_compaction]
doc_buffer_size = 524288
checkpoint_after = 5242880

[view_compaction]
keyvalue_buffer_size = 2097152

[smoosh]
db_channels = upgrade_dbs,ratio_dbs,slack_dbs
view_channels = upgrade_views,ratio_views,slack_views
cleanup_channels = index_cleanup
compaction_log_level = info
persist = true

[smoosh.ratio_dbs]
priority = ratio
min_priority = 2.0

[smoosh.ratio_views]
priority = ratio
min_priority = 2.0

[smoosh.slack_dbs]
priority = slack
min_priority = 536870912

[smoosh.slack_views]
priority = slack
min_priority = 536870912

[attachments]
compression_level = 9
compressible_types = text/*, application/javascript, application/json, application/xml

[uuids]
algorithm = sequential

[stats]
interval = 10

[ioq]
concurrency = 12
ratio = 0.01

[ioq.bypass]
os_process = true
read = true
write = true
view_update = true

[replicator]
max_jobs = 500
interval = 60000
max_churn = 20
max_history = 20
worker_batch_size = 500
worker_processes = 8
http_connections = 20
retries_per_request = 5
connection_timeout = 30000
socket_options = [{keepalive, true}, {nodelay, false}]
checkpoint_interval = 5000
use_checkpoints = true
use_bulk_get = true
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf

[chttpd]
bind_address = 0.0.0.0
buffer_response = true
max_http_request_size = 4294967296 ; 4 GB

[httpd]
server_options = [{backlog, 1024}, {acceptor_pool_size, 16}, {max, 8192}]

[chttpd_auth]
secret = f2336db1
allow_persistent_cookies = true
auth_cache_size = 50
timeout = 600

[rexi]
buffer_count = 2000
server_per_node = true
use_kill_all = true
stream_limit = 7

[prometheus]
additional_port = true
bind_address = 0.0.0.0
port = 9985

[admins]
admin = xxxxxx

• CouchDB version used: 3.3.3
• Browser name and version: N/A
• Operating system and version: RHEL 9

[rnewson]

Can you clarify step 5? I assume you meant that document "penny-003" did not appear in couchdb-A?

Further questions;

1. The one-shot replications return important information in the response like the number of documents written to the target. Can you please show those?
2. Are you running any proxy or other software in front of either couchdb server? If so, what is it and what is it doing?
3. In your 5 step example did you use a filter on the replication?
4. Your discovery of "not_found" missing is perhaps a failure to find the design document that your filter is in, rather than suggesting that the _changes endpoint itself is missing (which would indicate a broken installation).

[zhangjingcat]

Can you clarify step 5? I assume you meant that document "penny-003" did not appear in couchdb-A?

Further questions;

1. The one-shot replications return important information in the response like the number of documents written to the target. Can you please show those?
2. Are you running any proxy or other software in front of either couchdb server? If so, what is it and what is it doing?
3. In your 5 step example did you use a filter on the replication?
4. Your discovery of "not_found" missing is perhaps a failure to find the design document that your filter is in, rather than suggesting that the _changes endpoint itself is missing (which would indicate a broken installation).

Hi @rnewson , thanks a lot for your quick reply!

Yes the "penny-003" did not appear in couchdb-A.

1.I used UI to config the replications and did not keep the response. What I remembered is 0 Documents written and null pending. (I can try to re-produce that and copy the new result here for you later)

2.No.

3.I did not use the filter on replication and I just follow the UI instructions to use a source and target.

4.I have some more discoveries depending on the .ini files that might cause the replication to fail.

This is the special case for data syncing , I carried out some tests on replicating data from local to local (with another dummy name).

For example, we have a "sw360db" which is a small database, and a "sw360changelogs" which is a large database.

The replication job from local existing "sw360db" database to local new "penny1" database succeed.

The replication job from local existing "sw360changelogs" database to local new "penny2" database failed. (0 documents written and null pending)

I'm suspicious of the [replicator] configurations and carrying out some tests to prove my assumptions.

[couchdb]
max_document_size = 4294967296 ; bytes
uuid = ca6e9bf190c040ec9987fc5512563704
database_dir = ./data
view_index_dir = ./data
file_compression = snappy
max_document_size = 8000000 ; bytes
os_process_timeout = 7000 ; 7 seconds. for view servers.
max_document_size = 8000000 ; bytes
max_dbs_open = 500
attachment_stream_buffer_size = 8192

[purge]
max_document_id_number = 1000
max_revisions_number = 1000
index_lag_warn_seconds = 86400

[cluster]
q=8
n=3
seedlist = xxx,xxx,xxx

[database_compaction]
doc_buffer_size = 524288
checkpoint_after = 5242880

[view_compaction]
keyvalue_buffer_size = 2097152

[smoosh]
db_channels = upgrade_dbs,ratio_dbs,slack_dbs
view_channels = upgrade_views,ratio_views,slack_views
cleanup_channels = index_cleanup
compaction_log_level = info
persist = true

[smoosh.ratio_dbs]
priority = ratio
min_priority = 2.0

[smoosh.ratio_views]
priority = ratio
min_priority = 2.0

[smoosh.slack_dbs]
priority = slack
min_priority = 536870912

[smoosh.slack_views]
priority = slack
min_priority = 536870912

[attachments]
compression_level = 9
compressible_types = text/*, application/javascript, application/json, application/xml

[uuids]
algorithm = sequential

[stats]
interval = 10

[ioq]
concurrency = 12
ratio = 0.01

[ioq.bypass]
os_process = true
read = true
write = true
view_update = true

[replicator]
max_jobs = 500
interval = 60000
max_churn = 20
max_history = 20
worker_batch_size = 500
worker_processes = 8
http_connections = 20
retries_per_request = 5
connection_timeout = 30000
socket_options = [{keepalive, true}, {nodelay, false}]
checkpoint_interval = 5000
use_checkpoints = true
use_bulk_get = true
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf

[chttpd]
bind_address = 0.0.0.0
buffer_response = true
max_http_request_size = 4294967296 ; 4 GB

[httpd]
server_options = [{backlog, 1024}, {acceptor_pool_size, 16}, {max, 8192}]

[chttpd_auth]
secret = f2336db1
allow_persistent_cookies = true
auth_cache_size = 50
timeout = 600

[rexi]
buffer_count = 2000
server_per_node = true
use_kill_all = true
stream_limit = 7

[prometheus]
additional_port = true
bind_address = 0.0.0.0
port = 9985

[admins]
admin = xxxxxx

Steps:

1.Create the one-time replication job

2.Check the replication job status

3. See if any documents succeed, but unfortunately none

I got this error when creating the replication job.

Request:

curl --request POST
--url https://xxx:xxx@xxxxxx/_replicate
--header 'Content-Type: application/json'
--data '
{
"source": "https://xxx:xxx@xxxxxx/sw360changelogs",
"target": "https://xxx:xxx@xxxxxx/penny",
"create_target" : true
}'

Response:

{
"error": "changes_req_failed",
"reason": "{error,req_timedout}"
}

[rnewson]

Ok, so it sounds like the replication simply crashes at the start, which explains the state of the databases, we don't get as far as knowing which documents on the source need replicating to the target.

I don't see anything in your configuration that explains the issue.

I see you are using https but your configuration did not include the settings to enable that natively in couchdb. is there something else providing https or did you just omit those settings from the configuration above?

Can you test independently of the replicator? e.g, with ApacheBench, something like ab -n 5000 -A user:pass <url to source db/_changes> and see if they all succeed.

[zhangjingcat]

Hi @rnewson ,thanks for your suggestions. Actually we created the DNS pointing to the ip address in a separate place on cloud configuration panel. I tried use the ApacheBench to see the results ,here attached the response(70007). Actually for other database the replication were quite fast and not sure if the setup .ini params affects the efficiency.

I used local way to set replication job but also failed.

Request:

{
"source": "sw360changelogs",
"target": "penny",
"create_target" : true
}

Response:
{
	"error": "changes_req_failed",
	"reason": "{error,req_timedout}"
}

For sw360changelogs Test Results

(base) i545542@C02FD05EMD6T ~ % ab -n 5000 -A user:pass https://FQDN/sw360changelogs/_changes
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http:https://www.zeustech.net/
Licensed to The Apache Software Foundation, http:https://www.apache.org/

Benchmarking FQDN (be patient)
apr_pollset_poll: The timeout specified has expired (70007)

And I know the replication might be difficult for big databases and could we declare a small limit under [replicator] label and make it faster for us to get each partition for the whole changes, is it supported by setting this in [replicator] for .ini files now? Thank you very much!

[rnewson]

This does not sound like a replicator problem. Your couchdb is not contactable at all, focus on testing connectivity with curl and apachebench, figure out why that is not working first, and then the replication problem will also be solved.

There is no "local way" for replication, internally the "target":"penny" is converted to a url anyway, all replication is over http/https, so you should specify source and target as url's yourself for clarity.

try curl https://FQDN/sw360changelogs/_changes and tell me if that works. try it ten times and tell me if any fail.

[zhangjingcat]

Hi @rnewson ,thanks a lot for the hints!
Now I tried the "curl https://FQDN/sw360changelogs/_changes" several times and it took about 4minutes to respond as our sw360changelogs database is too huge and have 1818337 docs inside. We may try to increase the connect time out for replicator label in .ini first. It confused me is that when we reduce case to use very simple .ini file like below we do not face the replicator issue for sw360changelogs, but if we change back to use the complex .ini file for db setup then the timeout issue would appear.

The simple .ini file that works.

[admins]
xxx = xxx

[couchdb]
uuid = 9cfc779d1c23054f8e515f92042695c0
max_document_size = 4294967296 ; bytes
database_dir = ./data
view_index_dir = ./data
file_compression = snappy
os_process_timeout = 7000 ; 7 seconds. for view servers.
max_dbs_open = 500
attachment_stream_buffer_size = 8192

[couch_httpd_auth]
secret = fdccaf1f00c36ea9550a84b423b2e755

[chttpd]
bind_address = 0.0.0.0
port = 5984
max_http_request_size = 4294967296 ; 4 GB

[httpd]
max_http_request_size = 4294967296 ; 4 GB
server_options = [{backlog, 1024}, {acceptor_pool_size, 16}, {max, 8192}]

[cluster]
n = 1

[replicator]
max_jobs = 500
interval = 60000
max_churn = 20
max_history = 20
worker_batch_size = 500
worker_processes = 8
http_connections = 20
retries_per_request = 5
connection_timeout = 30000
socket_options = [{keepalive, true}, {nodelay, false}]
checkpoint_interval = 5000
use_checkpoints = true
use_bulk_get = true
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf

[prometheus]
additional_port = true
bind_address = 0.0.0.0
port = 9985

[rnewson]

you say 4 minutes to respond, is that accurate? I would expect retrieving the entire changes response body will take time, but that is not a problem (the replicator processes it as a stream, no matter how long it is).

the req timedout error is about whether the response even starts, not finishes.

Try curl again and add ?limit=1, tell me if that is quick and reliable.

[nickva]

I wonder if it's the buffer_response = true setting, noticed the simpler ini file didn't have that?

[zhangjingcat]

you say 4 minutes to respond, is that accurate? I would expect retrieving the entire changes response body will take time, but that is not a problem (the replicator processes it as a stream, no matter how long it is).

the req timedout error is about whether the response even starts, not finishes.

Try curl again and add ?limit=1, tell me if that is quick and reliable.

Yes with limit=1 it responds quick. Not sure whether the data length matters.

[zhangjingcat]

I wonder if it's the buffer_response = true setting, noticed the simpler ini file didn't have that?

Thanks for pointing that, I'll do some compare tests then.

[rnewson]

hm ok, one tip, you really should remove buffer_response = true setting. that is not the default and it causes couchdb to attempt to build the entire response in memory before starting the response.

[rnewson]

sidebar: we should exclude _changes response from that setting, or perhaps remove it entirely.

[zhangjingcat]

hm ok, one tip, you really should remove buffer_response = true setting. that is not the default and it causes couchdb to attempt to build the entire response in memory before starting the response.

Ok thanks a lot! Let me try the re-setup without the param.

[rnewson]

sorry I didn't notice it sooner, but thanks to @nickva for spotting it.