Process timeout error when view opens

[DoubleBoba]

"OS process timed out" error when trying to open big view. View size is hundreds of thousands documents and almost 400mb on disc.

Steps to Reproduce

1. upload big amount of data to some database
2. create view for this database
3. provide simple map function:

function (doc) {
    emit(doc._id, 1);
}

5. save view and wait for index to build
6. try to open the view
7. get the "OS process timed out" error

Expected Behaviour

view just opens

Your Environment

{
    "couchdb": "Welcome",
    "version": "3.2.1",
    "git_sha": "244d428af",
    "uuid": "6c8c861378cadc7a55a1e3565a456dc8",
    "features":
    [
        "access-ready",
        "partitioned",
        "pluggable-storage-engines",
        "reshard",
        "scheduler"
    ],
    "vendor":
    {
        "name": "The Apache Software Foundation"
    }
}

• Firefox 110.0
• couchDB runs on Ubuntu 20.04, kernel is 5.4.0-139-generic

Additional Context

Related logs:

[info] 2023-02-22T11:42:38.082498Z nonode@nohost <0.286.0> -------- couch_proc_manager <0.1161.25> died normal
[error] 2023-02-22T11:42:38.082591Z nonode@nohost <0.1112.25> -------- OS Process Error <0.1161.25> :: {os_process_error,"OS process timed out."}
[error] 2023-02-22T11:42:38.084320Z nonode@nohost <0.1100.25> 48dd004b0d rexi_server: from: nonode@nohost(<0.480.25>) mfa: fabric_rpc:map_view/5 throw:{os_process_error,"OS process timed out."} [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,138}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,82}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,262}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,141}]}]
[error] 2023-02-22T11:42:38.084764Z nonode@nohost emulator -------- Error in process <0.1112.25> with exit value:
{{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}

[error] 2023-02-22T11:42:38.085516Z nonode@nohost emulator -------- Error in process <0.1112.25> with exit value:
{{nocatch,{os_process_error,"OS process timed out."}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}

[error] 2023-02-22T11:42:38.086140Z nonode@nohost <0.480.25> 48dd004b0d req_err(157986948) os_process_error : OS process timed out.
    [<<"couch_mrview_util:get_view_index_state/5 L138">>,<<"couch_mrview_util:get_view/4 L82">>,<<"couch_mrview:query_view/6 L262">>,<<"rexi_server:init_p/3 L141">>]
[notice] 2023-02-22T11:42:38.087003Z nonode@nohost <0.480.25> 48dd004b0d 10.20.208.57:25984 10.31.32.38 admin GET /nvd/_design/test_view/_view/new-view?reduce=false&skip=0&limit=21 500 ok 24777

I already have tried to increase os_process_timeout but it didn't affect anything.

[big-r81]

Is the view getting fully build when you query it?

[nickva]

Try increasing the [query_server_config] os_process_limit to something higher, say 500 and see if that helps.

Do you see any other errors or crashes in the log? Or use any reduce functions that may produce large results.

[DoubleBoba]

@big-r81, I wait until all tasks in tasks view are done, and only then trying to open view.

@nickva, thank you for the suggestion. It doesn't help, i even tried 50000, but problem is still here. I don't use any reduce function at all. Tried _count, but result is same.

Also I've updated my couchdb installation to 3.3.1, but it doesn't help too.

[DoubleBoba]

Installed couchdb locally on my macbook and have got exactly same problem with same data. I found that there are document with more than 120mb of data. I have deleted this document from database and view was loaded successfully. But it doesn't eliminate the problem. I need this document to be in my database and i cannot see any problem for couch to hold and process it. My max_document_size is equals to 524288000 (500mb).

Fact that couchdb doesn't provide any meaningful error message is very frustrating. Process obviously doesn't time out, it just fails at some point. I've enabled debug logs and found the id of last processed document logged before error has happened and it was exactly same id that I've deleted. I tried to increase all possible process limits but it didn't affect situation at all. It continued to fail after 5-10 seconds when i opened view.

Question is still here: how to eliminate the error and put 120mb document to database?

[DoubleBoba]

After some research i've got much more ridiculous behavior. I've loaded big document back to reproduce error again and for some reason error is different now. It just crashes with out of memory after it eats all available 10GB of available RAM.

couchdb-couchdb-1  | out of memory
couchdb-couchdb-1  | [error] 2023-02-27T11:43:22.305878Z nonode@nohost <0.735.0> -------- OS Process Error <0.756.0> :: {os_process_error,{exit_status,1}}
couchdb-couchdb-1  | [info] 2023-02-27T11:43:22.305884Z nonode@nohost <0.274.0> -------- couch_proc_manager <0.756.0> died normal
couchdb-couchdb-1  | [error] 2023-02-27T11:43:22.306495Z nonode@nohost <0.719.0> 7f0651721d rexi_server: from: nonode@nohost(<0.562.0>) mfa: fabric_rpc:map_view/5 throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,208}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,153}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,286}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,146}]}]
couchdb-couchdb-1  | [error] 2023-02-27T11:43:22.306650Z nonode@nohost emulator -------- Error in process <0.735.0> with exit value:
couchdb-couchdb-1  | {{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,73}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,174}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,181}]}]}
couchdb-couchdb-1  |
couchdb-couchdb-1  | [error] 2023-02-27T11:43:22.306791Z nonode@nohost emulator -------- Error in process <0.735.0> with exit value:
couchdb-couchdb-1  | {{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,73}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,174}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,181}]}]}
couchdb-couchdb-1  |
couchdb-couchdb-1  | [error] 2023-02-27T11:43:22.306886Z nonode@nohost <0.562.0> 7f0651721d req_err(810725583) os_process_error : {exit_status,1}
couchdb-couchdb-1  |     [<<"couch_mrview_util:get_view_index_state/5 L208">>,<<"couch_mrview_util:get_view/4 L153">>,<<"couch_mrview:query_view/6 L286">>,<<"rexi_server:init_p/3 L146">>]
couchdb-couchdb-1  | [notice] 2023-02-27T11:43:22.307330Z nonode@nohost <0.562.0> 7f0651721d localhost:25984 172.18.0.1 admin GET /nvd/_design/test_view/_view/new-view?reduce=false&skip=0&limit=21 500 ok 13040
couchdb-couchdb-1  | [debug] 2023-02-27T11:45:08.436952Z nonode@nohost <0.723.0> -------- Index shutdown by monitor notice for db: shards/80000000-ffffffff/nvd.1677477593 idx: _design/test_view
couchdb-couchdb-1  | [debug] 2023-02-27T11:45:08.437287Z nonode@nohost <0.723.0> -------- Closing index for db: shards/80000000-ffffffff/nvd.1677477593 idx: _design/test_view sig: "9a0adb2bcc4e63b9bb9de044429dd869" because normal
couchdb-couchdb-1  | [debug] 2023-02-27T11:46:09.438794Z nonode@nohost <0.724.0> -------- Index shutdown by monitor notice for db: shards/00000000-7fffffff/nvd.1677477593 idx: _design/test_view
couchdb-couchdb-1  | [debug] 2023-02-27T11:46:09.439016Z nonode@nohost <0.724.0> -------- Closing index for db: shards/00000000-7fffffff/nvd.1677477593 idx: _design/test_view sig: "9a0adb2bcc4e63b9bb9de044429dd869" because normal

It happens while processing exactly same document. Checked my another server with same problem and memory usage is normal while i still get os process timed out error.

[big-r81]

Hi,

some suggestions:

1. You shouldn't put such large documents in your db.
2. Did your view process this large file? Can you exclude it from view-processing?
3. You can also try to increase [couchdb] os_process_timeout, but I'm really shure you hit an request timeout.
4. Try to increase [fabric] request_timeout.
5. Maybe you hit the memory limit of the Javascript view server (default 64MB).

[nickva]

@big-r81 good idea about couchjs memory limit.

Seeing {{nocatch,{os_process_error,{exit_status,1}}}, point to that as well.

With documents that large couchjs process will crash because it has a heap size limit of 64Mb by default. You can try to increase using the -S ... parameter. That parameter can be inserted in the COUCHDB_QUERY_SERVER_JAVASCRIPT=".../couchjs -S 536870912 .../main.js. Normally that's defined in bin/couchdb startup script but you can set it outside of it and it will take your value.

See https://docs.couchdb.org/en/stable/config/query-servers.html#query-servers-definition for more info about it

[DoubleBoba]

@nickva, thank you. It works as expected now.