Rewritten PUT of attachment hangs

[nkronlage]

Expected Behavior

Using a simple rewrite javascript function that maps to the non-rewrite path should allow uploading attachments.

Current Behavior

Using PUT to upload attachments hangs and eventually returns {"error":"unknown_error","reason":"undefined"}.

This appears to be very similar to issue #1612. @ermouth, in case you know about this.

Steps to Reproduce

This creates a database r and a design doc n that has a rewrite function that should pass the requests to the r database unmodified. Using the rewrite function to GET or PUT documents works fine. Using it to PUT an attachment causes the connection to hang for a while and then return an 'unknown_error'.

~$ curl http:https://127.0.0.1:5984
{"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"e07d95a0b0610b7bcb8e046fff5dd1b6","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

~$ curl -X PUT 'http:https://127.0.0.1:5984/r'
{"ok":true}

~$ curl -X PUT 'http:https://127.0.0.1:5984/r/_design/n' -d $'{"rewrites": "function (r) { return { path:\'../../\' + r.path.slice(4).join(\'/\') } } "}'
{"ok":true,"id":"_design/n","rev":"1-c19e01e83ea13c586401fa1fa66c6997"}

~$ curl -X PUT http:https://127.0.0.1:5984/r/_design/n/_rewrite/foo -d '{ }'
{"ok":true,"id":"foo","rev":"1-967a00dff5e02add41819138abb3284d"}

~$ curl -X PUT 'http:https://127.0.0.1:5984/r/_design/n/_rewrite/foo/test.jpg?rev=1-967a00dff5e02add41819138abb3284d' --data-binary @test.jpg -H "Content-Type: image/jpg"

HANG!!!!

{"error":"unknown_error","reason":"undefined"}

~$ curl -X PUT 'http:https://127.0.0.1:5984/r/foo/test.jpg?rev=1-967a00dff5e02add41819138abb3284d' --data-binary @test.jpg -H "Content-Type: image/jpg"
{"ok":true,"id":"foo","rev":"2-13c13abad5b38cf5d5a321a00ebf8b77"}

[wohali]

The JavaScript engine has not, and will not, have access to document attachments. Sorry.

[ermouth]

@nkronlage I‘ll look into it. @wohali, your verdict seems bit premature.

[wohali]

@ermouth let me know if you find a problem, i assumed it was related to attachments.

[ermouth]

JS rewrite definitely sees attachments sent in that way, which may be checked using ddoc returning request JSON:

{
  "_id": "_design/reflect",
  "rewrites": "function (req) {\n\t\treturn {\n\t\t\tcode:200,\n\t\t\tpath:'/',\n\t\t\tbody:JSON.stringify(req, null, '  ')\n\t\t};\n  }"
}

However, it’s unclear what happens when rewrite fn conveys result to Couch.

Explicitly defining {path, method, body, headers, query} of a rewrite result I see no hang, but I still can’t save an attach this way. Regardless how I pass revision – as query.rev, or as If-Match header – I get {"error":"not_found","reason":"missing_rev"}.

Gonna dig into it, smth interesting.

[ermouth]

@wohali I see this in logs:

[debug] 2019-01-28T11:00:38.326369Z [email protected] <0.2272.0> -------- OS Process #Port<0.8003> Input  :: ["ddoc","_design/clone",["rewrites"],[{"method":"PUT","requested_path":["test","_design","clone","_rewrite","foo","z.txt"],"path":["test","_design","clone","_rewrite","foo","z.txt"],"raw_path":"/test/_design/clone/_rewrite/foo/z.txt","query":{},"headers":{"Accept":"*/*","Content-Length":"4","Content-Type":"text/plain","Host":"127.0.0.1:5984","If-Match":"3-bc676df6ad1244ac5a888c629c908e68","User-Agent":"curl/7.47.0"},"body":"123\n","peer":"127.0.0.1","cookie":{},"userCtx":{"db":"test","name":null,"roles":[]},"secObj":{}}]]
[debug] 2019-01-28T11:00:38.327145Z [email protected] <0.2272.0> -------- OS Process #Port<0.8003> Output :: ["ok",{"path":"../../../test/foo/z.txt","method":"PUT","body":"123\u000a","headers":{"Accept":"*/*","Content-Length":"4","Content-Type":"text/plain","Host":"127.0.0.1:5984","If-Match":"3-bc676df6ad1244ac5a888c629c908e68","User-Agent":"curl/7.47.0"}}]
[debug] 2019-01-28T11:00:38.327312Z [email protected] <0.17577.6> 4afc9d08b5 rewrite to "/test/foo/z.txt"
[error] 2019-01-28T11:01:38.328979Z [email protected] <0.17577.6> 978d0bbb83 fabric_worker_timeout update_docs,'[email protected]',<<"shards/80000000-9fffffff/test.1548661440">>
[notice] 2019-01-28T11:01:38.329209Z [email protected] <0.17577.6> 978d0bbb83 127.0.0.1:5984 127.0.0.1 undefined PUT /test/foo/z.txt 500 ok 60002
[notice] 2019-01-28T11:01:38.329243Z [email protected] <0.17577.6> 978d0bbb83 127.0.0.1:5984 127.0.0.1 undefined PUT /test/_design/clone/_rewrite/foo/z.txt 500 ok 60006

Can you please give a hint what may cause fabric_worker_timeout update_docs?

[wohali]

It means the amount of time necessary to process the document update was exceeded. As you can see from the logfile it took 60s.

Is it taking JS longer than 60s to process the document, perhaps? That seems likely...

[ermouth]

Is it taking JS longer than 60s to process the document, perhaps? That seems likely...

According to logs JS processing took <1ms (38.326369—38.327145), which is very consistent with my experience about JS rewrites: they are fast.

60s timeout is a request timeout, not a special attachments timeout (which is 10min).

So I’m pretty sure JS is not the case here, this is smth more interesting.

[wohali]

no clue, and don't have time to look any deeper, sorry

[glrao]

is this issue resolved in any other next version 2.1.x ?

[ermouth]

@glrao afaik not