until parameter for change request

[ronag]

Would be nice to have an until parameter for change feeds that work similar to since but instead of start, it specifies the end of the feed, e.g.:

/{db}/_changes?since=0&until=now

[rnewson]

until=now is the default behaviour (that is, if you don't ask for feed=continuous)

For any other value of until there is no solution, as you cannot construct a future update sequence string externally.

[ronag]

until=now is the default behaviour (that is, if you don't ask for feed=continuous)

Exactly, would like to have it for feed=continuous which is much more efficient than normal.

[rnewson]

feed=continuous is as "efficient" as normal mode, it's the same code except it keeps reopening the database as it changes and sends the new updates. there is still no way for you to construct a future update sequence, and thus there's nothing you could pass as the until parameter.

I do not understand your need.

[ronag]

normal feeds require several thousand http requests while continuous is a single streaming request. There is a significant difference in both overhead and latency. Even with tricks like keep alive and pipelining (I'm unsure how well those are actually supported).

[rnewson]

keepalive (multiple sequential send req, read resps on the same tcp connection) is supported (and encouraged!). pipelining (sending multiple requests before reading the first response) is also supported.

I still don't see how you could know what value to pass to until. Have you seen that the changes endpoint supports a limit parameter? perhaps that is what you mean (`from here, return the next 10,000 updates and wait until they happen)

[ronag]

keepalive (multiple sequential send req, read resps on the same tcp connection) is supported (and encouraged!). pipelining (sending multiple requests before reading the first response) is also supported.

That's great! However, from our tests it's still a lot slower than a continuous feed in addition to that it takes more memory (at least on client side as we need a quite large batch size for optimal perf). Especially when running in clustered mode.

Feel free to close if you still don't think it makes sense.

I still don't see how you could know what value to pass to until.

I would have a previously saved sequence or just pass now

[rnewson]

are you perhaps buffering the entire response before processing it when not using feed=continuous? the response format is different, you see. for continuous we send a series of json objects separated by newlines. for normal, it's a single json object with all the docs within that. however we do intentionally put newlines between the items to make it easier to process in a streaming manner.

normal:

curl foo:bar@localhost:25984/db1/_changes
{"results":[
{"seq":"1-g1AAAABzeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____sxIZcCrJYwGSDA1A6j9IZQZzImMuUIDdzMIiNS0lDVNXFgCFqx2b","id":"doc1","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]},
{"seq":"2-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____szKYExlzgQLsycZJyQZmSZjKcRqRxwIkGRqA1H8Uk8wsLFLTUtIwdWUBAG6aJN4","id":"doc2","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]},
{"seq":"3-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____szKYE5lygQLsycZJyQZmSZjKcRqRxwIkGRqA1H-oSYxgk8wsLFLTUtIwdWUBAG8EJN8","id":"doc3","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]}
],
"last_seq":"3-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____szKYE5lygQLsycZJyQZmSZjKcRqRxwIkGRqA1H-oSYxgk8wsLFLTUtIwdWUBAG8EJN8","pending":0}

continuous

curl 'foo:bar@localhost:25984/db1/_changes?feed=continuous'
{"seq":"1-g1AAAABzeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____szKYExlzgQLsycZJyQZmSZjKcRqRxwIkGRqA1H-QSYkMWQBNRh2D","id":"doc2","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]}
{"seq":"2-g1AAAABzeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____szKYE5lygQLsycZJyQZmSZjKcRqRxwIkGRqA1H-QSYkMWQBNkB2E","id":"doc3","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]}
{"seq":"3-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTVyMDQy1zMAQsMckEQiQ1L9____szKYE5lygQLsycZJyQZmSZjKcRqRxwIkGRqA1H-oSYxgk8wsLFLTUtIwdWUBAG8EJN8","id":"doc1","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]}

[rnewson]

(obviously you'd also need to strip the comma to process "normal" format as a stream but this is simple.)

[ronag]

That's a good point!