-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes feed with descending=true is incorrect #4714
Comments
I had tried it in 2.3.1 and I see the same negative pending value behavior. Not clear if it was meant to do that (there is a plausible semantic explanation for it) or it was a genuine bug. |
Returning some of the same start sequence numbers comes as a side-effect of having individual shards. Each shard has an integer sequence number that's incrementing. 0 is a special "start" sequence value. When we emit a sequence row in the changes feed we combine all the sequences from individual shards, the numeric prefix is the sum of all the individual sequences in each range. In the regular, forward direction, all sequences initialize at 0. So if there are 2 shard ranges (q=2), 2 docs on one range and 3 on the other, we might get this sequence:
Could also be 0+1, 0+2, 0+3, 1+3, 2+3, etc. In the reverse case, we initialize first with the top sequence from each shard range (2 and 3) instead of 0. So then, if we don't hear from a row yet we assume it's has its top sequence (2 or 3), so we might get:
So we get 5 emitted twice. For a large Q we'd get a higher chance of emitting identical sum prefixes with a frequency proportional to Q or so. The prefix bit, turns out, is mostly cosmetic, it's usually thrown away anyway when we get back a sequence in Apache CouchDB. (Though, older version of CouchDB or other couches might parse it). We can probably improve this part and attempt to emit a decrementing sum, at the expense that it won't actually match the sum of sequences from the encoded per-node sequence part ( |
I guess a plausible interpretation of the pending number is that it reflects both the direction and offset from the end of the changes feed, regardless in which direction we're traversing. In a simple
I think it would make more sense to return it as a positive value, reflecting how many more rows are left till the end of the current direction of traversal. If we're traversing backwards it would be how many till we reach |
Previously, pending changes count for the `descending=true` didn't indicate the number of pending changes. Instead, it returned the negative value of the already emitted rows. So for `descending=true&limit=2`, it would return -2; for `descending=true&limit=5`, it would return -5. That wasn't a very useful value, so let's fix it to return the actual number of pending changes. When traversing the changes feed backwards, the number of pending changes until the start will be equal the total number of changes minus the number of changes from the current start sequence until the end. Issue: #4714
Let's try to fix the pending value to actually return something useful: #4715 |
Thanks for looking @nickva and for looking at the pending value 👍. The repeated values make it tricky, and impossible in some cases, to traverse the changes feed backwards, but it's a bit of a niche requirement (I've been using CouchDB for a decade and have never hit the changes feed with |
in theory it should be possible to fix _changes so it works in reverse order, I remember some previous fixes in this space. I suspect it's a bigger fix than this though. Just fixing the pending count but not addressing the encoded sequences doesn't seem useful. It is potentially misleading even. |
Hmm, I don't see how the pending count fix in #4715 is misleading. It shows the number of changes left to process in the direction of the traversal. It's should be about the same level of usefulness as the forward pending count. Perhaps the pending count overall is not a terribly used thing? But at lest they can now both work in about the same way. |
"pending" means "this is how many newer changes there are yet to be received" so it only means pending in one direction |
🤷 ok then. |
Previously, pending changes count for the `descending=true` didn't indicate the number of pending changes. Instead, it returned the negative value of the already emitted rows. So for `descending=true&limit=2`, it would return -2; for `descending=true&limit=5`, it would return -5. That wasn't a very useful value, so let's fix it to return the actual number of pending changes. When traversing the changes feed backwards, the number of pending changes until the start will be equal the total number of changes minus the number of changes from the current start sequence until the end. Issue: #4714
Description
A CouchDB changes feed from a database where
q
is > 1, when queried with?descending=true
provides a set of results with repeatedseq
values and a negativepending
value.Steps to Reproduce
Create a database and populate:
Fetch the changes feed with
?descending=true
curl "$COUCH_URL/mydb/_changes?descending=true"
Expected Behaviour
It is expected that fetching a CouchDB's changes feed with
descending=true
would provide an array ofresults
in descendingseq
order, alast_seq
and a positivepending
number indicating how many more changes are available to fetch.Your Environment
The text was updated successfully, but these errors were encountered: