Optimize and clean up couch_multidb_changes #4911
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jaydoane
approved these changes
Dec 12, 2023
| ok = couch_db:close(Db); | ||
| _Error -> | ||
| ok | ||
| case couch_server:exists(DbSuffix) of |
nickva
force-pushed
the
clean-up-multidb-changes
branch
from
8eef76b
to
72fc972
Compare
couch_multidb_changes is in charge of monitoring changes to multiple databases.
It is what drives the couch_replicator application when users update
replication docs. It starts off by scanning all the local db shards and
launches changes feeds for each of them. After it processes each changes feed,
it checkpoints where it stops. As the shards are updated, it starts change
feeds for those updated shards and checkpoints again. The checkpoints are kept
in an ets table and the main logic which decides when to start a changes feed
for a particular shard is in the resume_scan/2 function.
This change makes a few small optimizations:
* Use a map to track Pid -> DbName mappings. This avoids using a O(N)
operation for looking up exiting change feed processes. So `pids` is
switched to be a map of `#{Pid => DbName}` and the ets table has a 4th tuple
member to keep track of `dbname -> pid` mappings.
* Previously, when the plain "suffix" dbname (just <<"_replicator">>) was
found, we tried to open and close it to see if it exists. Change to use
`couch_server:exists/1` instead.
The are also a few clean-ups:
* Update tests to use ?TDEF_FE/?TDEF macros. This shortens them a bit and
saves one indentation level.
* Increase test coverage from 86% to 96%. To help create a few more test
scenarios switched to using a public ets table instead of a private one.
nickva
force-pushed
the
clean-up-multidb-changes
branch
from
72fc972
to
dc7e058
Compare
|
I caught a case missed where when a change feed process exits but it doesn't have to rescan again I didn't clear its old Pid from the ets table. So I added a clause for it: Also running on my slow laptop a few times in a row I noticed a few |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
couch_multidb_changes is in charge of monitoring changes to multiple databases. It is what drives the couch_replicator application when users update replication docs. It starts off by scanning all the local db shards and launches changes feeds for each of them. After it processes each changes feed, it checkpoints where it stops. As the shards are updated, it starts change feeds for those updated shards and checkpoints again. The checkpoints are kept in an ets table and the main logic which decides when to start a changes feed for a particular shard is in the resume_scan/2 function.
This change makes a few small optimizations:
Use a map to track Pid -> DbName mappings. This avoids using a O(N) operation for looking up exiting change feed processes. So
pidsis switched to be a map of#{Pid => DbName}and the ets table has a 4th tuple member to keep track ofdbname -> pidmappings.Previously, when the plain "suffix" dbname (just <<"_replicator">>) was found, we tried to open and close it to see if it exists. Change to use
couch_server:exists/1instead.The are also a few clean-ups:
Update tests to use ?TDEF_FE/?TDEF macros. This shortens them a bit and saves one indentation level.
Increase test coverage from 86% to 96%. To help create a few more test scenarios switched to using a public ets table instead of a private one.