[Enhancement] Clock-insensitive leader lock #25
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This updates the leader lock implementation so that it's no longer sensitive to
differences in system clock on Tasques servers.
It does this by making non-Leaders eagerly try to gain leadership after sleeping
for the leader lock period.
Changes:
* In CHECKER mode: stashes the current leader lock document even if
not leader, then *waits* for the leader lock lease period, and sets
mode to USURPER
* In USURPER mode: tries to jostle for leadership right away
* If there is a conflict, then someone else is leader (in normal cases, this is
likely existing leader); go into CHECKER mode and re-loop.
* When writing the leader lock document, we now also write the leader lock
lease period, which non-leaders will wait for before trying to USURP.
This allows config changes for leader_lock_lease to be rolled out
to servers incrementally without suddenly changing wait time on the original
leader.
Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #25 +/- ##
=======================================
Coverage 83.97% 83.97%
=======================================
Files 18 18
Lines 1142 1142
=======================================
Hits 959 959
Misses 121 121
Partials 62 62Continue to review full report at Codecov.
|
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates the leader lock implementation so that it's no longer sensitive to
differences in system clock on Tasques servers.
It does this by making non-Leaders eagerly try to gain leadership after sleeping
for the leader lock period.
Changes:
not leader, then waits for the leader lock lease period, and sets
mode to USURPER
likely existing leader); go into CHECKER mode and re-loop.
lease period, which non-leaders will wait for before trying to USURP.
This allows config changes for leader_lock_lease to be rolled out
to servers incrementally without suddenly changing wait time on the original
leader.
Credit:
https://aws.amazon.com/blogs/database/building-distributed-locks-with-the-dynamodb-lock-client/