Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cb_func is not triggered when cluster leader disappears #268

Open
faithware opened this issue Oct 21, 2021 · 3 comments
Open

cb_func is not triggered when cluster leader disappears #268

faithware opened this issue Oct 21, 2021 · 3 comments

Comments

@faithware
Copy link

Tested with quick_start.cxx example.
When a cluster leader is elected, the event nuraft::cb_func::Type::BecomeLeader is triggered on the leader and the event nuraft::cb_func::Type::BecomeFollower is triggered on the followers.
When the leader goes down, the new leader doesn't trigger BecomeLeader event.
Is this normal?
Cheers

@greensky00
Copy link
Contributor

greensky00 commented Oct 21, 2021

@faithware
BecomeLeader event is fired whenever a server becomes a leader, as it is directly called inside become_leader() function:

CbReturnCode rc = ctx_->cb_func_.call(cb_func::BecomeLeader, &param);

But that does not mean the event is fired as soon as the old leader goes down, since it will take some time to elect new leader according to the election timer.

@faithware
Copy link
Author

Thanks !@greensky00
In the quick_start example the election timer is initialized

params.election_timeout_lower_bound_ = 200;
params.election_timeout_upper_bound_ = 400;

The thing is the cluster I made is a two element cluster, when it is 3 element, this does not happen and one of the remaining two elements becomes a leader.
So if the cluster has two elements the only method to know that the leader is gone by the ConnectionClosed event.
Is there any way to force the only left element to be a leader so in case the other node comes back it joins as a follower?
Cheers

@greensky00
Copy link
Contributor

If there are 2 members in a cluster, the other member can't be a leader as you observed, because it cannot reach the consensus (quorum size is 2 for a 2-member cluster).

If you want to force the member become a leader, you can adjust the quorum size manually:
https://github.com/eBay/NuRaft/blob/master/docs/custom_quorum_size.md
but it is at your own risk -- split-brain may happen as a result of this.

There is also an option to adjust the quorum size automatically for a 2-member cluster. You can set this option to true.

/**
* If `true` and the size of the cluster is 2, the quorum size
* will be adjusted to 1 automatically, once one of two nodes
* becomes offline.
*/
bool auto_adjust_quorum_for_small_cluster_;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants