Is there a way to get a list of RaftEndpoints that are currently online?

[bootjp]

Hi, I use MicroRaft to create distributed storage (thanks for the great library)

I am now using MicroRaft to implement the Three Phase Commit and would like to "get a list of RaftEndpoints that are currently online."

Is there a way?

Specifically, if you have RaftEndpoints with nodes (or TERMINATED) that suddenly can't communicate, you want to get only the nodes that are currently online.

This is caused by the following issues when doing transactions using Three Phase Commit when some nodes have lost communication.

1. Replicate data using the StateMachine Operation
2. Ask the endpoint retrieved by RaftNode.getEffectiveMembers().getMembers() whether the commit is complete (the query provides an endpoint that works differently than Raft)
3. Endpoints that lost communication in (2) are also returned, so there is no data and the transaction always fails.

I would like to know if there is a good way.
Thank you and best regards,

[metanet]

HI @bootjp

Thank you for building your use case with MicroRaft.

In MicroRaft, followers only track the liveliness of the leader, and the leader tracks the liveliness of the followers. However, for replication, it is not clear to me why do you need liveness of each node. Normally, you just need to discover the leader node. Can you please elaborate why do you need to know liveliness of the individual nodes for committing your operations?

I am trying to understand your need. Do you need to identify the current leader to replicate your operation while there are some offline nodes in the system? Normally, you can query RaftNodeReport on some nodes and discover the leader via RaftTerm#getLeaderEndpoint.

2. Ask the endpoint retrieved by RaftNode.getEffectiveMembers().getMembers() whether the commit is complete (the query provides an endpoint that works differently than Raft)

About this point, RaftNode#replicate() returns a Future which is completed when your operation is committed. The returned object also provides the commit index of the operation. So if you want to check if some nodes also executed this commit, you can query their RaftNodeReport and check if their commit index is greater than or equal to the commit index of your operation.

If it will be useful, I can consider adding a method like <List<RaftEndpoint> RaftNodeReport#getLiveEndpoints() which will return a non-empty list only by the leader. Would that be useful for your use case?

Cheers,

[bootjp]

Hey, imo there is no such thing as "honest use of the Raft algorithm". Prod use cases will always have some extra requirements we need to build on top of the core algorithm to make it useful :)

👍

I will add the List getLiveEndpoints() method to the API. I think it is a reasonable API and might be useful for other use cases. You can subscribe to #38 for this.

Thank you, it's very helpful.

At POINT 2, your leader server can know the follower servers which appended your operation to their raft logs. However, you also want to ensure they apply the operation to the state machine then you will notify the lucene server processes running on your machines. So basically your leader server can make an RPC to follower servers to call RaftNode.query(no-op, QueryPolicy.EVENTUAL_CONSISTENCY, commitIndex) and if it throws LaggingCommitIndexException retry until it succeeds. Here, the gist is that Between POINT 1 and POINT 2, some of the followers may have crashed. If the RaftNode API also exposes the followers who recently have sent an heartbeat (#38), in your leader server you can eventually find out all alive nodes which applied your operation.

Does it make sense?

It looks very useful.

Actually, that was the implementation at first.

(Instead of using the Storage API, the implementation would wait for a success or timeout with the query RaftNode.query(no-op, QueryPolicy.EVENTUAL_CONSISTENCY, commitIndex).)

There are reasons for changing this implementation.

I couldn't figure out how to create an instance of a remote RaftNode in Java code that does RaftNode.query(…)

I thought that if I created a new RaftNode (RaftNode.newBuilder() ...), it would create a new RaftNode that was different from the remote node.

But is this my mistake and provide access to a remote existing node unless I do RaftNode.start()?

I have also have another question. Would you also benefit from adding a timeout parameter to the RaftNode#query() API? Basically once you have a commit index, on follower servers you can issue a no-op query to local Raft nodes with a timeout parameter to ensure that the local Raft node's state machine will have applied all the operations up to (including) the given commit index. It is similar to the Barrier discussion you had in hashicorp/raft#508.

Yes, this is very useful.
Having a feature like hashicorp/raft Barrier in MicroRaft makes it very easy to implement in my use case.

--

Thank you for a kind discussion.
I'm not a native English speaker, so it takes me a long time to respond. (Sorry)

[metanet]

I am not a native speaker either :)

I thought that if I created a new RaftNode (RaftNode.newBuilder() ...), it would create a new RaftNode that was different from the remote node.

But is this my mistake and provide access to a remote existing node unless I do RaftNode.start()

Yeah, calling RaftNode.newBuilder() will create a new Raft node. Instead of that, if you are using gRPC to communicate your servers, you can build a gRPC method to trigger RaftNode.query() on a remote server's Raft node.

If you want to continue with the solution described in your diagram, I would suggest you to do some perf testing based on your estimated throughput. You are making an RTT to commit an index update via MicroRaft, then wait until the alive followers apply the change locally. So it makes 2 RTTs. It might be very expensive to do this if you have high throughput and you may consider doing your MicroRaft commits in batches, i.e., collect a batch of index updates, pass them to RaftNode#replicate() one by one, wait for all of them to be completed, then verify that followers apply until the latest commit.

From another perspective, would it be possible to merge steps #3 and #5? On each server, when MicroRaft's state machine persists the Lucene index change to the storage, it can notify the Lucene server running on the same machine.

Just a few thoughts based on my understanding of your use case.

Anyway, I will ship #38 and make a new release in a few days.

Regards,

[metanet]

Code landed to the repo (917edc6#diff-87f60dee32360363ff0939e37cc15999a4acb755416917af931d08cd445ddb2e). I will make a new release tomorrow. Fyi.

[bootjp]

Yeah, calling RaftNode.newBuilder() will create a new Raft node. Instead of that, if you are using gRPC to communicate your servers, you can build a gRPC method to trigger RaftNode.query() on a remote server's Raft node.

I see, I understand.

If you want to continue with the solution described in your diagram, I would suggest you to do some perf testing based on your estimated throughput. You are making an RTT to commit an index update via MicroRaft, then wait until the alive followers apply the change locally. So it makes 2 RTTs. It might be very expensive to do this if you have high throughput and you may consider doing your MicroRaft commits in batches, i.e., collect a batch of index updates, pass them to RaftNode#replicate() one by one, wait for all of them to be completed, then verify that followers apply until the latest commit.

Yes, there should be very high latency.
(Is it a reply with that intention?)
However, this system does not update indexes frequently and can tolerate up to a few seconds of latency for a single update operation.
So you should be fine with this configuration.

But you're right that we should test, thank you!

From another perspective, would it be possible to merge steps #3 and #5? On each server, when MicroRaft's state machine persists the Lucene index change to the storage, it can notify the Lucene server running on the same machine.

Sure this looks mergeable, but if for some reason (DISK FULL, for example) one node fails to write, could it be inconsistent?
Since this configuration does not pass read requests through the reader, there should be two nodes, one that writes successfully and one that fails.

However, the solution is to write the index and then rename it before notification, which can also fail:)

Thank you so much for kindly thinking about how to solve the issue together. ❤️

Code landed to the repo (917edc6#diff-87f60dee32360363ff0939e37cc15999a4acb755416917af931d08cd445ddb2e). I will make a new release tomorrow. Fyi.

We are very grateful that you will be releasing a new API! Thank you so much.

[metanet]

Hey @bootjp

MicroRaft 0.5 is released. Please see it at https://repo.maven.apache.org/maven2/io/microraft/microraft/.

Release notes are at https://github.com/MicroRaft/MicroRaft/releases/tag/v0.5.

[metanet]

MicroRaft/microraft/src/main/java/io/microraft/report/RaftNodeReport.java

Lines 147 to 178 in 2487b9b

	* Returns timestamps of latest heartbeats sent by the non-leader nodes to the
	* leader Raft node, including both {@link RaftRole#FOLLOWER} and
	* {@link RaftRole#LEARNER} nodes. The leader node's RaftEndpoint is not present
	* in the map. This map is returned non-empty only by the leader Raft node.
	*
	* @return timestamps of latest heartbeats sent by the non-leader nodes
	*/
	@Nonnull
	Map<RaftEndpoint, Long> getHeartbeatTimestamps();
	/**
	* Returns earliest heartbeat timestamp of the replication quorum. This method
	* returns a non-empty value only for the leader Raft node. For instance, this
	* method returns 8 for the following heartbeat timestamps of 5 Raft nodes, A
	* (leader) ts = -, B (follower) ts = 10, C (follower) ts = 8, D (follower) ts =
	* 6, E (follower) ts = 4. Please note that {@link RaftRole#LEARNER} nodes and
	* their heartbeats are excluded in quorum calculations.
	*
	* @return earliest heartbeat timestamp of the replication quorum
	*/
	@Nonnull
	Optional<Long> getQuorumHeartbeatTimestamp();
	/**
	* Returns timestamp of the latest heartbeat received from the leader Raft node.
	* This method returns a non-empty value only from a non-leader Raft node if it
	* has ever received a heartbeat from the leader.
	*
	* @return timestamp of the latest heartbeat received from the leader Raft node
	*/
	@Nonnull
	Optional<Long> getLeaderHeartbeatTimestamp();

[bootjp]

Thank you so much!