Does the CompletableFuture of RaftNode#replicate guarantee that it has been processed in the StateMachine of all nodes?

[bootjp]

Thank you for maintaining the excellent library.

I am currently using MicroRaft to create distributed storage that is embedded into applications.

So I have a question.

When you call get or join on the CompletedFuture of RaftNode#replicate, do you guarantee that all nodes have completed their StateMachine processing?
This can be rephrased as Read After Write.

We created a simple StateMachine that stores replicated data to local storage and ran tests on it.

This behavior was also seen in hashicorp/raft in the past.
hashicorp/raft#508

Do I need to issue a query with QueryPolicy = LINEARIZABLE to achieve Write After Read after executing replicate in MicroRaft?

[metanet]

Oh, I love these questions!

Thanks @bootjp for using the library. I would appreciate if you share your insights and experience.

When you call get or join on the CompletedFuture of RaftNode#replicate, do you guarantee that all nodes have completed their StateMachine processing?
This can be rephrased as Read After Write.

When the future returned by RaftNode#replicate() is completed, it is guaranteed that the leader has replicated your operation to the majority nodes and executed it locally. The other followers may not have executed your operation yet, even though the operation is certainly appended to the Raft logs of the majority nodes.

If you issue a LINEARIZABLE read to the leader, you will definitely get read your writes guarantee. However you will pay the cost of a heartbeat between leader and followers for this. If you just want read your writes, there are other options for you. For e.g., you can also use QueryPolicy#LEADER_LEASE and to your query you can pass the commit index (https://github.com/MicroRaft/MicroRaft/blob/master/microraft/src/main/java/io/microraft/Ordered.java#L33) returned by your previous RaftNode#replicate() call. This will guarantee that your query will be executed on the leader only if the leader has executed a successful heartbeat recently and has executed the operation of the given commit index. This should protect you during leadership changes. Another option is querying followers using QueryPolicy#BOUNDED_STALENESS or QueryPolicy#EVENTUAL_CONSISTENCY and passing the same commit index. In this case, followers will run your query only if they have executed the given index but in this case you might need to retry few times until you find a follower which has done this.

I hope my answer helps.

Cheers,

[bootjp]

Oh, that's exactly what I wanted to know.
Thank you for your kind answer.

[metanet]

https://github.com/MicroRaft/MicroRaft/releases/tag/v0.6