Consensus algorithm
lastmod 2024-06-15created 2018-07-276 edits
When building a distributed system, unless you have designed a system in which every node runs independently, you need some way to agree on shared state. An algorithm that shares state in a distributed environment in this way is called a consensus algorithm.
Quorum
A quorum is needed not only for the cluster's management functions but also for maintaining fault tolerance. As long as a majority of nodes stay healthy and able to respond, the cluster keeps functioning, so even if some of the remaining nodes have problems, there is no risk of the whole cluster going down as long as their number is at most (N-1)/2. For example, if there are 3 nodes in total, even if one of them fails, the remaining 2 can guarantee that the cluster operates properly, so you can buy time to recover the failed node without downtime.
If this quorum is not met, the cluster cannot function properly. Let's look again at the case assumed in the last part of the previous post: a Docker Swarm cluster made up of 2 manager nodes. Here, the quorum needed to keep the cluster functioning is the natural number equal to or greater than (2+1)/2, that is, 2. So a problem occurs even if only one of the two stops. Electing a leader node becomes impossible, and so do updating and synchronizing the cluster, so in the end the management functions of the whole cluster collapse.
Optimal number of nodes for fault tolerance
In distributed systems that adopt a consensus algorithm, it is recommended to keep the total number of nodes at an odd number of 3 or more where possible. The reasons are as follows.
- A cluster needs at least 3 nodes to have fault tolerance.
- When the total number of nodes is odd, the proportion of failed nodes that can be tolerated is somewhat higher.