What is the correct amount of etcd nodes for my cluster?
This document (000021303) is provided subject to the disclaimer at the end of this document.
Environment
Rancher management (local) cluster running with RKE or RKE2.
The same tips can be used for managed downstream clusters.
Situation
When performing the installation/configuration of a cluster, you may wonder how many etcd nodes you should setup to ensure a good performance of the cluster.
Resolution
Three etcd nodes are generally sufficient for smaller and medium clusters, and five etcd nodes for large clusters.
One thing that we must consider is that etcd employs a quorum mechanism. This requires the presence of over half of the replicas to be operational before it can authorize any modifications to the database.
Therefore, in the case of a two-node etcd cluster, it won’t be sufficient to have only one healthy etcd node; both nodes must be functional. We always recommend having at least three etcd nodes. As per the functioning logic of etcd, it is recommended to always work with an odd number of primary nodes.
Adding more etcd nodes may also be possible, but we also need to consider replication performance, not only availability. Scaling up members will reduce performance. Our recommendation would be to add the necessary worker nodes and see how the cluster performs. If you notice high pressure after adding all the worker nodes, then you can go ahead and add two more etcd nodes. Always check the rule of having an odd number of etcd nodes.
See the below table for a better understanding:
Nodes with etcd role |
Majority | Failure Tolerance |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |
| 7 | 4 | 3 |
| 8 | 5 | 3 |
| 9 | 5 | 4 |
Additional Information
Please read more about this topic here:
- https://ranchermanager.docs.rancher.com/reference-guides/kubernetes-concepts
- https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/recommended-cluster-architecture
- https://etcd.io/docs/v3.4/faq/#what-is-failure-tolerance
- https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.