Skip to content

Known issue deleting etcd nodes in RKE clusters managed by Rancher v2.8.6 ≤ version < v2.9.0 or RKE CLI v1.5.11 ≥ version < v1.5.14

This document (000021636) is provided subject to the disclaimer at the end of this document.

Environment

  • A Rancher-provisioned RKE cluster managed using Rancher v2.8.6 ≤ version < v2.9.0
  • Or an RKE standalone cluster managed using RKE v1.5.11 ≥ version < v1.5.14

Situation

Due to a known bug in these versions of Rancher and RKE, after removing a node with at least the etcd role, it won't get removed from the etcd list, even if from the kubectl perspective, the node is not part of the cluster anymore.

The node won't be part of the cluster and won't appear using the Rancher UI, the Rancher API, or the kubectl command line.

However, when going into one of the other etcd nodes, and checking the etcd member list, the old etcd node will still appear:

docker exec etcd etcdctl member list

This inconsistent removal may lead you into unexpected scenarios, as the number of etcd nodes running in the cluster won't match the number you expect it to be.

Depending on the number of etcd nodes running, and the actions performed on the theoretically removed etcd nodes, you may even lose control over the cluster if it becomes unresponsive due to etcd quorum lost.

Resolution

To workaround this issue, you need to:

  1. Double-check the etcd member list, where the old node/s should still be listed:
docker exec etcd etcdctl member list
  1. Manually remove the desired etcd node, even if this node was already deleted via kubectl or editing the configuration YAML file.
docker exec etcdctl member remove <etcd node ID>

This bug won't affect you if you run an earlier or more modern version of Rancher or RKE. Please check the Environment section.

Cause

Due to a known bug in these versions of Rancher and RKE, after removing a node with at least the etcd role, it won't get removed from the etcd list, even if the node is not part anymore of the cluster from the kubectl perspective.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.