Skip to content

How to recover an RKE cluster when all control plane nodes have failed

Article Number: 000020695

Environment

Rancher-provisioned RKE1, RKE2 or K3s cluster

Situation

In a disaster recovery scenario, the control plane and etcd nodes managed by Rancher in a downstream cluster may no longer be available or functioning. The cluster can be rebuilt by adding control plane and etcd nodes again, followed by restoring from an available snapshot.

Resolution

Pre-requisites

  • Nodes to add to the cluster with control plane and etcd roles with adequate resources
  • An offline copy of a snapshot to be used as the recovery point, often stored in S3 or copied off node filesystems to a backup location
    Note: This article assumes that all control plane and etcd nodes are no longer functional and/or cannot be repaired via any other means, such as a VM snapshot restore.

Steps
To recover the downstream cluster, any existing nodes with the control plane and/or etcd roles must be removed. Worker nodes can remain in the cluster, and these may continue to operate with running workloads.
Please use the following steps as a guideline to recover the cluster; from this point, the cluster that has experienced the disaster will be referred to as the downstream cluster.

  1. As a precaution, it's recommended to take a snapshot of the Rancher management (local) cluster. Please see the documentation (RKE, RKE2, K3s (etcd)) for the appropriate distribution used for the local cluster.
    Alternatively, the rancher-backup operator can be used to back up all the related objects for restoration. This is useful when Rancher is deployed on a hosted provider (eg, EKS, GKE) or an external datastore is used with K3s.

  2. For the downstream cluster with the issue, delete all nodes with the control plane and/or etcd roles from Cluster Management in the Rancher dashboard.
    The delete action can fail when the downstream cluster is in this condition. If nodes do not get removed, follow the steps below to remove them from the cluster:

  3. Click on the node and select View in API, click the delete button for the object

  4. If this does not succeed, use kubectl or the Cluster Explorer for the Rancher local cluster, edit the corresponding nodes.management.cattle.io object in the namespace that matches the downstream cluster ID to remove the finalizers field
  5. Add a clean node back to the cluster with the all role (control plane, etcd, worker). The IP address does not have to match any of the previous nodes. If the node has previously been used in a cluster, use the appropriate clean-up step to remove any previous configuration from How to remove all Kubernetes components from nodes.
  6. The newly added node will fail to successfully register to the downstream cluster; it won't proceed past "Waiting to register with Kubernetes". This is normal.

  7. Copy the snapshot into place on the new node. The filename must match a snapshot name in the list of snapshots in the Rancher dashboard for the downstream cluster. Any snapshot should be usable; if the name is different, rename the file to match one of the known snapshots in the list. Default locations for each distribution are:

  8. RKE1 /opt/rke/etcd-snapshots

  9. RKE2 /var/lib/rancher/rke2/db/snapshots
  10. K3s /var/lib/rancher/k3s/db/snapshots
  11. Initiate a snapshot restore from the Rancher dashboard using the same snapshot name used in the previous step.
  12. Monitor the Rancher pod logs for progress.
    To follow all pod logs at once, a kubeconfig for the Rancher local cluster can be used with this kubectl command:

kubectl logs -n cattle-system -l app=rancher -f -c rancher
8. Once the new node reaches the active state, check the cluster and add additional nodes by repeating step 3. When ready, the additional nodes can be added with only the control plane and etcd roles if desired.

As a follow-up, once all desired nodes are added and the cluster is healthy, the control plane and etcd node roles can be configured as needed. For example, if the all role is not needed, update the node by removing and adding the node again in a rolling fashion.