Skip to content

Node draining in Rancher-provisioned RKE2 and K3s clusters upon Rancher version upgrade

Article Number: 000022311

Environment

  • A Rancher-provisioned RKE2 or K3s cluster, with Drain Nodes enabled in the Upgrade Strategy.
  • A Rancher version upgrade.

Situation

When performing a Rancher version upgrade, users may observe that Rancher-provisioned RKE2 or K3s clusters, with Drain Nodes enabled in the Upgrade Strategy, are updated after the Rancher upgrade and node drains are performed as a result.

Cause

As noted in the Rancher documentation on enabling node drains:

During an upgrade, nodes may be drained even when no user-visible YAML changes are present. This can occur if non-dynamic configuration files are updated or if a new system-agent-installer image is introduced. In such cases, Rancher generates a new upgrade plan, resulting in a new plan hash. When Upgrade Strategy is set to Drain nodes, this plan change can trigger node draining.

Resolution

If users wish to avoid the possibility of nodes drains - which will cause temporary workload unavailability due to rescheduling - when upgrading Rancher, set Drain Nodes to No in the Upgrade Strategy of the clusters, before the Rancher upgrade:

  1. Navigate to Cluster Management in the Rancher UI and click Edit Config for the relevant RKE2/K3s cluster.
  2. Under Cluster Configuration select Upgrade Strategy.
  3. Set the Drain Nodes option to No for both Control Plane and Worker Nodes.
  4. Click Save to apply the changes.

After the Rancher upgrade, you can revert this change, to set Drain Nodes back to Yes.