Skip to content

Rancher Upgrade Checklist

Article Number: 000020061

Situation

This article details the high-level steps required when planning and performing a Rancher and/or Kubernetes upgrade.

Note on versioning

The versions contained in this document were current at the time of writing and are meant only as an example.

Rancher and Kubernetes use the semantic versioning format, where the version number is referred to major.minor.patch, for example: v2.13.x, x is the patch version.

Resolution

Planning

  • The Rancher Support Matrix gives a full list of the versions that are certified by Rancher and supported together.

  • The recommended order of upgrades is: Rancher, Kubernetes, and then OS.

  • It is important to remain within the supported version combinations listed in the support matrix. For large version jumps, when planning the upgrade steps it is best to split the upgrade into smaller portions, completing all of the areas in each portion before upgrading further.

  • For example, do not upgrade Rancher to a newer version if the Kubernetes versions of your management (local) and downstream clusters are not supported by the upgraded version of Rancher. Instead, upgrade Kubernetes to a version supported by both the current and upgraded versions of Rancher.
  • All upgrades should be performed in a lab, testing, or non-prod environment first.

  • Adding a grace period between upgrades is recommended, to run workloads and confirm that the single upgrade did not cause any issues.

  • For example: add 24 hours between upgrading Rancher, the local cluster, and downstream clusters.

  • This reduces the number of changes occurring in a short timeframe and can aid in troubleshooting if an unexpected issue occurs.
  • It is recommended to pause any application deployments that use the Rancher API during an upgrade of Rancher.

  • You should perform on-demand snapshots before each and every upgrade. Without a snapshot you will be unable to perform a rollback in the event of an issue resulting from the upgrade. In a worst case scenario this could result in complete loss of a cluster or the environment.

  • Rancher: Before each Rancher version upgrade perform a snapshot using the Rancher Backup operator, per the Rancher documentation. If the Rancher local cluster is running in an RKE, RKE2 or K3s cluster, we would optionally recommend that you take an on-demand snapshot of the local cluster, per the RKE, RKE2 and K3s documentation.

  • Kubernetes: Before each Kubernetes version upgrade perform a snapshot of the cluster. Optionally also take an additional Rancher backup, per the process described above.

    • For the Rancher local and standalone/imported clusters, follow the snapshot steps in the RKE, RKE2 and K3s documentation.
    • For Rancher-provisioned clusters, follow the snapshot steps in the Rancher documentation.

Requirements

Rancher:

  • Do not skip minor versions when upgrading. For example:

  • When upgrading from v2.11.x -> v2.13.x you should upgrade v2.11.x -> v2.12.x -> v2.13.x.

  • You should upgrade from the newest patch version of the current release to the newest patch version of the next minor release. For example:

  • When upgrading from Rancher Prime v2.12.2 -> v2.13.x, you should first upgrade to v2.12.7 (newest Prime v2.12.x patch version at the time of writing) as an intermediary step.

  • Then upgrade from v2.12.7 to v2.13.3 (newest Prime v2.13.x patch version at the time of writing) and not any earlier v2.13.x patch version, for example, v2.13.1.

  • Do not upgrade to a pre-release or non-stable version, as they are not yet fully tested and supported.

  • Pre-release versions can be identified by a -rc# or -alpha# following the Rancher version number (eg, v2.13.0-rc2)

  • Approximately three Rancher minor version releases are planned per year, so it is good practice to also plan a Rancher minor version upgrade multiple times throughout the year.

Kubernetes:

Data collection

Before you start your upgrade, please collect the following pieces of information to best prepare yourself in case you need to open a support ticket.

  • Scheduled change window:
  • Current Rancher version (shown bottom left in the Rancher dashboard):
  • Target Rancher version:
  • Installation option (single install/HA):
  • Current Kubernetes version of Rancher local cluster (use kubectl version):

Rancher pre-upgrade steps

  • Check if the Rancher UI is accessible
  • Check if all clusters in UI are in an Active state
  • Check if all pods in kube-system and cattle-system namespaces are running (in both Rancher and downstream clusters):

kubectl get pods -n kube-system
kubectl get pods -n cattle-system
- Verify the datastore has scheduled snapshots configured, and that these are working.

  • RKE: If Rancher is deployed on a Kubernetes cluster built with RKE, verify etcd snapshots are enabled and working, on etcd nodes you can confirm with the following:

    ls -l /opt/rke/etcd-snapshots
docker logs etcd-rolling-snapshots
    - RKE2: If Rancher is deployed on a RKE2 Kubernetes cluster, ensure scheduled backups are configured and working. Please see the RKE2 documentation pages for further information on this. - K3s: If Rancher is deployed on a K3s Kubernetes cluster, ensure scheduled backups are configured and working. Please see the K3s documentation pages for further information on this.
    - Create a snapshot with the Rancher Backup operator, and optionally an on-demand cluster snapshot of the Rancher local cluster, per the backup notes in the Planning section above.

Rancher upgrade steps

The Rancher upgrade process is detailed in the Rancher documentation.

Rancher post-upgrade verification steps

After the upgrade is completed, go through the following checklist to verify your environment is in working order.

  • Check if the Rancher UI is accessible:

  • You should be able to login into Rancher, view clusters, and browse to workloads.

  • Verify the Rancher version has changed in UI:

  • After logging into Rancher, review the version in the bottom left corner of the page.

  • Check if all clusters in UI are in an Active state.
  • Check if all pods in kube-system and cattle-system are running (in both Rancher and downstream clusters).

  • Check the cattle-cluster-agent and cattle-node-agent pods are running in all downstream clusters and running the latest version.

  • The rollout of the updated agent versions can take some time if there are a lot of downstream clusters or nodes

  • Create a fresh snapshot with the Rancher Backup operator, and optionally an on-demand cluster snapshot of the local cluster, per the backup notes in the Planning section above.

Rancher rollback steps

The Rancher rollback process is detailed in the Rancher documentation.

Follow-up tasks (optional)

  • Upgrade the Rancher management cluster, this is often a follow-up to the Rancher upgrade. Refer to the RKE, RKE2 , and K3s upgrade documentation for the upgrade process. Please pay attention to the requirements and planning sections above.
  • Upgrade the downstream clusters, refer to the Rancher documentation for more information. Please pay attention to the requirements and planning sections above.
  • For OS or Docker upgrades, refer to the separate article on performing rolling changes to nodes.

Other useful KBs