Skip to content

Rancher Upgrade Checklist

This document (000020061) is provided subject to the disclaimer at the end of this document.

Situation

This article details the high-level steps required when planning and performing a Rancher and/or Kubernetes upgrade.

Note

The versions contained in this document were current at the time of writing and are meant only as an example

Rancher and Kubernetes use the semantic versioning format, where the version number is referred to major.minor.patch, for example: v2.11.x, x is the patch version

Resolution

Planning

  • The support matrix gives a full list of the versions that are certified by Rancher and work best together

  • The recommended order of upgrades is: Rancher, Kubernetes, and then OS

  • It is important to remain within the supported version combinations in the support matrix. For large version jumps, when planning the upgrade steps it may be best to split the upgrade into smaller portions, completing all of the areas in each portion before upgrading further.

  • For example, do not upgrade Rancher to a newer version if the Kubernetes versions of your management (local) and downstream clusters are not supported by the upgraded version of Rancher. Instead, upgrade Kubernetes to a version supported by both the current and upgraded versions of Rancher.
  • All upgrades should be performed in a lab, testing, or non-prod environment first
  • Adding a grace period between upgrades is recommended, to run workloads and confirm that the single upgrade did not cause any issues
  • For example: add 24 hours between upgrading Rancher, the local cluster, and downstream clusters
  • This reduces the number of changes occurring in a short timeframe and can aid in troubleshooting if an unexpected issue occurs
  • It is recommended to pause any application deployments that use the Rancher API during an upgrade of Rancher

Please see the following recommendations when planning version upgrades:

  • Rancher:

  • Avoid skipping minor versions when upgrading

    • For example: when upgrading from v2.9.x -> v2.11.x we encourage upgrading v2.9.x -> v2.10.x -> v2.11.x
  • It is recommended to upgrade from the newest patch version of the current release to the newest patch version of the next minor release, particularly if the current patch version is many releases behind and may be missing important fixes. For example:
    • When upgrading from v2.9.2 -> v2.10.x, we encourage you to first upgrade to v2.9.10 (newest v2.9.x patch version at the time of writing) as an intermediary step
    • Then to upgrade from v2.9.10 to v2.10.6 (newest v2.10.x patch version at the time of writing) and not any earlier v2.10.x patch version, for example, v2.10.1

  • Avoid upgrading to a pre-release or non-stable version as they are not yet fully tested and supported. This is generally recommended for any cluster/environment, however particularly important for production

    • Pre-release versions can be identified by a -rc# or -alpha# following the Rancher version number (eg, v2.12.0-rc2)

  • Kubernetes:

  • It is recommended to upgrade incrementally and avoid skipping minor versions. This helps minimize potential issues by introducing gradual changes

    • For example: when upgrading from v1.29.x -> v1.32.x we encourage upgrading v1.29.x -> v1.30.x -> v1.31.x -> v1.32.x
    • Kubernetes plans approximately three minor version releases a year, so it is a good practice to also plan a minor version upgrade multiple times throughout the year

  • RKE1 CLI:

  • Perform one minor CLI version jump at a time

    • For example: when upgrading from v1.5.x -> v1.8.x instead do v1.5.x -> v1.6.x -> v1.7.x -> v1.8.x

Data collection

Before you start your upgrade, please collect the following pieces of information to best prepare yourself in case you need to open a support ticket.

  • Scheduled change window:
  • Current Rancher version ( `` shown bottom left in the Rancher dashboard):
  • Target Rancher version:
  • Installation option (single install/HA):
  • Current Kubernetes version of Rancher local cluster (use kubectl version):

Rancher Pre-Upgrade

  • Check if the Rancher UI is accessible
  • Check if all clusters in UI are in an Active state
  • Check if all pods in kube-system and cattle-system namespaces are running (in both Rancher and downstream clusters)
kubectl get pods -n kube-system
kubectl get pods -n cattle-system
  • Verify the datastore has scheduled snapshots configured, and these are working.

  • RKE: If Rancher is deployed on a Kubernetes cluster built with RKE, verify etcd snapshots are enabled and working, on etcd nodes you can confirm with the following:

    ls -l /opt/rke/etcd-snapshots
docker logs etcd-rolling-snapshots

  • k3s: If Rancher is deployed on a k3s Kubernetes cluster, ensure scheduled backups are configured and working. Please see the k3s documentation pages for further information on this.

  • RKE2: If Rancher is deployed on a RKE2 Kubernetes cluster, ensure scheduled backups are configured and working. Please see the RKE2 documentation pages for further information on this.
  • Create a one-time datastore snapshot, please see the following documentation for RKE, RKE2, and k3s , and the single node Docker install options for more information

Rancher Upgrade steps

Rancher Review/Verify

After the upgrade is completed, go through the following checklist to verify your environment is in working order.

  • Check if the Rancher UI is accessible
  • You should be able to login into Rancher, view clusters, and browse to workloads
  • Verify the Rancher version has changed in UI
  • After logging into Rancher, review the version in the bottom left corner of the page
  • Check if all clusters in UI are in an Active state
  • Check if all pods in kube-system and cattle-system are running (in both Rancher and downstream clusters)

  • Check the cattle-cluster-agent and cattle-node-agent pods are running in all downstream clusters and running the latest version

  • The rollout of the updated agent versions can take some time if there are a lot of downstream clusters or nodes

  • Create a one-time datastore snapshot, please see the following documentation for RKE,  RKE2, and k3s, and the single node Docker install options for more information

Rancher Rollback steps

The Rancher rollback process is detailed in the rollback documentation, please follow the relevant link for Rancher installed on a Kubernetes cluster, or Docker

Follow-up tasks (optional)

  • Upgrade the Rancher management cluster, this is often a follow-up to the Rancher upgrade. Please see the RKE, RKE2 , and k3s upgrade documentation for the upgrade process, as mentioned it is best to avoid skipping minor versions
  • Upgrade the downstream clusters, please see the documentation for more information. A snapshot of both the local and downstream clusters before the upgrade is recommended to provides the maximum amount of recoverability options in the event of a rollback
  • Docker/OS upgrades, please our article on performing rolling changes to nodes

Other useful KBs

[Rancher] Can Rancher Support validate our planned upgrade?

[Rancher] Can Rancher Support join me as I do my upgrade?

Additional Information

Below is an example plan for upgrading from Rancher 2.9.5 to 2.11.2, and upgrading the local and downstream clusters from 1.27 to 1.31.

Note Refer to the planning section above to plan each Rancher and Kubernetes upgrade. For example, between each upgrade (bullet point) take an etcd snapshot of the related cluster(s), add a grace period of 24 hours or longer between each upgrade, and check the critical applications/components in the cluster are healthy

1) Upgrade Rancher to the latest version of the current release (2.9.9 at the time of writing)

  • Upgrade Rancher 2.9.5 to Rancher 2.9.9
  • Test for a minimum of 24 hours, preferably longer

2) Upgrade local and downstream clusters to the maximum supported Kubernetes version on Rancher 2.9.x

  • Upgrade Kubernetes on the local cluster from 1.27 to 1.28
  • Upgrade Kubernetes on the local cluster from 1.28 to 1.29
  • Upgrade Kubernetes on the local cluster from 1.29 to 1.30
  • Upgrade Kubernetes on all downstream clusters from 1.27 to 1.28
  • Upgrade Kubernetes on all downstream clusters from 1.28 to 1.29
  • Upgrade Kubernetes on all downstream clusters from 1.29 to 1.30

3) Upgrade Rancher to the latest version of the next release (2.10.6 at the time of writing)

  • Upgrade Rancher 2.9.9 to Rancher 2.10.6

4) Upgrade local and downstream clusters to the maximum supported Kubernetes version on Rancher 2.10.x

  • Upgrade Kubernetes on the local cluster from 1.30 to 1.31
  • Upgrade Kubernetes on all downstream clusters from 1.30 to 1.31

5) Upgrade Rancher to the latest version of the next release (2.11.2 at the time of writing)

  • Upgrade Rancher 2.10.6 to Rancher 2.11.2

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.