Failed snapshot restore for RKE2 clusters with misplaced mountpoint for the RKE2 binaries/config files

This document (000021719) is provided subject to the disclaimer at the end of this document.

Environment

A standalone or Rancher-provisioned RKE2 cluster, in which a rollback is performed from:

• a v1.22 patch version >= v1.22.16+rke2r1,
• a v1.23 patch version >= v1.23.15+rke2r1,
• a v1.24 patch version >= v1.24.9,
• a v1.25 patch version >= v1.25.5+rke2r1,
• or any RKE2 release >= v1.26.0+rke2r1

to:

• a v1.22 patch version < v1.22.16+rke2r1,
• a v1.23 patch version < v1.23.15+rke2r1,
• a v1.24 patch version < v1.24.9,
• a v1.25 patch version < v1.25.5+rke2r1,
• or any RKE2 release <= v1.21.14+rke2r1

Situation

The etcd snapshot restore will fail.

Resolution

Workaround:

Execute on all nodes of the cluster before initiating the restore

1. `rm /etc/systemd/system/rke2-*.service`
2. `systemctl daemon-reload`

Cause

• The RKE2 install script for v1.24.4 does not install to /opt/rke2 if /usr/local is on a separate mount.
• The RKE2 install script for v1.24.17 does install to /opt/rke2 if /usr/local is on a separate mount.
• As part of the copy process, the systemd unit files are stored into an override in /etc/system/systemd, with /opt/rke2 defined in the exec path
• When you rollback from v1.24.17 to v1.24.4, the binaries are written to /usr/local, but the systemd unit file override remains in place pointing to /opt/rke2 and the v1.24.17 binaries. The restore will fail as a result of not finding the right binaries and systemd config files.
• This behavior is the result of an earlier bug in the RKE2 install script ( https://github.com/rancher/system-agent-installer-rke2/blob/v1.24.4%2Brke2r1/package/run.sh#L9) that was fixed in ( https://github.com/rancher/system-agent-installer-rke2/blob/v1.24.9%2Brke2r1/package/run.sh)

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.