Skip to content

Failed snapshot restore for RKE2 clusters with misplaced mountpoint for the RKE2 binaries/config files

This document (000021719) is provided subject to the disclaimer at the end of this document.

Environment

A standalone or Rancher-provisioned RKE2 cluster, in which a rollback is performed from:

  • a v1.22 patch version >= v1.22.16+rke2r1,
  • a v1.23 patch version >= v1.23.15+rke2r1,
  • a v1.24 patch version >= v1.24.9,
  • a v1.25 patch version >= v1.25.5+rke2r1,
  • or any RKE2 release >= v1.26.0+rke2r1

to:

  • a v1.22 patch version < v1.22.16+rke2r1,
  • a v1.23 patch version < v1.23.15+rke2r1,
  • a v1.24 patch version < v1.24.9,
  • a v1.25 patch version < v1.25.5+rke2r1,
  • or any RKE2 release <= v1.21.14+rke2r1

Situation

The etcd snapshot restore will fail.

Resolution

Workaround:

Execute on all nodes of the cluster before initiating the restore.

rm /etc/systemd/system/rke2-*.service
systemctl daemon-reload

Cause

  • The RKE2 install script for v1.24.4 does not install to /opt/rke2 if /usr/local is on a separate mount.
  • The RKE2 install script for v1.24.17 does install to /opt/rke2 if /usr/local is on a separate mount.
  • As part of the copy process, the systemd unit files are stored into an override in /etc/system/systemd, with /opt/rke2 defined in the exec path
  • When you rollback from v1.24.17 to v1.24.4, the binaries are written to /usr/local, but the systemd unit file override remains in place pointing to /opt/rke2 and the v1.24.17 binaries. The restore will fail as a result of not finding the right binaries and systemd config files.
  • This behaviour is the result of an earlier bug in the RKE2 install script that was fixed in the following script.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.