Skip to content

Rolling back from Rancher v2.13.0 to v2.12.3

Article Number: 000022186

Environment

  • Rancher v2.13.0
  • Using the Rancher Backup & Restore Operator

Situation

There is a known bug faced when performing a rollback from Rancher v2.13.0 to Rancher v2.12.3 using BRO (the Backup & Restore Operator), which prevents the Restore from completing successfully.

The Backup & Restore Operator logs look similar to this:

ERRO[2025/11/17 16:26:58] Error restoring cluster-scoped resources [error restoring cattle-globalrole-user-base of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner error restoring cattle-globalrole-users-manage of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner error restoring cattle-globalrole-user of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner error restoring cattle-globalrole-clusters-create of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner]
ERRO[2025/11/17 16:26:58] error syncing 'restore-migration': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing

Resolution

The bug will be officially fixed in the v2.14.0 version of Rancher as part of improvements tracked by this Github issue. There is, however, a workaround solution to prevent this bug from happening.

Disabling the Rancher Webhook during a Restore

The official documentation on how to perform a rollback can be found here. Essentially, the usual steps will look like this (summarized from the documentation):

  1. In a cluster running Rancher v2.13.0, make sure you have access to a Backup taken for Rancher v2.12.3
  2. Create a BRO Restore CR referencing the desired Backup, wait for it to be completed
  3. Perform the Helm Rollback as referenced in the docs, usually helm rollback rancher -n cattle-system

To prevent the bug from happening

Two extra steps are needed (added as points 2 and 3 below):

  1. In a cluster running Rancher v2.13.0, make sure you have access to a Backup taken for Rancher v2.12.3
  2. Scale down Rancher to 0 replicas kubectl scale deploy/rancher -n cattle-system --replicas=0
  3. Uninstall the Rancher webhook with helm uninstall helm uninstall rancher-webhook -n cattle-system. Make sure it was uninstalled correctly with helm list -n cattle-system
  4. Create a BRO Restore CR (via kubectl, as Rancher is down) referencing the desired Backup, wait for it to be completed
  5. Perform the Helm Rollback as referenced in the docs, usually helm rollback rancher -n cattle-system