Skip to content

Tigera-operator Pod operations failing due to rancher-webhook unauthorized errors in a Rancher-managed RKE2 cluster

This document (000021994) is provided subject to the disclaimer at the end of this document.

Environment

An RKE2 cluster, with the Calico CNI, managed by a Rancher v2.7.4+ instance

Situation

The tigera-operator fails when attempting to apply Pod Security Admission (PSA) labels to the calico-system Namespace, with the requests denied by the Rancher admission webhook (rancher-webhook). The tigera-operator Pod in the tigera-operator Namespace logs errors of the following format:

{"level":"info","ts":"2025-08-22T09:57:08Z","logger":"controller_installation","msg":"Failed to update object.","Name":"calico-system","Namespace":"","Kind":"Namespace","key":{"name":"calico-system"}}
{"level":"error","ts":"2025-08-22T09:57:08Z","logger":"controller_installation","msg":"Failed to create or update object","component":"*render.namespaceComponent","key":{"name":"calico-system"},"error":"admission webhook \"rancher.cattle.io.namespaces\" denied the request: Unauthorized","stacktrace":"github.com/tigera/operator/pkg/controller/utils.(*componentHandler).CreateOrUpdateOrDelete\n\t/go/src/github.com/tigera/operator/pkg/controller/utils/component.go:347\ngithub.com/tigera/operator/pkg/controller/installation.(*ReconcileInstallation).Reconcile\n\t/go/src/github.com/tigera/operator/pkg/controller/installation/core_controller.go:1499\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
{"level":"error","ts":"2025-08-22T09:57:08Z","logger":"controller_installation","msg":"Error creating / updating resource","Request.Namespace":"calico-system","Request.Name":"active-operator","reason":"ResourceUpdateError","error":"admission webhook \"rancher.cattle.io.namespaces\" denied the request: Unauthorized","stacktrace":"github.com/tigera/operator/pkg/controller/status.(*statusManager).SetDegraded\n\t/go/src/github.com/tigera/operator/pkg/controller/status/status.go:356\ngithub.com/tigera/operator/pkg/controller/installation.(*ReconcileInstallation).Reconcile\n\t/go/src/github.com/tigera/operator/pkg/controller/installation/core_controller.go:1500\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}
{"level":"error","ts":"2025-08-22T09:57:08Z","msg":"Reconciler error","controller":"tigera-installation-controller","object":{"name":"active-operator","namespace":"calico-system"},"namespace":"calico-system","name":"active-operator","reconcileID":"66cddc2d-4337-4c12-949c-22122d342ded","error":"admission webhook \"rancher.cattle.io.namespaces\" denied the request: Unauthorized","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249"}

The rancher-webhook admission controller rejections can also be seen in the Kubernetes API server logs for the cluster:

W0822 10:00:54.333194       1 dispatcher.go:225] rejected by webhook "rancher.cattle.io.namespaces": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"rancher.cattle.io.namespaces\" denied the request: Unauthorized", Reason:"Unauthorized", Details:(*v1.StatusDetails)(nil), Code:403}}

If the typha-certs in the calico-system Namespace have expired, the inability of the tigera-operator to add these PSA labels to the calico-system Namespace, blocks it from successfully rotating the certificates. As a result, errors of the following format may be observed in the calico-node Pods, and cluster Pod networking will not function correctly:

2025-08-022 08:14:02.163 [WARNING][16782] tunnel-ip-allocator/sync_client.go 158: error connecting to typha endpoint (3 of 3) 10.43.9.174:5473 connID=0x0 error=x509: certificate has expired or is not yet valid: current time 2025-08-22T08:14:02Z is after 2025-06-11T06:15:14Z type="tunnel-ip-allocation"

Resolution

Create the following ClusterRole and ClusterRolebinding, in the affected cluster, to provide the tigera-operator with permission to update PSA labels:



apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tigera-operator-psa
rules:
- apiGroups:
  - management.cattle.io
  resources:
  - projects
  verbs:
  - updatepsa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tigera-operator-psa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tigera-operator-psa
subjects:
- kind: ServiceAccount
  name: tigera-operator
  namespace: tigera-operator

Cause

The tigera-operator requires the updatepsa permission to apply the PSA labels "pod-security.kubernetes.io/enforce: privileged" and "pod-security.kubernetes.io/enforce-version: latest" to the calico-system Namespace, in a Rancher-managed RKE2 cluster (where the rancher-webhook is deployed by Rancher).

By default, these labels are successfully applied to the calico-system Namespace by the installation of the rke2-calico chart, during initial cluster provisioning, before the subsequent installation of the rancher-webhook.

If the cluster was initially provisioned with a much older version of RKE2, using a version of calico that did not apply the labels (e.g. v1.22.4+rke2r2), or the labels are manually removed from the calico-system Namespace after initial cluster provisioning, this issue is encountered. In this instance, applying the ClusterRole and ClusterRoleBinding, as documented here, is required to permit the tigera-operator to successfully apply the PSA labels.

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.