Skip to content

How to rotate rancher-webhook (cattle-webhook-tls) certificates?

This document (000020415) is provided subject to the disclaimer at the end of this document.

Environment

Rancher versions ≤ 2.6.2

Situation

An upgrade of Rancher or editing roles in Rancher UI fails with the below error.

Internal error occurred: failed calling webhook "rancherauth.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": x509: certificate has expired or is not yet valid: current time 2021-10-25T07:43:50Z is after 2021-10-06T20:20:47Z

Resolution

  • Set the kubectl context for the Rancher management cluster (local cluster).
  • Take the backup of existing secret
kubectl get secret -n cattle-system cattle-webhook-tls -o yaml > cattle-webhook-tls.yaml
  • Delete the secret that contains expired certificate
    kubectl delete secret -n cattle-system cattle-webhook-tls 
    - Delete the `rancher.cattle.io` mutating webhook
    kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io 
    - Delete the rancher webhook Pod to regenerate the expired certificate.
    kubectl delete pod -n cattle-system -l app=rancher-webhook

Cause

This issue is caused by the expired certificate of the rancher webhook.

Additional Information

In Rancher v2.6.3 and up, rancher-webhook deployments will automatically renew their TLS certificate when it is within 30 or fewer days of its expiration date.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.