How to rotate rancher-webhook (cattle-webhook-tls) certificates?
Article Number: 000020415
Environment
Rancher versions ≤ 2.6.2
Situation
An upgrade of Rancher or editing roles in Rancher UI fails with the below error.
Internal error occurred: failed calling webhook "rancherauth.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": x509: certificate has expired or is not yet valid: current time 2021-10-25T07:43:50Z is after 2021-10-06T20:20:47Z
Cause
This issue is caused by the expired certificate of the rancher webhook.
Resolution
- Set the kubectl context for the Rancher management cluster (local cluster).
- Take the backup of existing secret
kubectl get secret -n cattle-system cattle-webhook-tls -o yaml > cattle-webhook-tls.yaml
Delete the secret that contains expired certificate
kubectl delete secret -n cattle-system cattle-webhook-tls
kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io
- Delete the `rancher.cattle.io` mutating webhook
kubectl delete pod -n cattle-system -l app=rancher-webhook- Delete the rancher webhook Pod to regenerate the expired certificate.