Skip to content

Rancher managed RKE2 clusters stuck "Waiting for kube-controller-manager/kube-scheduler probes" due to certificate expiration

This document (000021371) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Rancher <2.7.5

RKE2 managed cluster

Situation

This article explains a problem where RKE2 clusters managed by Rancher get stuck in a "Waiting for probes: kube-controller-manager, kube-scheduler" state. This happens because the certificates used by the kube-controller-manager and kube-scheduler components expire.

Resolution

This issue is fixed in Rancher version 2.7.5. Starting with this version, the kube-controller-manager and kube-scheduler certificates in Rancher-managed RKE2 clusters are now part of the certificate rotation. When you use Rancher's Rotate Certificates feature, these certificates will also be renewed, which stops the problem described earlier.

Workaround

For versions older than 2.7.5, as well as in exceptional cases, the suggested workaround would be as follows:

  1. Stop the RKE2 server:
  2. systemctl stop rke2-server
  3. Remove the .crt and .key file in the respective tls directories:
  4. rm /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.{crt,key}
  5. rm /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.{crt,key}
  6. Perform a certificate rotation:
  7. rke2 certificate rotate
  8. Restart RKE2 server: ``
  9. systemctl start rke2-server

Cause

RKE2's regular certificate rotation doesn't manage the certificates for the kube-controller-manager and kube-scheduler in other clusters. Because of this, these certificates don't automatically renew when other cluster certificates are rotated. If these certificates expire, it can stop communication between these parts and the kube-apiserver, leading to the "Waiting for probes" state.

Additional Information

https://github.com/rancher/rancher/issues/41125

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.