Skip to content

Unused Machine Configs (rke-machine-config.cattle.io) are automatically cleaned up in Rancher v2.10+

Article Number: 000022049

Environment

  • SUSE Rancher Prime v2.10.x till v2.11.x
  • RKE2
  • VMware vSphere
  • AWS EC2

Situation

  • After upgrading to Rancher v2.10, VmwarevsphereConfigs created via Terraform (rancher2_machine_config_v2) are automatically deleted after one day if they are not associated with any downstream cluster. This occurs when a machine config is not bound in the Terraform code. There is no alert or notification prior to deletion.
  • A similar scenario occurs when a Machine Pool is configured through the UI but left without nodes for an extended period. In such cases, Rancher automatically deletes some underlying resources (e.g., vmwarevsphereconfig, amazonec2configs). This causes the pool to become uneditable in the UI and blocks the creation of new pools.

Cause

The automatic deletion of orphaned machine configs was introduced in Rancher v2.10 as a resource optimization measure. A cronjob was deployed to clean up machine configs that lack owner references and are older than one hour.

Resolution

  • The deletion of unbound machine configs is an expected behavior in Rancher v2.10+. A cronjob, which runs every 24 hours, deletes machine configs that have no owner references and are older than 1 hour.
  • The cronjob responsible for cleaning up unused machine configs is:
kubectl get cronjob -n fleet-default
NAME                                 SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
rke2-machineconfig-cleanup-cronjob   5 0 * * *   False     0        5h53m           67d
  • However, it will become a problem when an existing pool is scaled down to zero, as it prevents updates to the pool or the creation of new machine pools.
  • Staring Rancher v2.12.1 this behavior has been changed. Machine configs in rke-machine-config.cattle.io are added back with ownerReferences when the cluster is edited to modify the node pool.
  • As a workaround in older versions, this issue can be mitigated by suspending the cleanup cronjob.
  • The suspension will remain effective until Rancher is restarted and can be applied via:
    Rancher → Explore "Local" cluster → More Resources → Batch → Cronjobs → Select "fleet-default" namespace → Suspend

Caution: Suspending the cronjob may leave orphaned configurations in the cluster for an extended period.