FailedCreatePodSandBox "cannot allocate new block due to per host block limit" events in an RKE2 cluster with the Calico CNI

Article Number: 000022137

Environment

A Rancher-provisioned or standalone RKE2 cluster with the Calico CNI

Situation

Creating new Pods is failing with a FailedCreatePodSandBox event containing a message of the following format:

failed to setup network for sandbox "<hash>": plugin type="calico" failed (add): cannot allocate new block due to per host block limit 

Cause

If endpoints are not cleanly removed from Calico, this can result in leaked addresses, which consume IP space within the cluster. Following the process documented above will release these addresses, ensuring the IPAM handles are consistent, and all unused IP addresses are available to the cluster.

Resolution

The calicoctl CLI can be used to query for leaked addresses and release these, freeing IP space within the cluster.

1. First, install calicoctl locally with a version matching the Calico version in the affected RKE2 cluster, per the Calico documentation.
2. Source a kubeconfig, with cluster admin permissions, for the affected RKE2 cluster: `export KUBECONFIG=<path to kubeconfig>`
3. Query and release any leaked addresses:

calicoctl datastore migrate lock
calicoctl ipam check -o report.json
calicoctl ipam release --from-report report.json
calicoctl datastore migrate unlock