Large volume of cattle impersonation token created for the same service account on downstream cluster

Article Number: 000021553

Environment

Rancher v2.7.x; v2.8.x < v2.8.8; v2.9.x < v2.9.2

Situation

A large volume of cattle impersonation tokens are created for the same service account on a downstream cluster resulting in strange intermediate issues with user permissions and logins. The issue may also result in significant growth in memory usage by Rancher Pods.

Resolution

This is a known issue and is addressed in Rancher versions v2.8.8, and v2.9.2.

NOTE: This will not be back-ported to v2.7, as it is end of maintenance since March 2024 already.

For those unable to upgrade right now there is a a job manifest available to deploy and clean up the secrets here. This must be executed on each downstream cluster where the issue (a large number of impersonation tokens for the same service account) is present.

Instructions from the README:

1. Modify the image defined in resources.yaml to point to your private registry kubectl image, or leave as the default Docker Hub-hosted image bitnami/kubectl:latest
2. Apply resources.yaml using 

kubectl apply -f resources.yaml

or the equivalent using fleet/argo etc. 3. Monitor the jobs until completion 4. Clean up the resources using 

kubectl delete -f resources.yaml