Skip to content

Large volume of cattle impersonation token created for the same service account on downstream cluster

This document (000021553) is provided subject to the disclaimer at the end of this document.

Environment

Rancher v2.7.x; v2.8.x < v2.8.8; v2.9.x < v2.9.2

Situation

A large volume of cattle impersonation tokens are created for the same service account on a downstream cluster resulting in strange intermediate issues with user permissions and logins. The issue may also result in significant growth in memory usage by Rancher Pods.

Resolution

This is a known issue and is addressed in Rancher versions v2.8.8, and v2.9.2.

NOTE: This will not be back-ported to v2.7, as it is end of maintenance since March 2024 already.

For those unable to upgrade right now there is a a job manifest available to deploy and clean up the secrets here. This must be executed on each downstream cluster where the issue (a large number of impersonation tokens for the same service account) is present.

Instructions from the README:

  1. Modify the image defined in resources.yaml to point to your private registry kubectl image, or leave as the default Docker Hub-hosted image bitnami/kubectl:latest
  2. Apply resources.yaml using
kubectl apply -f resources.yaml

or the equivalent using fleet/argo etc. 3. Monitor the jobs until completion 4. Clean up the resources using

kubectl delete -f resources.yaml

Additional Information

  • Github issue: https://github.com/rancher/rancher/issues/44347
  • Clean up job README: https://github.com/ryanelliottsmith/cleanup-impersonation-secrets/blob/main/README.md

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.