Skip to content

Cattle-cluster-agent flapping between versions

Article Number: 000021769

Environment

Any Rancher-managed downstream cluster.

Situation

A Rancher-managed downstream Kubernetes cluster shows the cattle-cluster-agent repeatedly being recreated or changing between different rancher/rancher-agent image versions.

For example:

kubectl -n cattle-system get deployment cattle-cluster-agent -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

may show an agent image that does not match the expected Rancher version, or the image may change repeatedly between versions.

The cluster may also intermittently report connectivity problems in Rancher while the agent is being replaced.

Cause

The cattle-cluster-agent deployment is managed by Rancher. Changes made directly to the Deployment in the downstream cluster are therefore not authoritative and may be overwritten when Rancher reconciles the agent configuration.

When the cattle-cluster-agent repeatedly alternates between different rancher/rancher-agent image versions, the important diagnostic step is to determine which component is updating the Deployment and why different desired configurations are being applied.

This behavior can occur when the downstream cluster receives or retains conflicting Rancher agent configuration. Scenarios that should be investigated include:

  • A Rancher upgrade, rollback, restore, or migration that did not complete cleanly.
  • The downstream cluster retaining connectivity or configuration associated with a previous Rancher management environment.
  • DNS, proxy, ingress, or load-balancer configuration routing Rancher agent traffic to unintended Rancher endpoints.
  • An explicit CATTLE_AGENT_IMAGE override that does not match the currently running Rancher version.
  • Rancher management resources retaining agent configuration associated with a previous Rancher version.

To identify the actual cause, determine which component is modifying the cattle-cluster-agent Deployment when the image changes. Review the Deployment's managedFields, ReplicaSet history, events, CATTLE_SERVER, CATTLE_SERVER_VERSION, and Rancher agent logs.

Do not assume that the Rancher agent image itself is the root cause. The changing image is generally a symptom of the Deployment being reconciled with different configurations. The component or configuration responsible for those changes must be identified before applying a permanent resolution.

Resolution

1. Determine the currently deployed agent version

On the downstream cluster:

kubectl -n cattle-system get deployment cattle-cluster-agent -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Also inspect the Rancher server information provided to the agent:

kubectl -n cattle-system get deployment cattle-cluster-agent -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' | grep -E 'CATTLE_SERVER=|CATTLE_SERVER_VERSION='

The output should identify the Rancher server URL and Rancher server version that the agent expects to use.

2. Check whether the Deployment is actually being replaced

Run:

kubectl -n cattle-system get deployment cattle-cluster-agent -w

In another terminal:

kubectl -n cattle-system get rs -l app=cattle-cluster-agent --sort-by=.metadata.creationTimestamp

Check the images used by the ReplicaSets:

kubectl -n cattle-system get rs -l app=cattle-cluster-agent -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image,CREATED:.metadata.creationTimestamp'

If ReplicaSets containing different Rancher agent versions are continually being created, determine what is updating the Deployment before attempting to force an image manually.

3. Inspect the Deployment history and metadata

Run:

kubectl -n cattle-system describe deployment cattle-cluster-agent

and:

kubectl -n cattle-system get deployment cattle-cluster-agent -o yaml

Review:

  • container image
  • CATTLE_SERVER
  • CATTLE_SERVER_VERSION
  • annotations
  • labels
  • ReplicaSet history
  • recent events

These values can help determine whether Rancher is continuously reconciling the Deployment.

4. Verify that the Rancher endpoint resolves consistently

From the downstream cluster, verify the Rancher hostname:

getent hosts <RANCHER_FQDN>

or:

nslookup <RANCHER_FQDN>

If multiple addresses are returned, confirm that all addresses belong to the intended Rancher environment.

Also verify connectivity:

curl -vk https://<RANCHER_FQDN>/ping

A successful Rancher endpoint normally returns: pong

If Rancher was recently migrated, restored, or moved to another cluster, verify that DNS, load balancers, proxies, and ingress configuration no longer route traffic to the previous Rancher environment.

5. Check the cattle-cluster-agent logs

Run:

kubectl -n cattle-system logs deployment/cattle-cluster-agent

If the pod has restarted, also inspect the previous container:

kubectl -n cattle-system logs deployment/cattle-cluster-agent --previous

Look for messages showing:

  • Rancher agent version
  • Rancher server URL
  • connection or WebSocket failures
  • TLS/CA validation failures
  • authentication or registration failures

For example, the startup log normally identifies the agent version:

Rancher agent version <VERSION> is starting

Compare this with the expected Rancher version.

6. Check for an explicit Rancher agent image override

If Rancher is installed with Helm, review its configured values:

helm get values rancher -n cattle-system

Check for an environment variable such as:

CATTLE_AGENT_IMAGE5

An agent image override should only be retained when it is required for the Rancher version andconfiguration in use.

If an override was introduced as a temporary workaround for an earlier Rancher release, verify whether it isstill required before removing it.

7. Allow Rancher to regenerate the agent configuration

After correcting the underlying Rancher, DNS, load-balancer, migration, or image-override problem, force Rancher to redeploy the downstream cluster agent.

First identify the downstream cluster ID from the Rancher management cluster:

kubectl get clusters.management.cattle.io

Then annotate the appropriate cluster:

kubectl annotate clusters.management.cattle.io <CLUSTER_ID>io.cattle.agent.force.deploy=true --overwrite

This command must be run against the Rancher local/management cluster, not the downstream cluster.

Monitor the downstream cluster:

kubectl -n cattle-system rollout status deployment/cattle-cluster-agent

Then verify the resulting image:

kubectl -n cattle-system get deployment cattle-cluster-agent -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

8. Verify that the version remains stable

Monitor the Deployment and pods:

kubectl -n cattle-system get deployment,pods -l app=cattle-cluster-agent -w

The cattle-cluster-agent should remain on the version managed by the current Rancher server and should no longer alternate between versions.

Important

Do not use a manual modification of the downstream cattle-cluster-agent Deployment as the permanent resolution.

For example:

kubectl set image deployment/cattle-cluster-agent ...

or:

kubectl edit deployment cattle-cluster-agent

may temporarily change the running agent, but Rancher manages this Deployment and can reconcile the modification.

The underlying Rancher-side configuration or connectivity issue should be corrected instead.

Additional troubleshooting

If the agent continues to alternate between versions, collect the following information:

kubectl -n cattle-system get deployment cattle-cluster-agent -o yaml

kubectl -n cattle-system get rs -l app=cattle-cluster-agent -o yaml

kubectl -n cattle-system get pods -l app=cattle-cluster-agent -o wide

kubectl -n cattle-system logs deployment/cattle-cluster-agent

Also collect:

  • Rancher version
  • Rancher installation method
  • Rancher Helm values, if applicable
  • downstream Kubernetes distribution and version
  • Rancher server URL
  • whether Rancher was recently upgraded, rolled back, restored, or migrated
  • DNS resolution for the Rancher server hostname
  • load-balancer or ingress configuration in front of Rancher