Skip to content

system-upgrade-controller failing with "exec /opt/rancher-system-agent-suc/run.sh: argument list too long"

This document (000021385) is provided subject to the disclaimer at the end of this document.

Environment

  • Rancher v2.6+
  • A Rancher-managed RKE2 or K3s cluster

Situation

The system-upgrade-controller Pods in a Rancher-managed RKE2 or K3s cluster fail with the error "exec /opt/rancher-system-agent-suc/run.sh: argument list too long"

Resolution

The Rancher TLS certificate may be signed by a private CA or by intermediate certificate(s) signed by a public root CA. In this instance, the tls-ca secret generated in the Rancher local cluster should contain the root CA, followed by any required intermediate CA certificates.

If the secret contains additional certificates that are not required, it might be too large, and the system-upgrade-controller Pods will fail.

To solve this issue, you must recreate the tls-ca secret containing only the Rancher root, and any required intermediate, CA certificates. To do so, you can follow this documentation: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/add-tls-secrets#using-a-private-ca-signed-certificate

Then, perform a restart of the Rancher deployment:

kubectl rollout restart deploy/rancher -n cattle-system

Afterwards, the system-upgrade-controller Job Pods should be able to restart successfully.

Cause

This issue is caused by the tls-ca secret in the cattle-system Namespace of the Rancher local cluster. The secret is passed down to the system-upgrade-controller Job Pods as an environment variable (via the stv-aggregation secret in the cattle-system Namespace of the downstream clusters). If the environment variable is too large, it will result in the "exec /opt/rancher-system-agent-suc/run.sh: argument list too long" error.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.