Skip to content

The Repositories of Air-gapped Downstream Clusters in Rancher v2.8.3. fail to pull charts.

Article Number: 000021441

Environment

SUSE Rancher v.2.8.3

Situation

There is a bug affecting Rancher v2.8.3 Downstream clusters in AirGapped environments. The Repositories try to pull from "git.rancher.io", when they should be using the bundledSystemCharts (assuming the system-catalog setting's value is "bundled" in the Upstream cluster). This action fails, as the AirGapped clusters do not have connection to the exterior.

Cause

The "useBundledSystemChart=true" setting is not being properly shared from upstream to downstream clusters in Rancher v2.8.3. This will cause Downstream Clusters not to contain the "bundled" value for their system-catalog setting. As such, they will try to pull the charts from git.rancher.io. In the case of AirGapped clusters, this will fail, and the fetch action will throw a "Context Deadline Exceeded" error.

Resolution

You can check if you are hitting this issue by running the following command in the Upstream cluster:

kubectl get settings.management.cattle.io system-catalog

If the value is set to "bundled", run the same command on the Downstream Cluster. If the value is empty, that is a confirmation that you hit the issue. 

There is a manual workaround available to update the system-catalog setting for Downstream clusters. 

  • On Rancher Manager, navigate to the Downstream clusters with the issue.
  • Go to Workloads -> Pods  (with All Namespaces selected in the upper-right dropdown menu).
  • Find the cattle-cluster-agent-######## pod (it is in the cattle-system namespace) -> click on the 3 vertical dots to the right -> Execute Shell
  • Inside this shell, execute:
kubectl edit settings.management.cattle.io system-catalog
  • Change the value to "bundled".

The Repositories should work normally after the change is done and a few minutes pass or you Refresh the repositories.