Skip to content

The Rancher Apps repositories of air-gapped clusters in Rancher v2.8.3 - v2.8.5 fail to pull charts

Article Number: 000021441

Environment

Rancher v2.8.3, v2.8.4 and v2.8.5

Situation

Air-gapped clusters in a Rancher v2.8.3, v2.8.4 or v2.8.5 environment fail to pull the Rancher and Partners Apps Repositories.

The issue affects air-gapped Rancher local clusters where the Helm chart value useBundledSystemChart=true has not been set, and any air-gapped downstream clusters where the settings.management.cattle.io/system-catalog resource has not been manually updated to bundled.

In affected clusters the Rancher and Partners App Repositories remain in a Downloading state, with errors similar to the following:

Ensure failure: git-C/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/817acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- 8a753a4d6549a40ebeb725151cbbcc7e9065fce7 error: exit status 128, detail: fatal: unable to access https://git.rancher.io/partner-charts/': Failed to connect to git.rancher.io port 443 after 260784 ms: Couldn't connect to server:update failure: Repo [https://git.rancher.io/partner-charts] is not accessible: Get "https://git.rancher.io/repos/partner-charts/commits/main": dial tcp 148.113.221.144:443: i/o timeout (Client Timeout exceeded while awaiting headers)

Ensure failure: git-C/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 08898fb85b7aa56e754073a007844b3ecdae703 error: exit status 128, detail: fatal: unable to access https://git.rancherio/charts/': Failed to connect to git.rancher.io port 443 after 260685 ms: Couldn't connect to server:update failure: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/release-v2.8": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Cause

The issue was tracked in GitHub Issue #45420. Due to a change introduced in Rancher v2.8.3 the built-in Rancher and Partners Apps Repositories did not automatically fallback to using the Helm charts bundled within Rancher, in the instance that git.rancher.io was unreachable. As a result, in air-gapped clusters Rancher attempts and fails to pull the repositories from git.rancher.io.

Resolution

This issue was fixed in Rancher v2.8.6+ and users should upgrade to take advantage of the fix.