Fleet SSH GitRepo Connections Fail After Rancher Upgrade Because of Unknown Host Key

Article Number: 000022146

Environment

Rancher v2.12, Fleet v0.13

Situation

After upgrading Rancher to v2.12, connections to SSH-based Git repositories fail with an error as below

ssh: handshake failed: knownhosts: key is unknown

OR

Ssh: handshake failed: knownhosts: key is unknown:failed to clone repo from branch repo="git@..." branch="main" revision="" path="/workspace": ssh: handshake failed: knownhosts: key is unknown

Cause

The issue is caused by a security enhancement in Fleet v0.13, which enforces SSH host key verification. This change requires users to explicitly trust the host key of their Git repositories, preventing potential man-in-the-middle attacks.

Resolution

1. Add the SSH host key fingerprint of the Git repository to the `known_hosts` field of each secret in the Git repository configuration. This is the recommended solution.

kubectl create secret generic customkey  -n fleet-default --from-file=ssh-privatekey=private_key --from-file=known_hosts=known_hosts --type=kubernetes.io/ssh-auth

2. When you don't specify an individual secret for gitrepo and you rely on "gitcredential" secret, known_host must be added in that secret, located in either the fleet-default or fleet-local namespace, depending on where the gitrepo is added.

kubectl create secret generic gitcredential  -n fleet-local --from-file=ssh-privatekey=private_key --from-file=known_hosts=known_hosts --type=kubernetes.io/ssh-auth

3. Alternatively, add the private repository's host key fingerprint manually to the `known-hosts` configmap after backing up the existing configmap. This is not a recommended solution

kubectl -n cattle-fleet-system get cm known-hosts  -o yaml > known-hosts.yaml
kubectl -n cattle-fleet-system edit cm known-hosts

Note: Host key fingerprints are already added for popular cloud-based Git repositories like GitHub, Bitbucket etc.