Skip to content

Downstream cluster provisioning fails due to unexpected newline characters in Kubernetes secrets

Article Number: 000021957

Environment

  • Rancher v2.11.x
  • Rancher v2.12.x

Situation

  • In downstream custom clusters with a registry pull secret, you see the error:
system-agent-installer-rke2%3Apull&service=myregistry.test.io: : Bad credentials"
  • An example of the registry pull secret is below.
apiVersion: v1
data:
  password: SGVsbG8K
  username: V29ybGQK
kind: Secret
metadata:
  name: k8s-secret-clusterpull-hello-world
  namespace: fleet-default
type: kubernetes.io/basic-auth
  • This results in "/etc/rancher/agent/registries.yaml" file on the downstream server nodes containing the unexpected newline characters ('\n'). Hence the error.

Cause

The issue is caused by the echo command appending a newline character ('\n') when encoding strings to base64, which is then used in Kubernetes secrets.

Resolution

  • When encoding strings for Kubernetes secrets, use the "-n" flag with the echo command to suppress the trailing newline character. For example, instead of "echo mypassword" | base64", use "echo -n "mypassword" | base64".
  • This prevents the newline character from being included in the encoded string, causing authentication issues. Here, in the above example, the registries.yaml file, the encoded string, i.e, SGVsbG8K = decodes to Hello\n and V29ybGQK = decodes to World\n.
  • Unfortunately, the "echo | base64 -d" command does not show the extra newline character. Instead, you can use the 'hexdump' command to confirm any unexpected newline character in encoded strings.

- The "0a" character in the command output above indicates the new line. See the ASCII character chart here. - As a solution, use the "-n" flag as shown below when encoding the strings. The "-n" flag suppresses the trailing. \n. 

From the man page of the echo command.

#man echo | grep -i 'newline'
-n     do not output the trailing newline