Skip to content

Downstream cluster provisioning fails due to unexpected newline characters in Kubernetes secrets

This document (000021957) is provided subject to the disclaimer at the end of this document.

Environment

  • Rancher v2.11.x
  • Rancher v2.12.x

Situation

  • In downstream custom clusters with a registry pull secret, you see the error:
system-agent-installer-rke2%3Apull&service=myregistry.test.io: : Bad credentials"
  • An example of the registry pull secret is below.
apiVersion: v1
data:
  password: SGVsbG8K
  username: V29ybGQK
kind: Secret
metadata:
  name: k8s-secret-clusterpull-hello-world
  namespace: fleet-default
type: kubernetes.io/basic-auth
  • This results in "/etc/rancher/agent/registries.yaml" file on the downstream server nodes containing the unexpected newline characters ('\n'). Hence the error.

Resolution

  • When encoding strings for Kubernetes secrets, use the "-n" flag with the echo command to suppress the trailing newline character. For example, instead of " echo mypassword" | base64", use "echo -n "mypassword" | base64".

  • This prevents the newline character from being included in the encoded string, causing authentication issues. Here, in the above example, the registries.yaml file, the encoded string, i.e, SGVsbG8K = decodes to Hello\n and V29ybGQK = decodes to World\n.

  • Unfortunately, the "echo | base64 -d" command does not show the extra newline character. Instead, you can use the 'hexdump' command to confirm any unexpected newline character in encoded strings.

  • The "0a" character in the command output above indicates the new line. See the ASCII character chart here.

  • As a solution, use the "-n" flag as shown below when encoding the strings. The "-n" flag suppresses the trailing. \n.

From the man page of the echo command.

#man echo | grep -i 'newline'
-n     do not output the trailing newline

Cause

The issue is caused by the echo command appending a newline character ('\n') when encoding strings to base64, which is then used in Kubernetes secrets.

Additional Information

This issue is irrespective of the downstream cluster provisioning. To avoid the unexpected "\ n" character, always use the "-n" flag while encoding the string with the ' echo' command.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.