Skip to content

Activating CIS on the Rancher local cluster breaks Rancher related deployments

This document (000021724) is provided subject to the disclaimer at the end of this document.

Environment

Rancher 2.8+

Situation

You have applied a CIS profile to an RKE2 Rancher local cluster, and your Rancher and Fleet deployments are unable to scale pods, displaying this error message:

Error creating: pods "rancher-677d78b948-d7m9s" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "rancher", "rancher-audit-log" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "rancher", "rancher-audit-log" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "rancher", "rancher-audit-log" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "rancher", "rancher-audit-log" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost").

Another detected symptom is the impossibility to connect on the local cluster through the Kubectl UI shell.

Resolution

Copy this PSA from Rancher security guide to for example /etc/rancher/rke2/rke2-pss_rancher.yaml and add the pod-security-admission-config-file variable pointing to this file to ensure that Kubelet applies it.

RKE2 configuration file will then looks like this:

# cat /etc/rancher/rke2/config.yaml
profile: cis
pod-security-admission-config-file: "/etc/rancher/rke2/rke2-pss_rancher.yaml"

Restart rke2-server systemd service for this new configuration to take effect:

systemctl restart rke2-server

To ensure that your new PSA configuration is applied by Kubelet, you have two options:

  • Check the kubelet process arguments:
# ps -ef | grep [a]dmission-control-config-file
root        3633    3580  6 09:44 ?        00:02:28 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true ...

If multiple --admission-control-config-file options are provided, the last one will take precedence and be applied.

  • Create a dummy pod in the cattle-system namespace:
$ kubectl run dummy-pod --image='busybox' -n cattle-system
pod/dummy-pod created

If the correct PSA is not applied, pod creation in the cattle-system namespace will be forbidden.

Cause

The PSA exemptions on namespace pushed by RKE2 on file /etc/rancher/rke2/rke2-pss.yaml are not enough to let Rancher components to correctly work.

You can find in the Rancher security guide the correct PodSecurityConfiguration containing all the required Rancher namespace exemptions for a rancher-restricted cluster to run properly.

Additional Information

Link to PSA Rancher exemption:

https://ranchermanager.docs.rancher.com/reference-guides/rancher-security/psa-restricted-exemptions

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.