Error when enabling CIS profile on RKE2 cluster post-provisioning

Article Number: 000021600

Environment

A Kubernetes cluster provisioned by the RKE2 CLI or Rancher v2.x

Situation

To enable the CIS profile on an already provisioned RKE2 cluster, users must first manually stop the rke2-server process on etcd nodes. This step is necessary because the etcd database remains memory-mapped and open for writing, which can lead to issues.

Although this requirement is mentioned in the documentation, it is not clearly communicated in the UI. As a result, many users overlook this step and enable the CIS profile directly, causing issues with the etcd database, specifically the "permission denied" error mentioned below:

{"level":"panic","ts": "caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/db","error":"open /var/lib/rancher/rke2/server/db/etcd/member/snap/db: permission denied"}

Cause

By design, the RKE2 server service must be in a stopped state before enabling the CIS Profile. This step is required because the etcd database remains memory-mapped and open for writing while the service is running, which can lead to etcd permission denied issue as mentioned above.

Resolution

Manually update the file permissions for the etcd database directory to grant etcd access, allowing it to start successfully:

chown -R etcd:etcd /var/lib/rancher/rke2/server/db/etcd

Note: This can only be done for 'test' or non-critical clusters if one accidentally enables this post-cluster provisioning and is stuck with this issue.

Please note that the UI improvement request is fixed in rancher v2.12.0, this enhancement will provide clarity for users who enable the CIS profile through the UI.