Skip to content

Inability to attach/detach vSphere CNS block volumes

Article Number: 000021286

Environment

  • Rancher 2.6 / 2.7
  • RKE1/RKE2
  • Kubernetes v1.19+
  • vSphere 6.7 U3+ or vSphere 7.0+

  • Vsphere cloud provider:

  • Vsphere CPI: rancher-vsphere-cpi:100.3.0+up1.2.1+

  • Vsphere CSI: rancher-vsphere-csi:100.3.0+up2.5.1-rancher1+

Situation

Inability to detach/attach CNS block volumes:

Customers can create CNS block volumes in the RKE1/RKE2 cluster using the Vsphere CSI.

However, when scaling down a workload (deployment,statefulset), the block volume does not get detached automatically from the nodes. Scaling up the workload, the following error appears in the cluster events:

rpc error: code = Internal desc = queryVolume failed for volumeID: "5db7cc3c-62b9-427d-823b-87729fcef771" with err=ServerFaultCode: NoPermission

Cause

The user account within vSphere must be granted the following permissions:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-C04F1605-D158-4B65-810F-6F5B109BCDEC.html

Resolution

This error indicates the user is missing the permission "Cns.Searchable" at the root vCenter level and Datastore level.

To grant the user account the Cns.Searchable permission in vSphere, see the following documentation:
https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials