Skip to content

How to create a custom cluster role in Rancher v2.x to grant permission on the metrics endpoint of the kube-apiserver in a Rancher managed cluster

This document (000020052) is provided subject to the disclaimer at the end of this document.

Environment

  • A Rancher v2.x instance
  • A Rancher-managed Kubernetes cluster, with Metrics Server deployed. This is deployed by default in Rancher-provisioned clusters.

Situation

This article details how to create a cluster role to grant users access to the /metrics endpoint of the Kubernetes API Server, in Rancher-managed Kubernetes clusters.

In Rancher v2.4.x - v2.7.x it should be possible to define a non-resource URL grant via role creation within the Rancher UI. However, this was affected by the issue tracked in Issue #30321, and use of the Rancher v3 API is therefore required to create the role. This bug was solved in Rancher v2.8.0. Then, since that version, it is possible to create the role using the UI or the Rancher API, as explained here.

  • Rancher v2.4.x - v2.7.x: creating the role is only possible using the Rancher API.
  • Rancher 2.8.x and above: it is possible to create the role using the Rancher API or the Rancher UI.

Resolution

  1. The first step is to create a custom cluster role within Rancher, that grants get permission on the non-resource URL /metrics endpoint.

As an admin user, generate an un-scoped Rancher API token, and execute the following API request via cURL, to create the required role. You will need to set CATTLE_ACCESS_KEY, CATTLE_SECRET_KEY and RANCHER_URL to reflect the generated API token and your Rancher URL. You can also edit the role name, as desired, which is set to kube-api metrics in this example.

export CATTLE_ACCESS_KEY=token-8jn92
export CATTLE_SECRET_KEY=l2r4nq9sx6pdhpm4bgwntvgk49qn6rvvmtsvlvkmjk9rjsfd7n65fz
export RANCHER_URL=rancher.example.com
curl -k -u "${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}" \
   -X POST \
   -H 'Accept: application/json' \
   -H 'Content-Type: application/json' \
   -d '{"context":"cluster","clusterCreatorDefault":false," projectCreatorDefault":false,"name":"kube-api metrics","rules":[{"nonResourceURLs":["/metrics"],"type":"/v3/schemas/policyRule","verbs":["get"]}]}' \
"https://${RANCHER_URL}/v3/roletemplates"

  1. After creating the cluster role, you can then grant this for a user or group. To do so, follow the steps in the Rancher documentation on assigning a cluster role to a user or group.

  2. Once the role is granted to a user, they will be able to test their access to the /metrics endpoint.

The user can access the endpoint, with the applicable cluster ID, via the Rancher proxied Kubernetes API Server endpoint, by generating a cluster-scoped or un-scoped API token. The user will need to set CATTLE_ACCESS_KEY, CATTLE_SECRET_KEY, RANCHER_URL and CLUSTER_ID to reflect the generated API token, Rancher URL and cluster ID.

export CATTLE_ACCESS_KEY=token-8jn92
export CATTLE_SECRET_KEY=l2r4nq9sx6pdhpm4bgwntvgk49qn6rvvmtsvlvkmjk9rjsfd7n65fz
export RANCHER_URL=rancher.example.com
export CLUSTER_ID=c-wwdjc
curl -k https://${RANCHER_URL}/k8s/clusters/${CLUSTER_ID}/metrics \
   -H "Authorization: Bearer ${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}"

For Rancher-provisioned Kubernetes clusters with Authorized Cluster Endpoint enabled, the user can also query the endpoint by connecting to the Kubernetes API Server on the cluster's control plane nodes directly, using a cluster-scoped API token. The user will need to set CATTLE_ACCESS_KEY, CATTLE_SECRET_KEY and AUTHORIZED_ENDPOINT_ADDRESS to reflect the generated API token, and the authorized endpoint address.

export CATTLE_ACCESS_KEY=token-d6cls
export CATTLE_SECRET_KEY=b6gk6lmgrhsb4rjccktzkwxn5df7tm87msggq87lpmls2pkbpc5t5r
export AUTHORIZED_ENDPOINT_ADDRESS=controlplane-01.example.com
curl -k https://${AUTHORIZED_ENDPOINT_ADDRESS}:6443/metrics \
   -H "Authorization: Bearer ${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}"

Additional Information

Further reading

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.