How to Include additional resource labels in kube-state-metrics for prometheus

Article Number: 000021620

Environment

A Kubernetes cluster provisioned by the RKE2 CLI or Rancher v2.x

Situation

By default, kube-state-metrics in Rancher Monitoring exports only a limited set of Kubernetes labels to help optimize resource usage and avoid excessive data collection in Prometheus. If you need additional labels to be included in the metrics for better monitoring or alerting, you can easily customize the values.yaml file of the rancher-monitoring chart to export those labels.

Resolution

Why Customize Labels?

Sometimes, you may need more granular insights into your Kubernetes cluster. For example, you might want to track node group labels, application versions, or specific zone information. By exporting additional labels, you can enrich your metrics and enable more precise monitoring, alerting, and troubleshooting.

How to Export Additional Labels

To export additional labels, you can modify the metricLabelsAllowlist setting in the values.yaml file of the rancher-monitoring chart. This configuration lets you specify which labels to include for specific Kubernetes resources like nodes, pods, or namespaces.

Go to Rancher UI ☰ > Apps > Installed apps > rancher-monitoring, click on Edit/Upgrade option and then edit the helm values.

Example: Include All Labels for Nodes

If you need to include all labels for nodes (which can be useful but should be used cautiously due to performance considerations), you can use the wildcard *:

kube-state-metrics:
metricLabelsAllowlist:
- nodes=[*]

Warning: Using * will include all labels for nodes, which may increase the resource usage in Prometheus. 

If you want to add specific labels for namespaces and pods, you can configure it like this:

kube-state-metrics:
metricLabelsAllowlist:
- namespaces=["k8s-label-1", "k8s-label-2"]
- pods=["app", "version"]