How to add custom labels to Alerts in Monitoring v2

This document (000020302) is provided subject to the disclaimer at the end of this document.

Environment

A Rancher v2.5+ managed cluster, with the rancher-monitoring chart installed

Situation

Alerts in Monitoring v2 contain standard labels. In some cases, users may want to inject custom labels, such as Kubernetes cluster names to easily identify the environment or other custom labels for Alert routing.

Resolution

Use the value defaultRules.additionalRuleLabels  in the rancher-monitoring chart to inject custom labels.

For example, to inject cluster name, open the Monitoring App in Apps from Cluster explorer. Under "Edit as YAML", add the custom label as below:

defaultRules:
  additionalRuleLabels:
    cluster: "My_Test_cluster"

Then click on "Deploy" or "Upgrade" if App is already installed.

If your receiver is webhook, then the alerts will have the custom labels as shown in the below example alert.

...
...
status":"firing",
"labels":{
  "alertname":"NodeClockNotSynchronising",
  "cluster":"My_Test_cluster", <<<------
  "container":"node-exporter",
  "endpoint":"metrics",
  "instance":"192.168.110.157:9796",
  "job":"node-exporter",
  "namespace":"cattle-monitoring-system",
  "pod":"rancher-monitoring-prometheus-node-exporter-lg2g6",
  "prometheus":"cattle-monitoring-system/rancher-monitoring-prometheus",
  "service":"rancher-monitoring-prometheus-node-exporter",
...
...

Additional Information

GitHub issue #3325 is opened to add additional labels via UI.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.