How to update your etcd space alerts for better etcd monitoring
This document (000020187) is provided subject to the disclaimer at the end of this document.
Situation
Task
An alert you may have seen is Database usage close to the quota 500M in your Rancher 2.x cluster.
This is a default etcd alert built into Rancher, you can find more info on the default alerts here: Rancher v2.x Default Alerts
Upon further examination of the alert, you see the description of
A warning alert is triggered when the size of etcd exceeds 500M.
This alert is somewhat misleading as the default etcd size is 2GB. This alert is also at a severity level of Warning. Below, we suggest configuring your etcd alerts to better utilize the alert thresholds and to avoid the default alert constantly going off.
Pre-requisites
- Running Rancher v2.x
Resolution
You can access the alerts by going to Tools -> Alerts at the cluster level. From here, clone the default alert three times. Once the 3 clones are created, disable the default alert so you always retain a clean reference alert.
Alert for 1.5GB usage - WARNING
Alert for 1.75GB usage - CRITICAL
If you are running out of space, please reference the following two links. There will be an upcoming KB article walking through the resize process for etcd.
etcd space quota documentation
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.