Rancher Monitoring: Known Issues with NFS Persistent Storage for Prometheus
This document (000021332) is provided subject to the disclaimer at the end of this document.
Environment
- Rancher v2.x, from v2.5 and above
- rancher-monitoring deployed with a Prometheus persistent storage configuration using NFS
Situation
The Prometheus server is unstable, with storage-related errors in the Prometheus Pod logs:
2022-05-26T07:15:19.251601279Z level=error ts=2022-05-26T07:15:19.251Z caller=db.go:745 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/01F8Q3MT8CJXMNMPKW7G86M6M5/meta.json: no such file or directory"
and/or a warning about NFS storage not being supported:
2022-05-25T10:15:40.601740700Z level=warn ts=2022-05-25T10:15:40.601Z caller=main.go:756 fs_type=NFS_SUPER_MAGIC msg="This filesystem is not supported and may lead to data corruption and data loss. Please carefully read https://prometheus.io/docs/prometheus/latest/storage/ to learn more about supported filesystems."
Resolution
As described in the Prometheus docs, NFS is not a supported filesystem for Prometheus database storage:
CAUTION: Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may happen. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.
You should therefore remove the persistent storage configuration using NFS in the rancher-monitoring chart options or switch to a supported persistent storage type, such as a block storage volume provisioned (e.g. AWS EBS, Longhorn, vSphere volumes).
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.