Skip to content

Rancher Monitoring: Known Issues with NFS Persistent Storage for Prometheus

This document (000021332) is provided subject to the disclaimer at the end of this document.

Environment

  • Rancher v2.x, from v2.5 and above
  • rancher-monitoring deployed with a Prometheus persistent storage configuration using NFS

Situation

The Prometheus server is unstable, with storage-related errors in the Prometheus Pod logs:

2022-05-26T07:15:19.251601279Z level=error ts=2022-05-26T07:15:19.251Z caller=db.go:745 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/01F8Q3MT8CJXMNMPKW7G86M6M5/meta.json: no such file or directory"

and/or a warning about NFS storage not being supported:

2022-05-25T10:15:40.601740700Z level=warn ts=2022-05-25T10:15:40.601Z caller=main.go:756 fs_type=NFS_SUPER_MAGIC msg="This filesystem is not supported and may lead to data corruption and data loss. Please carefully read https://prometheus.io/docs/prometheus/latest/storage/ to learn more about supported filesystems."

Resolution

As described in the Prometheus docs, NFS is not a supported filesystem for Prometheus database storage:

CAUTION: Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may happen. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.

You should therefore remove the persistent storage configuration using NFS in the rancher-monitoring chart options or switch to a supported persistent storage type, such as a block storage volume provisioned (e.g. AWS EBS, Longhorn, vSphere volumes).

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.