Skip to content

'SIGSEGV: segmentation violation' in prometheus container of the prometheus-project-monitoring-0 Pod when enabling Project monitoring on the System Project

This document (000020219) is provided subject to the disclaimer at the end of this document.

Situation

Issue

Enabling project monitoring in a Rancher v2.2 cluster, in which cluster monitoring is enabled, fails with the Prometheus Pod in a CrashLoopBackOff.

The prometheus container in the prometheus-project-monitoring StatefulSet fails with an error of the following format:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x669c12]
goroutine 437 [running]:
net/http.(*Client).deadline(0x0, 0xc005381070, 0x40bb8f, 0xc0055e3600)
/usr/local/go/src/net/http/client.go:187 +0x22
net/http.(*Client).do(0x0, 0xc005cdaa00, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:527 +0xab
net/http.(*Client).Do(0x0, 0xc005cdaa00, 0x23, 0xc002802230, 0x9)
/usr/local/go/src/net/http/client.go:509 +0x35
github.com/prometheus/prometheus/scrape.(*targetScraper).scrape(0xc0060fa960, 0x1fd4a60, 0xc00010ec60, 0x1fb2760, 0xc0002eb110, 0x0, 0x0, 0x0, 0x0)
/app/scrape/scrape.go:471 +0x111
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc00616a100, 0xdf8475800, 0x2540be400, 0x0)
/app/scrape/scrape.go:813 +0x487
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
/app/scrape/scrape.go:336 +0x45d

Pre-requisites

  • A cluster managed by Rancher v2.2
  • Cluster monitoring enabled and Project monitoring enabled on the System project

Resolution

Project monitoring is not compatible with the Rancher System project and should not be enabled in the System project. Starting with Rancher v2.3.0 monitoring of the System project is performed by cluster monitoring, when this is enabled, and the UI prevents enabling of project monitoring on the System project.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.