Performance considerations for a large number of CRDs in Rancher-managed clusters

This document (000021521) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Rancher 2.7.x+

Situation

Under the Practical Considerations documentation, it is mentioned we recommend not having above 100 CRDs in downstream clusters.

Resolution

If performance issues are seen in the Rancher UI (such as the loading time taking several seconds) with a number of CRDs above 100 on a downstream cluster, it may be prudent to check the number of objects that each CRD has in order to verify if there is any concern. If so, consider cleaning unneeded objects that relate to CRDs with the highest object counts, or review distributing these to other clusters.

You can check the number of objects by CRDs in the Rancher UI. To do so, click on the Hamburger menu in the top-left of the Rancher UI, then click About -> Diagnostics. You will see a Resource Counts by Cluster section which can be expanded on each managed cluster to see the number of objects on each CRD.

Cause

A high number of CRDs increases the chances for introducing latency when interacting with etcd. For example, if a CRD has a high amount of total objects, an API request for the objects will be chunked into batches, increasing the response time to retrieve the request data from etcd. If the additional CRDs only have a small number of objects each, the impact may not be observable, however, if they contain thousands of objects, that could be a concern.

Additional Information

Performance in Kubernetes is multi-dimensional, so it is important to consider the needs of workloads sharing the same cluster, and the growth of these over time. When planning for multiple large-scale workloads that have unique resource needs, it is recommended to consider splitting loosely-coupled workloads into separate clusters so they can scale without contradicting the needs of others.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.