Performance considerations for a large number of CRDs in Rancher-managed clusters

Article Number: 000021521

Environment

SUSE Rancher 2.7.x+

Situation

Under the Practical Considerations documentation, it is mentioned we recommend not having above 100 CRDs in downstream clusters.

Cause

A high number of CRDs increases the chances for introducing latency when interacting with etcd. For example, if a CRD has a high amount of total objects, an API request for the objects will be chunked into batches, increasing the response time to retrieve the request data from etcd. If the additional CRDs only have a small number of objects each, the impact may not be observable, however, if they contain thousands of objects, that could be a concern.

Resolution

If performance issues are seen in the Rancher UI (such as the loading time taking several seconds) with a number of CRDs above 100 on a downstream cluster, it may be prudent to check the number of objects that each CRD has in order to verify if there is any concern. If so, consider cleaning unneeded objects that relate to CRDs with the highest object counts, or review distributing these to other clusters.

You can check the number of objects by CRDs in the Rancher UI. To do so, click on the Hamburger menu in the top-left of the Rancher UI, then click About -> Diagnostics. You will see a Resource Counts by Cluster section which can be expanded on each managed cluster to see the number of objects on each CRD.