How to configure Pod Topology Spread Constraints in RKE2?

This document (000021901) is provided subject to the disclaimer at the end of this document.

Environment

RKE2

Situation

This article explains how to configure pod topology spread constraints in RKE2 to ensure better distribution of pods and improve fault tolerance.

Resolution

Note: Ensure to back up any of the files mentioned below if they already exist, especially if they contain other customized configurations.

1]  Create a file named scheduler.config in the /var/lib/rancher/rke2/server/manifests/ directory with the following content:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
profiles:
  - schedulerName: default-scheduler
    pluginConfig:
      - name: PodTopologySpread
        args:
          defaultConstraints:
            - maxSkew: 1
              topologyKey: topology.kubernetes.io/region
              whenUnsatisfiable: DoNotSchedule
          defaultingType: List

This configuration sets default constraints for the PodTopologySpread plugin, ensuring pods are spread across different topology.kubernetes.io/region with a maximum skew of 1. The configuration can be modified as per requirement.

2] Edit the RKE2 configuration file, located at /etc/rancher/rke2/config.yaml, and add the following argument under kube-scheduler-arg:

kube-scheduler-arg:
  - "config=/var/lib/rancher/rke2/server/manifests/scheduler.config"

This tells the kube-scheduler to use the custom configuration file.

3] Apply the changes by restarting the rke2-server service:

systemctl restart rke2-server

4] Finally, deploy some test pods and observe their distribution to ensure the custom topology spread constraints are being applied as expected.

5] Below is an example deployment used for testing PodTopologySpread at a region level:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dev
  labels:
    app: nginx-dev
spec:
  replicas: 6 # Adjust this based on your desired total Pod count
  selector:
    matchLabels:
      app: nginx-dev
  template:
    metadata:
      labels:
        app: nginx-dev
    spec:
      # --- PodTopologySpread Configuration Starts Here ---
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/region # This is the key: spread across regions
        whenUnsatisfiable: DoNotSchedule # If the constraint cannot be met, do not schedule the Pod
        labelSelector:
          matchLabels:
            app: nginx-dev # Only count Pods belonging to this Deployment
      # --- PodTopologySpread Configuration Ends Here ---

      containers:
      - name: nginx-container
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"

7] Topology spread constraints rely on node labels to identify the topology domains that each node is in. Hence, its important to label the nodes accordingly.

Additional Information

In RKE2 clusters, pods may get scheduled unevenly across nodes, regions or zones, which can lead to poor availability or resource usage. To fix this, Kubernetes offers Pod Topology Spread Constraints, which help spread pods evenly across nodes, regions, or other topology domains.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.