Skip to content

How to configure Pod Topology Spread Constraints in RKE2?

Article Number: 000021901

Environment

RKE2

Situation

This article explains how to configure pod topology spread constraints in RKE2 to ensure better distribution of pods and improve fault tolerance.

Resolution

Note: Ensure to back up any of the files mentioned below if they already exist, especially if they contain other customized configurations.

1]  Create a file named scheduler.config in the /var/lib/rancher/rke2/server/manifests/ directory with the following content:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
profiles:
  - schedulerName: default-scheduler
    pluginConfig:
      - name: PodTopologySpread
        args:
          defaultConstraints:
            - maxSkew: 1
              topologyKey: topology.kubernetes.io/region
              whenUnsatisfiable: DoNotSchedule
          defaultingType: List

This configuration sets default constraints for the PodTopologySpread plugin, ensuring pods are spread across different topology.kubernetes.io/region with a maximum skew of 1. The configuration can be modified as per requirement.

2] Edit the RKE2 configuration file, located at /etc/rancher/rke2/config.yaml, and add the following argument under kube-scheduler-arg:

kube-scheduler-arg:
  - "config=/var/lib/rancher/rke2/server/manifests/scheduler.config"

This tells the kube-scheduler to use the custom configuration file.

3] Apply the changes by restarting the rke2-server service:

systemctl restart rke2-server

4] Finally, deploy some test pods and observe their distribution to ensure the custom topology spread constraints are being applied as expected. 

5] Below is an example deployment used for testing PodTopologySpread at a region level:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dev
  labels:
    app: nginx-dev
spec:
  replicas: 6 # Adjust this based on your desired total Pod count
  selector:
    matchLabels:
      app: nginx-dev
  template:
    metadata:
      labels:
        app: nginx-dev
    spec:
      # --- PodTopologySpread Configuration Starts Here ---
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/region # This is the key: spread across regions
        whenUnsatisfiable: DoNotSchedule # If the constraint cannot be met, do not schedule the Pod
        labelSelector:
          matchLabels:
            app: nginx-dev # Only count Pods belonging to this Deployment
      # --- PodTopologySpread Configuration Ends Here ---

      containers:
      - name: nginx-container
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"

7] Topology spread constraints rely on node labels to identify the topology domains that each node is in. Hence, its important to label the nodes accordingly.