Skip to content

How to configure custom kube-scheduler profiles in a Rancher-provisioned RKE2 cluster

This document (000021703) is provided subject to the disclaimer at the end of this document.

Environment

  • Rancher v2.6+
  • A Rancher-provisioned RKE2 cluster

Situation

This article details how to configure the Kubernetes scheduler by injecting custom kube-scheduler profiles in a Rancher-provisioned RKE2 cluster.

Resolution

This solution employs a ConfigMap and machineSelectorFiles to deploy the necessary kube-scheduler configuration to the control-plane nodes within a Rancher-provisioned RKE2 cluster. The following steps are needed:

  1. ConfigMap for kube-scheduler configuration:
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-scheduler-config
  namespace: fleet-default
  annotations:
    rke.cattle.io/object-authorized-for-clusters: rke2custom
data:
  config: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    clientConnection:
      kubeconfig: /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
    profiles:
    - schedulerName: default-scheduler
    - schedulerName: no-scoring-scheduler
      plugins:
        preScore:
          disabled:
          - name: '*'
            score:
  1. Referencing the ConfigMap in the RKE2 cluster configuration: This is done within the cluster.provisioning.cattle.io rkeConfig.
machineSelectorFiles:
  - fileSources:
    - configMap:
      items:
       - key: config
        path: /kube-scheduler-config
        permissions: '644'
      name: kube-scheduler-config
   machineLabelSelector:
    matchLabels:
     rke.cattle.io/control-plane-role: 'true'
 machineSelectorConfig:
  - config:
    kube-scheduler-arg: 'config=/kube-scheduler-config'
   machineLabelSelector:
    matchLabels:
     rke.cattle.io/control-plane-role: 'true'

Note: The /kube-scheduler-config path is arbitrary and was used for testing purposes

Cause

To customise the Kubernetes scheduler in an RKE2 cluster a custom kube-scheduler configuration can be injected to achieve this desired result.

Additional Information

  • Key Considerations:

  • The clientConnection.kubeconfig configuration is essential, as the presence of the --config parameter in the kube-scheduler arguments overrides the --kubeconfig parameter automatically configured by RKE2.

  • Do not manually add the /kube-scheduler-config file as an additional mount via kube-apiserver-extra-mount. RKE2 handles this automatically when referenced in the arguments. Adding it manually will result in a double-mount attempt and kube-scheduler pod startup failures.
  • Configuring custom scheduling policies like NodeResourcesBalancedAllocation is possible in RKE2. However, creating and troubleshooting such policies are outside the standard support scope. For tailored guidance, users should contact their CSM for consulting services.
  • References:
  • Kubernetes Scheduler Configuration
  • RKE2 Server Configuration
  • Rancher RKE2 Cluster Configuration - machineSelectorFiles
  • Rancher RKE2 Cluster Configuration - machineSelectorConfig

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.