How to run workloads on etcd or controlplane nodes, without the worker role, in a Rancher Kubernetes Engine (RKE) cluster

This document (000020116) is provided subject to the disclaimer at the end of this document.

Environment

A Rancher Kubernetes Engine (RKE) cluster, provision with the RKE CLI or Rancher v2.x

Situation

Although it is usually not advised to run workloads on your controlplane and etcd nodes, there are occasionally scenarios when this is necessary. A few common examples are virus scanning, monitoring, and log collection workloads.

Resolution

Both the controlplane and etcd nodes, which are not additionally designated the worker role, have taints. When RKE or Rancher provisions these nodes, it adds these taints automatically. Workloads that need to run on these nodes require tolerations for these taints. For Rancher managed clusters, you can see these taints within the Rancher UI on the cluster node view. The following kubectl command will also list the taints for each node.

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
NAME           TAINTS
ip-10-0-2-10   [map[effect:NoExecute key:node-role.kubernetes.io/etcd value:true]]
ip-10-0-2-11   [map[effect:NoSchedule key:node-role.kubernetes.io/controlplane value:true]]
ip-10-0-2-12   <none>

Per this output, each etcd node has the NoExecute taint node-role.kubernetes.io/etcd=true and each controlplane node has the NoSchedule taint node-role.kubernetes.io/controlplane=true.

The Rancher UI does not have fields for adding tolerations, so you must specify the tolerations directly in the workload's YAML manifest. You can use the Import YAML button to deploy your workload, and make sure to add the following tolerations block in your manifest:

spec:
...
  template:
...
    spec:
...
      tolerations:
      - operator: Exists
...

If you have an existing workload, you can also select the View/Edit YAML option for the workload and apply the above change. This toleration will allow you to run the workload on any nodes with taints, so use with caution. If you are using Helm charts, you can also specify the same YAML in your Helm chart.

Additional Information

For more information on how taints and tolerations work in Kubernetes, see: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.