How to drain a node in an RKE1 cluster from the local node using a docker command

This document (000020373) is provided subject to the disclaimer at the end of this document.

Environment

An RKE CLI or Rancher-provisioned RKE1 cluster

Situation

Need a way to cordon or drain a node when OS patching is automated, and the automation doesn't have access to the Rancher API.

Resolution

Integrate the below command in the automation to drain the node in the OS patching workflow.

Rancher v2.7.14+/Rancher v2.8.5+, RKE v1.4.19+/RKE v1.5.10+ :

docker exec kubelet bash -c 'kubectl --kubeconfig <(kubectl --kubeconfig /etc/kubernetes/ssl/kubecfg-kube-node.yaml get secret -n kube-system full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r .currentState.certificatesBundle.\"kube-admin\".config | sed -e "/^[[:space:]]*server:/ s_:.*_: \"https://127.0.0.1:6443\"_") drain $(hostname -s) --delete-local-data=true --force=true --grace-period=60 --ignore-daemonsets=true --timeout=120s'

Earlier versions of Rancher and RKE:

docker exec kubelet bash -c 'kubectl --kubeconfig <(kubectl --kubeconfig /etc/kubernetes/ssl/kubecfg-kube-node.yaml get configmap -n kube-system full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .currentState.certificatesBundle.\"kube-admin\".config | sed -e "/^[[:space:]]*server:/ s_:.*_: \"https://127.0.0.1:6443\"_") drain $(hostname -s) --delete-local-data=true --force=true --grace-period=60 --ignore-daemonsets=true --timeout=120s'

Please note that the below flags need to be changed according to your requirements.

--delete-local-data=true
--force=true
--grace-period=60
--ignore-daemonsets=true
--timeout=120s

Cause

Attempting to drain nodes using the kubeconfig file " /etc/kubernetes/ssl/kubecfg-kube-node.yaml" will result in errors like below, since the " system:node" role is not authorized to access the needed API groups.

cannot delete daemonsets.apps "nginx-ingress-controller" is forbidden: User "system:node" cannot get resource "daemonsets" in API group "apps" in the namespace "ingress-nginx"

But this kubeconfig file can access the secret/config-map "full-cluster-state" in the " kube-system" namespace, containing the kubeconfig file that has the privilege to do the drain operation.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.