Skip to content

How to add a kubelet configuration file to nodes in a Rancher-provisioned RKE2 or K3s cluster

Article Number: 000022176

Environment

  • Rancher v2.7.2+
  • A Rancher-provisioned RKE2 or K3s cluster

Procedure

This article details how to configure the Kubernetes kubelet parameters by providing a custom configuration file in a Rancher-provisioned RKE2/K3s cluster. Whilst it is still possible to pass kubelet parameters as arguments, even though this is noted as deprecated in the Kubernetes documentation, the following steps enable deployment via a configuration file.

Per the RKE2 and K3s documentation there are two methods for passing a kubelet configuration file:

  1. In Kubernetes v1.32+ a drop-in configuration file can be written to the directory /var/lib/rancher/<rke2/k3s>/agent/etc/kubelet.conf.d/
  2. The kubelet argument config can be set to reference a specific kubelet configuration file at another location

The steps below make use of a machineSelectorFiles block on the cluster resource, to enable deployment of a file across nodes in a Rancher-provisioned cluster.

In this example the imageMaximumGCAge parameter is configured to 12h.

1. Create a ConfigMap or Secret in the Rancher local cluster

Create a ConfigMap or Secret within the fleet-default Namespace in the Rancher local cluster, containing your custom kubelet configuration.

The Secret or ConfigMap must meet the following requirements:

  1. It must exist within the same fleet workspace as the cluster itself, for Rancher-deployed clusters this is fleet-default by default.
  2. It must have the annotation rke.cattle.io/object-authorized-for-clusters: <cluster-name1>,<cluster-name2>with a comma separated list of the clusters which are permitted to use it.

In the example below a ConfigMap is created, with access granted to the cluster named rke2custom, and a kubelet configuration to define imageMaximumGCAge.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-kubelet-config
  namespace: fleet-default
  annotations:
    rke.cattle.io/object-authorized-for-clusters: rke2custom
data:
  kubelet-config.yaml: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    imageMaximumGCAge: 12h

2a. In Kubernetes v1.32+

The recommended method for passing a custom kubelet configuration in Kubernetes v1.32+ is the use of a drop-in file, which removes the requirement to pass an additional config argument to instruct kubelet to read the file.

In the example below, the kubelet configuration ConfigMap created above is written to the kubelet configuration drop-in directory on the nodes of an RKE2 cluster (/var/lib/rancher/rke2/agent/etc/kubelet.conf.d/).

To define this machineSelectorFiles block in a Rancher-provisioned RKE2 or K3s cluster, navigate to Cluster Management in the Rancher UI and click Edit Config for the relevant cluster. Then click Edit as YAML to enter the machineSelectorFiles block, before clicking Save to apply the change.

[...]
spec:
  [...]
  rkeConfig:
    [...]
    machineSelectorFiles:
      - fileSources:
          - configMap:
              items:
                - key: kubelet-config.yaml
                  path: /var/lib/rancher/rke2/agent/etc/kubelet.conf.d/01-custom.conf
              name: custom-kubelet-config
[...]

2b. In Kubernetes <v1.32

In Kubernetes < v1.32, it is not possible to use the drop-in file feature, so the file must be written to a location (via a machineSelectorFiles block) which is then explicitly passed to the kubelet via the config argument (using a machineSelectorConfig block).

In the example below, the kubelet configuration ConfigMap created above is written to the file /etc/rancher/rke2/kubelet-config-custom.yaml on the nodes of an RKE2 cluster, which is then referenced in the config argument passed to the kubelet.

To define the machineSelectorFiles and machineSelectorConfig blocks in a Rancher-provisioned RKE2 or K3s cluster, navigate to Cluster Management in the Rancher UI and click Edit Config for the relevant cluster. Then click Edit as YAML to enter the blocks, before clicking Save to apply the change.

[...]
spec:
  [...]
  rkeConfig:
   [...]
    machineSelectorConfig:
      - config:
          kubelet-arg:
            - config=/etc/rancher/rke2/kubelet-config-custom.yaml
    [...]
    machineSelectorFiles:
      - fileSources:
          - configMap:
              items:
                - key: kubelet-config.yaml
                  path: /etc/rancher/rke2/kubelet-config-custom.yaml
              name: custom-kubelet-config
[...]