Skip to content

How to customise container log rotation in an RKE2 or K3s cluster

Article Number: 000020071

Environment

A Rancher-provisioned or standalone RKE2 or K3s cluster

Situation

In RKE2 or K3s clusters, which utilize containerd via the Container Runtime Interface (CRI), the kubelet is responsible for managing the rotation of container logs.

This log rotation is configured via two kubelet options:

  • --container-log-max-files : "the maximum number of container log files that can be present for a container" (default: 5).
  • --container-log-max-size : "the maximum size (e.g. 10Mi) of container log file before it is rotated" (default: 10Mi).

In this default configuration, with a maximum of 5 log files of 10MiB each, each container can retain up to 50MiB of logs.

Resolution

Note: The examples below use 4 log files of 50MiB each (totaling ~200MiB per container). Please adjust these values to meet your specific retention and disk space requirements.

Standalone RKE2/K3s clusters

In standalone clusters, you must apply the configuration to the RKE2/K3s config file on every node in the cluster.

  1. Configure the kubelet's container-log-max-files and container-log-max-size arguments in the the RKE2 or K3s configuration file (/etc/rancher/rke2/config.yaml or /etc/rancher/k3s/config.yaml) per the following example:

kubelet-arg:
  - "container-log-max-files=4"
  - "container-log-max-size=50Mi"
2. After updating the configuration file, you will need to restart the RKE2/K3s service to apply the changes:

  • On RKE2 server nodes: systemctl restart rke2-server
  • On RKE2 worker nodes: systemctl restart rke2-agent
  • On K3s server nodes: systemctl restart k3s
  • On K3s worker nodes: systemctl restart k3s-agent

Rancher-provisioned RKE2/K3s cluster

In a Rancher-provisioned cluster, the configuration can be managed at the cluster-level within Rancher:

  1. Navigate to Cluster Management within the Rancher UI and click Edit Config for the relevant RKE2/K3s cluster
  2. Under Cluster Configuration click the Advanced tab
  3. Scroll down to For all machines, use the Kubelet args: in the section Additional Kubelet Args
  4. Click Add Argument and enter container-log-max-files=4
  5. Click Add Argument again and enter container-log-max-size=50Mi
  6. Click Save to apply the changes

Rancher will perform a rolling update of the cluster nodes to apply the new kubelet arguments.