Skip to content

Hard eviction settings are no longer configured by default in RKE1 v1.26+

Article Number: 000021816

Environment

  • Rancher v2.x
  • A Rancher-provisioned or standalone RKE1 v1.26+ cluster

Situation

In RKE1 v1.26 and above, the kubelet hard eviction settings are blank.

Previously, in RKE1 v1.25 and below, the kubelet hard eviction settings were configured by default. To check the kubelet settings, use kubectl get --raw "/api/v1/nodes/<nodename>/proxy/configz"

  • RKE v1.25: 

 kubectl get --raw "/api/v1/nodes/wk1/proxy/configz" 
{"kubeletconfig":{"enableServer":true,"syncFrequency":"1m0s",:true, ...
"evictionHard":{"imagefs.available":"15%","memory.available":"100Mi","nodefs.available":"10%","nodefs.inodesFree":"5%"} ...
,"evictionPressureTransitionPeriod":"5m0s","enableControllerAttachDetach":true,"makeIPTablesUtilChains":true,"iptablesMasqueradeBit":14,"iptablesDropBit":15,"failSwapOn":false,"memorySwap":}>
- RKE v1.26+ (there is no evictionHard section):

kubectl get --raw "/api/v1/nodes/wk1/proxy/configz" 
{"kubeletconfig":{"enableServer":true,"syncFrequency":"1m0s","fileCheckFrequency":"20s","httpCheckFrequency":"20s","address":"0.0.0.0","port":10250,"tlsCertFile":"/var/lib/kubelet/pki/kubelet.crt","tlsPrivateKeyFile":"/var/lib/kubelet/pki/kubelet.key","tlsCipherSuites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"],"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/ssl/kube-ca.pem"},"webhook":{"enabled":true,"cacheTTL":"2m0s"},"anonymous":{"enabled":false}},"authorization":{"mode":"Webhook","webhook":{"cacheAuthorizedTTL":"5m0s","cacheUnauthorizedTTL":"30s"}},"registryPullQPS":5,"registryBurst":10,"eventRecordQPS":0,"eventBurst":10,"enableDebuggingHandlers":true,"healthzPort":10248,"healthzBindAddress":"127.0.0.1","oomScoreAdj":-999,"clusterDomain":"cluster.local","clusterDNS":["10.43.0.10"],"streamingConnectionIdleTimeout":"30m0s","nodeStatusUpdateFrequency":"10s","nodeStatusReportFrequency":"5m0s","nodeLeaseDurationSeconds":40,"imageMinimumGCAge":"2m0s","imageGCHighThresholdPercent":85,"imageGCLowThresholdPercent":80,"volumeStatsAggPeriod":"1m0s","cgroupsPerQOS":true,"cgroupDriver":"systemd","cpuManagerPolicy":"none","cpuManagerReconcilePeriod":"10s","memoryManagerPolicy":"None","topologyManagerPolicy":"none","topologyManagerScope":"container","runtimeRequestTimeout":"2m0s","hairpinMode":"promiscuous-bridge","maxPods":110,"podPidsLimit":-1,"resolvConf":"/run/systemd/resolve/resolv.conf","cpuCFSQuota":true,"cpuCFSQuotaPeriod":"100ms","nodeStatusMaxImages":50,"maxOpenFiles":1000000,"contentType":"application/vnd.kubernetes.protobuf","kubeAPIQPS":5,"kubeAPIBurst":10,"serializeImagePulls":true,"evictionPressureTransitionPeriod":"5m0s","enableControllerAttachDetach":true,"makeIPTablesUtilChains":true,"iptablesMasqueradeBit":14,"iptablesDropBit":15,"failSwapOn":false,"memorySwap":{},"containerLogMaxSize":"10Mi","containerLogMaxFiles":5,"configMapAndSecretChangeDetectionStrategy":"Watch","enforceNodeAllocatable":["pods"],"volumePluginDir":"/var/lib/kubelet/volumeplugins","logging":{"format":"text","flushFrequency":5000000000,"verbosity":2,"options":{"json":{"infoBufferSize":"0"}}},"enableSystemLogHandler":true,"shutdownGracePeriod":"0s","shutdownGracePeriodCriticalPods":"0s","enableProfilingHandler":true,"enableDebugFlagsHandler":true,"seccompDefault":false,"memoryThrottlingFactor":0.8,"registerNode":true,"localStorageCapacityIsolation":true}}>

Node pressure is never detected, even if the disk reaches the previous default of 90% usage and above or the node runs low on memory, due to the kubelet hard eviction settings missing.

Cause

This is happening due to an upstream change in behavior. It can be confirmed by checking the hard eviction defaults in the kubelet’s test files:

v1.25 – The hard eviction settings are set to the Default
https://github.com/kubernetes/kubernetes/blob/v1.25.16/pkg/kubelet/apis/config/v1beta1/defaults_test.go#L102

v1.26 – The hard eviction settings are set to "Nil"
https://github.com/kubernetes/kubernetes/blob/v1.26.15/pkg/kubelet/apis/config/v1beta1/defaults_test.go#L102

v1.26 introduces the default hard eviction settings at a different level, but this doesn't apply to RKE1 since it does not pass a config file. If a config file is passed, the default is picked up from this part of the code, which is only reached when the len(ConfigFile) > 0
https://github.com/kubernetes/kubernetes/blob/v1.26.15/cmd/kubelet/app/server.go#L357

Resolution

To add the hard eviction settings to a Rancher-provisioned RKE1 v1.26+ cluster, navigate to Cluster Management => Edit Config => Edit as YAML, then add the following configuration:

[...]
services:
  kubelet:
    extra_args:
      eviction-hard: "nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%,memory.available<100Mi"
[...]

For an RKE1 standalone cluster, apply the above changes to the cluster configuration YAML file and invoke rke up to apply the changes.