Skip to content

Configuring kube-proxy to use the nftables iptables backend with RKE2

Article Number: 000022178

Environment

RKE2

Situation

In some circumstances, the auto-detection of the iptables backend used by kube-proxy may switch between the legacy and nftables backends. This can result in inconsistent or missing NAT rules, causing network communication issues between pods, services, or nodes.
This article explains how to explicitly configure kube-proxy to use the nftables (modern) backend by setting the IPTABLES_MODE environment variable.

You may observe one or more of the following symptoms:

  • The iptables NAT table appears incomplete or missing expected rules
  • Traffic between pods or services intermittently fails
  • The node's OS uses the nftables backend (iptables-nft package), but kube-proxy is managing rules with the legacy backend

This can be confirmed on RKE2 nodes where the issue is suspected:

  • List all of the kube-proxy pods:
kubectl get pods -n kube-system -l component=kube-proxy -o wide
  • Update the commands to run the kubectl exec commands on the kube-proxy pod of the related node(s):
kubectl exec -it -n kube-system kube-proxy-xxxx -- /usr/sbin/iptables-legacy-save
  # Compare with nft
kubectl exec -it -n kube-system kube-proxy-xxxx -- /usr/sbin/iptables-nft-save
  • The output for legacy should either not contain any rules, or only the default chains with no rules. The threshold for auto-detection is 10 rules, so there should at least be fewer than 10 legacy rules
  • Review any legacy rules, if the rules are not added by any Kubernetes components, these custom rules should be updated to ensure they are added to the NFT backend. Avoiding these legacy rules may be related to the auto-detection switching to the legacy backend

Cause

CNI's, runtimes and kube-proxy rely on internal logic to detect the iptables mode (nft or legacy) in use on the OS. In some corner cases, this auto-detection can fail and use the incorrect backend — for example:

  • When both legacy and nft iptables binaries are installed (/usr/sbin/iptables and /usr/sbin/iptables-legacy) and may be used by different scripts, applications or incorrectly symlinked. As mentioned above, the 10 rule threshold could be met if custom rules are added to the incorrect backed
  • When the system alternates between iptables providers during OS updates or security patching
  • If the OS is heavily contended for resources the auto-detection logic may fail during startup and default to the incorrect backend

This can lead to inconsistent NAT rule management — kube-proxy writes rules to one backend (e.g., legacy) while the kernel actually uses nftables rules already in place.

Resolution

If the auto-detection has switched, as a preventative measure it is recommended to ensure consistent networking behaviour and explicitly pin kube-proxy to use the nftables backend in all nodes in the cluster, this can be accomplished by setting the environment variable IPTABLES_MODE=nft.

Steps (via Rancher UI)

  1. In the Rancher UI, go to Cluster Management → Clusters
  2. Select the affected downstream cluster
  3. Click ⋮ → Edit Config
  4. Choose Edit as YAML
  5. Locate the machineGlobalConfig section and add the following:

machineGlobalConfig:
  # existing configuration
  kube-proxy-extra-env:
    - IPTABLES_MODE=nft
6. Click Save
The configuration will roll out to each node, and the kube-proxy pods will automatically restart with the updated environment variable

Note the same change can be accomplished on standalone clusters that aren't managed by Rancher, by adding the same kube-proxy-extra-env field in the config.yaml of each node. The rke2-agent/server service will need to be restarted to apply the change

Verification

After the configuration is applied:

  1. Check that kube-proxy is running with the correct environment:
kubectl -n kube-system exec -it kube-proxy-xxxx -- env | grep IPTABLES_MODE

Expected output:

IPTABLES_MODE=nft 2. Confirm that NAT rules are populated and accurate under nftables:

kubectl exec -it -n kube-system kube-proxy-xxxx -- /usr/sbin/iptables-nft -nvL

or on the node directly

nft list ruleset | grep KUBE-