Issue with DNS while using NodeLocalDNS and network policy in RKE2

Article Number: 000022141

Environment

Rancher, RKE2, NetworkPolicy

Situation

Network policies that worked in RKE2 are not functioning correctly after NodeLocalDNS is enabled. Specifically, DNS resolution fails for pods even when network policies are configured to allow DNS traffic on port 53. The issue occurs despite explicitly allowing such traffic in the network policy.

For example, the following egress rule was present in the network policy for allowing outgoing connections to coredns

  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP

Cause

The issue occurred due to the absence of proper NetworkPolicies for NodeLocalDNS. When NodeLocalDNS is enabled, DNS queries are handled locally on each node, and the network policies must be configured to allow traffic to the NodeLocalDNS IP.

Resolution

Using podSelector( k8s-app: node-local-dns) to allow egress in policy will not work because NodeLocalDNS pods run in hostnetwork and bind 10.43.0.10 on the host, traffic destined for 10.43.0.10 (169.254.20.10) directly reaches those IPs and is not DNATed to the NodeLocalDNS (host network IP) or CoreDNS pod IPs. So essentially, after NodeLocalDNS is installed, the IP 10.43.0.10 is no longer the Kubernetes service IP. 

To resolve the DNS resolution issues with network policies in RKE2 clusters, modify the network policies to include the following egress rules:

- to:
    - ipBlock:
        cidr: 10.43.0.10/32
    ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP