Pod level DNS resolution failed after modifying the node's dns servers on /etc/resolv.conf

Article Number: 000021861

Environment

RKE2

Situation

The newly created pods are unable to communicate with core-dns after updating the node's DNS server details in /etc/resolv.conf. The new pods fail to resolve DNS.

Cause

On Linux, some dns services like systemd-resolved, nscd, dnsmaq etc. caches dns queries in memory and may not immediately pick up changes to /etc/resolv.conf unless restarted or flushed. If a dns query was previously resolved and cached, the system might continue using old IP address even after updating the /etc/resolv.conf

Resolution

Flush the dns cache in Linux node to ensure, the system or the application stop using stale dns resolution that was cached before the changes. 

The following steps need to be executed. 

Restart the rke2-server service if it is a control plane+worker node 

systemctl restart rke2-server.service

Restart the rke2-agent service if the node is only a worker 

systemctl restart rke2-agent.service

To flush the dns cache, different commands can be used, depends on the Linux distro. 

If using nscd

service nscd restart or systemctl restart nscd

If using dnsmaq

systemctl restart dnsmaq

If using systemd-resolvd

systemd-resolve --flush-caches