Pod level DNS resolution failed after modifying the node's dns servers on /etc/resolv.conf

This document (000021861) is provided subject to the disclaimer at the end of this document.

Environment

RKE2

Situation

The newly created pods are unable to communicate with core-dns after updating the node's DNS server details in /etc/resolv.conf. The new pods fail to resolve DNS.

Resolution

Flush the dns cache in Linux node to ensure, the system or the application stop using stale dns resolution that was cached before the changes.

The following steps need to be executed.

Restart the rke2-server service if it is a control plane+worker node

systemctl restart rke2-server.service

Restart the rke2-agent service if the node is only a worker

systemctl restart rke2-agent.service

To flush the dns cache, different commands can be used, depends on the Linux distro.

If using nscd

service nscd restart or systemctl restart nscd

If using dnsmaq

systemctl restart dnsmaq

If using systemd-resolvd

systemd-resolve --flush-caches

Cause

On Linux, some dns services like systemd-resolved, nscd, dnsmaq etc. caches dns queries in memory and may not immediately pick up changes to /etc/resolv.conf unless restarted or flushed. If a dns query was previously resolved and cached, the system might continue using old IP address even after updating the /etc/resolv.conf

Status

Top Issue

Additional Information

To flush the dns cache, different commands can be used, depends on the Linux distro, and the dns flushing many not be necessary for all the Linux distributions.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.