Skip to content

"log unreadable. It is excluded and would be examined next time." warning messages, for kubelet and kube-proxy, in rancher-logging-fluentd Pod logs of worker nodes

This document (000020201) is provided subject to the disclaimer at the end of this document.

Situation

Issue

In a Rancher v2.x provisioned Kubernetes cluster, with Rancher Cluster Logging configured, the rancher-logging-fluentd Pod logs on Linux worker role only nodes show warning messages of the following format:

2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kubelet_5c47838dd4af749a7a0d1c457b04a6d7b905e680157718063c8e5d9eb61268fa.log unreadable. It is excluded and would be examined next time.
2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kube-proxy_d2beb2e667eefbd6d95355082af4bc61c367fc4c220d9f1d165d15a8c8be2ab1.log unreadable. It is excluded and would be examined next time.

The output of ls /var/lib/rancher/rke/log/ on affected workers shows that these files referenced in the warning log messages are broken symlinks. In addition, docker ps output shows the container ID for the currently running kubelet and kube-proxy containers does not match the IDs in these filenames.

Pre-requisites

Root cause

These warning level messages in the rancher-logging-fluentd Pod are the result of the issue tracked in Rancher GitHub issue #22549.

The container log symlinks in /var/lib/rancher/rke/log/ for cluster component containers ( kubelet, kube-proxy, nginx-proxy) on worker nodes in Rancher launched clusters are not cleaned up when these components are re-created, i.e. due to a Kubernetes version upgrade, or other configuration update for these components.

As a result these broken symlinks persist and cause the log unreadable warning messages when the rancher-logging-fluentd Pod attempts to parse files in the /var/lib/rancher/rke/log/ directory.

This warning message itself is harmless and can be ignored.

Resolution

The request to handle automatic clean-up of these log symlinks on worker nodes, in Rancher provisioned clusters, is tracked in Rancher GitHub issue #22549.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.