Basic troubleshooting steps for the Logging Operator

This document (000020946) is provided subject to the disclaimer at the end of this document.

Environment

Rancher 2.6.x

Rancher 2.7.x

Rancher 2.8.x

Logging Operator V2

Situation

Logs may not be sent from the operator to the chosen destination (ELK, Splunk, etc)

Resolution

1. On the Rancher menu, go to Workload, Pods.

Check the rancher-logging-root-fluentd-configcheck pod. Its status should be `Completed`.

If it is in a different state, check its logs to understand the error on your configuration.

You may do so by clicking on the three dots menu and 'View Logs'.

2. Check the fluentd logs.

a) When using logging-operator version +up3.17.10 and earlier, they are not written to standard output. Executing on the container rancher-logging-root-fluentd-0, navigate to /fluentd/out, and check the log file there:

$ kubectl -n cattle-logging-system exec -it rancher-logging-root-fluentd-0 -- sh
$ cat /fluentd/out/log

b) When using logging-operator version +up4.4.0 and later, the fluentd logs are written to standard output, in the container versus a file, i.e. they are written to the fluentd Pod logs (and excluded from collection by the "fluentbit.io/exclude: 'true'" annotation on the fluentd Pod). To check the fluentd logs you can therefore check the Pod logs:

$ kubectl logs rancher-logging-root-fluentd-0 -n cattle-logging-system

If there are any errors, you can try to fix them or open a case if you need more help.

3. If there are no errors in the two first steps, a simple way to check if the logs are being scrapped is to send them to a file. An output YAML would look like this:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: fileoutput
  namespace: default
spec:
  file:
    path: /tmp/${tag}

Note the ${tag} on the path, required by Fluentd.

In this case, the logs will be available on the /tmp folder of the Pod rancher-logging-root-fluentd-0

If logs are not written to the /tmp folder, open a case with us to help you.

4. If logs are flowing to the file, you need to check your Outputs.

Tweaking the buffer values might be a solution.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.