Skip to content

How to test rancher-logging Flows and Outputs with local file output

This document (000021317) is provided subject to the disclaimer at the end of this document.

Environment

  • A Rancher v2.5+ managed Kubernetes clusters
  • rancher-logging installed in the cluster

Situation

When configuring rancher-logging on a cluster, it may be necessary to troubleshoot Flows and Outputs. This guide provides a way to test these without any third-party tools involved (e.g. an external logging destination such as Elasticsearch).

Resolution

  1. The first step will be to create an Output on the cluster-logging-system namespace. You can do so by exploring the cluster within the Rancher UI, navigating to Logging > Outputs in the resources menu on the left side, and then clicking Create(alternatively, you can apply the manifest via the kubectl CLI). The following example manifest will create a basic Output that you can customize later. It will send the resulting logs to the /tmp/logs/ folder on the FluentD pod (rancher-logging-root-fluentd-0) in the cluster-logging-system Namespace:
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
    name: test-output
    namespace: cattle-logging-system
spec:
    file:
       path: /tmp/logs/${tag}/%Y/%m/%d/%H.%M
       buffer:
         timekey: 1m
         timekey_wait: 10s
         timekey_use_utc: true
  1. The next step is to create a Flow that targets the pod created in step 3, using a label selector, and routes logs to the Output created in step 1. Again, this can be done from the UI by clicking on the Logging > Flows tab and clicking Create(or via kubectl):
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
     name: test-flow
     namespace: cattle-logging-system
spec:
     localOutputRefs:
    - test-output
match:
    - select:
        labels:
          logging-test: 'true'
  1. Next, a dummy pod will be deployed to generate log messages with which to test the Flow and Output. In this example the swiss-army-knife debug image created by Rancher Support is used, but you can replace this as required by your testing. You can create this Pod via the Import YAML function at the top-right of Rancher the UI or via the kubectl CLI:

``` apiVersion: v1 kind: Pod metadata: name: test-logging-pod namespace: cattle-logging-system   labels:     logging-test: "true" spec: containers: - name: sak image: rancherlabs/swiss-army-knife:latest

4. With the Output, Flow, and Pod in place, you can now test the setup by executing a shell in the test pod and sending any desired test log messages to STDOUT (before doing this, be sure that the fluentd-configcheck pod in the cattle-logging-system Namespace is in status Completed):
$ kubectl -n cattle-logging-system exec -it test-logging-pod -- /bin/bash

echo 'Hello World from Rancher Logging!' > /proc/1/fd/1

5. To view the results, you have to go to the /tmp/logs folder that will be created in the Fluentd pod (you might have to wait up to a few minutes for the log file to be populated, due to container log scraping and buffering by Fluent Bit). Inside the /tmp/logs directory will be a directory for the test Pod, and the log file will be in a child directory under the day, month and year:
6. ```
$ kubectl -n cattle-logging-system exec -it rancher-logging-root-fluentd-0 -- /bin/sh

$ ls -lrt /tmp/logs
total 4
drwxr-xr-x    3 fluent   fluent        4096 Feb 28 15:06 kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log

$ cat 15.05_0.log
2024-02-28T15:05:50+00:00       kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log   {"log":"Hello World from Rancher Logging!\n","stream":"stdout","time":"2024-02-28T15:05:50.766535534Z","kubernetes":{"pod_name":"test-logging-pod","namespace_name":"cattle-logging-system","pod_id":"1d16a19e-f565-447b-b9f0-39bdf559eaf6","labels".....
  1. You can now freely change your Output and Flow configuration to test and troubleshoot configurations, filters and parsers etc. The results will appear in the /tmp/logs/ folder of the Fluentd pod.

Additional Information

Provided you followed the steps above, you can now change the created Output and Flow to test any custom configurations that you need. You should be able to see the results right away and quickly spot any problems. Remember that you can use the following KB to guide you in troubleshooting issues: How to troubleshoot rancher-logging | Support | SUSE

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.