How to test rancher-logging Flows and Outputs with local file output

Article Number: 000021317

Environment

A Rancher v2.5+ managed Kubernetes clusters
rancher-logging installed in the cluster

Situation

When configuring rancher-logging on a cluster, it may be necessary to troubleshoot Flows and Outputs. This guide provides a way to test these without any third-party tools involved (e.g. an external logging destination such as Elasticsearch).

Resolution

The first step will be to create an Output on the cluster-logging-system namespace. You can do so by exploring the cluster within the Rancher UI, navigating to Logging > Outputs in the resources menu on the left side, and then clicking Create (alternatively, you can apply the manifest via the kubectl CLI). The following example manifest will create a basic Output that you can customize later. It will send the resulting logs to the /tmp/logs/ folder on the FluentD pod (rancher-logging-root-fluentd-0) in the cluster-logging-system Namespace:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
 name: test-output
 namespace: cattle-logging-system
spec:
 file:
    path: /tmp/logs/${tag}/%Y/%m/%d/%H.%M
    buffer:
      timekey: 1m
      timekey_wait: 10s
      timekey_use_utc: true

2. The next step is to create a Flow that targets the pod created in step 3, using a label selector, and routes logs to the Output created in step 1. Again, this can be done from the UI by clicking on the Logging > Flows tab and clicking Create (or via kubectl):

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: test-flow
  namespace: cattle-logging-system
spec:
  localOutputRefs:
    - test-output
  match:
    - select:
        labels:
          logging-test: 'true'

3. Next, a dummy pod will be deployed to generate log messages with which to test the Flow and Output. In this example the swiss-army-knife debug image created by Rancher Support is used, but you can replace this as required by your testing. You can create this Pod via the Import YAML function at the top-right of Rancher the UI or via the kubectl CLI:

apiVersion: v1
kind: Pod
metadata:
  name: test-logging-pod
  namespace: cattle-logging-system
  labels:
    logging-test: "true"
spec:
  containers:
  - name: sak
    image: rancherlabs/swiss-army-knife:latest

4. With the Output, Flow, and Pod in place, you can now test the setup by executing a shell in the test pod and sending any desired test log messages to STDOUT (before doing this, be sure that the fluentd-configcheck pod in the cattle-logging-system Namespace is in status Completed):

$ kubectl -n cattle-logging-system exec -it test-logging-pod -- /bin/bash
# echo 'Hello World from Rancher Logging!' > /proc/1/fd/1

5. To view the results, you have to go to the /tmp/logs folder that will be created in the Fluentd pod (you might have to wait up to a few minutes for the log file to be populated, due to container log scraping and buffering by Fluent Bit). Inside the /tmp/logs directory will be a directory for the test Pod, and the log file will be in a child directory under the day, month and year: 6. ``` $ kubectl -n cattle-logging-system exec -it rancher-logging-root-fluentd-0 -- /bin/sh

$ ls -lrt /tmp/logs total 4 drwxr-xr-x 3 fluent fluent 4096 Feb 28 15:06 kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log

$ cat 15.05_0.log 2024-02-28T15:05:50+00:00 kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log {"log":"Hello World from Rancher Logging!\n","stream":"stdout","time":"2024-02-28T15:05:50.766535534Z","kubernetes":{"pod_name":"test-logging-pod","namespace_name":"cattle-logging-system","pod_id":"1d16a19e-f565-447b-b9f0-39bdf559eaf6","labels"..... ``` 7. You can now freely change your Output and Flow configuration to test and troubleshoot configurations, filters and parsers etc. The results will appear in the /tmp/logs/ folder of the Fluentd pod.