Using the Fluentd concat filter plugin to concatenate multiline logs in rancher-logging

Article Number: 000021670

Environment

A Kubernetes cluster managed by Rancher v2.6+ with the rancher-logging chart installed

Situation

Logs of multiple lines are separated across multiple log events within Pod logs and there is a need to combine them into a single event before forwarding them to a logging solution.

Resolution

The rancher-logging operator provides a wide range of Fluentd filter plugins. The concat filter can be used to concatenate multiline logs into a single event. 

Filters are defined in a Cluster or ClusterFlow, per the example below:

spec:
  filters:
    - concat:
        flush_interval: 10
        multiline_start_regexp: /\d{4}-\d{1,2}-\d{1,2}/
        use_first_timestamp: true

Users should refer to the Fluentd concat filter plugin documentation, as well as the kube-logging documentation, to determine the appropriate concat filter parameters to concatenate their application logs.