fluentbit k8s filter plugin configs

K8s filter plugin is used to add extra information in log while tailing the container logs using fluenbit. In fluentbit version 3, there are certain things to consider.

Here is an example for conf for fluentbit in k8s.

fluent-bit.conf: |
    [INPUT]
        Name tail
        Path /var/log/containers/ingress-nginx-controller-*.log
        multiline.parser docker, cri
        Tag hox.ingress-nginx-controller.*
        Mem_Buf_Limit 50MB
        Skip_Long_Lines Off
        DB /var/log/db/ingress-nginx-controller.db
    [FILTER]
        Name                kubernetes
        Match               hox.ingress-nginx-controller.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        Keep_Log            Off
        Kube_Tag_Prefix     hox.ingress-nginx-controller.var.log.containers.
        Labels              On
        Annotations         On
        Namespace_Labels    On
        Namespace_Annotations On
        Buffer_Size         0
        Use_Kubelet         false
        Kubelet_Port        10250

For example to work make sure folder Path /var/log/containers is similar to Kube_Tag_Prefix var.log.containers.

Tag hox.ingress-nginx-controller.* in INPUT and Kube_Tag_Prefix hox.ingress-nginx-controller.var.log.containers. [in FILTER], with Path /var/log/containers, makes the fluentbit extract relevant fields to make api calls on k8s api-server.

If someone wants to change the relevant fields, please make sure to change the fields accordingly.

Example: If Path /etc/log/hoxserver/hoxserver-.log then Tag will be nox.hoxserver. and Kube_Tag_Prefix will be nox.hoxserver.etc.log.hoxserver.

fluentbit uses regex to extract information about namespace, pod_name, container_name etc for parsing, similar to below:

[Example]:

[PARSER]
    Name kubernetes_container_logs_metadata
    Format regex
    Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$