HostTailer includes Fluent-bit logs

[sebastiangaiser]

Hey, we're using a HostTailer for sending Kubernetes cluster audit logs to Opensearch. The logs are getting parsed as json and everything works like like a charm. But unfortunately, the HostTailer Fluent-bit is producing some [ info] [input:tail:tail.0] inotify_fs_remove(): inode=11024910 watch_fd=20 messages.
Is there an option to exclude/remove these logs? Currently they get send to the stderr stream and I can simply exclude them via a grep filter in FluentD but that feels a bit odd.

[csatib02]

Well I've just tested this out, and unfortunately sinceHostTailer logs to stdout adding:

annotations:
    fluentbit.io/exclude: "true"

would exclude the tailed logs themselves. (Deleting previous comments.)

[sebastiangaiser]

Indeed, you're right. Maybe I was a bit to euphoric, yesterday

[aslafy-z]

Maybe one of these way can help:

• reduce the log level to error with Log_Level error or -qq cli flag
• disable all logging altogether with Log_Level off (risky)

[aslafy-z]

I guess we may want to configure this as a default behavior at https://github.com/kube-logging/logging-operator/blob/593d7193195c117e6800cd65b7fe81b4d18719e5/pkg/sdk/extensions/api/v1alpha1/filetailer.go. Shouldn't we?

[pepov]

Makes sense to me too

[aslafy-z]

@sebastiangaiser FYI, the default log level was increased to error with #1957. You can opt out of this new behavior using the verbose config attribute. It will be part of the next release.

Edit: Published as 5.3.0

[pepov]

Currently this works as intended. With the telemetry controller the plan is to send host logs directly to the opentelemetry collector instead of writing them to stdout.