Allow disabling and configuring the add_x_metadata default global processors

[gizas]

Edited by @cmacknz to generalize to all add_x_metadata processors and include the suggested approach from #4670 (comment)

Describe the enhancement:

The default set of processors for each Beat, when they are started by agent, is defined in the code and is not configurable. Here is the definition for Metricbeat. Same default config stands for rest of beats.

For example the add_kubernetes_metadata processor is used to enrich kubernetes events with extra metadata fields. This results in the number of k8s API calls being proportional to the number of inputs in the agent policy. In this case the same functionality is also present through enrichers and start by default in code of beats as described here-see enrichers.

This enhancement will allow disabling or configuring any of the default beat global processors. This will be done by defining a new type of provider , an event_data provider, which controls the behavior of the event metadata processors added by each beat.

We will additionally expose the entire contents of the providers section of the agent policy in Fleet to allow configuring this. Here is a sample of the intended configuration:

providers:
     event_metadata:
         # The default value would be true to avoid a breaking change when they aren't specified.
         host: # controls add_host_metadata
             enabled: true
         kubernetes: # controls add_kubernetes_metadata
             enabled: false
         cloud: # controls add_cloud_metadata
             enabled: true
         docker: # controls add_docker_metadata 
             enabled: false

Describe a specific use case for the enhancement or feature:

All the elastic-agent Kubernetes deployments experience the default enablement of processor in the background, which can be "expensive" (as far as extra memory and number of API calls) that the agent has to do, especially in big Kubernetes deployments

Related Issues:

• Investigate deployment and cronjob enrichement integrations#8176
• https://github.com/elastic/sdh-beats/issues/3863

Definition of Done:

• Allow disabling the add_kubernetes_metadata processor for all Beats run by Elastic Agent.
• Provide tests that showcase the same functionality and enrichment with/without processor enabled
• Demonstrate the event metadata processors can be disabled from Fleet.
• Document the new functionality

[joshdover]

I'm still unfamiliar with how enrichers work, so bear with me here. As far as I understand, they only apply to the metricsets that use them directly and are not configurable. I think we still need an option for users to be able to easily add the same metadata as before with any input type.

The problem we're trying to solve here is especially problematic on Elastic Agent, where we start a new Beat instance for each unique input type. Each of these Beats will run their own instance of add_kubernetes_metadata in addition to the Agent's built-in kubernetes provider. Each of these essentially makes the same K8s API requests which puts a lot of load on the K8s API. This problems scales horizontally both with number of K8s nodes in the cluster and with the number of integrations added to the policy.

I was thinking that for Agent users we could add the same exact properties that the processor gives, but isntead through the Kubernetes provider by updating integrations and standalone config examples to populate add_fields processors. An example would be like:

inputs:
- id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
  type: filestream
  streams:
    - id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
      data_stream:
        dataset: kubernetes.container_logs
        type: logs
      prospector.scanner.symlinks: true
      parsers:
        - container: ~
      paths:
        - /var/log/containers/*${kubernetes.container.id}.log

      # 👇 Here is what is new
      processors:
        - add_fields:
            target: kubernetes
            fields:
              pod.name: ${kubernetes.pod.name}
              pod.id: ${kubernetes.pod.id}
              namespace: ${kubernetes.namespace}
              labels: ${kubernetes.labels}

Ideally this is actually something that Agent could pull off automatically without having to change integrations and configurations. This would help avoid a breaking change. I see a couple options to solve this:

• Change how the add_kubernetes_metadata processor works when running under Agent to somehow fetch this info from the agent core process instead of K8s API directly. This may have some of the same memory issues though, not sure. This might set us up to more easily have a cluster-level agent that provides this info for all agents later.
• Disable the add_kubernetes_metadata by default when running Beats under Agent, but have Agent insert this add_fields processor block automatically when the provider detects it's on K8s.

[gizas]

they only apply to the metricsets that use them directly and are not configurable

Small correction on this is that yes they start by default but are configurable based on add_resource_metadata block . See below example from managed agent (same from standalone as well):

So at the moment you can only disable namespace enrichement (this is the main one that gives us labels and annotations), node enrichment (that adds node and hostname), cronjob (that adds in pods ony cronjob name) and deployments (that adds in pods deployment name)
See all options here in add_resource_metadata block

So what you describe above for the processor can still happen in the enricher and be configured with the options you specify. We maybe need to specify extra user scenarios but it is ok.

The disablement of the processor needs to happen as both start and this is not correct.

Agent to somehow fetch this info from the agent core process instead of K8s API directly.

I understand your point. To see if agent has this info already and not to ask again from them. We will need to see the code and if the cache of agent can be utlised from the integration side.
cc @ChrsMark and @MichaelKatsoulis that have worked in the past in this part of the code to see if they remember anything more

[ChrsMark]

Hey folks, I'm not sure if the components are completely distinguished here. So here is a quick list of the related components:

1) k8s provider of Elastic Agent

• Only configurable in standalone mode and enabled by default.
• Has no interaction with any other components since it only discovers Pods/Nodes/Services and emit mappings for the compo-sable controller.
• code ref: https://github.com/elastic/elastic-agent/tree/main/internal/pkg/composable/providers/kubernetes

2) k8s enrichers of Metricbeat

• Only configurable through the metricset specific configurations. Hence it's configurable through the Agent's policy/input.
• Again has no interaction with any other components
• code ref: https://github.com/elastic/beats/blob/a55ae5857c0d80e0834c9f3e98fd1b833675f12c/metricbeat/module/kubernetes/state_pod/state_pod.go#L95 (one example metricset that enables the enricher)

3) add_kubernetes_metadata processor

⚠️ This is what the issue is about ⚠️

This processor is started by default by all Beats because the default Beat's configuration includes it: https://github.com/elastic/beats/blob/b8a241b0703d252c3ff1ba9b417aea585f811da1/metricbeat/metricbeat.yml#L127 (same for other Beats like Filebeat etc).
I'm not sure how https://github.com/elastic/beats/blob/main/x-pack/metricbeat/cmd/root.go#L74-L78 interacts with this tbh.

From my perspective in Beats I should be able to remove the add_kubernetes_metadata: ~ processor from the Beat's default config and disable the processor. Isn't that correct @gizas ?
But Elastic Agent does not provide that option which leads to the following issue ---->

In the past we had a related issue at elastic/beats#27216 when Elastic Agent was starting (still does :)) Filebeat with the default config file (filebeat.yml) and as a result we had the processor (add_kubernetes_metadata) to produce tons of warnings because the defaults were not valid.
That's why we introduced elastic/beats#27689 to skip the enrichment whenever we detect that the k8s metadata are already present on the event (from the k8s autodiscovery provider for example).

However this only makes the processor to skip the enrichment part not the "watching" part since the processor starts properly and makes all the required API calls so as to populate the caches. This is why we need to completely disable it if we don't need it specifically.

⚠️ However the original issue is still there and is filed at #90. So I think this is why we are still hitting this @joshdover @gizas and this what we should focus on since it's a generic problem of how Elastic Agent starts and configures the underlying Beats. ⚠️

Note that the default enablement of the processor in Beats was a "nice-to-have" but in Agent it's completely useless since other components already do the job and eventually if anyone needs the processor they can add it into the policy manually.

I hope that's clear why it's super crucial to either disable the processor by default on Beats level or provide the option to do it from Elastic Agent through #90.
All the information about the problematic existence of the add_kubernetes_metadata processor can be found in the linked issues above so please have a look but happy to elaborate more on this if needed.

ps1: Reusing metadata is a valid point in general but not applicable here so let's try to keep the focus on the existent issue :).

[gizas]

Thanks @ChrsMark , so trying to summarise your info and Josh's comments:

• Yes this issue is to remove add_kubernetes_metadata processor to run by default in beats and agents. Also to test if this leaves any issue comparing to what users see today
• @joshdover (correct me here) was trying to suggest more granural disablement of existing metadata if makes sense
• Suggestion whether we could reuse metadata from agents provider is not feasible in metricbeat as the provider is not used at all .

[ChrsMark]

@gizas 👍🏽

I had filed an issue regarding the memory utilization improvement by using common shared caches etc: elastic/beats#35591. But as I said this is out of scope for the current problem.

So yes let's try to see if we can completely disable the add_kubernetes_metadata processor on Beat's startup (ideally if we detect that we run under Elastic Agent.
We could either introduce an ENV var check at https://github.com/MichaelKatsoulis/beats/blob/3955e717adb8a1204464de756fcba91d7df89d75/libbeat/processors/add_kubernetes_metadata/kubernetes.go#L142 so as to skip the whole initialization if the specific variable is present. Sth similar to what we did for add_host_metadata processor @gizas .

[MichaelKatsoulis]

+1 to what @ChrsMark mentioned.
Also regarding @joshdover comment #4670, in case of container_logs integration in managed agent or standalone agent, the use of dynamic variables (/var/log/containers/*${kubernetes.container.id}.log) starts the kubernetes provider.
The provider enrichers the log events with metadata. So adding the add_fields processor won't add anything new.

[MichaelKatsoulis]

More on the ENV var approach , a variable worth investigating is BEAT_SETUID_AS which under agent is set to elastic-agent by the dockerfile. When beat runs autonomously the var is unset as far as I have seen until now.