containerTemplate for all Pod Containers

[syepes]

Summary

There should be a single way of establishing a global containerTemplate that will be applied on all the deployed Pod's and all its underlining Containers. These global containerTemplates would also be possible to overwrite by specifying the specific settings, just in case we need to change the debug level or other settings on a single vertice.

Today the containerTemplate are not applyed to all container:

numaflow/pkg/apis/numaflow/v1alpha1/container_template.go

Line 94 in 2b21145

if containers[i].Name == CtrMain || containers[i].Name == CtrInit || containers[i].Name == CtrInitSideInputs {

It could also be possible to add a setting like containerTemplateApplyTo, to not change the current behavior an let the user overwrite the default list of numa, int and init-side-inputs

apiVersion: numaflow.numaproj.io/v1alpha1
kind: Pipeline
metadata:
  name: processor
  namespace: numaflow
spec:
  templates:
    vertex:
      containerTemplateApplyTo: ["numa","transformer"..]
      containerTemplate:
        env:
          - name: NUMAFLOW_DEBUG
            value: "false"
          - name: RUST_LOG
            value: "warn"

Use Cases

This should facilitate the deployment and large scale management by simplefying the number is specific settings.

Related

#3090

[whynowy]

containerTemplate is only intended to apply to numa container, the code does not work as you understand.

I don't think it makes sense to apply the same template settings to all the containers, across all the vertices in a pipeline. The proposal of adding containerTemplateApplyTo: ["numa","transformer"..] is also leaky.

[syepes]

ok then what would the the best solution to set for example LOG_LEVEL=Xx (Just as example) a cross all deployed containers?

[whynowy]

ok then what would the the best solution to set for example LOG_LEVEL=Xx (Just as example) a cross all deployed containers?

I don't think we support LOG_LEVEL for any built in containers. I assume it's rare that there are common template settings across built in containers and user defined containers?

[syepes]

It was just as an example but it could be RUST_LOG to set the global logging level

[whynowy]

It was just as an example but it could be RUST_LOG to set the global logging level

If you do have the requirements to set global env vars, considering it's not common, you could also use kustomize to manage them.

[magelisk]

I think would be very useful. In practice, a lot of things like image pull secrets or serviceAccountName, and securityContext will be identical accross all pods. So a lot of our pipelines have a lot of boilerplate to set these.

Similarly, there are often more business logic level things that tend to be the same across all vertices in a pipeline - endpoints and auth info for external services, volumes+mounts for data access. Having a since place like the existing templates would be quite useful.

Internally to my own work, we wrote our own operator to wrap Pipelines+custom logic to basically do this behavior for me and have it submit a filled-in Pipeline. I'd love to have it more built in for quality of life purposes. I solve the 'what to apply it to' by using annotations as 'selectors' for the cases where I wanted thing to be not-as-global.

[whynowy]

I think would be very useful. In practice, a lot of things like image pull secrets or serviceAccountName, and securityContext will be identical accross all pods. So a lot of our pipelines have a lot of boilerplate to set these.

Similarly, there are often more business logic level things that tend to be the same across all vertices in a pipeline - endpoints and auth info for external services, volumes+mounts for data access. Having a since place like the existing templates would be quite useful.

Internally to my own work, we wrote our own operator to wrap Pipelines+custom logic to basically do this behavior for me and have it submit a filled-in Pipeline. I'd love to have it more built in for quality of life purposes. I solve the 'what to apply it to' by using annotations as 'selectors' for the cases where I wanted thing to be not-as-global.

@magelisk - thanks for sharing your thoughts!

Just want to clarify - For image pull secrets, serviceAccountName and securityContext, they are pod level configuration, which are already available in spec.templates.vertex.xxx (unless you meant the container level securityContext).

For volumes, it's not available today, if enabling it, it should go to existing spec.templates.vertex.volumes. But it has to be used together with volumeMount (which needs to be specified at container level), I'm kind of struggling how to make the configuration generic while elegant. Could you please share more detail of your approach of using annotations?

[magelisk]

I think I actually missed that the templates.vertex had imagepullsecrets, serviceaccount and such. So that actually is helpful to realize and simplifies things (a bit)

The main approach we've taken is to have a CRD that will just take the numaflow Pipeline fields as-is. We then added a new section that is just 'annotatedTemplates'. Basically it's the below structure. We did originally just put these helpers into various helm helper functions so that adding those 10 lines of env vars was just a a single include, but we ultimately found it just became a bit annoying to remember each of these "boiler plate items" whenever we added a new vertex. Things like volumes became more annoying because they need to cross two sections. Generally each vertex definition wound up being a third this boilerplate, so having these single line ways of configuring items just slimmed down the actual vertices section quite a bit and made tweaking fields just a lot easier.

Our custom operator simply matches annotations, and merges these directly into the vertex definitions and submits a Pipeline and lets numaflow handle it from there.

annotatedTemplates:
# For those that need to connect to shared message broker
- selector:
    # Can do multiple selector types if necessary
    myapp/broker: central-activemq
  containerTemplate:
    env:
    - name: ACTIVEMQ_ENDPOINT
      value: amqp-svc.amqp.svc.cluster.local
    - name: ACTIVEMQ_USERNAME
      valueFrom:
        secretKeyRef:
          name: activemq-secret
          key: username
    - name: ACTIVEMQ_PASSWORD
      valueFrom:
        secretKeyRef:
          name: activemq-secret
          key: password

# For those that need to access 
- selector:
     myapp/broker: file-access
  volumes:
  - name: data-vol
      persistentVolumeClaim:
        claimName: my-data-vol
  containerTemplate:
    volumeMounts:
        - name: my-data-vol
          mountPath: /mnt/data
...
templates:
- name: mysource
  metadata:
    annotations:
      # Uses both annotations - reads messages, and then can verify files exist on disk
      myapp/broker: central-activemq
      myapp/broker: file-access
 udsource:
  ...
- name: myudf:
  metadata:
    annotations:
      myapp/broker: file-access

- name: mysink
  metadata:
    annotations:
      myapp/broker: central-activemq
  udsink:
    ...