Sidecar without image specification causes all PostgreSQL pods to shutdown

[ovnozdrach]

• Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.14.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

It's possible to add a sidecar specification without providing any image (by mistake) for that additional container.
This leads to a cascading failure where all cluster instances are shut down sequentially during sync intervals (30 minutes) until all pods are terminated, resulting in complete PostgreSQL unavailability.

New pods cannot start due to the Kubernetes error: "create Pod ... in StatefulSet ... failed error: Pod ... is invalid: spec.containers[2].image: Required value"

The operator logs show this -

{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"performing rolling update","pkg":"cluster","time":"2025-10-20T09:10:42Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"there are 2 pods in the cluster to recreate","pkg":"cluster","time":"2025-10-20T09:10:42Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"subscribing to pod \"dev-mdb-mq-clusters/test-no-image-1\"","pkg":"cluster","time":"2025-10-20T09:10:42Z","worker":2}
...
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"unsubscribing from pod \"dev-mdb-mq-clusters/test-no-image-1\" events","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"error","msg":"could not sync statefulsets: could not recreate pods: could not recreate replica pod \"dev-mdb-mq-clusters/test-no-image-1\": pod label wait timeout","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"syncing roles","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"db ready status: true","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"closing database connection","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}

Next operator sync kills last available pod -

{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"found rolling update flag on pod \"test-no-image-0\"","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"1 / 1 pod(s) still need to be rotated","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"performing rolling update","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"there are 1 pods in the cluster to recreate","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"warning","msg":"cannot perform switch over before re-creating the pod: no replicas","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"recreating old master pod \"dev-mdb-mq-clusters/test-no-image-0\"","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"subscribing to pod \"dev-mdb-mq-clusters/test-no-image-0\"","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
...
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"unsubscribing from pod \"dev-mdb-mq-clusters/test-no-image-0\" events","pkg":"cluster","time":"2025-10-20T09:46:24Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"warning","msg":"error while syncing cluster state: could not sync statefulsets: could not recreate pods: could not recreate old master pod \"dev-mdb-mq-clusters/test-no-image-0\": pod label wait timeout","pkg":"cluster","time":"2025-10-20T09:46:24Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"error","msg":"could not sync cluster: could not sync statefulsets: could not recreate pods: could not recreate old master pod \"dev-mdb-mq-clusters/test-no-image-0\": pod label wait timeout","pkg":"controller","time":"2025-10-20T09:46:25Z","worker":2}

The operator should validate the StatefulSet configuration before triggering updates.
At minimum, the sidecar image field should be marked as required in the CRD validation schema.

[FxKu]

Sounds bad. I would expect only one pod to go down and the rotation to stop because the replaced pod does not come back. Are you sure it takes down every instance after subsequent syncs? For single-pod clusters you would be doomed, indeed.

Unfortunately, we are not validating the full sidecar spec in our CRDs. Same for init containers. I guess that could be improved.