Retention configuration is not respected

[mboronin]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

We see a strange behaviour on 3.7.1, where retention config is not respected.
We have a clear indication of workflow removals, but the retention amounts do not match the configuration, even thought logs indicate thet the newest configmap is used

Version(s)

v3.7.1

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

time="2025-12-15T10:05:37.988Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-s4d6s for delete due to max rention(50 workflows)"
time="2025-12-15T13:33:27.424Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-m6x27 for delete due to max rention(50 workflows)"
time="2025-12-15T16:11:46.595Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-fhj56 for delete due to max rention(50 workflows)"
time="2025-12-15T16:42:35.977Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-gzk8j for delete due to max rention(50 workflows)"
time="2025-12-15T18:39:06.072Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-6f5zd for delete due to max rention(50 workflows)"
time="2025-12-16T06:13:31.442Z" level=info msg="Queueing Succeeded workflow sre-india/amf-traffic-shaping-dag-q27xv for delete due to max rention(50 workflows)"
time="2025-12-16T08:47:43.066Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-8r8rm for delete due to max rention(50 workflows)"
time="2025-12-16T10:32:51.767Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete due to max rention(50 workflows)"


However, that does not comply with the configuration that we have 

Data
====
config:
----
parallelism: 10
resourceRateLimit:
  burst: 5
  limit: 20
metricsConfig:
  enabled: true
  path: /metrics
  port: 9090
  ignoreErrors: false
  secure: false
workflowDefaults:
  metadata:
  spec:
    podMetadata:
    serviceAccountName: argo-workflows-sa
    ttlStrategy:
      secondsAfterCompletion: 2592000
      secondsAfterFailure: 2592000
      secondsAfterSuccess: 2592000
sso:
  issuer: <issuer>
  clientId:
    name: argo-workflows-okta-client-credentials
    key: client-id
  clientSecret:
    name: argo-workflows-okta-client-credentials
    key: client-secret
  redirectUrl: <url>
  rbac:
    enabled: true
  scopes:
    - openid
    - profile
    - email
  sessionExpiry: 12h
  customGroupClaimName: gt_groups
retentionPolicy:
  completed: 200
  errored: 200
  failed: 200
nodeEvents:
  enabled: true
workflowEvents:
  enabled: true


In the same time we have logs confirming that configuration is successfully fetched from the configmap

time="2025-12-16T13:30:08.082Z" level=info msg="Received Workflow Controller config map argo-workflows/argo-workflows-workflow-controller-configmap update"
time="2025-12-16T13:30:08.109Z" level=info msg="Configuration:\nartifactRepository: {}\ninitialDelay: 0s\nmetricsConfig:\n  enabled: true\n  path: /metrics\n  port: 9090\n  secure: false\nnodeEvents:\n  enabled: true\nparallelism: 10\npodSpecLogStrategy: {}\nresourceRateLimit:\n  burst: 5\n  limit: 20\nretentionPolicy:\n  completed: 200\n  errored: 200\n  failed: 200\nsso:\n  clientId:\n    key: client-id\n    name: argo-workflows-okta-client-credentials\n  clientSecret:\n    key: client-secret\n    name: argo-workflows-okta-client-credentials\n  customGroupClaimName: gt_groups\n  issuer: https://sso.wbd.com/oauth2/aus1di7v8x4SIaRwj0x8\n  rbac:\n    enabled: true\n  redirectUrl: https://argo-workflows.prd.idp.discomax.com/oauth2/callback\n  scopes:\n  - openid\n  - profile\n  - email\n  sessionExpiry: 12h0m0s\ntelemetryConfig: {}\nworkflowDefaults:\n  metadata:\n    creationTimestamp: null\n    labels:\n      omd_business_service: boltcloud\n  spec:\n    arguments: {}\n    podMetadata:\n      labels:\n        omd_business_service: boltcloud\n    serviceAccountName: argo-workflows-sa\n    ttlStrategy:\n      secondsAfterCompletion: 2592000\n      secondsAfterFailure: 2592000\n      secondsAfterSuccess: 2592000\n  status:\n    finishedAt: null\n    startedAt: null\nworkflowEvents:\n  enabled: true\n"

Logs from the workflow controller

kubectl logs deploy/argo-workflows-workflow-controller | grep test-workflows/create-new-svc-v3-hkxd4
Found 2 pods, using pod/argo-workflows-workflow-controller-7d4c8b7478-58kdt
time="2025-12-11T20:46:40.415Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 638h23m32s due to TTL"
time="2025-12-11T21:06:40.423Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 638h3m32s due to TTL"
time="2025-12-11T21:26:40.453Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 637h43m32s due to TTL"
time="2025-12-11T21:46:40.508Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 637h23m32s due to TTL"
time="2025-12-11T22:06:40.389Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 637h3m32s due to TTL"
time="2025-12-11T22:26:40.478Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 636h43m32s due to TTL"
time="2025-12-11T22:46:40.497Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 636h23m32s due to TTL"
time="2025-12-11T23:06:40.419Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 636h3m32s due to TTL"
time="2025-12-11T23:26:40.461Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 635h43m32s due to TTL"
time="2025-12-11T23:46:40.510Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 635h23m32s due to TTL"
time="2025-12-12T00:06:40.482Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 635h3m32s due to TTL"
time="2025-12-12T00:26:40.509Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 634h43m32s due to TTL"
time="2025-12-12T00:46:40.502Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 634h23m32s due to TTL"
time="2025-12-12T01:06:40.497Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 634h3m32s due to TTL"
time="2025-12-12T01:26:40.502Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 633h43m32s due to TTL"
time="2025-12-12T01:46:40.465Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 633h23m32s due to TTL"
time="2025-12-12T02:06:40.444Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 633h3m32s due to TTL"
time="2025-12-12T02:26:40.513Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 632h43m32s due to TTL"
time="2025-12-12T02:46:40.505Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 632h23m32s due to TTL"
time="2025-12-12T03:06:40.455Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 632h3m32s due to TTL"
time="2025-12-12T03:26:40.405Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 631h43m32s due to TTL"
time="2025-12-12T03:46:40.515Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 631h23m32s due to TTL"
time="2025-12-12T04:06:40.407Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 631h3m32s due to TTL"
time="2025-12-12T04:26:40.502Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 630h43m32s due to TTL"
time="2025-12-12T04:46:40.530Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 630h23m32s due to TTL"
time="2025-12-12T05:06:40.420Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 630h3m32s due to TTL"
time="2025-12-12T05:26:40.468Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 629h43m32s due to TTL"
time="2025-12-12T05:46:40.468Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 629h23m32s due to TTL"
time="2025-12-12T06:06:40.413Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 629h3m32s due to TTL"
time="2025-12-12T06:26:40.427Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 628h43m32s due to TTL"
time="2025-12-12T06:46:40.451Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 628h23m32s due to TTL"
time="2025-12-12T07:06:40.415Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 628h3m32s due to TTL"
time="2025-12-12T07:26:40.447Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 627h43m32s due to TTL"
time="2025-12-12T07:46:40.417Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 627h23m32s due to TTL"
time="2025-12-12T08:06:40.417Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 627h3m32s due to TTL"
time="2025-12-12T08:26:40.532Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 626h43m32s due to TTL"
time="2025-12-12T08:46:40.416Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 626h23m32s due to TTL"
time="2025-12-12T09:06:40.424Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 626h3m32s due to TTL"
time="2025-12-12T09:26:40.504Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 625h43m32s due to TTL"
time="2025-12-12T09:46:40.503Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 625h23m32s due to TTL"
time="2025-12-12T10:06:40.589Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 625h3m32s due to TTL"
time="2025-12-12T10:26:40.420Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 624h43m32s due to TTL"
time="2025-12-12T10:46:40.513Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 624h23m32s due to TTL"
time="2025-12-12T11:06:40.423Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 624h3m32s due to TTL"
time="2025-12-12T11:26:40.416Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 623h43m32s due to TTL"
time="2025-12-12T11:46:40.449Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 623h23m32s due to TTL"
time="2025-12-12T12:06:40.487Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 623h3m32s due to TTL"
time="2025-12-12T12:26:40.526Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 622h43m32s due to TTL"
time="2025-12-12T12:46:40.450Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 622h23m32s due to TTL"
time="2025-12-12T13:06:40.435Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 622h3m32s due to TTL"
time="2025-12-12T13:26:40.523Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 621h43m32s due to TTL"
time="2025-12-12T13:46:40.512Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 621h23m32s due to TTL"
time="2025-12-12T14:06:40.484Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 621h3m32s due to TTL"
time="2025-12-12T14:26:40.462Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 620h43m32s due to TTL"
time="2025-12-12T14:46:40.470Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 620h23m32s due to TTL"
time="2025-12-12T15:06:40.506Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 620h3m32s due to TTL"
time="2025-12-12T15:26:40.475Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 619h43m32s due to TTL"
time="2025-12-12T15:46:40.485Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 619h23m32s due to TTL"
time="2025-12-12T16:06:40.475Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 619h3m32s due to TTL"
time="2025-12-12T16:26:40.449Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 618h43m32s due to TTL"
time="2025-12-12T16:46:40.477Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 618h23m32s due to TTL"
time="2025-12-12T17:06:40.429Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 618h3m32s due to TTL"
time="2025-12-12T17:26:40.439Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 617h43m32s due to TTL"
time="2025-12-12T17:46:40.494Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 617h23m32s due to TTL"
time="2025-12-12T18:06:40.445Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 617h3m32s due to TTL"
time="2025-12-12T18:26:40.550Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 616h43m32s due to TTL"
time="2025-12-12T18:46:40.442Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 616h23m32s due to TTL"
time="2025-12-12T19:06:40.541Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 616h3m32s due to TTL"
time="2025-12-12T19:26:40.476Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 615h43m32s due to TTL"
time="2025-12-12T19:46:40.515Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 615h23m32s due to TTL"
time="2025-12-12T20:06:40.554Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 615h3m32s due to TTL"
time="2025-12-12T20:26:40.488Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 614h43m32s due to TTL"
time="2025-12-12T20:46:40.522Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 614h23m32s due to TTL"
time="2025-12-12T21:06:40.458Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 614h3m32s due to TTL"
time="2025-12-12T21:26:40.518Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 613h43m32s due to TTL"
time="2025-12-12T21:46:40.484Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 613h23m32s due to TTL"
time="2025-12-12T22:06:40.499Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 613h3m32s due to TTL"
time="2025-12-12T22:26:40.461Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 612h43m32s due to TTL"
time="2025-12-12T22:46:40.531Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 612h23m32s due to TTL"
time="2025-12-12T23:06:40.573Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 612h3m32s due to TTL"
time="2025-12-12T23:26:40.454Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 611h43m32s due to TTL"
time="2025-12-12T23:46:40.455Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 611h23m32s due to TTL"
time="2025-12-13T00:06:40.504Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 611h3m32s due to TTL"
time="2025-12-13T00:26:40.447Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 610h43m32s due to TTL"
time="2025-12-13T00:46:40.539Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 610h23m32s due to TTL"
time="2025-12-13T01:06:40.455Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 610h3m32s due to TTL"
time="2025-12-13T01:26:40.468Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 609h43m32s due to TTL"
time="2025-12-13T01:46:40.549Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 609h23m32s due to TTL"
time="2025-12-13T02:06:40.496Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 609h3m32s due to TTL"
time="2025-12-13T02:26:40.459Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 608h43m32s due to TTL"
time="2025-12-13T02:46:40.460Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 608h23m32s due to TTL"
time="2025-12-13T03:06:40.537Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 608h3m32s due to TTL"
time="2025-12-13T03:26:40.544Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 607h43m32s due to TTL"
time="2025-12-13T03:46:40.463Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 607h23m32s due to TTL"
time="2025-12-13T04:06:40.571Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 607h3m32s due to TTL"
time="2025-12-13T04:26:40.476Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 606h43m32s due to TTL"
time="2025-12-13T04:46:40.532Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 606h23m32s due to TTL"
time="2025-12-13T05:06:40.478Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 606h3m32s due to TTL"
time="2025-12-13T05:26:40.483Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 605h43m32s due to TTL"
time="2025-12-13T05:46:40.566Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 605h23m32s due to TTL"
time="2025-12-13T06:06:40.531Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 605h3m32s due to TTL"
time="2025-12-13T06:26:40.595Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 604h43m32s due to TTL"
time="2025-12-13T06:46:40.520Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 604h23m32s due to TTL"
time="2025-12-13T07:06:40.506Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 604h3m32s due to TTL"
time="2025-12-13T07:26:40.485Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 603h43m32s due to TTL"
time="2025-12-13T07:46:40.527Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 603h23m32s due to TTL"
time="2025-12-13T08:06:40.547Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 603h3m32s due to TTL"
time="2025-12-13T08:26:40.477Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 602h43m32s due to TTL"
time="2025-12-13T08:46:40.502Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 602h23m32s due to TTL"
time="2025-12-13T09:06:40.602Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 602h3m32s due to TTL"
time="2025-12-13T09:26:40.587Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 601h43m32s due to TTL"
time="2025-12-13T09:46:40.497Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 601h23m32s due to TTL"
time="2025-12-13T10:06:40.494Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 601h3m32s due to TTL"
time="2025-12-13T10:26:40.561Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 600h43m32s due to TTL"
time="2025-12-13T10:46:40.506Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 600h23m32s due to TTL"
time="2025-12-13T11:06:40.507Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 600h3m32s due to TTL"
time="2025-12-13T11:26:40.571Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 599h43m32s due to TTL"
time="2025-12-13T11:46:40.486Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 599h23m32s due to TTL"
time="2025-12-13T12:06:40.527Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 599h3m32s due to TTL"
time="2025-12-13T12:26:40.509Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 598h43m32s due to TTL"
time="2025-12-13T12:46:40.567Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 598h23m32s due to TTL"
time="2025-12-13T13:06:40.498Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 598h3m32s due to TTL"
time="2025-12-13T13:26:40.536Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 597h43m32s due to TTL"
time="2025-12-13T13:46:40.537Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 597h23m32s due to TTL"
time="2025-12-13T14:06:40.599Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 597h3m32s due to TTL"
time="2025-12-13T14:26:40.501Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 596h43m32s due to TTL"
time="2025-12-13T14:46:40.526Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 596h23m32s due to TTL"
time="2025-12-13T15:06:40.575Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 596h3m32s due to TTL"
time="2025-12-13T15:26:40.503Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 595h43m32s due to TTL"
time="2025-12-13T15:46:40.565Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 595h23m32s due to TTL"
time="2025-12-13T16:06:40.549Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 595h3m32s due to TTL"
time="2025-12-13T16:26:40.566Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 594h43m32s due to TTL"
time="2025-12-13T16:46:40.590Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 594h23m32s due to TTL"
time="2025-12-13T17:06:40.546Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 594h3m32s due to TTL"
time="2025-12-13T17:26:40.563Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 593h43m32s due to TTL"
time="2025-12-13T17:46:40.570Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 593h23m32s due to TTL"
time="2025-12-13T18:06:40.569Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 593h3m32s due to TTL"
time="2025-12-13T18:26:40.553Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 592h43m32s due to TTL"
time="2025-12-13T18:46:40.515Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 592h23m32s due to TTL"
time="2025-12-13T19:06:40.564Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 592h3m32s due to TTL"
time="2025-12-13T19:26:40.615Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 591h43m32s due to TTL"
time="2025-12-13T19:46:40.508Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 591h23m32s due to TTL"
time="2025-12-13T20:06:40.516Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 591h3m32s due to TTL"
time="2025-12-13T20:26:40.519Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 590h43m32s due to TTL"
time="2025-12-13T20:46:40.515Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 590h23m32s due to TTL"
time="2025-12-13T21:06:40.582Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 590h3m32s due to TTL"
time="2025-12-13T21:26:40.560Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 589h43m32s due to TTL"
time="2025-12-13T21:46:40.570Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 589h23m32s due to TTL"
time="2025-12-13T22:06:40.602Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 589h3m32s due to TTL"
time="2025-12-13T22:26:40.570Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 588h43m32s due to TTL"
time="2025-12-13T22:46:40.605Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 588h23m32s due to TTL"
time="2025-12-13T23:06:40.520Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 588h3m32s due to TTL"
time="2025-12-13T23:26:40.592Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 587h43m32s due to TTL"
time="2025-12-13T23:46:40.626Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 587h23m32s due to TTL"
time="2025-12-14T00:06:40.629Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 587h3m32s due to TTL"
time="2025-12-14T00:26:40.584Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 586h43m32s due to TTL"
time="2025-12-14T00:46:40.520Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 586h23m32s due to TTL"
time="2025-12-14T01:06:40.560Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 586h3m32s due to TTL"
time="2025-12-14T01:26:40.516Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 585h43m32s due to TTL"
time="2025-12-14T01:46:40.583Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 585h23m32s due to TTL"
time="2025-12-14T02:06:40.515Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 585h3m32s due to TTL"
time="2025-12-14T02:26:40.603Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 584h43m32s due to TTL"
time="2025-12-14T02:46:40.539Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 584h23m32s due to TTL"
time="2025-12-14T03:06:40.529Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 584h3m32s due to TTL"
time="2025-12-14T03:26:40.639Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 583h43m32s due to TTL"
time="2025-12-14T03:46:40.572Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 583h23m32s due to TTL"
time="2025-12-14T04:06:40.636Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 583h3m32s due to TTL"
time="2025-12-14T04:26:40.552Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 582h43m32s due to TTL"
time="2025-12-14T04:46:40.627Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 582h23m32s due to TTL"
time="2025-12-14T05:06:40.602Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 582h3m32s due to TTL"
time="2025-12-14T05:26:40.521Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 581h43m32s due to TTL"
time="2025-12-14T05:46:40.609Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 581h23m32s due to TTL"
time="2025-12-14T06:06:40.592Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 581h3m32s due to TTL"
time="2025-12-14T06:26:40.585Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 580h43m32s due to TTL"
time="2025-12-14T06:46:40.538Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 580h23m32s due to TTL"
time="2025-12-14T07:06:40.643Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 580h3m32s due to TTL"
time="2025-12-14T07:26:40.600Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 579h43m32s due to TTL"
time="2025-12-14T07:46:40.622Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 579h23m32s due to TTL"
time="2025-12-14T08:06:40.628Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 579h3m32s due to TTL"
time="2025-12-14T08:26:40.594Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 578h43m32s due to TTL"
time="2025-12-14T08:46:40.571Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 578h23m32s due to TTL"
time="2025-12-14T09:06:40.528Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 578h3m32s due to TTL"
time="2025-12-14T09:26:40.549Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 577h43m32s due to TTL"
time="2025-12-14T09:46:40.582Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 577h23m32s due to TTL"
time="2025-12-14T10:06:40.653Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 577h3m32s due to TTL"
time="2025-12-14T10:26:40.626Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 576h43m32s due to TTL"
time="2025-12-14T10:46:40.580Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 576h23m32s due to TTL"
time="2025-12-14T11:06:40.551Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 576h3m32s due to TTL"
time="2025-12-14T11:26:40.552Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 575h43m32s due to TTL"
time="2025-12-14T11:46:40.537Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 575h23m32s due to TTL"
time="2025-12-14T12:06:40.628Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 575h3m32s due to TTL"
time="2025-12-14T12:26:40.610Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 574h43m32s due to TTL"
time="2025-12-14T12:46:40.574Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 574h23m32s due to TTL"
time="2025-12-14T13:06:40.583Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 574h3m32s due to TTL"
time="2025-12-14T13:26:40.606Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 573h43m32s due to TTL"
time="2025-12-14T13:46:40.553Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 573h23m32s due to TTL"
time="2025-12-14T14:06:40.559Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 573h3m32s due to TTL"
time="2025-12-14T14:26:40.592Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 572h43m32s due to TTL"
time="2025-12-14T14:46:40.646Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 572h23m32s due to TTL"
time="2025-12-14T15:06:40.546Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 572h3m32s due to TTL"
time="2025-12-14T15:26:40.609Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 571h43m32s due to TTL"
time="2025-12-14T15:46:40.661Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 571h23m32s due to TTL"
time="2025-12-14T16:06:40.651Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 571h3m32s due to TTL"
time="2025-12-14T16:26:40.599Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 570h43m32s due to TTL"
time="2025-12-14T16:46:40.615Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 570h23m32s due to TTL"
time="2025-12-14T17:06:40.605Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 570h3m32s due to TTL"
time="2025-12-14T17:26:40.620Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 569h43m32s due to TTL"
time="2025-12-14T17:46:40.578Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 569h23m32s due to TTL"
time="2025-12-14T18:06:40.670Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 569h3m32s due to TTL"
time="2025-12-14T18:26:40.591Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 568h43m32s due to TTL"
time="2025-12-14T18:46:40.611Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 568h23m32s due to TTL"
time="2025-12-14T19:06:40.568Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 568h3m32s due to TTL"
time="2025-12-14T19:26:40.571Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 567h43m32s due to TTL"
time="2025-12-14T19:46:40.637Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 567h23m32s due to TTL"
time="2025-12-14T20:06:40.646Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 567h3m32s due to TTL"
time="2025-12-14T20:26:40.684Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 566h43m32s due to TTL"
time="2025-12-14T20:46:40.614Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 566h23m32s due to TTL"
time="2025-12-14T21:06:40.590Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 566h3m32s due to TTL"
time="2025-12-14T21:26:40.604Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 565h43m32s due to TTL"
time="2025-12-14T21:46:40.597Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 565h23m32s due to TTL"
time="2025-12-14T22:06:40.578Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 565h3m32s due to TTL"
time="2025-12-14T22:26:40.625Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 564h43m32s due to TTL"
time="2025-12-14T22:46:40.664Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 564h23m32s due to TTL"
time="2025-12-14T23:06:40.660Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 564h3m32s due to TTL"
time="2025-12-14T23:26:40.668Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 563h43m32s due to TTL"
time="2025-12-14T23:46:40.617Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 563h23m32s due to TTL"
time="2025-12-15T00:06:40.579Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 563h3m32s due to TTL"
time="2025-12-15T00:26:40.621Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 562h43m32s due to TTL"
time="2025-12-15T00:46:40.671Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 562h23m32s due to TTL"
time="2025-12-15T01:06:40.612Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 562h3m32s due to TTL"
time="2025-12-15T01:26:40.615Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 561h43m32s due to TTL"
time="2025-12-15T01:46:40.587Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 561h23m32s due to TTL"
time="2025-12-15T02:06:40.675Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 561h3m32s due to TTL"
time="2025-12-15T02:26:40.625Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 560h43m32s due to TTL"
time="2025-12-15T02:46:40.654Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 560h23m32s due to TTL"
time="2025-12-15T03:06:40.627Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 560h3m32s due to TTL"
time="2025-12-15T03:26:40.621Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 559h43m32s due to TTL"
time="2025-12-15T03:46:40.699Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 559h23m32s due to TTL"
time="2025-12-15T04:06:40.690Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 559h3m32s due to TTL"
time="2025-12-15T04:26:40.671Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 558h43m32s due to TTL"
time="2025-12-15T04:46:40.591Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 558h23m32s due to TTL"
time="2025-12-15T05:06:40.687Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 558h3m32s due to TTL"
time="2025-12-15T05:26:40.593Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 557h43m32s due to TTL"
time="2025-12-15T05:46:40.645Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 557h23m32s due to TTL"
time="2025-12-15T06:06:40.596Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 557h3m32s due to TTL"
time="2025-12-15T06:26:40.611Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 556h43m32s due to TTL"
time="2025-12-15T06:46:40.672Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 556h23m32s due to TTL"
time="2025-12-15T07:06:40.657Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 556h3m32s due to TTL"
time="2025-12-15T07:26:40.585Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 555h43m32s due to TTL"
time="2025-12-15T07:46:40.672Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 555h23m32s due to TTL"
time="2025-12-15T08:06:40.709Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 555h3m32s due to TTL"
time="2025-12-15T08:26:40.615Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 554h43m32s due to TTL"
time="2025-12-15T08:46:40.585Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 554h23m32s due to TTL"
time="2025-12-15T09:06:40.640Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 554h3m32s due to TTL"
time="2025-12-15T09:26:40.638Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 553h43m32s due to TTL"
time="2025-12-15T09:46:40.703Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 553h23m32s due to TTL"
time="2025-12-15T10:06:40.631Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 553h3m32s due to TTL"
time="2025-12-15T10:26:40.639Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 552h43m32s due to TTL"
time="2025-12-15T10:46:40.622Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 552h23m32s due to TTL"
time="2025-12-15T11:06:40.642Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 552h3m32s due to TTL"
time="2025-12-15T11:26:40.664Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 551h43m32s due to TTL"
time="2025-12-15T11:46:40.588Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 551h23m32s due to TTL"
time="2025-12-15T12:06:40.683Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 551h3m32s due to TTL"
time="2025-12-15T12:26:40.632Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 550h43m32s due to TTL"
time="2025-12-15T12:46:40.631Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 550h23m32s due to TTL"
time="2025-12-15T13:06:40.616Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 550h3m32s due to TTL"
time="2025-12-15T13:26:40.702Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 549h43m32s due to TTL"
time="2025-12-15T13:46:40.653Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 549h23m32s due to TTL"
time="2025-12-15T14:06:40.600Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 549h3m32s due to TTL"
time="2025-12-15T14:26:40.701Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 548h43m32s due to TTL"
time="2025-12-15T14:46:40.641Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 548h23m32s due to TTL"
time="2025-12-15T15:06:40.718Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 548h3m32s due to TTL"
time="2025-12-15T15:26:40.666Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 547h43m32s due to TTL"
time="2025-12-15T15:46:40.625Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 547h23m32s due to TTL"
time="2025-12-15T16:06:40.632Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 547h3m32s due to TTL"
time="2025-12-15T16:26:40.620Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 546h43m32s due to TTL"
time="2025-12-15T16:46:40.638Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 546h23m32s due to TTL"
time="2025-12-15T17:06:40.600Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 546h3m32s due to TTL"
time="2025-12-15T17:26:40.654Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 545h43m32s due to TTL"
time="2025-12-15T17:46:40.667Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 545h23m32s due to TTL"
time="2025-12-15T18:06:40.631Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 545h3m32s due to TTL"
time="2025-12-15T18:26:40.718Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 544h43m32s due to TTL"
time="2025-12-15T18:46:40.643Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 544h23m32s due to TTL"
time="2025-12-15T19:06:40.635Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 544h3m32s due to TTL"
time="2025-12-15T19:26:40.669Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 543h43m32s due to TTL"
time="2025-12-15T19:46:40.685Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 543h23m32s due to TTL"
time="2025-12-15T20:06:40.728Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 543h3m32s due to TTL"
time="2025-12-15T20:26:40.726Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 542h43m32s due to TTL"
time="2025-12-15T20:46:40.662Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 542h23m32s due to TTL"
time="2025-12-15T21:06:40.730Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 542h3m32s due to TTL"
time="2025-12-15T21:26:40.681Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 541h43m32s due to TTL"
time="2025-12-15T21:46:40.622Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 541h23m32s due to TTL"
time="2025-12-15T22:06:40.630Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 541h3m32s due to TTL"
time="2025-12-15T22:26:40.674Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 540h43m32s due to TTL"
time="2025-12-15T22:46:40.731Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 540h23m32s due to TTL"
time="2025-12-15T23:06:40.673Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 540h3m32s due to TTL"
time="2025-12-15T23:26:40.653Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 539h43m32s due to TTL"
time="2025-12-15T23:46:40.714Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 539h23m32s due to TTL"
time="2025-12-16T00:06:40.678Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 539h3m32s due to TTL"
time="2025-12-16T00:26:40.652Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 538h43m32s due to TTL"
time="2025-12-16T00:46:40.648Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 538h23m32s due to TTL"
time="2025-12-16T01:06:40.667Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 538h3m32s due to TTL"
time="2025-12-16T01:26:40.746Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 537h43m32s due to TTL"
time="2025-12-16T01:46:40.676Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 537h23m32s due to TTL"
time="2025-12-16T02:06:40.675Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 537h3m32s due to TTL"
time="2025-12-16T02:26:40.638Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 536h43m32s due to TTL"
time="2025-12-16T02:46:40.636Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 536h23m32s due to TTL"
time="2025-12-16T03:06:40.752Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 536h3m32s due to TTL"
time="2025-12-16T03:26:40.709Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 535h43m32s due to TTL"
time="2025-12-16T03:46:40.699Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 535h23m32s due to TTL"
time="2025-12-16T04:06:40.696Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 535h3m32s due to TTL"
time="2025-12-16T04:26:40.719Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 534h43m32s due to TTL"
time="2025-12-16T04:46:40.730Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 534h23m32s due to TTL"
time="2025-12-16T05:06:40.752Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 534h3m32s due to TTL"
time="2025-12-16T05:26:40.646Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 533h43m32s due to TTL"
time="2025-12-16T05:46:40.629Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 533h23m32s due to TTL"
time="2025-12-16T06:06:40.630Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 533h3m32s due to TTL"
time="2025-12-16T06:26:40.677Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 532h43m32s due to TTL"
time="2025-12-16T06:46:40.715Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 532h23m32s due to TTL"
time="2025-12-16T07:06:40.630Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 532h3m32s due to TTL"
time="2025-12-16T07:26:40.691Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 531h43m32s due to TTL"
time="2025-12-16T07:46:40.752Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 531h23m32s due to TTL"
time="2025-12-16T08:06:40.743Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 531h3m32s due to TTL"
time="2025-12-16T08:26:40.696Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 530h43m32s due to TTL"
time="2025-12-16T08:46:40.742Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 530h23m32s due to TTL"
time="2025-12-16T09:06:40.765Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 530h3m32s due to TTL"
time="2025-12-16T09:26:40.654Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 529h43m32s due to TTL"
time="2025-12-16T09:46:40.766Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 529h23m32s due to TTL"
time="2025-12-16T10:06:40.754Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 529h3m32s due to TTL"
time="2025-12-16T10:26:40.758Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete in 528h43m32s due to TTL"
time="2025-12-16T10:32:51.767Z" level=info msg="Queueing Succeeded workflow test-workflows/create-new-svc-v3-hkxd4 for delete due to max rention(50 workflows)"
time="2025-12-16T10:32:51.767Z" level=info msg="Deleting garbage collected workflow 'test-workflows/create-new-svc-v3-hkxd4'"
time="2025-12-16T10:32:51.855Z" level=info msg="Successfully request 'test-workflows/create-new-svc-v3-hkxd4' to be deleted"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[Joibel]

Is this only if you modify the controller-configmap without restarting the controller?

[mboronin]

Is this only if you modify the controller-configmap without restarting the controller?

that is our assumption, yes, but we are verifying it currently, whether the restart leads to correct behavior

[mboronin]

Can confirm that the issue was resolved by restarting the controller, however I still believe this should be considered as a bug due to misleading logs showing a different configuration and indicating that configuration is being updated from the configmap

[Joibel]

Can confirm that the issue was resolved by restarting the controller, however I still believe this should be considered as a bug due to misleading logs showing a different configuration and indicating that configuration is being updated from the configmap

Ok, thanks. I do agree it's a bug. I'm not convinced we should have ever implemented configmap reload though, because it results in problems like this.

[jswxstw]

time="2025-12-16T13:30:08.082Z" level=info msg="Received Workflow Controller config map argo-workflows/argo-workflows-workflow-controller-configmap update"

The logs you provided do not contain any entries after 2025-12-16T13:30:08.082Z, when the workflow-controller-configmap was updated.

By the way, updating workflowDefaults will only affect workflows created after the update; workflows created before will still use the old configs.

[mboronin]

@jswxstw thanks for looking into it. We've actually had this issue for a few days, so we have plenty of misleading logs for this, but yes, I guess it is not clear in my original message, here are the first entries for this

time="2025-12-11T00:51:22.137Z" level=info msg="Configuration:\nartifactRepository: {}\ninitialDelay: 0s\nmetricsConfig:\n  enabled: true\n  path: /metrics\n  port: 9090\n  secure: false\nnodeEvents:\n  enabled: true\nparallelism: 10\npodSpecLogStrategy: {}\nresourceRateLimit:\n  burst: 5\n  limit: 20\nretentionPolicy:\n  completed: 200\n  errored: 200\n  failed: 200\nsso:\n  clientId:\n    key: client-id\n    name: argo-workflows-okta-client-credentials\n  clientSecret:\n    key: client-secret\n    name: argo-workflows-okta-client-credentials\n  customGroupClaimName: gt_groups\n  issuer: https://sso.wbd.com/oauth2/aus1di7v8x4SIaRwj0x8\n  rbac:\n    enabled: true\n  redirectUrl: https://argo-workflows.prd.idp.discomax.com/oauth2/callback\n  scopes:\n  - openid\n  - profile\n  - email\n  sessionExpiry: 12h0m0s\ntelemetryConfig: {}\nworkflowDefaults:\n  metadata:\n    creationTimestamp: null\n    labels:\n      omd_business_service: boltcloud\n  spec:\n    arguments: {}\n    podMetadata:\n      labels:\n        omd_business_service: boltcloud\n    serviceAccountName: argo-workflows-sa\n    ttlStrategy:\n      secondsAfterCompletion: 2592000\n      secondsAfterFailure: 2592000\n      secondsAfterSuccess: 2592000\n  status:\n    finishedAt: null\n    startedAt: null\nworkflowEvents:\n  enabled: true\n"

time="2025-12-11T00:51:22.131Z" level=info msg="Received Workflow Controller config map argo-workflows/argo-workflows-workflow-controller-configmap update"

We have noticed the change of the behaviour straight after the restart, when workflows were not deleted anymore.
Are you sure that retention policy based on the amounts of workflows does not affect the workflows created before the change? I had the impression that it is the case only with ttl value

[jswxstw]

Sorry, I misunderstood your question before, but I think I know what the problem is now.
Although workflow-controller-configmap supports dynamic updates，gcCtrl did not re-inited here, so new Retention configurations could not be loaded.

argo-workflows/workflow/controller/controller.go

Line 249 in 7fd533a

gcCtrl := gccontroller.NewController(ctx, wfc.wfclientset, wfc.wfInformer, wfc.metrics, wfc.Config.RetentionPolicy)