[VPA] OOM event will be discarded - it is too old

[R-Studio]

Which component are you using?:

• recommender
• update
• admission

/area vertical-pod-autoscaler

What version of the component are you using?:

VPA: 1.4.1

Component version:

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: v1.34.0
Kustomize Version: v5.7.1
Server Version: v1.33.0

What environment is this in?:

OnPrem VMs with Talos

What did you expect to happen?:

VPA will increase the memory by our memory bump settings:

recommender:
  extraArgs:
    oom-bump-up-ratio: 1.5
    oom-min-bump-up-bytes: 104857600 # 100Mi

What happened instead?:

OOM event will be discarded - it is too old:
Here an example (the first timestamp is form Loki and is converted to my timezone):

2025-09-18 10:50:22.358	I0918 08:50:22.357719       1 cluster_feeder.go:528] "OOM detected" oomInfo={"Timestamp":"2025-09-18T08:49:46Z","Memory":195989341,"ContainerID":{"Namespace":"authentik","PodName":"authentik-server-c4544c967-ftqx6","ContainerName":"server"}}
2025-09-18 10:50:22.358	I0918 08:50:22.357757       1 cluster_feeder.go:530] "Failed to record OOM" oomInfo={"Timestamp":"2025-09-18T08:49:46Z","Memory":195989341,"ContainerID":{"Namespace":"authentik","PodName":"authentik-server-c4544c967-ftqx6","ContainerName":"server"}} error="error while recording OOM for {{authentik authentik-server-c4544c967-ftqx6} server}, Reason: OOM event will be discarded - it is too old (2025-09-18 08:49:46 +0000 UTC)"

How to reproduce it (as minimally and precisely as possible):

Try creating OOM events for a pod that is handled by VPA.

Anything else we need to know?:

Similar issue: #4152

[adrianmoisey]

How to reproduce it (as minimally and precisely as possible):

Try creating OOM events for a pod that is handled by VPA.

Can you give more details on how to reproduce this? Specifically, using kind.

I've setup a test and I can't reproduce this locally.

[adrianmoisey]

/triage needs-information

[R-Studio]

@adrianmoisey Unfortunately, I am not sure how to reproduce this issue. I only looked at the logs, and I don't know why the memory request for my pod isn't being increased. When I look at the timestamps, there is no reason for "OOM event will be discarded - it is too old"

[adrianmoisey]

Unfortunately, I am not sure how to reproduce this issue.

I think this is what happened on the linked issue, no maintainers could reproduce it, so it can't get fixed.

I only looked at the logs, and I don't know why the memory request for my pod isn't being increased. When I look at the timestamps, there is no reason for "OOM event will be discarded - it is too old"

Not too sure what you mean here, there is a reason, the OOM is too old, so it's discarded.
I'm not too familiar with that logic or why it exists.

Could there be a delay between the event being created and the VPA receiving it? Is the VPA getting throttled to the api-server? How many VPAs are you running? Could this be a scale problem?

[adrianmoisey]

What do you have --memory-aggregation-interval set to?

[adrianmoisey]

Here an example (the first timestamp is form Loki and is converted to my timezone):

2025-09-18 10:50:22.358 I0918 08:50:22.357719 1 cluster_feeder.go:528] "OOM detected" oomInfo={"Timestamp":"2025-09-18T08:49:46Z","Memory":195989341,"ContainerID":{"Namespace":"authentik","PodName":"authentik-server-c4544c967-ftqx6","ContainerName":"server"}}
2025-09-18 10:50:22.358 I0918 08:50:22.357757 1 cluster_feeder.go:530] "Failed to record OOM" oomInfo

Can you also confirm the correct timezones here?

The log message was made at 10:50:22.358, but the error states 08:50:22.357757, which is 2 hours difference. I'm unsure what adjustments you made for your timezone.

[R-Studio]

@adrianmoisey

Could there be a delay between the event being created and the VPA receiving it? Is the VPA getting throttled to the api-server? How many VPAs are you running? Could this be a scale problem?

I don't think there is a delay because on that cluster we don't really have big workloads running.
I don't see any api-server throttling at this time and also the VPA was not throlled and uses extremly less cpu:

We run 1x recommender, 1x updater & 2x admission-controller

I think this is what happened on the linked issue, no maintainers could reproduce it, so it can't get fixed.

I can only say that, I wanted to rollout a new version of my authentik-server-c4544c967-ftqx6 pod and it gets OOM killed multiple times. My expectation would be that the VPA then bumps the pod's memory in the predefined steps (oom-bump-up-ratio=1.5, oom-min-bump-up-bytes=104857600 ).

What do you have --memory-aggregation-interval set to?

So I don't set this argument, and I think the default is 24h0m0s.

Can you also confirm the correct timezones here?

The log message was made at 10:50:22.358, but the error states 08:50:22.357757, which is 2 hours difference. I'm unsure what adjustments you made for your timezone.

Yes it is the correct timezone +2h.
I don't understand why VPA says "it is too old" because

• time difference between 2025-09-18 08:49:46 +0000 UTC and 2025-09-18T08:49:46Z -> 0
• time difference between 2025-09-18 08:49:46 +0000 UTC and 08:50:22.357757 -> is only 36s (or is 36s already too old?)

Is there maybe an argument to configure this?

I set the following extra arguments on the recommender and the updater:

recommender:
  extraArgs:
    v: "4"
    recommender-name: efficiency-prom
    memory-saver: true
    recommendation-margin-fraction: 0.0
    pod-recommendation-min-cpu-millicores: 10                                                                                                                
    pod-recommendation-min-memory-mb: 10
    history-length: 4d
    ## CPU
    recommendation-lower-bound-cpu-percentile: 0.40
    target-cpu-percentile: 0.60
    recommendation-upper-bound-cpu-percentile: 0.70
    ## Memory
    recommendation-lower-bound-memory-percentile: 0.50
    target-memory-percentile: 0.70
    recommendation-upper-bound-memory-percentile: 0.80
    oom-bump-up-ratio: 1.5
    oom-min-bump-up-bytes: 104857600 # 100Mi
    humanize-memory: true
    ## Prometheus
    storage: prometheus
    prometheus-address: http://prometheus-operated.kube-prometheus-stack:9090
    prometheus-cadvisor-job-name: kubelet
    metric-for-pod-labels: kube_pod_labels{job="kube-state-metrics"}[4d]
    pod-label-prefix: label_
    pod-namespace-label: namespace
    pod-name-label: pod
    container-namespace-label: namespace
    container-pod-name-label: pod
    container-name-label: container

updater:
  extraArgs:
    updater-interval: 5m
    in-recommendation-bounds-eviction-lifetime-threshold: 24h
    pod-update-threshold: 0.2

# We don't set any arguments for the recommender

[adrianmoisey]

My expectation would be that the VPA then bumps the pod's memory in the predefined steps

This expectation makes sense, as that is how it should operate.

Is there maybe an argument to configure this?

It seems to be related to the --memory-aggregation-interval argument, which is why I asked about that.

I set the following extra arguments on the recommender and the updater:

Which Helm chart are you using here?

Unfortunately, without the ability to reproduce this, it's hard to know exactly what is going wrong or what we can do to fix it.

[R-Studio]

It seems to be related to the --memory-aggregation-interval argument, which is why I asked about that.

Should we configure the --memory-aggregation-interval

Which Helm chart are you using here?

We use the Helm chart from Fairwinds (v4.8.0): https://artifacthub.io/packages/helm/fairwinds-stable/vpa/4.8.0

Maybe you are able to reproduce this, by running a pod for hours or days to generate metrik history, and then increase the memory usage that the pods needs so that it gets OOM killed multiple times.

[adrianmoisey]

Maybe you are able to reproduce this, by running a pod for hours or days to generate metrik history, and then increase the memory usage that the pods needs so that it gets OOM killed multiple times.

Unfortunately I don't have time to figure out how to reproduce this at the moment. The best way to move this issue forward is to get reliable reproduction steps written down, so that we understand the problem and can look into a fix.

/help

[k8s-ci-robot]

@adrianmoisey:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Maybe you are able to reproduce this, by running a pod for hours or days to generate metrik history, and then increase the memory usage that the pods needs so that it gets OOM killed multiple times.

Unfortunately I don't have time to figure out how to reproduce this at the moment. The best way to move this issue forward is to get reliable reproduction steps written down, so that we understand the problem and can look into a fix.

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.