[Bug]: Heavy RAM Usage over time

[TSJasonH]

What happened?

We issue many requests daily through both the proxy and pass-through in testing of an application we're building. After a few days the RAM usage by litellm is significant and it never seems to let it go until issuing a restart. Eventually our monitoring software triggers a ram usage threshold alert and I go in and restart the container.

Here's a graph over a week - you can clearly see where I restarted it ;-)

Running in docker on Almalinux 9 and generally I update when there is a new stable pushed out, but seemingly always the same thing happens.

Relevant log output

Nothing especially telling in the logs. I see a number of these repeating all the time:

litellm-1     | Traceback (most recent call last):
litellm-1     |   File "/usr/lib/python3.13/site-packages/litellm/proxy/hooks/proxy_track_cost_callback.py", line 178, in _PROXY_track_cost_callback
litellm-1     |     raise Exception(
litellm-1     |         "User API key and team id and user id missing from custom callback."
litellm-1     |     )
litellm-1     | Exception: User API key and team id and user id missing from custom callback.
litellm-1     | ESC[92m10:34:50 - LiteLLM Proxy:ERRORESC[0m: proxy_track_cost_callback.py:215 - Error in tracking cost callback - User API key and team id and user id missing from custom ca
llback.

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.73.6-stable.patch.1

Twitter / LinkedIn details

No response

[chiruno-9]

How do you configure your config.yaml?
We managed to solve a similar RAM issue by adding:

environment_variables:
  MAX_IN_MEMORY_QUEUE_FLUSH_COUNT: "5000"
  MAX_SIZE_IN_MEMORY_QUEUE: "500"

[BigFoxMedia]

I experienced the same issue, one of my production servers went down when eventually it consumed almost 20GB of RAM alone, and that's being idle... totally crashed the server.

[xmcp]

I deployed litellm on a server with 8GB RAM and observed similar memory leak. I had to restart the docker once a week or it will eventually exhaust all RAMs. My workload is typically <20k calls per day.

I will try the env vars suggested by @chiruno-9 and see if it helps.

[TSJasonH]

I'm also trying out those env options - I'll monitor the usage for a few days to see what happens.

[jugaldb]

We are investigating this internally, if you all can share some profiler logs, that would be really helpful to check with the data we have internally.

[BigFoxMedia]

myself i made a cron to restart the container every 12h. What exactly do you need in terms of "profile logs"?

…

On Sat, Jul 26, 2025, 16:14 Jugal D. Bhatt ***@***.***> wrote: *jugaldb* left a comment (BerriAI/litellm#12685) <#12685 (comment)> We are investigating this internally, if you all can share some profiler logs, that would be really helpful to check with the data we have internally. — Reply to this email directly, view it on GitHub <#12685 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2S6VR2S2ZX7MYL6AKOCIL3KPOS5AVCNFSM6AAAAACBXSFJIWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMRTGE4DSMRWG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[404Huang]

We are investigating this internally, if you all can share some profiler logs, that would be really helpful to check with the data we have internally.

glad to hear that, this problem is quite severe. Even our proxy receives requests slowly, RAM will run out eventually. Hope to see a solution

[amend-gerke]

I am having the same issue as above. I currently host this as a portainer stack using a docker compose config. I have managed to contain it by adding the following to the litellm service:

deploy: resources: limits: memory: 14G reservations: memory: 2G

this prevents the one service from taking the entire server down but it will go from 2GB Memory to 14GB memory in 12 hours and then hold steady there and never letting up. If it is sitting at 14GB and it gets a rapid fire set of proxy calls, it will send the litellm container into an unhealthy status and I will have to manually restart.

I don't have a high volume of traffic right now but I am hoping to scale up soon. If it is helpful to know, I do have callbacks to a langfuse instance configured as well.

Happy to provide any data I can to help run down this issue.

Currently running 1.74.15

[wblinux]

litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
I found there is a memory leak issue in the [initialize_pass_through_endpoints] function that allows duplicate routes to be registered repeatedly, leading to memory accumulation. When [initialize_pass_through_endpoints] is called multiple times(dynamic endpoint updates), the same routes are added to FastAPI's internal routing table without checking if they already exist. @jugaldb

[wblinux]

litellm/proxy/pass_through_endpoints/pass_through_endpoints.py I found there is a memory leak issue in the [initialize_pass_through_endpoints] function that allows duplicate routes to be registered repeatedly, leading to memory accumulation. When [initialize_pass_through_endpoints] is called multiple times(dynamic endpoint updates), the same routes are added to FastAPI's internal routing table without checking if they already exist. @jugaldb

[jugaldb]

Hi @wblinux, Is it possible for you to try with ghcr.io/berriai/litellm:litellm_dev_memory_fix-litellm_v1.73.0-dev_memory_fix_2 ? Added a fix for proxy pass through endpoints.

[wblinux]

Hi @wblinux, Is it possible for you to try with ghcr.io/berriai/litellm:litellm_dev_memory_fix-litellm_v1.73.0-dev_memory_fix_2 ? Added a fix for proxy pass through endpoints.

Hi @jugaldb , I've tested the ghcr.io/berriai/litellm:litellm_dev_memory_fix-litellm_v1.73.0-dev_memory_fix_2 image, and I can confirm that the memory leak issue with the pass-through endpoints is resolved.

Thank you so much for the quick fix!

[ishaan-jaff]

@wblinux how did you find the issue - we're hiring someone to help us with these types of issues would you be interested in joining us - either as a contractor or full time.

If yes - please book some time here

[SAhmadrezaAnaami]

I have the same problem

and I can not remove pass through endpoints