My Python service is leaking memory in production but not in dev. From where should I start?

Python's garbage collector is like a diligent janitor who is legally forbidden from throwing away anything you are still touching. The most counterintuitive reality of Python memory leaks is that, technically, **Python rarely "leaks" memory** in the traditional sense. Instead, your code is likely "hoarding" it—holding onto references to objects that it will never use again, but which the janitor isn't allowed to toss. ### Why Production is Different The reason your service fails in production but thrives in development is often a matter of **scale and time**. In a dev environment, a small leak might consume 10MB over an hour. In production, under 1,000 times the traffic, that same leak consumes 10GB in minutes. Furthermore, production environments often use different web servers (like Gunicorn or uWSGI) and monitoring tools that may interact with Python's memory management differently than your local debugger. ### Your Starting Point: The Snapshot Strategy To find the source, you shouldn't just look at the total memory; you need to look at the **growth over time**. 1. **Use `tracemalloc`**: This is a built-in Python library that acts as a high-tech camera for your RAM. You take a "snapshot" of memory at Point A, wait for the leak to occur, and take another at Point B. By comparing them, you can see exactly which line of code allocated the most new memory. 2. **Check for "Global Bloat"**: In many cases, developers accidentally use a list or dictionary as a global cache. In development, the cache stays small. In production, it grows infinitely until the OS kills the process. 3. **Inspect C-Extensions**: If you are using libraries for data science or image processing (like NumPy or Pillow), the leak might not be in Python at all, but in the underlying C code where memory is managed manually. As the [Python Documentation on Garbage Collection](https://docs.python.org/3/library/gc.html) explains, Python primarily uses **Reference Counting**: > "Python’s memory management is based on reference counting... when an object’s reference count reaches zero, the object is deallocated." If your memory is growing, it means you have "zombie" references—objects that are logically dead but technically still have a reference count higher than zero. ### Recommended Tooling If `tracemalloc` doesn't give you the answer, look toward [Memray](https://bloomberg.github.io/memray/), a powerful profiler from Bloomberg. It can track memory allocations in both Python code and C extensions, providing a "flame graph" that visually shows where your memory is being eaten. *** ### Explore Further - **The Instagram Strategy**: Why did the engineering team at Instagram famously decide to [disable the Python Garbage Collector](https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-479733910b51) to improve performance? - **Circular References**: What happens when Object A points to Object B, and Object B points back to Object A? How does Python eventually solve this "deadlock" of memory? - **Fragmentation**: Can a program use up all its memory even if it has plenty of "free" space available inside it?