How to find a memory leak in production

TL;DR

• First prove it is a leak: steady growth across GC cycles under flat load, on a trend line that never plateaus. Caches, fragmentation, and runtime headroom all impersonate leaks.
• Capture two heap snapshots hours apart on the same instance and diff them. The leak is whatever class of object grows monotonically between them.
• Tooling: Chrome DevTools heap snapshots for Node, tracemalloc or objgraph for Python, heap dumps plus Eclipse MAT for the JVM, pprof heap profiles for Go.
• Mitigate first (restart cadence, memory limits with graceful drain, roll back the suspect deploy), then fix the actual retention path.
• The leak is almost always a reference someone forgot: an unbounded cache or map, listeners never removed, closures captured by something long-lived, or connections never released.

Step 1: Confirm it is actually a leak

Before profiling anything, spend twenty minutes ruling out the impostors.

1. Plot memory over days, not hours. A sawtooth that climbs, drops on GC or restart, and climbs back to the same peak is normal. A sawtooth whose floor rises every cycle is a leak.
2. Correlate with load. Memory that tracks request volume and falls when traffic falls is working memory. A leak keeps its gains at 3 a.m.
3. Know your runtime’s habits. The JVM keeps heap up to -Xmx and rarely gives it back to the OS. Go returns memory lazily (GOGC and the OS reclaim policy shape RSS). Python’s allocator holds onto freed blocks, so RSS overstates live objects. In all three, RSS is a symptom metric, not a proof metric; use the runtime’s own live-heap number.
4. Check for the step change. If the growth started at a specific deploy, git log between the two releases is your fastest profiler. A rollback that flattens the curve has already isolated the bug to one diff.

Signals that say “real leak, keep going”: OOM kills in dmesg or kubectl describe pod (OOMKilled, exit code 137), heap-after-full-GC trending up in JVM GC logs, Node process.memoryUsage().heapUsed climbing after global.gc() in a test harness.

Step 2: Stabilize production before you investigate

You need the service alive long enough to study it.

• Set a restart cadence as a stopgap. A nightly rolling restart, or a Kubernetes memory limit with proper preStop draining, converts an outage into a maintenance fact. Say out loud that it is temporary.
• Keep one instance leaking. Pull one pod or VM out of the restart schedule (and, if snapshots will pause it, out of the load balancer). A freshly restarted fleet has no evidence in it.
• Turn on the cheap telemetry now: GC logs on the JVM (-Xlog:gc*), heap stats scraped into your metrics, expvar/pprof endpoint enabled in Go.

Step 3: Capture the evidence

The universal technique is the two-snapshot diff: capture the heap early, let the process leak for a few hours, capture again, and diff. Whatever grew is your suspect list, ranked.

Runtime	Capture	Analyze with
Node.js	node --inspect + Chrome DevTools, or v8.writeHeapSnapshot(), or kill -USR2 with heapdump	DevTools Memory tab, “Comparison” view
Python	tracemalloc.take_snapshot() at two points	snapshot2.compare_to(snapshot1, 'lineno'); objgraph.growth() for object counts
JVM	jcmd <pid> GC.heap_dump /tmp/heap.hprof (or jmap -dump:live)	Eclipse MAT “Leak Suspects” report, dominator tree
Go	curl :6060/debug/pprof/heap at two points	go tool pprof -base heap1.pb.gz heap2.pb.gz, then top and list

Practical notes:

• Node: a heap snapshot temporarily needs roughly as much memory as the heap itself and pauses the event loop. Take it on the drained instance. In the comparison view, sort by “Delta” and look at retainer chains, not just counts.
• Python: start tracemalloc early (it only records allocations made after it starts). compare_to(..., 'traceback') gives you the exact allocation site. If counts grow but tracemalloc looks flat, suspect a C extension leaking outside the Python heap.
• JVM: always analyze the dump off-box; MAT’s dominator tree answers “what single object is keeping 4 GB alive” directly. Heap dumps of -Xmx32g processes are slow and huge, plan for that.
• Go: remember pprof’s heap profile shows live heap, and goroutine leaks are memory leaks too: check /debug/pprof/goroutine for counts in the tens of thousands. Each parked goroutine pins its stack and everything it references.

Step 4: Isolate the source

Reading the diff, you are looking for the retention path: the chain of references from a GC root to the growing objects. The growing object type is rarely the bug; the thing holding it is.

The usual suspects, in rough order of frequency:

1. Unbounded caches and maps. A memoization dict, a Map keyed by user or request id, a metrics registry with unbounded label cardinality. Fix: bound it (LRU, TTL) or key it correctly.
2. Listeners and callbacks never removed. Event emitters, subscriptions, timers (setInterval with a closure), signal handlers registered per request.
3. Closures captured by something long-lived. A per-request closure appended to a module-level array, or a logger holding the last N request contexts where N is “all of them”.
4. Resources without release paths. Connections, cursors, file handles, streams that error before close(). Often visible as fd growth (ls /proc/<pid>/fd | wc -l) alongside memory.
5. Framework-specific classics: JVM ThreadLocal in thread pools, Node promises that never settle, Python default mutable arguments accumulating, Go tickers without Stop().

If the snapshot diff is ambiguous, bisect by behavior: replay one endpoint at a time against a canary and watch which traffic makes the line climb. A leak you can reproduce in ten minutes on a laptop is a solved leak.

Step 5: Fix, then prove the fix

• Ship the fix behind normal review; leaks love “obvious one-line fixes” that move the retention rather than remove it.
• Prove it with the same graph that convicted it: flat heap-after-GC over at least a few days at production load, on instances excluded from any restart cadence.
• Then remove the restart stopgap deliberately, and add a regression tripwire: an alert on heap-after-GC slope, not just on absolute memory.

When to call for help

If the fleet is OOM-looping faster than you can capture a snapshot, if the diff points into a dependency or a C extension you do not own, or if the leak only reproduces at full production traffic, it is reasonable to want senior eyes on it. Erzon takes exactly this kind of incident from triage, first response within one business hour, works in your own repos and infrastructure with least-privilege access, and every engagement ends with a written root-cause report and a prevention list so the same leak does not come back wearing a different class name.