Fix: Out of memory: Killed process (Linux OOM killer)

Your process died and the log shows a line like Out of memory: Killed process 12345 (mysqld). That message comes from the Linux kernel, not from your application. The kernel ran out of usable memory, decided the system could not continue safely, and terminated a process to reclaim RAM. This guide walks through confirming the event, freeing memory quickly so the service comes back, and then finding and fixing what actually consumed the memory so it does not recur.

What this error means

Linux allows processes to reserve more memory than physically exists, on the assumption that most reservations are never fully used. This is called overcommit. When processes actually try to use more memory than the machine (plus swap) can provide, the kernel cannot honor the demand. Rather than freeze, it invokes the out-of-memory (OOM) killer, which selects one process and sends it a fatal signal.

The kernel does not pick randomly. It calculates an oom_score for each process, weighted by how much memory the process is using and adjusted by that process’s oom_score_adj. The process with the highest score becomes the victim. Because databases and application servers tend to hold large resident sets, they are frequent targets even when a smaller runaway process triggered the shortage.

Common causes

• No swap configured, or swap too small to absorb a transient spike, so the kernel hits the OOM condition immediately.
• A memory leak or an unbounded in-memory cache in an application that grows until it exhausts RAM.
• A database configured to use more memory than the host has, for example an oversized innodb_buffer_pool_size or a high maintenance_work_mem in Postgres.
• A JVM or other runtime started without a heap ceiling, or with a ceiling larger than the machine can support.
• A cgroup or container memory limit that is smaller than the workload needs, triggering an OOM inside the limit even though the host still has free RAM.
• An undersized instance for the real, peak workload rather than the average.

How to fix it

Stabilize

1. Restart the killed service so traffic recovers, then watch memory while it warms up.

sudo systemctl restart mysql
watch -n 2 free -m

2. If memory is already tight and there is no swap, add a temporary swap file as a cushion. This is a stopgap, not a fix.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
free -m

Diagnose

1. Confirm the OOM event and read the kernel summary, which lists the victim and each process’s memory usage.

sudo journalctl -k | grep -i -A 20 "oom-killer"
sudo dmesg -T | grep -i "killed process"

2. Find which processes hold the most resident memory right now.

free -m
ps aux --sort=-rss | head -n 15
top -o %MEM

3. If the workload runs in a container or cgroup, check its limit and current usage. A value close to the limit points to a cgroup OOM rather than host exhaustion.

cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current

Fix

1. Cap the process that dominates memory. For MySQL, size the buffer pool to fit the host with headroom left for the OS and connections.

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
# innodb_buffer_pool_size = 2G
sudo systemctl restart mysql

2. For a JVM service, set an explicit heap ceiling so the runtime cannot grow past what the machine can give it.

java -Xmx2g -jar app.jar

3. For a container, set a memory limit that matches the workload and lets the orchestrator reschedule cleanly.

docker run --memory=2g --memory-swap=2g myimage

4. If a specific process must survive pressure, lower its oom_score_adj so the kernel prefers other victims. Do this only after addressing the real cause.

echo -900 | sudo tee /proc/$(pgrep -o mysqld)/oom_score_adj

How to prevent it

• Alert on memory before it saturates, for example at 80 percent used, so you act before the kernel does.
• Right-size instance memory and per-process limits against peak load, not average load.
• Set explicit ceilings on every memory-hungry process: database buffers, runtime heaps, and container limits.
• Load test the path that grows memory, so caches and connection pools reveal their true footprint.
• Keep a small swap cushion on production hosts to absorb brief spikes.
• Track resident memory over time to catch a slow leak long before it kills anything.

When to call a senior engineer

If the OOM killer keeps firing after you have right-sized limits, the memory is likely leaking somewhere that basic tools do not surface, and you need heap analysis or allocation profiling under real traffic. Call for help when a database is the repeat victim and downtime is compounding, when the OOM happens inside a cgroup limit you cannot safely raise, or when you cannot reproduce the growth outside production. Erzon can join mid-incident, stabilize the service, and find the true source of the pressure rather than restarting on a loop.