Fix: fork: Cannot allocate memory (Linux)

fork: Cannot allocate memory means the kernel refused to create a new process. It is worse than a normal out-of-memory error because process creation itself is broken: cron jobs fail, SSH sessions cannot spawn shells, and monitoring agents go quiet exactly when you need them. The name misleads slightly, because both memory exhaustion and PID or thread exhaustion produce this exact message. Any shell you already have open is now a precious resource, so use built-ins where you can.

fork: Cannot allocate memory

What this error means

fork creates a new process by duplicating the caller. The kernel must allocate a task structure, a PID, kernel stacks, and page tables, and account the child’s address space against its overcommit budget. If any of that fails, fork returns EAGAIN or ENOMEM and the C library prints “Cannot allocate memory”. Genuine memory pressure is the classic cause, but hitting kernel.pid_max, a cgroup pids.max limit (systemd’s TasksMax), or per-user process limits (ulimit -u) fails the same way. Since every thread on Linux consumes a PID, a thread-leaking application can exhaust PIDs while memory looks fine.

Common causes

• Real memory exhaustion: a leaking or oversized process consumed RAM and swap, so the kernel cannot budget a new process.
• PID exhaustion: a fork bomb, a thread leak, or zombie accumulation used up kernel.pid_max or a cgroup’s pids.max.
• systemd’s TasksMax on one service capping its threads and children, so only that service sees the error.
• Strict overcommit (vm.overcommit_memory=2) with little or no swap, where a large process cannot fork even though the box is idle, because fork briefly doubles committed address space.
• Per-user nproc limits from /etc/security/limits.conf hit by an app spawning workers.

How to fix it

STABILIZE first. From an existing shell, free the resource. If it is memory, kill the biggest consumer; if it is a fork loop, kill the offending process group. Shell built-ins like echo and kill do not fork, which matters when nothing else runs.

ps aux --sort=-%mem | head -10
kill -9 <pid>

1. DIAGNOSE which resource is exhausted. Check process and thread counts against the PID ceiling, and memory including the commit limit.

ps -eLf | wc -l
cat /proc/sys/kernel/pid_max
free -h
grep -E 'CommitLimit|Committed_AS' /proc/meminfo

2. If the count is near pid_max, find what owns all the tasks. Thousands of identical entries name the fork or thread leak.

ps -eo comm= | sort | uniq -c | sort -rn | head -10
ps -eo nlwp,pid,comm --sort=-nlwp | head -10

3. If one systemd service hits the error while the host is healthy, check and raise its task limit.

systemctl status myservice | grep Tasks
systemctl edit myservice

[Service]
TasksMax=8192

4. FIX the ceiling only when the workload is legitimate. Raising pid_max is safe on modern systems, and persists via sysctl:

sysctl -w kernel.pid_max=131072
echo 'kernel.pid_max = 131072' >> /etc/sysctl.d/99-pidmax.conf

5. If strict overcommit blocks forks from a large process, add swap headroom or revisit the overcommit policy. Check the current mode first:

cat /proc/sys/vm/overcommit_memory
fallocate -l 4G /swapfile && chmod 600 /swapfile
mkswap /swapfile && swapon /swapfile

Then treat the real incident: whatever leaked memory, leaked threads, or forked in a loop must be fixed, or the same wall returns at the new limit.

How to prevent it

• Alert on process and thread counts per host and per service, not just on memory, so PID leaks are visible early.
• Set TasksMax deliberately on services that spawn workers, sized to the design, so one service cannot starve the host.
• Give service managers restart backoff so a crashing process cannot become a fork loop.
• Keep modest swap on memory-tight hosts to absorb fork-time commit spikes.
• Reap children properly in anything that spawns processes, so zombies do not accumulate PIDs.

When to call a senior engineer

Call for help when the host locks up faster than you can diagnose it, when thread or process counts climb daily and the owning code cannot be found, or when overcommit and cgroup limits interact in ways nobody on the team can predict. Erzon engineers can identify the exact leak from kernel accounting, fix the spawning logic at the source, and set limits and alerts so process exhaustion never again takes out SSH along with the app.