Fix: bash: fork: retry: Resource temporarily unavailable

You typed a command and the shell answered with bash: fork: retry: Resource temporarily unavailable. The system is refusing to create new processes. This is not a disk or memory error in the usual sense. It is the kernel returning EAGAIN when a process calls fork(), because some limit on the number of processes or threads has been reached. Until you free up slots or raise the limit, almost nothing that launches a new program will run.

What this error means

Every time you start a program, the shell calls fork() to create a new process. When that call fails with EAGAIN, the kernel is saying it cannot create another process right now. The cause is a ceiling being hit: a per-user process or thread limit, a system-wide PID limit, or a container cgroup limit. The word “retry” in the message is bash trying the fork a few times before giving up. The system is not necessarily out of RAM. It is out of process slots.

Common causes

• A per-user process and thread limit reached (ulimit -u, set via nproc in /etc/security/limits.conf or a file under /etc/security/limits.d/).
• A systemd unit hitting TasksMax, or the host hitting DefaultTasksMax in /etc/systemd/system.conf.
• System-wide PID exhaustion, where the live process count approaches kernel.pid_max.
• A thread leak in an application: a service keeps spawning threads without joining them, and every thread counts against nproc.
• A runaway loop or a fork bomb spawning processes faster than they exit.
• A container hitting its cgroup pids.max, which caps processes inside that container independently of the host.

How to fix it

Stabilize

You may not be able to run external commands, so lean on shell builtins that do not fork.

1. If you have a second terminal already open, especially a root session, use it. Root usually falls under a different limit than the user that exhausted its quota.
2. Use builtins that run inside the current shell. echo and the kill builtin do not fork:

# kill is a bash builtin here, so this does not require a fork
kill -TERM %1

3. If you know the offending process group, signal it directly. Sending a signal to a negative PID targets the whole process group:

kill -TERM -12345

4. As a last resort on a leaking service, restarting or stopping its unit frees its slots at once.

Diagnose

Once you can run commands again, find what filled the table.

1. Count total threads on the system (each line is a thread):

ps -eLf | wc -l

2. Find which processes hold the most threads. nlwp is the thread count per process:

ps -eo pid,user,nlwp,comm --sort=-nlwp | head

3. Count processes per user to spot the offender:

ps -eo user= | sort | uniq -c | sort -rn | head

4. Check your current per-user limit and the system PID ceiling:

ulimit -u
cat /proc/sys/kernel/pid_max

5. For a systemd service, check its task accounting:

systemctl status your-service.service
systemctl show your-service.service -p TasksCurrent -p TasksMax

Fix

1. If the limit is genuinely too low, raise the per-user limit. Add a line to /etc/security/limits.conf (or a file in /etc/security/limits.d/):

# username  type  item   value
appuser     soft  nproc  8192
appuser     hard  nproc  16384

Users must log out and back in for this to take effect.

2. For a systemd service, raise its cap instead of the global one:

systemctl edit your-service.service
# add under [Service]:
#   TasksMax=16384
systemctl daemon-reload
systemctl restart your-service.service

3. If you are truly running out of PIDs, raise kernel.pid_max, and make it persistent:

sysctl -w kernel.pid_max=131072
echo 'kernel.pid_max = 131072' >> /etc/sysctl.d/99-pids.conf

4. If a container hit its limit, raise pids.max on the container (for Docker, docker run --pids-limit).

5. Then fix the application that leaked threads or processes. Raising a limit only delays the next failure if the leak is still there.

How to prevent it

• Alert on process and thread counts per host and per service, well before the limit.
• Set sane per-service limits with systemd TasksMax rather than one large global ceiling.
• Fix thread and process leaks at the source: ensure threads are joined and child processes are reaped.
• Use connection and worker pools with fixed sizes instead of spawning a thread or process per request.
• In containers, set an explicit pids.max so one container cannot starve the host.

When to call a senior engineer

If the process count climbs back to the limit within minutes of a restart, you have an active leak or a fork loop that needs to be traced in the application, not just contained at the OS. If the host itself is unresponsive and you cannot get a shell even as root, or if raising limits keeps buying only a short reprieve, that is the point to bring in help. Erzon engineers do this under incident conditions: get you a working shell, identify the leaking process, and fix the code path that spawns without bound so the limit stops being reached.