Fix: Too many open files (Linux)

Too many open files means a process asked the kernel for a new file descriptor and was refused because it already holds its maximum. Sockets count, so on servers this usually surfaces as failed connections, not failed file opens: accepts fail, outbound requests error, and log lines pile up. The limit is per process, and there are two questions to answer: is the limit simply too low for a legitimately busy service, or is something leaking descriptors?

Too many open files

What this error means

Everything a Linux process interacts with through the kernel, files, sockets, pipes, epoll instances, gets a file descriptor. Each process has a soft limit on how many it can hold at once (RLIMIT_NOFILE, historically 1024), and the kernel returns EMFILE, printed as “Too many open files”, when an open, socket, accept, or pipe call would exceed it. There is also a system-wide ceiling, fs.file-max, but the per-process limit is almost always the one you hit. Crucially, limits are inherited at process start: your shell’s ulimit -n says nothing about what systemd or a container runtime gave your service.

Common causes

• A default limit of 1024 on a service that legitimately handles thousands of concurrent sockets.
• A descriptor leak: code opens files, sockets, or HTTP responses and never closes them on some path, so usage climbs until the cap.
• Connection pools or keep-alive settings that hold far more idle sockets than intended.
• The limit raised in the shell or in /etc/security/limits.conf, but the service runs under systemd or in a container that never sees those settings.
• Watchers and epoll-heavy tooling (log shippers, file watchers) multiplying descriptors across many files.

How to fix it

STABILIZE first. Restarting the affected process releases every descriptor it holds and restores service instantly. If it is a leak, this buys you hours; do the diagnosis before those hours are gone.

systemctl restart myservice

1. DIAGNOSE the real limit and real usage for the running process, from /proc, not from your shell.

pgrep -f myservice
cat /proc/<pid>/limits | grep 'open files'
ls /proc/<pid>/fd | wc -l

2. See what the descriptors actually are. Thousands of sockets to one destination, or thousands of the same log file, names the leak.

ls -l /proc/<pid>/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20
ss -tnp | grep '<pid>' | awk '{print $5}' | sort | uniq -c | sort -rn | head

3. FIX the limit where the process actually starts. For a systemd service:

systemctl edit myservice

[Service]
LimitNOFILE=65536

systemctl daemon-reload
systemctl restart myservice

4. For containers, set the limit at the runtime. With Docker:

docker run --ulimit nofile=65536:65536 myimage

5. If step 2 showed a leak (usage climbing with no matching traffic growth), the fix is in code: close descriptors on every path, including errors. In most languages that means with-blocks, try-with-resources, or defer around every open and every HTTP response body. Raise the limit anyway so the service survives while the fix ships, and watch the count:

watch -n5 'ls /proc/<pid>/fd | wc -l'

Flat under load means capacity was the issue. Steadily climbing means the leak is still there.

How to prevent it

• Set explicit LimitNOFILE in every service unit and container spec for network-facing services; never rely on distribution defaults.
• Monitor per-process descriptor counts and alert at around 80 percent of the limit, so you see the climb before the errors.
• Close resources with language constructs that guarantee it (context managers, defer, try-with-resources), and lint for unclosed handles.
• Bound connection pools and idle keep-alive counts to intentional numbers.
• Load test at production connection counts, where descriptor exhaustion shows up long before it does in staging.

When to call a senior engineer

Call for help when descriptor usage climbs and nobody can find which code path leaks, when limits look right everywhere but the process still hits the wall, or when the errors arrive with load spikes and take revenue with them. Erzon engineers can map every descriptor a process holds back to the code that opened it, fix the leak rather than the symptom, and set limits and monitoring so this error stops being how you discover growth.