Fix: socket() failed (24: Too many open files) while connecting to upstream

If Nginx is logging socket() failed (24: Too many open files) while connecting to upstream, a worker process tried to open a new socket to your backend and the kernel refused because the process already holds as many file descriptors as it is allowed. Requests that need a fresh upstream connection fail while the worker is at its ceiling, so users see intermittent 502s under load even though the backend is healthy. The fix is to raise the descriptor limit at every layer that caps it, not just inside nginx.conf.

2026/07/13 14:22:09 [crit] 2831#2831: *148213 socket() failed (24: Too many open files) while connecting to upstream, client: 10.0.4.12, server: app.example.com, request: "GET /api/orders HTTP/1.1", upstream: "http://127.0.0.1:3000/api/orders"

What this error means

Every socket, open file, and pipe a process holds is a file descriptor, and the operating system caps how many each process may have open at once. Errno 24 (EMFILE, “Too many open files”) is the kernel telling the Nginx worker it has reached that per-process cap and cannot open another descriptor. Because a reverse proxy needs a descriptor for each client connection and another for each upstream connection, a worker under real load consumes descriptors quickly. When it hits the limit, socket() fails and the request that triggered it cannot reach the backend.

This is a resource ceiling, not a bug in your backend. The backend is usually fine; Nginx simply cannot open the door to it.

Common causes

• The soft file descriptor limit for the Nginx worker is the default 1024, which is far too low for a busy proxy.
• worker_rlimit_nofile is unset or set below the concurrency the proxy actually handles.
• The systemd unit for Nginx has a low or default LimitNOFILE, so the hard limit caps whatever worker_rlimit_nofile tries to raise.
• A high number of keepalive connections to upstreams holds descriptors open longer than expected.
• A descriptor leak or a slow upstream keeps connections open, so descriptors accumulate faster than they are released.
• The value was raised in /etc/security/limits.conf only, which does not apply to services started by systemd.

How to fix it

STABILIZE first. If the proxy is dropping requests right now, reload Nginx to reset workers, which frees descriptors and buys a short window. This is a reset, not a cure; the limit is still low and the error returns under load.

sudo nginx -t && sudo systemctl reload nginx

DIAGNOSE the real ceiling. Confirm the limit the running worker actually has, so you know which layer is capping you. Read the worker’s own limits from the kernel:

cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

The “Soft Limit” and “Hard Limit” columns show what the worker is running with. Also confirm what the systemd unit grants and the current system-wide maximum:

systemctl show nginx -p LimitNOFILE
cat /proc/sys/fs/file-max

If the soft limit reads 1024 while your load needs far more, that is the ceiling you are hitting. Compare against actual usage to size the new value:

ls -1 /proc/$(pgrep -f 'nginx: worker' | head -1)/fd | wc -l

FIX at every layer. First, raise the limit in the systemd unit that launches Nginx, because worker_rlimit_nofile cannot exceed the hard limit set here. Use a drop-in so your change survives package upgrades:

sudo mkdir -p /etc/systemd/system/nginx.service.d
sudo tee /etc/systemd/system/nginx.service.d/limits.conf >/dev/null <<'EOF'
[Service]
LimitNOFILE=65535
EOF
sudo systemctl daemon-reload

Next, set worker_rlimit_nofile in the main context of nginx.conf, at or below the systemd hard limit, and give each worker enough connections to use it:

# nginx.conf (main context, not inside http or server)
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
}

Then restart Nginx so the new systemd limit and configuration both take effect. A reload is not enough here because the process-level limit is inherited at start:

sudo systemctl restart nginx

Finally, verify the worker now runs with the raised limit:

cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

If worker_connections is what you are exhausting instead, the log line differs (worker_connections are not enough); in that case raise worker_connections, but remember it must stay below the descriptor limit or you circle back to errno 24.

How to prevent it

• Set worker_rlimit_nofile explicitly in every deployment rather than trusting the 1024 default.
• Keep the systemd LimitNOFILE at or above worker_rlimit_nofile, applied through a drop-in that survives upgrades.
• Size both worker_connections and the descriptor limit to measured peak concurrency, with headroom.
• Monitor open descriptor counts per worker and alert before they approach the limit.
• Watch for descriptor leaks from slow or hung upstreams, which quietly consume the budget over time.
• Re-verify limits after any base image, kernel, or init system change, since defaults differ across distributions.

When to call a senior engineer

Call for help when the limit is already high but descriptors still climb without bound, which points to a leak or a stuck upstream rather than a sizing problem, or when raising limits pushes the bottleneck somewhere else and the proxy still drops requests under load. Erzon engineers can profile descriptor usage against real traffic, find the connection that never closes, and set limits that match how your proxy is actually used so the error stops recurring instead of moving.