Fix: 1024 worker_connections are not enough

Nginx writes this to the error log when one of its worker processes has used every connection slot it is allowed and a new connection arrives anyway. New requests get dropped or stall, so the site feels down under load even though the machine is mostly idle. The default of 1024 connections per worker is a decades-old conservative default, not a recommendation.

1024 worker_connections are not enough

What this error means

Nginx handles traffic with a small number of worker processes, each serving many connections concurrently. The worker_connections directive caps how many simultaneous connections one worker may hold, and that count includes everything: client connections, keepalive connections idling between requests, and, on a reverse proxy, the connection to the upstream for every in-flight request. A proxied request therefore costs two slots, not one.

When a worker is at its cap, it cannot accept new connections, and Nginx logs this line. Clients experience refused connections, hangs, or 500-range errors from any load balancer in front. Because the ceiling is per worker, the error can appear while overall server resources look fine.

Common causes

• Real traffic growth past the default: 1024 connections per worker is tiny for a production proxy.
• A slow upstream making requests pile up, so each request holds its two connection slots far longer than normal.
• Long-lived connections (WebSockets, server-sent events, long polling) that occupy slots for minutes or hours.
• Generous keepalive settings holding thousands of idle client connections open.
• worker_processes set to a fixed low number instead of auto, shrinking total capacity on multi-core machines.
• A traffic spike or load test that simply exceeds the configured ceiling.

How to fix it

STABILIZE first. Raise worker_connections and the file descriptor limit together, then reload. This is the whole fix for the immediate incident.

1. Confirm the current limits and how close the workers are to them.

grep -E "worker_connections|worker_processes|worker_rlimit_nofile" /etc/nginx/nginx.conf
ss -s

2. Check the file descriptor limit actually applied to a running worker, because it silently caps whatever worker_connections promises.

ps -o pid= -C nginx | head -2
cat /proc/$(pgrep -f "nginx: worker" | head -1)/limits | grep "open files"

DIAGNOSE the shape of the load before raising anything. If ss -s shows tens of thousands of established connections and the upstream is answering fast, this is genuine concurrency and a bigger ceiling is correct. If connections are mostly waiting on a slow upstream, the ceiling is filling because requests are not completing, and the upstream latency is the real incident.

FIX by raising the connection ceiling and the descriptor limit together in nginx.conf:

worker_processes auto;

# per process descriptor limit; on a proxy allow 2x worker_connections
worker_rlimit_nofile 16384;

events {
    worker_connections 8192;
    multi_accept on;
}

If Nginx runs under systemd, the unit’s descriptor limit also applies. Set it and restart:

mkdir -p /etc/systemd/system/nginx.service.d
printf "[Service]\nLimitNOFILE=65535\n" > /etc/systemd/system/nginx.service.d/limits.conf
systemctl daemon-reload
nginx -t && systemctl restart nginx

Verify the new limit took effect on a fresh worker with the /proc/<pid>/limits check from step 2. With worker_processes auto on an 8-core machine, 8192 connections per worker gives you 65536 total slots, which covers roughly 32000 concurrent proxied requests plus keepalive idle connections.

How to prevent it

• Size worker_connections from measured peak concurrency times two (client plus upstream), with at least 50 percent headroom.
• Always set worker_rlimit_nofile alongside it; a raised ceiling without descriptors is a no-op.
• Use worker_processes auto so capacity scales with the cores you pay for.
• Alert on the ratio of active connections to total slots (stub_status makes this a one-line scrape) before it reaches 100 percent.
• Keep an eye on upstream latency; connection exhaustion is often the first visible symptom of a slowing backend.

When to call a senior engineer

Call for help when the error returns after limits are raised, when connection counts climb without matching request volume (a leak or an upstream that stopped answering), or when WebSocket-heavy traffic needs capacity planning across multiple proxy nodes. Erzon engineers can profile where connections actually spend their time, tune the full chain from kernel limits to upstream pools, and leave you with alerting that warns before the ceiling is hit rather than after.