Fix: 499 Client Closed Request

A 499 in the Nginx access log means the client closed the connection before Nginx could return a response. Nothing with this status is ever sent over the wire; it exists purely so the log records how the request ended. Scattered 499s are normal life on the internet (users close tabs). A cluster of them is a latency alarm: your backend is answering slower than callers are willing to wait.

499 Client Closed Request

What this error means

Nginx accepted the request and forwarded it to the upstream, and while the upstream was still working, the client hung up. Nginx logs the request with status 499 and, by default, closes the upstream connection too. Every 499 is therefore a race that latency lost: the caller’s timeout expired before your response arrived.

The important shift in thinking is that the client’s timeout is data. Browsers wait a long time, but programmatic callers do not: mobile apps, SDKs, and internal services typically time out between 1 and 30 seconds. When your p99 latency crosses a caller’s timeout, that caller’s requests start showing up as 499s, and often as retries right behind them.

Common causes

• A slow endpoint: a missing database index, an N+1 query pattern, or a slow third-party call inside the request path.
• A downstream caller with a shorter timeout than your actual response time, so it hangs up and usually retries.
• Retry storms: timeouts trigger retries, retries add load, load adds latency, latency adds more 499s.
• A load balancer or proxy in front of Nginx with an idle or response timeout shorter than your slowest endpoint.
• Users abandoning genuinely long operations (large exports, report generation) that should be asynchronous jobs.
• Deploy-time stalls, such as cold starts or connection pool warmup, clustering 499s right after each release.

How to fix it

STABILIZE first. If 499s are spiking together with load, pause or slow aggressive retriers if you control them, and scale out or restart the slow backend if it is degrading. Breaking the timeout-retry loop stops the amplification.

1. Find where the 499s concentrate and how long those requests ran before the client left. With $request_time in your log format:

awk '$9 == 499 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
awk '$9 == 499 {sum += $(NF); n++} END {if (n) print n " reqs, avg " sum/n "s before client gave up"}' /var/log/nginx/access.log

2. Reproduce the slow endpoint directly against the upstream and measure server-side latency without any client timeout in the way.

curl -s -o /dev/null -w "total: %{time_total}s ttfb: %{time_starttransfer}s\n" http://127.0.0.1:3000/api/report

DIAGNOSE by comparing three numbers: how long the endpoint actually takes, how long clients waited before hanging up (the request time on 499 lines), and the configured timeout of whatever calls you. If clients drop at a suspiciously uniform time such as 10.0 seconds, you have found a caller’s timeout, not human impatience. Uniform drops at a round number from one internal service means its HTTP client timeout is shorter than your endpoint’s real latency.

FIX the latency, because that is the root cause. Profile the endpoint, add the missing index, cache the repeated lookup, or move the slow third-party call out of the request path. Then align the timeout chain so each layer waits slightly less than the layer in front of it, giving your server a chance to answer or fail cleanly before the caller walks away:

location /api/ {
    proxy_pass http://app;
    proxy_connect_timeout 5s;
    proxy_send_timeout    15s;
    proxy_read_timeout    15s;
}

For operations that legitimately take longer than any sane HTTP timeout, return 202 Accepted with a job ID immediately and let the client poll or receive a webhook, instead of holding a connection open for a minute. Finally, make sure abandoned requests stop consuming the backend: leave proxy_ignore_client_abort at its default of off, and have the application honor cancellation when the socket closes.

How to prevent it

• Alert on p95 and p99 latency per endpoint, with thresholds below your callers’ known timeouts.
• Document the timeout of every hop (client, CDN, load balancer, Nginx, app, database) and keep them descending toward the origin.
• Give internal callers explicit timeouts and capped, jittered retries so a slow endpoint cannot trigger a storm.
• Convert anything slower than a few seconds into an asynchronous job with a status endpoint.
• Watch the 499 rate per caller; a step change usually means someone changed a timeout, and it is cheaper to hear about it from a graph than an incident.

When to call a senior engineer

Call for help when 499s spike under load and feed a retry storm you cannot break, when latency lives somewhere unclear in a chain of services and every team’s dashboard looks green, or when a critical flow needs redesigning from a long blocking request into an asynchronous pipeline. Erzon engineers can trace where the milliseconds actually go, align the full timeout chain, and rework the slow paths so callers stop hanging up on you.