Fix: upstream prematurely closed connection while reading response header from upstream

If your proxy is logging upstream prematurely closed connection while reading response header from upstream, the backend application closed the TCP connection before it sent even the first line of an HTTP response. The proxy had a request in flight and the socket went away underneath it, so it returns a 502 to the client. This is almost never a proxy bug. Something in the application, its memory budget, its timeouts, or its connection handling ended the connection early, and the fix is to find which one.

2026/07/13 15:04:41 [error] 3120#3120: *90412 upstream prematurely closed connection while reading response header from upstream, client: 203.0.113.9, server: api.example.com, request: "POST /v1/reports HTTP/1.1", upstream: "http://127.0.0.1:8000/v1/reports", host: "api.example.com"

What this error means

The proxy opened (or reused) a connection to the upstream, sent the request, and began waiting for the response headers. Before those headers arrived, the upstream closed the connection: it sent a TCP FIN or RST, or the process holding the socket went away. From the proxy’s point of view the exchange ended with nothing usable, so it cannot forward a real response and serves a 502.

The key fact is timing. The connection did not stay open too long; it ended too soon. That points at the upstream deciding, or being forced, to hang up mid-request rather than at any slowness on the wire.

Common causes

• The application process crashed or was restarted mid-request, so every in-flight connection was severed.
• The kernel OOM killer terminated the app worker because it exceeded available memory, visible in dmesg.
• The application’s own request timeout (for example a Gunicorn, Puma, or PHP-FPM worker timeout) is shorter than the request and kills the worker before it responds.
• A keepalive mismatch: the proxy reuses an upstream connection that the app had already decided to close, so the first byte it sends lands on a dying socket.
• A worker recycle setting (max requests per worker) fired at the wrong moment and closed the connection during a request.
• A request body larger than the app or proxy allows caused the app to close the connection instead of responding.

How to fix it

STABILIZE first. If requests are failing now, confirm the app process is actually up and restart it if it is crash looping. That restores service while you find why connections are being closed.

sudo systemctl status your-app
sudo systemctl restart your-app

DIAGNOSE by reading the application’s side of the story, because the proxy log only sees the closed socket. Start with the app logs and the system journal, looking for crashes, worker timeouts, or restarts that line up with the 502s:

sudo journalctl -u your-app --since "15 minutes ago" --no-pager

Check for an OOM kill, which is the most commonly missed cause. If the kernel killed a worker, it is recorded here:

sudo dmesg -T | grep -i -E 'killed process|out of memory'

Confirm the app answers directly, bypassing the proxy, to prove the close comes from the app and to see how long a real request takes:

curl -w '\nstatus=%{http_code} time=%{time_total}s\n' -s -o /dev/null -X POST http://127.0.0.1:8000/v1/reports

If a slow request is being cut off, compare how long it runs against the app’s worker timeout. For example, with Gunicorn:

ps -o cmd= -C gunicorn | tr ' ' '\n' | grep -A1 -- --timeout

FIX according to the cause.

1. Crash or OOM. Raise the worker memory budget or reduce per-request memory, and set the process manager to restart cleanly. If it is an OOM kill, give the service more memory or lower concurrency so total usage fits.

2. App timeout shorter than the request. Raise the application worker timeout above the real request duration. For Gunicorn, the worker timeout must exceed the slowest legitimate request:

gunicorn app:app --timeout 120 --graceful-timeout 30 --workers 4

3. Keepalive mismatch. Make the application’s keepalive idle timeout longer than the proxy’s, and let the proxy retry idempotent requests on a closed connection. In Nginx:

upstream app {
    server 127.0.0.1:8000;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_next_upstream error timeout http_502;
    }
}

4. Worker recycling mid-request. If you cap requests per worker (for example --max-requests), add jitter so all workers do not recycle together, and ensure the process manager drains in-flight requests before closing.

5. Request body too large. If large uploads trigger the close, raise client_max_body_size in Nginx and the matching body limit in the app so the app can accept and respond instead of hanging up.

How to prevent it

• Alert on application crashes, restarts, and OOM kills, not just on the proxy’s 502 count.
• Keep the application worker timeout comfortably above the slowest real request, and set graceful shutdown so deploys drain connections.
• Align keepalive idle timeouts so the app never closes a connection the proxy is about to reuse; keep the app’s longer than the proxy’s.
• Give workers enough memory headroom and cap concurrency so the OOM killer never fires.
• Enable proxy retries for idempotent methods so a single stale connection does not surface as a user-visible 502.
• Load test at realistic concurrency so keepalive and recycling edge cases appear before production does.

When to call a senior engineer

Call for help when premature closes recur under load with no obvious crash in the app logs, when they only appear behind the proxy and never against the app directly, or when memory, timeouts, and keepalive all look correct yet 502s persist. Erzon engineers can correlate proxy and application logs, capture the connection close on the wire, and identify whether the app is dying, timing out, or being handed a dead keepalive socket, then leave you with a configuration that holds under real traffic.