Fix: 502 Bad Gateway (Nginx)

A 502 Bad Gateway from Nginx means it forwarded a request to your backend and got nothing usable back: the connection was refused, reset, closed before a full response arrived, or the response violated what Nginx will accept. The status page users see is generic by design. The specific failure is written to the Nginx error log, and reading that one line is the difference between a five minute fix and an hour of guessing.

502 Bad Gateway

What this error means

Nginx is acting as a reverse proxy. For every request it opens (or reuses) a connection to an upstream, sends the request, and expects a well formed HTTP response. A 502 is Nginx reporting that this exchange failed at the protocol level. The error log entry that accompanies it names the precise mode: connect() failed (111: Connection refused) means nothing is listening at the upstream address, no live upstreams means every server in the upstream block is marked failed, upstream prematurely closed connection means the backend died or hung up mid-response, and upstream sent too big header means the response headers exceeded the proxy buffers.

The 502 itself is a symptom. The upstream, or the path to it, is the problem.

Common causes

• The upstream process is dead or crashing: connection refused because nothing listens on the port or socket.
• The upstream is overloaded: its listen backlog is full or its workers are exhausted, so it refuses or resets new connections.
• Wrong upstream address or port in proxy_pass or the upstream block, or a DNS name that resolved to a stale IP at startup.
• The upstream closes connections early: an application timeout shorter than the request, a worker killed by the OOM killer, or a keep-alive connection reused after the backend already closed it.
• Response headers larger than the proxy buffers, common with large cookies or verbose frameworks, logged as upstream sent too big header while reading response header from upstream.
• A protocol mismatch, such as proxying HTTP to a port that speaks HTTPS or FastCGI.

How to fix it

STABILIZE first. If every request is failing, confirm whether the backend process is up and restart it if it is down. That alone often restores service while you investigate why it died.

1. Read the error log. This names the exact failure and the upstream address involved.

tail -50 /var/log/nginx/error.log

2. Confirm what is actually listening, and test the upstream directly, bypassing Nginx.

ss -ltnp | grep 3000
curl -i http://127.0.0.1:3000/healthz

DIAGNOSE based on the log line. Connection refused plus nothing in ss means the app is down: check its own logs and the system journal for a crash, and check dmesg for an OOM kill. If the app is up but Nginx still fails, compare the upstream definition to the real bind address and port. upstream prematurely closed connection points at the backend dying mid-request: look for crashes correlated with specific URLs or payload sizes. upstream sent too big header is a buffer problem in Nginx itself.

FIX according to the cause. Make sure the upstream block matches reality and let Nginx retry the next server on connection errors:

upstream app {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=10s;
    server 127.0.0.1:3001 max_fails=3 fail_timeout=10s;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_next_upstream error timeout http_502;
    }
}

For the too big header case, raise the buffer that holds the response headers:

proxy_buffer_size   16k;
proxy_buffers       8 16k;
proxy_busy_buffers_size 32k;

If the upstream is crashing under load, no Nginx setting fixes that. Restart it to stabilize, then treat the crash as the real incident: check memory limits, connection pool exhaustion, and unhandled errors in the application.

How to prevent it

• Run more than one upstream instance and let proxy_next_upstream fail over on connection errors.
• Deploy with a zero-downtime pattern: start the new process, health check it, shift traffic, then stop the old one.
• Alert on the upstream error rate in the Nginx error log, not just on the 502 count in access logs.
• Keep proxy buffers sized for your largest real response headers, and keep cookies small.
• Monitor upstream process restarts and OOM kills so a crash loop cannot hide behind a load balancer.

When to call a senior engineer

Call for help when 502s recur under load and the upstream keeps dying without an obvious crash log, when failures only appear behind Nginx and never against the app directly, or when a deploy pipeline keeps causing brief outages nobody can eliminate. Erzon engineers can trace the failing hop with packet captures and log correlation, find the process that is actually failing, and leave you with a proxy and deploy setup that does not serve 502s again.