Fix: upstream timed out (110: Connection timed out) (Nginx 504)

A 504 with upstream timed out (110: Connection timed out) while reading response header from upstream means Nginx forwarded a request to your backend and gave up waiting because the backend did not respond in time. Nginx is healthy here. It is telling you, accurately, that something behind it is too slow or unreachable. Treat the 504 as a report about the upstream, not a problem with Nginx itself.

What this error means

Nginx is acting as a reverse proxy. When a request arrives it opens a connection to an upstream (your application server, another service, or a socket) and waits for a response. Three timeouts govern that wait: proxy_connect_timeout (time to establish the connection), proxy_send_timeout (time to send the request body), and proxy_read_timeout (time to receive the response). If any of these elapses, Nginx stops waiting and returns 504.

The exact phase in the log message matters. “while connecting to upstream” means the connection never opened. “while reading response header from upstream” means the connection opened but the application did not answer in time. That single phrase tells you whether to look at reachability or at slow code.

Common causes

• A slow upstream response: a slow database query, a blocking external API call, or a long garbage collection pause holds the request past proxy_read_timeout.
• The upstream is at capacity: all worker threads or processes are busy, or the database connection pool is exhausted, so requests queue and time out under load.
• The upstream is down or unreachable, or the address in the upstream block is wrong, which produces a connect timeout rather than a read timeout.
• A genuinely long request, such as report generation or a large export, that needs more time than the default allows.
• Keepalive misconfiguration between Nginx and the upstream, causing connection churn or reuse of dead connections.

How to fix it

Stabilize

1. Confirm the blast radius. Check whether every endpoint is timing out or only one, and whether it correlates with traffic.

sudo tail -f /var/log/nginx/error.log | grep "upstream timed out"

2. Check whether the upstream is even alive from the Nginx host.

curl -sv -o /dev/null http://127.0.0.1:8080/health

If this hangs or refuses the connection, you have a reachability or capacity problem, not a tuning problem.

Diagnose

1. Read the error log and note the phase and the upstream address.

sudo grep "upstream timed out" /var/log/nginx/error.log | tail -n 20

2. Measure real latency end to end so you know how slow “slow” is.

curl -w "connect:%{time_connect} ttfb:%{time_starttransfer} total:%{time_total}\n" \
  -o /dev/null -s https://example.com/slow-endpoint

3. Check the backend. Read the application logs at the same timestamps as the 504 entries, and inspect slow queries in your database (for example the slow query log or pg_stat_activity). The goal is to find what the request was waiting on.

Fix

1. Fix the slow work first. Add the missing index, cache the expensive call, or make the blocking dependency non-blocking. This resolves most 504s without touching Nginx.

2. Add capacity where the upstream is saturated: more application workers, a larger database connection pool, or another backend instance in the upstream block.

3. Configure keepalive and sensible timeouts. Only after the backend is healthy should you raise a timeout, and only for endpoints that truly need it.

upstream app_backend {
    server 127.0.0.1:8080;
    keepalive 32;
}

server {
    location /api/ {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 5s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;
    }

    location /api/reports/ {
        proxy_pass http://app_backend;
        proxy_read_timeout 120s;
    }
}

4. Test the configuration and reload without dropping connections.

sudo nginx -t && sudo systemctl reload nginx

Note that keepalive in the upstream requires proxy_http_version 1.1 and an empty Connection header, otherwise Nginx opens a new connection per request.

How to prevent it

• Monitor upstream latency and error rate, and alert on rising p95 response time before it crosses the timeout.
• Plan capacity against real concurrency: size worker counts and database pools for peak load, not average load.
• Move long running work (reports, exports, emails) to a background queue and return quickly, rather than holding an HTTP request open.
• Add upstream health so traffic drains from a failing backend instead of timing out against it.
• Keep the global proxy_read_timeout modest and grant longer timeouts per location only where justified.

When to call a senior engineer

If the 504s are intermittent under production load, if raising timeouts only shifts the failure, or if the slow path crosses several services and you cannot isolate which one is stalling, that is a systemic capacity or concurrency problem rather than a config typo. If it is happening live and affecting customers, bring in someone who can read the logs, the metrics, and the backend together and fix the root cause safely.