Fix: recv() failed (104: Connection reset by peer)

A recv() failed (104: Connection reset by peer) error, ECONNRESET in most runtimes, means the process tried to read from a TCP connection and the other end answered with an RST packet instead of data. The peer did not close the connection politely with a FIN. It either crashed, was told by its kernel that the connection no longer exists, or deliberately aborted it. Resets are almost never random. They follow a pattern, and the pattern identifies the cause.

recv() failed (104: Connection reset by peer) while reading response header from upstream

What this error means

TCP connections are supposed to end with a FIN handshake. An RST is the abrupt alternative: it means “I have no such connection, stop talking to me.” You receive Connection reset by peer when your side still believed the connection was alive and the packet you sent, or the read you attempted, hit a peer that disagreed. The three classic reasons the peer disagrees: its process died while the connection was open, its kernel or an intermediary (load balancer, NAT, firewall) discarded the connection state after an idle period, or it closed the socket with unread data in the buffer, which the kernel converts into an RST.

Which side logs the error matters. Nginx logging recv() failed from an upstream means the backend reset the proxy. Clients logging ECONNRESET against your API means your side, or something in front of it, reset them.

Common causes

• The upstream process crashed mid-request: a segfault, an unhandled exception that killed the worker, or the OOM killer. The kernel resets every open connection the process held.
• Keep-alive mismatch: the client or proxy pool keeps idle connections longer than the server does, then reuses a connection the server already closed. Node’s default keepAliveTimeout of 5 seconds behind a load balancer with a 60 second idle timeout is the textbook case.
• Load balancer or NAT idle timeout: the middlebox silently drops connection state after its idle window, and the next packet on that flow gets an RST or a black hole followed by a reset.
• The server closed the connection while the request body was still arriving, for example rejecting an oversized upload, which surfaces to the sender as a reset.
• A full listen backlog on an overloaded server, which on some configurations resets new connections instead of queueing them.

How to fix it

STABILIZE first. If resets are spiking right now, check whether the upstream process is crash looping and restart or roll back the deploy that started it. Enable retries for idempotent requests at the proxy so users stop seeing errors while you dig.

1. Establish the pattern. Constant rate, bursts aligned with deploys or crashes, or a regular interval that matches an idle timeout?

grep "104: Connection reset by peer" /var/log/nginx/error.log | awk '{print $1, $2}' | uniq -c | tail -20

2. Check whether the upstream died at those timestamps.

journalctl -u myapp -S "1 hour ago" | grep -iE "killed|panic|fatal|segfault"
dmesg -T | grep -i "out of memory"

DIAGNOSE the mismatch if nothing crashed. List the idle and keep-alive timeouts at every hop: client pool, load balancer, proxy, application server. The rule that must hold is that each server-side timeout is longer than the idle timeout of whatever sits in front of it, so the side that opened the connection is always the side that closes it. If the intervals in the reset log match one of these timeouts, you have found the hop. When the path is still unclear, a packet capture shows exactly who sends the RST:

tcpdump -i any -nn 'tcp[tcpflags] & tcp-rst != 0 and port 3000'

FIX the specific cause. For a crashed upstream, fix the crash; resets are just the shrapnel. For the keep-alive mismatch, raise the application’s keep-alive window above the load balancer’s idle timeout. With an AWS ALB at its default 60 second idle timeout, a Node server needs:

const server = app.listen(3000);
server.keepAliveTimeout = 65000;
server.headersTimeout   = 66000;

On the proxy side, keep Nginx’s pooled upstream connections fresher than the backend’s timeout so Nginx never reuses a dead connection:

upstream app {
    server 127.0.0.1:3000;
    keepalive 32;
    keepalive_timeout 60s;
}

location / {
    proxy_pass http://app;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_next_upstream error timeout;
}

proxy_next_upstream error lets Nginx transparently retry a reset connection against another upstream, which absorbs the rare legitimate reset without surfacing it to users.

How to prevent it

• Document the idle and keep-alive timeout at every hop and enforce the ordering: each layer’s server timeout exceeds the idle timeout of the layer in front of it.
• Alert on upstream process restarts and OOM kills, since crashes always produce reset storms.
• Drain connections on shutdown: stop accepting, finish in-flight requests, then exit, so deploys do not reset live connections.
• Enable retries for idempotent requests at the proxy or client, with a small budget so retries cannot amplify an outage.
• Watch the reset rate as a metric. A low steady baseline is normal on the internet; a step change is a real event.

When to call a senior engineer

Call for help when resets persist after the timeouts are aligned, when packet captures are needed across cloud load balancers you cannot tap directly, or when the resets are the visible edge of an upstream that keeps dying under load. Erzon engineers can capture and read the traffic, map every timeout on the path, and stop the resets at the hop that actually causes them.