Fix: SSL SYSCALL error: EOF detected

SSL SYSCALL error: EOF detected means the TCP connection under your TLS session to PostgreSQL was closed abruptly, with no proper shutdown from either side. The client was waiting to read and the socket simply ended. This is not really a TLS problem: the encryption layer is only reporting that the pipe beneath it vanished. The cause is either the server side dying (a crashed or killed backend) or something in between (a load balancer, firewall, or NAT gateway) cutting the connection.

SSL SYSCALL error: EOF detected

What this error means

libpq raises this error when a read on the encrypted socket returns end of file unexpectedly. Three broad stories produce it. First, the backend process serving your session died: it crashed on a bug, was killed by the kernel OOM killer, or the whole server restarted. Second, the server deliberately ended the session, for example a firing idle_in_transaction_session_timeout or an administrator running pg_terminate_backend. Third, a middlebox on the network path (cloud load balancer, NAT gateway, stateful firewall, VPN) dropped the TCP connection, usually because it sat idle past the device’s timeout during a long query or an idle pool slot. The client cannot tell these apart, but the server and kernel logs can.

Common causes

• The kernel OOM killer terminated a PostgreSQL backend during a memory hungry query, taking the connection with it.
• A backend crash (segfault or assertion) that forces PostgreSQL to reinitialize and drop all sessions.
• A load balancer or NAT gateway with an idle timeout shorter than your longest query or your pool’s idle time.
• Server side timeouts such as idle_in_transaction_session_timeout closing sessions the client thought were alive.
• A database restart or failover (maintenance window, managed service patching) while clients held open connections.
• Connection pools handing out stale connections that were closed on the server side while sitting idle.

How to fix it

STABILIZE first. If errors are continuous, confirm the server is up and accepting connections; if a backend crash caused a restart, the immediate priority is that PostgreSQL came back cleanly.

1. DIAGNOSE: check the PostgreSQL log at the exact time of the client error. Crashes and terminations are explicit there.

tail -200 /var/log/postgresql/postgresql-*.log | grep -E "terminated|crash|FATAL|reinitializing"

2. DIAGNOSE: check for the OOM killer on the database host.

dmesg -T | grep -i -E "out of memory|oom|killed process"

3. DIAGNOSE: if the server log is silent at that timestamp, the connection died in transit. Test whether failures correlate with connection idle time or query duration, and compare a direct connection against the path through the proxy.

psql "host=db.internal port=5432 sslmode=require" -c "SELECT pg_sleep(600), 1;"

If the direct connection survives ten idle minutes but the proxied one does not, the middlebox timeout is your cause.

4. FIX a killed backend: if the OOM killer fired, lower the memory a single query can consume and give the host headroom. work_mem applies per sort or hash node, so a handful of big queries can multiply it dramatically.

ALTER SYSTEM SET work_mem = '32MB';
SELECT pg_reload_conf();

5. FIX the idle drop: enable TCP keepalives aggressive enough to beat the middlebox timeout, on the server and in the client connection string.

ALTER SYSTEM SET tcp_keepalives_idle = 60;
ALTER SYSTEM SET tcp_keepalives_interval = 10;
ALTER SYSTEM SET tcp_keepalives_count = 6;
SELECT pg_reload_conf();

psql "host=db.internal dbname=app sslmode=require keepalives=1 keepalives_idle=60 keepalives_interval=10"

Also raise the idle timeout on the load balancer itself where you control it.

6. FIX the pool: configure your connection pool to test or recycle connections older than the shortest timeout on the path, so the application never picks up a dead socket.

How to prevent it

• Keep TCP keepalives (server, client, and pooler) below the idle timeout of every device on the path.
• Set pool max_lifetime and idle recycle times shorter than load balancer timeouts.
• Give the database host enough memory headroom that the OOM killer never chooses PostgreSQL, and consider disabling memory overcommit on dedicated database hosts.
• Alert on backend crashes and server reinitializations, not just on availability.
• Make applications treat dropped connections as retryable: reconnect and rerun idempotent work instead of surfacing the raw error.
• During planned failovers and patching, drain connections first so clients reconnect cleanly.

When to call a senior engineer

Call for help when connections keep dying and neither the server log nor the kernel log explains it, when failures follow a pattern nobody can reproduce, or when an OOM crash loop points at queries whose memory use you cannot pin down. Erzon engineers can correlate client, server, and network evidence, capture the failing path with tcpdump when needed, and leave you with keepalive, pooling, and memory settings that hold up in production.