Fix: could not extend file: No space left on device (PostgreSQL)

Your database has stopped accepting writes and the log is full of could not extend file ... No space left on device. The data volume is full, and the priority is to free space without touching anything PostgreSQL owns.

What this error means

PostgreSQL tried to grow a relation file (a table, index, or temp file) and the filesystem refused because the volume holding PGDATA has no free blocks. Once this happens, most write transactions fail, but reads and connections usually still work. The cluster is not corrupted by this error itself. The danger comes from what people do next: deleting files inside the data directory to reclaim space, which does corrupt the cluster.

Common causes

• WAL accumulation because a replication slot is inactive but still holding segments PostgreSQL is not allowed to recycle.
• A failing archive_command (wrong path, full archive target, bad credentials), so WAL cannot be archived and therefore cannot be removed.
• A long running or idle-in-transaction session pinning an old transaction ID, which prevents WAL recycling and blocks vacuum from cleaning dead rows.
• Table and index bloat from heavy updates and deletes with vacuum falling behind.
• Large temporary files in base/pgsql_tmp from oversized sorts, hash joins, or index builds spilling to disk.
• Genuine data growth or an undersized volume, plus logs or backups written to the same disk.

How to fix it

Stabilize first

1. Confirm which volume is full and how full. Do not guess.

df -h
df -h /var/lib/pgsql/data   # point at your actual PGDATA mount

2. Find the largest consumers outside PGDATA. Old logs, core dumps, stale backups, and package caches are safe to remove and often free enough space to get writes flowing again.

du -h -d1 /var | sort -h | tail -20
journalctl --disk-usage
sudo journalctl --vacuum-size=200M

Do NOT delete anything inside PGDATA, especially pg_wal/ or base/. Removing a WAL segment or a relation file by hand can leave the cluster unable to start or silently corrupt data. On RDS, Aurora, and Cloud SQL you have no shell access to the data directory anyway; use the console to raise storage instead (see below).

3. If nothing outside is safe to delete and the volume can be grown, expand it. On a self-managed cloud VM, grow the block device, then extend the filesystem live.

sudo growpart /dev/nvme1n1 1
sudo resize2fs /dev/nvme1n1p1     # ext4
sudo xfs_growfs /var/lib/pgsql/data   # xfs

Diagnose the real cause

4. Check WAL size and whether a replication slot is pinning it.

SELECT pg_size_pretty(sum(size)) AS wal_total FROM pg_ls_waldir();

SELECT slot_name, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots
ORDER BY retained DESC;

An inactive slot with a large retained value is the usual culprit.

5. Check for archiver failures and long transactions.

SELECT * FROM pg_stat_archiver;   -- look at failed_count and last_failed_time

SELECT pid, state, age(backend_xid) AS xid_age, query_start, left(query,80)
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY xid_age DESC NULLS LAST;

Fix it properly

6. If an unused replication slot is the cause and you are certain no replica or CDC consumer needs it, drop it so PostgreSQL can recycle WAL.

SELECT pg_drop_replication_slot('old_replica_slot');

If a replica does need it, fix the replica or its connectivity instead of dropping the slot.

7. If the archive_command is failing, fix the target or command. Once archiving succeeds, run a checkpoint so WAL can be recycled.

CHECKPOINT;

8. If pg_wal itself needs to live on a different volume long term, stop PostgreSQL, move the directory to a larger disk, and replace it with a symlink. Never do this while the server is running.

sudo systemctl stop postgresql
sudo rsync -a /var/lib/pgsql/data/pg_wal/ /mnt/waldisk/pg_wal/
sudo mv /var/lib/pgsql/data/pg_wal /var/lib/pgsql/data/pg_wal.old
sudo ln -s /mnt/waldisk/pg_wal /var/lib/pgsql/data/pg_wal
sudo chown -h postgres:postgres /var/lib/pgsql/data/pg_wal
sudo systemctl start postgresql

Remove pg_wal.old only after the cluster starts cleanly and confirms the symlink.

9. Once there is headroom, address bloat. Plain VACUUM (VERBOSE) tablename; reclaims dead rows for reuse but does not shrink the file. VACUUM FULL returns space to the OS but needs free room equal to the table size and takes an exclusive lock, so run it during a window, not mid-incident.

On RDS and Aurora, enable storage autoscaling and raise the max threshold, or modify the instance to add storage. Aurora storage grows automatically, so a full-volume error there points to local temp or log storage, not the main data volume.

How to prevent it

• Monitor free disk, WAL directory size, and replication slot lag with alerts well before 100 percent.
• Set max_slot_wal_keep_size so an abandoned slot cannot fill the disk indefinitely.
• Keep logs, backups, and archives on a separate volume from PGDATA.
• Ensure autovacuum is keeping up; tune it on high-churn tables rather than disabling it.
• On RDS and Cloud SQL, enable storage autoscaling with a sane ceiling.
• Alert on pg_stat_archiver.failed_count increasing.

When to call a senior engineer

Call for help if the cluster will not restart, if you see any relation or WAL file errors after freeing space, or if the only slot holding WAL is feeding a replica you cannot lose. Recovering a cluster where files were deleted from PGDATA is delicate and time sensitive, and the wrong next command can turn a full disk into data loss. Erzon can join a live incident and stabilize the database without risking integrity.