Fix: No space left on device (Linux)

You have a service that just stopped writing, a failed deploy, or an application throwing ENOSPC, and the message is the same: No space left on device. The instinct is to start deleting files. Resist that for two minutes. Deleting the wrong thing during an incident is how a recoverable problem becomes data loss. First find where the space went, then reclaim it deliberately.

What this error means

Linux raises ENOSPC when a write cannot be completed because the target filesystem has no room. There are two distinct conditions that produce the exact same message, and they require different fixes.

The first is the obvious one: the filesystem is out of data blocks. The second is subtler: the filesystem is out of inodes. Every file, directory, and symlink consumes one inode, and a filesystem is created with a fixed number of them. If you have millions of tiny files you can exhaust the inode table while df -h still reports plenty of free gigabytes. Your first diagnostic job is to tell these two cases apart.

Common causes

• Runaway or unrotated logs filling /var/log, often from a service in a crash loop.
• The systemd journal (journald) growing without a size cap.
• Docker images, stopped containers, dangling volumes, and build cache accumulating over time.
• Package manager caches (apt, dnf, yum) and old cached packages never cleaned.
• Core dumps or large temporary files piling up in /tmp or a working directory.
• Inode exhaustion from millions of small files (session files, cache shards, mail queues) even when byte usage looks fine.

How to fix it

Stabilize

1. Confirm which condition you have. Check bytes and inodes for every mount:

df -h
df -i

If a filesystem shows 100 percent under Use in the first command, it is a byte problem. If it shows 100 percent under IUse in the second, it is an inode problem. Note the exact mount point that is full before touching anything.

2. If it is a byte problem, find the largest directories on that filesystem. The -x flag keeps du on a single filesystem so it does not wander into other mounts:

du -xh --max-depth=1 / | sort -h

Repeat, descending into the largest directory each time, or use ncdu -x / for an interactive view. This tells you where the space actually is instead of guessing.

Diagnose

3. Before deleting, check for deleted-but-open files. Space is not freed while a process still holds a deleted file open, which is the single most confusing part of this incident:

sudo lsof +L1

sudo lsof | grep deleted

If you see a large file listed as deleted with a live PID, the fix is to restart that process, not to delete more files.

4. If df -i showed inode exhaustion, locate the directory holding the files. Count entries per top-level directory:

for d in /*; do echo "$d"; find "$d" -xdev | wc -l; done

Descend into the offending path the same way until you find the directory with millions of entries.

Fix

5. Reclaim the space at the source, matched to what you found:

# Cap and vacuum the systemd journal
sudo journalctl --vacuum-size=200M

# Reclaim Docker space (review first, then prune)
docker system df
docker system prune -a --volumes

# Truncate a large active log without deleting it
sudo truncate -s 0 /var/log/large-service.log

6. If you are genuinely out of capacity and cleanup is not enough, grow the volume. On LVM with an ext4 filesystem:

sudo lvextend -L +10G /dev/mapper/vg-root
sudo resize2fs /dev/mapper/vg-root

For XFS, replace the last step with sudo xfs_growfs / (XFS grows online by mount point). On a cloud instance, expand the underlying disk in the provider console first, then run these to consume the new space.

How to prevent it

• Alert on disk usage by trend, not just a static threshold, so you get warning days ahead of full.
• Add a separate inode alert (df -i), since byte monitoring alone will miss inode exhaustion entirely.
• Enforce log rotation with logrotate and set SystemMaxUse in journald.conf to cap the journal.
• Schedule Docker cleanup and package cache cleanup on hosts that build or pull regularly.
• Give logs, /tmp, and application data their own volumes so a runaway writer cannot take down the root filesystem.

When to call a senior engineer

Call for help if the root filesystem is full and the box will not accept SSH sessions, if a database refused writes and you are unsure whether it is consistent, or if inode exhaustion is coming from a path you do not recognize and deleting feels risky. Erzon puts a senior engineer on the incident with you, finds the space safely, and leaves you with monitoring so it does not recur.