Fix: ENOSPC: System limit for number of file watchers reached

ENOSPC: System limit for number of file watchers reached is one of the most misleading errors on Linux, because it names disk space when the real limit is the number of files the kernel will let a user watch through inotify. It shows up when a dev server, bundler, or log tailer tries to watch a large tree of files and crosses fs.inotify.max_user_watches. Your disk is fine. The fix is to raise the watch limit and persist it.

ENOSPC: System limit for number of file watchers reached

What this error means

inotify is the kernel facility that lets a program get notified when files change. Every watched path consumes one watch slot, and each user has a cap on the total, controlled by the sysctl fs.inotify.max_user_watches. When a process asks to add a watch beyond that cap, the kernel refuses with the errno ENOSPC, which shares the “no space” error code even though nothing is out of disk. Tools that watch whole project trees, webpack, vite, nodemon, chokidar, and log shippers, add thousands of watches at once, so they blow through a low default (often 8192 on older systems) quickly. There is a second related limit, fs.inotify.max_user_instances, which caps how many separate inotify instances one user may open.

Common causes

• fs.inotify.max_user_watches left at a low distribution default while a bundler or dev server watches a large source tree.
• A monorepo or a project with a huge node_modules that is being watched recursively.
• Many pods or containers on one node, all running watchers, sharing the host’s inotify limits under one user id.
• Multiple watch-heavy tools running at once (a dev server, a test watcher, and a log tailer) adding up past the cap.
• Hitting fs.inotify.max_user_instances instead, when many small tools each open their own inotify instance.

How to fix it

STABILIZE first. The value applies immediately once set, so raising the limit and restarting the affected process clears the error without a reboot.

1. DIAGNOSE the current limit and how close you are to it.

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances

2. See who is consuming watches, so you know whether the limit is genuinely too low or one process is watching far more than it should.

find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | \
  cut -d/ -f3 | sort | uniq -c | sort -rn | head

3. FIX the limit at runtime to unblock immediately.

sudo sysctl fs.inotify.max_user_watches=524288

4. Persist it so a reboot does not revert to the low default. Write a drop-in file under /etc/sysctl.d, then reload:

echo 'fs.inotify.max_user_watches=524288' | \
  sudo tee /etc/sysctl.d/60-inotify.conf
sudo sysctl -p /etc/sysctl.d/60-inotify.conf

5. If step 1 showed max_user_instances was the constraint (many tools, each opening its own instance), raise that too and persist it the same way:

echo 'fs.inotify.max_user_instances=1024' | \
  sudo tee -a /etc/sysctl.d/60-inotify.conf
sudo sysctl -p /etc/sysctl.d/60-inotify.conf

6. Restart the process that failed (your dev server, watcher, or log shipper) so it picks up the new ceiling.

How to prevent it

• Ship a raised fs.inotify.max_user_watches in your base image and developer setup scripts, not just on the one machine that broke.
• Exclude directories you never edit (like node_modules or build output) from watch globs so tools request fewer watches.
• On nodes running many watcher pods, size the host inotify limits for the total across all containers, since they share the same per-user cap.
• Bake the sysctl drop-in into your provisioning (cloud-init, Ansible, Dockerfile) so new hosts start with the right value.

When to call a senior engineer

Call for help when raising the watch limit does not stop the errors, when watch consumption keeps climbing without matching project growth, or when many containers on a shared node exhaust host limits in ways that are hard to attribute. Erzon engineers can trace inotify usage back to the exact process and config that opened it, set the right limits across your images and hosts, and stop this error from recurring across your fleet.