Fix: The node was low on resource: ephemeral-storage. Pod evicted

This eviction message means the kubelet detected disk pressure on the node: free ephemeral storage fell below its eviction threshold, so it started killing pods to reclaim space. Evicted pods are rescheduled elsewhere, which often pushes the same disk-hungry workload onto the next node and repeats the problem. Treat it as a node filling up, not as a pod problem, and find what is eating the disk.

The node was low on resource: ephemeral-storage. Pod evicted

What this error means

Ephemeral storage is the node-local disk that backs container writable layers, emptyDir volumes, and container logs. The kubelet watches free space on that filesystem, and when it crosses the eviction threshold (by default 10 percent free for the hard threshold on nodefs) it marks the node with a DiskPressure condition, stops accepting new pods, and evicts running pods, starting with those using the most ephemeral storage relative to their requests. Eviction is the kubelet protecting the node: if the disk actually reaches 100 percent, the container runtime and kubelet themselves start failing, which takes out every pod at once.

Common causes

• Container logs growing without bound: a chatty app writing gigabytes to stdout, with generous or misconfigured log rotation on the node.
• Applications writing large files into the container filesystem or emptyDir: caches, temp files, downloaded artifacts, media processing scratch space.
• Accumulated container images and stopped containers, especially on nodes with many deploys and small disks.
• A pod with no ephemeral-storage request or limit consuming far more disk than the scheduler accounted for.
• Undersized node disks for the workload, so normal operation sits close to the threshold and any spike tips it over.

How to fix it

STABILIZE first. Reclaim space on the pressured node so the eviction storm stops, beginning with unused images and dead containers.

kubectl get nodes -o wide
kubectl describe node <node-name> | grep -A5 Conditions
crictl rmi --prune

1. DIAGNOSE what is consuming the disk. On the node, check the filesystem and then the usual suspects: logs, container writable layers, and emptyDir directories.

df -h /var/lib/kubelet /var/lib/containerd
du -sh /var/log/pods/* 2>/dev/null | sort -rh | head -10
du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* \
  2>/dev/null | sort -rh | head -10

2. If one pod’s logs or scratch data dominate, that workload is the cause. Restart or fix it, and clean up the evicted pod objects left behind.

kubectl delete pods --field-selector=status.phase=Failed -A

3. FIX the workload by declaring honest ephemeral-storage requests and limits, so the scheduler places it on nodes with room and the kubelet kills only the offender instead of evicting neighbors:

resources:
  requests:
    ephemeral-storage: "2Gi"
  limits:
    ephemeral-storage: "4Gi"

4. Cap container log growth at the runtime level. With containerd, set log limits in the kubelet configuration:

# kubelet config
containerLogMaxSize: 50Mi
containerLogMaxFiles: 3

5. Move real data off ephemeral storage. Anything the app writes that is large or must survive belongs on a PersistentVolume, and pure scratch space can be a size-limited emptyDir:

volumes:
  - name: scratch
    emptyDir:
      sizeLimit: 1Gi

How to prevent it

• Set ephemeral-storage requests and limits on every workload that writes to disk, including logs-heavy services.
• Enforce log rotation via kubelet settings and keep application log volume sane at the source.
• Run image garbage collection with headroom, and size node disks for peak image plus log plus scratch usage, not the average.
• Alert on node disk usage well before the kubelet’s eviction threshold, for example at 75 percent.
• Use PersistentVolumes for datasets, uploads, and caches instead of the container filesystem.

When to call a senior engineer

Call for help when evictions cascade across nodes faster than you can clean disks, when you cannot identify which workload is filling ephemeral storage, or when the cluster’s node sizing and storage architecture were never designed for the current write volume. Erzon engineers can trace disk consumption to the exact pods, set resource policies that make eviction boringly rare, and re-architect storage so node disks stop being the failure point.