Linux / shell

Pod evicted because the node is under disk pressure

Written and reviewed by Sahil Srivastav

KubernetesEphemeral storageNode pressure

Evicted
The node was low on resource: ephemeral-storage.

What this error actually means

Kubelet is reclaiming a scarce node resource by terminating pods. For disk pressure, the scarce resource can be free bytes or free inodes on a filesystem used by the node or runtime. A container can be healthy at the application level and still lose its pod because the shared machine cannot safely support the accumulated storage demand.

The failed pod remains an Evicted record; it does not move to another node. A Deployment or other controller may create a replacement, and the scheduler places that new pod. Repeatedly deleting Evicted records removes visible history without necessarily reclaiming the storage responsible for ongoing pressure.

Separate node pressure from a pod exceeding its own ephemeral-storage limit. Both can lead to eviction, but their messages and remedies differ. A pod limit violation calls for bounding that workload’s files or revisiting its measured budget. Node pressure requires examining all writers and the actual backing filesystem.

Causes, most common first

1Logs and temporary files accumulate without a bound. Verbose errors can fill container logs precisely when a dependency fails. Disk-backed emptyDir contents and files written into container layers also consume local storage. Rotating stdout logs does not clean a separate application log file inside the container.
2Runtime image storage shares a constrained filesystem. Large image rollouts need room for new layers while old containers still reference previous ones. Garbage collection cannot reclaim layers that remain in use. A node sized only for steady state can fail during an otherwise normal deployment overlap.
3Tiny files exhaust inodes before bytes. Caches, extracted archives and per-request scratch files can consume millions of inode entries with modest total size. Free gigabytes in df -h do not clear an inode-pressure event; inspect inode availability on the relevant mount.
4Local storage demand is not represented in scheduling. Workloads without realistic ephemeral-storage requests can be placed together even when their normal scratch and log footprints do not fit. CPU and memory reservations do not automatically reserve enough disk for the same pods.

When you see it

Multiple unrelated workloads are evicted from the same node
The node reports DiskPressure and replacement pods prefer other eligible nodes
Image pulls or application writes fail around the same time as evictions
A log burst or batch job precedes a sudden decline in disk or inode headroom

How to diagnose it

Step 1

Preserve the eviction message and affected node

Replace demo and app-pod with the evicted pod’s identity. Read the full status message to separate a workload limit from low node resources. Controller-created replacements have different names and may already be on a healthy node.

kubectl -n demo get pod app-pod -o jsonpath='{.spec.nodeName}{"\n"}{.status.reason}{"\n"}{.status.message}{"\n"}'

Step 2

Correlate node conditions with neighbouring workloads

Use the node name from the previous step in place of worker-1. Review pressure transitions and events. A transient condition may already have cleared after reclamation, so retain historical node metrics alongside the eviction timestamp.

kubectl describe node worker-1
kubectl get pods -A --field-selector spec.nodeName=worker-1 -o wide

Step 3

Inspect bytes and inodes on the real backing mounts

Run these read-only commands on the affected Linux node using the approved access path. Adjust directories for its runtime layout. nodefs and image storage may be separate filesystems; a free root partition says nothing about a full dedicated runtime disk.

df -h /var/lib/kubelet /var/log
df -i /var/lib/kubelet /var/log
findmnt -T /var/lib/kubelet

Step 4

Attribute growth before cleaning it

Use node monitoring and runtime-supported storage inspection to identify logs, image layers, writable data and scratch volumes. Check deleted-but-open files when df shows usage that directory totals cannot explain. Broad recursive scans can add I/O pressure, so start with the known growing mount and service.

The fix

Stop the writer responsible for uncontrolled growth: reduce repetitive error logging, cap scratch files and remove temporary data when work finishes. Preserve files required for recovery before cleanup. Deleting random directories under the runtime data root risks breaking running containers and their metadata.

Restore node headroom through the platform’s supported image and container garbage collection or by adding capacity. When draining or replacing a node, account for stateful workloads and the storage demand that moves with them. Moving every pod to equally full nodes merely relocates the failure.

Set realistic ephemeral-storage requests and limits, and bound disk-backed emptyDir volumes where appropriate. Requests help scheduling; limits support enforcement and can trigger eviction. Neither turns local scratch space into durable storage or preallocates a private disk partition.

Use a suitably provisioned persistent volume or object storage for data that must survive pod loss. Changing emptyDir to memory-backed storage transfers pressure to memory accounting and can introduce OOM kills, so it is not a general disk-pressure fix.

# Illustrative pod spec fragment; size from measured scratch usage.
containers:
  - name: app
    image: registry.example.com/team/app:release-42
    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "3Gi"
    volumeMounts:
      - name: scratch
        mountPath: /app/tmp
volumes:
  - name: scratch
    emptyDir:
      sizeLimit: "2Gi"

How to stop it coming back

Alert on byte and inode headroom for every filesystem used by kubelet and the runtime
Budget for simultaneous old and new image versions during rolling deployments
Measure local log and scratch growth during downstream failure, not only healthy traffic
Test that interruption and retry clean temporary files rather than accumulating abandoned work

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Does Guaranteed QoS protect a pod from disk eviction?

No. CPU and memory QoS classification does not provide a disk reservation or immunity to DiskPressure. Local storage requests, actual usage, priority and the kind of pressure matter.

Will a PodDisruptionBudget stop node-pressure eviction?

Do not rely on it. Kubelet’s resource-pressure eviction is different from an API-initiated voluntary disruption and does not provide the same PodDisruptionBudget protection.

Why is disk space available inside my persistent volume?

That volume may use a different filesystem from node logs, image layers or writable container storage. Inspect the filesystem named by the pressure signal rather than assuming all paths share one capacity pool.

Container-layer no space left on device Linux inode exhaustion OOMKilled from memory pressure

Other errors engineers hit next to this one

Full error and symptom index →