Linux / shell

No space left on device from container image or writable layers

Written and reviewed by Sahil Srivastav

ContainersFilesystem layersDisk capacity
no space left on device

What this error actually means

A write failed at the storage boundary backing the container operation. That boundary may be the daemon’s image store, a build cache, the container’s writable layer, a mounted volume or a quota. The free space visible in the application’s working directory is not necessarily the space needed to download and unpack an image on the daemon host.

Layered filesystems introduce a non-obvious multiplier. With OverlayFS, modifying a file from a lower read-only layer can copy that file into the writable upper layer. A small change to a large file can therefore require substantially more free space than the changed bytes suggest. Concurrent containers modifying the same bundled file can each create their own copy.

Deleting a file that belongs to an image layer normally hides it from the container’s merged view; it does not remove bytes from that immutable lower layer. Similarly, deleting a large build artifact in a later Dockerfile layer does not erase it from an earlier one. A smaller visible directory can coexist with an unchanged image-store footprint.

Causes, most common first

  1. 1The runtime’s backing filesystem is full. Images, stopped containers, writable layers and logs share a finite host or virtual-machine disk. On Docker Desktop, the VM’s storage limit can be reached while the laptop filesystem still has free space. Identify the daemon location before choosing a cleanup target.
  2. 2Writable-layer growth or copy-up amplifies a workload. An application writes caches or data into its root filesystem, or updates a large file baked into the image. The write belongs in explicitly sized runtime storage when it is expected to grow; embedding mutable data in a base layer makes both capacity and lifecycle harder to reason about.
  3. 3Build cache and old image versions exceed rollout headroom. Repeated builds retain intermediate results, and active containers keep old image layers in use. Shared layers mean that summing displayed image sizes overcounts some usage, while assuming every old tag is fully reclaimable overestimates the space cleanup will free.
  4. 4Inodes or a storage quota are exhausted. A filesystem can reject new files with free bytes remaining, and a project or volume quota can enforce a smaller budget than the containing disk. Compare the failing path’s mount and quota before expanding a different partition.

When you see it

  • A small write fails although the application has only changed a few bytes
  • Image pulls fail while an unrelated mounted data volume still has space
  • Removing files inside a container does not noticeably reduce daemon disk usage
  • Builds succeed initially and fail after many revisions accumulate cache and images

How to diagnose it

Step 1

Identify which daemon and storage driver own the operation

These commands target Docker Engine; Kubernetes commonly uses another runtime and requires that runtime’s tooling. A remote Docker context means the relevant disk is on the remote daemon, not the machine running this CLI.

docker context show
docker info --format '{{.DockerRootDir}} {{.Driver}}'
docker system df -v

Step 2

Separate image storage from container writes and mounts

Inspect the affected container, here app, for its writable-layer size and mount destinations. A bind mount or volume at the failing path changes the storage owner. Container size reporting is not a complete accounting of logs, mounted volumes or every backend’s metadata.

docker ps -a --size
docker inspect app --format '{{json .Mounts}}'

Step 3

Check the actual daemon-host filesystem

Run df on the Linux daemon host, using the DockerRootDir reported above instead of assuming /var/lib/docker. Check both bytes and inodes. With Docker Desktop, inspect the VM disk allocation through its supported management interface rather than treating the macOS path as the Linux backing filesystem.

df -h /var/lib/docker
df -i /var/lib/docker
findmnt -T /var/lib/docker

Step 4

Look for lower-layer files that become writable

Review application writes and the Dockerfile’s layer history. A large bundled database or cache modified on startup is a copy-up candidate. For unexplained host usage, investigate deleted-but-open files and storage-driver accounting before assuming directory totals fully describe allocated space.

The fix

Stop uncontrolled growth before reclaiming space. Bound local caches and logs, clean per-job scratch data on completion and failure, and place mutable runtime data on storage with an explicit capacity and retention policy. A Docker volume still consumes storage somewhere; it is not unlimited capacity.

Use runtime-supported cleanup for identified unused objects after checking their owners and recovery requirements. Start with measured build-cache or obsolete image usage. Avoid broad volume-pruning commands as a reflex, because a volume unused by a currently running container may still contain data required for recovery.

Never manually remove directories inside the runtime’s layer store to make df improve. The runtime maintains references and metadata that ordinary file deletion bypasses. If storage must move or expand, use the runtime and platform’s supported migration procedure with an explicit workload plan.

Reduce image-layer waste with multi-stage builds and by creating and removing temporary build artifacts within the same build step. Keep mutable large files out of read-only image layers when normal operation rewrites them. Verify the resulting image-store size and rollout peak rather than only checking the final container directory listing.

How to stop it coming back

  • Monitor the daemon’s backing filesystem and inode headroom independently of application volumes
  • Budget storage for old and new image versions to coexist during deployment
  • Apply deliberate build-cache retention to CI builders with many revisions
  • Treat large startup writes as a capacity event and test copy-up behaviour under limited storage

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Why did deleting a file inside the container not free space?

If the bytes live in a read-only image layer, deletion hides the file in the merged view instead of altering that layer. An open deleted writable file can also retain allocated blocks until its process closes it.

Is docker system df the same as df?

No. Docker reports its object accounting; df reports filesystem allocation. Logs, shared layers, open deleted files and other host data can make the numbers differ. Use both to identify the owner of the missing headroom.

Will mounting a volume always fix this?

Only if it addresses the actual storage boundary and lifecycle. A named volume on the same full host disk may not add capacity. A separately provisioned volume can isolate application data, while image pulls still need space in the runtime store.

Related

Kubernetes disk-pressure evictionLinux inode exhaustion despite free spaceImagePullBackOff diagnosis

Other errors engineers hit next to this one

Full error and symptom index →