Linux / shell

No space left on device — when bytes are free but inodes are not

Written and reviewed by Sahil Srivastav

Filesystem capacityInodesSmall-file growth
touch: cannot touch 'probe': No space left on device

What this error actually means

A filesystem needs metadata as well as data blocks to create a file. On filesystems with a fixed inode supply, each new file or directory consumes an inode even if its contents occupy almost no space. Millions of tiny cache entries can exhaust that supply while df -h still reports many gigabytes available.

The decisive comparison is df -h and df -i for the same target path, from the same mount namespace as the writer. Looking at the host’s root filesystem says little about a separate volume mounted under an application directory. Container overlays and temporary filesystems make this mistaken comparison particularly easy.

ENOSPC is broader than inode exhaustion. If inode availability is healthy, continue investigating the actual filesystem, block allocation and storage layer. Deleted files held open usually explain why du sees less space than df reports; they are not the default explanation for a filesystem that genuinely has plenty of usable blocks and no available inodes.

Causes, most common first

  1. 1A file-per-item cache or spool grows without retention. Byte limits underestimate the cost of tiny objects. A workload can stay below its content-size budget while creating one metadata object per event. Failed cleanup jobs let this accumulate until new file creation stops.
  2. 2Temporary files survive error and cancellation paths. A job may remove its scratch files only after success. Repeated failures leave many small files that normal happy-path tests never retain. A restart can make matters worse if startup creates more temporary files without cleaning stale owned artifacts.
  3. 3The filesystem was sized for a different file distribution. A volume suitable for a few large artifacts may have insufficient inodes for millions of small records. This is a workload-layout mismatch even if total stored bytes remain within the original capacity estimate.
  4. 4The capacity check inspected the wrong mount. A path beneath /var or a container directory can be a separate filesystem with its own limits. Host-level aggregate storage figures are not evidence about the allocation that returned ENOSPC.

When you see it

  • Creating an empty file fails although ordinary byte-capacity dashboards look healthy
  • A directory contains very large numbers of tiny cache, spool or session files
  • Deleting a small number of obsolete files permits a similar number of new files
  • The failure is isolated to one mounted volume rather than every path on the machine

How to diagnose it

Step 1

Compare blocks and inodes for the failing directory

Replace /var/lib/app with the actual destination or its existing parent. A near-full inode count alongside free blocks supports the inode diagnosis. Some filesystems allocate metadata differently, so interpret the output using the filesystem type.

df -h /var/lib/app
df -i /var/lib/app
findmnt -T /var/lib/app -o TARGET,SOURCE,FSTYPE,OPTIONS

Step 2

Find where inode consumption is concentrated

GNU du can summarise inode usage by directory. This walks the tree and can be expensive on the very workload causing the problem; run it during a controlled investigation. -x avoids crossing into other filesystems.

du --inodes -x --max-depth=2 /var/lib/app | sort -n | tail -20

Step 3

Inspect retention and object counts

For a known cache directory, count regular files without emitting their names. Compare creation patterns with application and cleanup logs. GNU find’s constant-format output avoids ambiguity from embedded newlines in filenames.

find /var/lib/app/cache -xdev -type f -printf '.' | wc -c

Step 4

Investigate deleted open files only if block accounting points there

If blocks are full but directory totals are unexpectedly small, lsof can identify unlinked files that remain open. This is a different branch of the diagnosis: space is released when the final reference closes, not by deleting the already-unlinked name again.

lsof +L1

The fix

Identify a set of expired or disposable files owned by the application, pause the producer if necessary, and use its documented cleanup procedure. Verify inode availability after cleanup before restarting the full workload. Do not remove arbitrary database, queue or container-runtime files just because their directory has a high count.

Repair retention in both success and failure paths. Put an explicit object-count bound beside the byte bound, and make cleanup failures observable. For a cache, ensure expired entries are actually removed rather than merely ignored on reads. For a durable spool, investigate why processing is stalled instead of deleting unprocessed work.

If the required retained population is legitimate, redesign storage or provision a filesystem layout suited to it. On ext4, inode density is largely chosen at filesystem creation; blindly enlarging a quota or changing permissions does not create the needed metadata capacity. Plan migration or expansion according to the filesystem’s supported procedure.

Validate recovery by exercising the operation that failed: creating and writing a new file as the application account. A successful df command is only an observation. Keep enough free metadata capacity for deployment and recovery operations while the application catches up.

How to stop it coming back

  • Alert on both block and inode utilisation for every writable application filesystem.
  • Track file creation and deletion rates, plus the age of the oldest pending spool item, to distinguish retention from stalled processing.
  • Include aborted jobs and cleanup outages in storage-capacity estimates; steady-state successful traffic is not the worst case.

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Will deleting one large file solve inode exhaustion?

It generally releases only the inode for that file when its references are gone, even if it frees many data blocks. Reclaim the excess object population identified by the investigation, not whichever file happens to be largest.

Can I fix this with chmod?

No. Permission failures and allocation failures are different mechanisms. Preserve the errno and examine capacity on the destination filesystem. Permission changes cannot replenish an exhausted inode supply.

Why do df and du disagree?

They measure different things: filesystem allocation versus reachable file-tree usage. Open deleted files, metadata and mount boundaries can contribute to differences. First decide whether your shortage is blocks or inodes before using that discrepancy as evidence.

Related

Safely batch large file setsLinux log parsing interview

Other errors engineers hit next to this one

Full error and symptom index →