Linux / shell

Argument list too long — the command never received your files

Written and reviewed by Sahil Srivastav

ARG_MAXShell expansionBatch processing

bash: /usr/bin/du: Argument list too long

What this error actually means

Before an external command starts, the shell expands a glob such as ./logs/* into separate arguments. Linux must copy those argument strings and the exported environment into the new process. If the combined payload exceeds the execution limit, process creation fails. The program you intended to run has not had an opportunity to process even the first filename.

The limit is about bytes and argument overhead, not just file count. Longer paths, a larger environment and process limits can make a previously successful command fail against the same number of files. Linux also limits individual argument strings, so one enormous JSON argument is a different case from many small filenames.

The repair is to stream names into bounded invocations, or to give the program a file or standard input for large data. A pipeline helps only if the oversized expansion is removed from the command that launches the pipeline. Writing xargs command ./logs/* still asks the shell to expand the whole directory first.

Causes, most common first

1An unbounded glob becomes the argument vector. A command such as du ./logs/* sends every matching name in one invocation. Quoting the glob prevents expansion but usually changes the requested operation: the receiving utility gets a literal asterisk instead of a stream of paths.
2Command substitution materialises the entire input. Using command $(find ...) recreates the same argument-volume problem and adds word-splitting bugs. It also loses filename boundaries. A producer-consumer pipeline or find -exec avoids constructing one giant shell word list.
3The environment consumes the execution budget. Exported configuration blobs travel with every child. A large environment can make even a small command fail to launch. Keep bulk data in a file or other explicit input channel, rather than exporting it for every subprocess.
4One argument is itself oversized. Batching does not split a single large argument into a valid one. If a request body or generated configuration is passed as one string, use the tool’s file-input or stdin interface so that exec does not have to carry the content.

When you see it

A maintenance command fails as the directory grows, before doing any work
The same command succeeds on a smaller directory or with shorter paths
A job starts failing after exporting a large variable even with unchanged input files
Passing one very large payload as a quoted argument fails despite correct quoting

How to diagnose it

Step 1

Inspect the current execution limit

getconf reports ARG_MAX for the current environment. Treat it as a boundary to avoid, not a target batch size: the environment, pointers and platform details consume part of the available space.

getconf ARG_MAX
getconf PAGESIZE

Step 2

Measure exported bytes without printing secrets

This counts environment bytes without displaying variable values. If a new exported blob dominates, remove it from the launcher and pass the content through an explicit file. An already enormous environment may prevent env itself from launching.

env -0 | wc -c

Step 3

Ask GNU xargs about its batching limits

GNU xargs reports the environment cost and the command buffer it plans to use. This Linux-oriented option is not universal across xargs implementations. Empty input keeps the diagnostic from consuming filenames or starting real work.

xargs --show-limits </dev/null

Step 4

Try bounded, read-only processing

The example processes regular log files directly beneath ./logs and reports their sizes. The directory must exist. NUL delimiters preserve names containing whitespace or newlines; -r prevents an empty GNU xargs invocation. A successful run proves the oversized launch can be avoided.

find ./logs -maxdepth 1 -type f -print0 | xargs -0 -r -n 100 du -h --

The fix

Prefer find -exec command {} + when the task naturally applies one command to groups of discovered files. find builds bounded invocations without serialising names through whitespace. Use xargs -0 with find -print0 when a pipeline is more convenient, and include the receiving command’s end-of-options marker when it supports one.

Choose the traversal deliberately. A recursive find is not equivalent to a top-level glob, and filtering with -type f excludes directories and symlinks that the original command may have included. Confirm selection with a read-only operation before applying a destructive maintenance action. Fixing argument limits must not silently broaden the operation.

For payloads, use stdin or a named file supported by the receiving utility. Do not split JSON or SQL into arbitrary fragments merely to fit ARG_MAX. Raising the stack limit to move an argument ceiling leaves input volume unbounded and makes the script depend on launcher-specific limits.

Batching changes failure semantics: some invocations may succeed before another fails. Make the operation resumable, check the pipeline’s status, and avoid declaring success merely because the producer finished. Start with sequential batches; parallel xargs adds concurrency and can overload the storage you are trying to maintain.

# GNU find batches paths without a shell-expanded file list.
find ./logs -maxdepth 1 -type f -exec du -h -- {} +

How to stop it coming back

Use streaming or batched interfaces anywhere directory size or payload length can grow with production data.
Test filename handling with spaces, wildcard characters and embedded newlines, alongside an empty input set.
Keep exported environment variables small and make partial batch failure visible to the scheduler.

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Does quoting the variable solve ARG_MAX?

Quoting preserves an argument boundary; it does not remove byte limits. One huge quoted value can still exceed the per-string or combined limit. File or stdin input is the appropriate interface for large content.

Why not pipe ls into xargs?

Line-oriented output is ambiguous for filenames containing newlines, and default xargs parsing also treats quotes and backslashes specially. find -print0 with xargs -0 preserves actual filename boundaries instead of attempting to reconstruct them.

Will xargs -n 100 always stay below the byte limit?

It caps the number of arguments, not their total size. xargs also accounts for its command buffer and system limit. A single oversized argument still needs a different input interface.

Unquoted variables corrupting arguments Bash script hardening interview

Other errors engineers hit next to this one

Full error and symptom index →