Thread pool starvation with nested tasks

What this error actually means

Every worker in the pool is occupying a thread while waiting for work that can only run on a thread from that same pool. The queue holds the sub-tasks, the sub-tasks need a worker, and every worker is blocked waiting for a sub-task. Nothing will break it and nothing will time out, because nobody is waiting on a lock — they are waiting on a `Future` that is queued behind them.

This is a deadlock, but it is invisible to the JVM deadlock detector. The detector finds cycles in the *lock* wait-for graph, and there is no lock here: the cycle runs through a queue and a scheduler, which the JVM does not model. So `Found one Java-level deadlock` never appears, and the incident presents as a silent, complete loss of throughput with clean error metrics.

The reason it survives review is that the dependency is not visible at the call site. A handler calls a helper, the helper fans out to an executor and joins, and the fact that the handler is *itself* running on that executor is three layers away in a different file. The bug is a property of the composition, not of either piece — which is why it appears after an unrelated refactor moves a caller onto a pooled thread.

Thread count is what determines whether you see it at all. With `n` workers and a task that spawns one sub-task and waits, the pool survives until `n` such tasks run concurrently. A pool of 32 hides the bug under normal load and fails at peak; a pool of 2 fails immediately. That is why "it only happens in production" is the usual framing, and why shrinking the pool is the fastest way to reproduce it.

How to diagnose it

jcmd <pid> Thread.print | grep -A 10 "batch-worker" | grep -c "FutureTask.awaitDone"

Metrics.gauge("pool.active", pool, ThreadPoolExecutor::getActiveCount);
Metrics.gauge("pool.queue", pool, p -> p.getQueue().size());
Metrics.gauge("pool.completed", pool, ThreadPoolExecutor::getCompletedTaskCount);

var pool = Executors.newFixedThreadPool(1);  // nested join hangs immediately

grep -rn -E "\\.get\\(\\)|\\.join\\(\\)|invokeAll|allOf\\(.*\\)\\.join" src/main/java | grep -v test

Executors.newFixedThreadPool(8, Thread.ofPlatform().name("batch-worker-", 1).factory());

The fix

Remove the join rather than resize the pool. Restructure the fan-out so the outer task submits the sub-tasks and *returns*, with the aggregation done by a continuation that runs when the children complete. `CompletableFuture.allOf(...).thenApply(...)` expresses the join without any thread waiting for it — nothing occupies a worker while the children queue, so the cycle cannot form regardless of pool size or fan-out width.

If the code must block, give the blocking wait a different pool from the work it waits for. Two pools with a strict one-directional dependency — outer waits on inner, inner never submits to outer — cannot starve, because the inner pool’s workers are never held by a waiter. Make the direction explicit in the wiring and enforce it, or you will reintroduce the cycle in six months.

Never run a parallel stream, or a defaulted `CompletableFuture` stage, inside work that is itself on the common ForkJoinPool. Submit the stream into a dedicated `ForkJoinPool` you own, and always pass an explicit executor to `supplyAsync`, `thenApplyAsync`, and friends. Relying on the default executor is how unrelated subsystems end up sharing one small pool.

Make fan-out width bounded and independent of input size. A task that spawns one sub-task per row needs a slot count proportional to the data; chunking the work into a fixed number of batches makes the concurrency requirement a constant you can reason about.

Where a scarce resource is held across the wait — a connection, a permit — release it before submitting and re-acquire after. Holding a pool slot *and* a connection while waiting for a task that needs both is two overlapping cycles, and fixing only the pool leaves the other.

Add a bounded `get(timeout, unit)` at every remaining blocking join once the structure is right. This does not fix starvation, but it converts an eternal silent hang into a timeout you can alarm on — the difference between a five-minute incident and a five-hour one.

// Deadlocks once `n` outer tasks run at the same time: each holds a worker
// while its sub-task sits in the queue behind it.
var pool = Executors.newFixedThreadPool(4);

Future<Report> outer = pool.submit(() -> {
    Future<Rows> inner = pool.submit(this::loadRows);   // same pool
    return render(inner.get());                         // holds a worker, waits
});

// Fix 1 (preferred): no thread ever waits — the join becomes a continuation
CompletableFuture<Report> outer =
    CompletableFuture.supplyAsync(this::loadRows, ioPool)
                     .thenApplyAsync(this::render, cpuPool);

// Fix 2: blocking is fine if the waiter and the worker are different pools,
// with a strictly one-directional dependency.
var orchestrator = Executors.newFixedThreadPool(4, named("orchestrator-"));
var workers      = Executors.newFixedThreadPool(8, named("worker-"));

orchestrator.submit(() -> {
    var parts = workers.invokeAll(chunks(order));   // workers never submit back
    return merge(parts);
});

How to stop it coming back

• Make it an architectural rule that a task never blocks on a task in its own pool, and record for every pool which pools it is allowed to depend on
• Always pass an explicit executor to `CompletableFuture` async stages; treat a defaulted stage in application code as a review finding
• Name every pool through a thread factory — diagnosis time for this failure is dominated by whether the threads are identifiable
• Run a CI suite with every pool sized to one thread: structural nesting hangs immediately, and a timeout in the test is a clear failure signal
• Alarm on completed-task count being flat while queue depth is non-zero, which detects starvation without needing a dump
• Keep fan-out width constant rather than proportional to input, so the concurrency requirement does not depend on the data

FAQ

Related

Other errors engineers hit next to this one

• SIGTERM misses the application — shutdown ends in SIGKILL
• Redis OOM command not allowed above maxmemory
• MISCONF Redis is configured to save RDB snapshots
• READONLY You can’t write against a read only replica
• LOADING Redis is loading the dataset in memory
• CROSSSLOT Keys in request don’t hash to the same slot
• MOVED and ASK replies from Redis Cluster
• Redis clients stall during KEYS on a large keyspace