ExecutorService shutdown hangs in awaitTermination

What this error actually means

`shutdown()` is a request to stop accepting new work, not a request to stop current work. It marks the executor as shutting down, refuses further submissions with `RejectedExecutionException`, and then lets every already-submitted task — running *and queued* — run to completion. `awaitTermination` returns only when all of them have finished. So a pool with a thousand queued tasks will not terminate until it has executed all thousand, no matter how long that takes.

`shutdownNow()` is stronger but still not a kill. It drains and returns the queued tasks, then calls `Thread.interrupt()` on each worker. Interruption is cooperative: it sets a flag and, for a thread parked in a *interruptible* operation, causes that operation to throw. A thread that is not in such an operation, or that catches `InterruptedException` and carries on, is unaffected. There is no mechanism in Java to forcibly stop a thread — `Thread.stop` was deprecated and then removed because it unlocked monitors at arbitrary points and left shared state corrupted.

The practical consequence is the trap in the dump above: a task blocked reading from a plain `Socket` shows as `RUNNABLE`, and interrupting it does nothing. Operations on `InterruptibleChannel` respond to interrupt by closing the channel and throwing `ClosedByInterruptException`; classic socket I/O, most JDBC drivers, and native calls do not. The only thing that unblocks them is a read timeout or someone closing the underlying resource.

On top of all this, pool threads created by `Executors` factories are **non-daemon**, so a JVM that has finished `main` will not exit while one survives. The visible failure is a container that ignores SIGTERM, sits out the grace period, and is SIGKILLed — losing in-flight work, skipping cleanup, and producing the deployment that takes thirty seconds longer per pod than it should.

How to diagnose it

jcmd <pid> Thread.print | grep -A 8 "batch-worker" | grep -E "Thread.State|sun.nio.ch|java.net|Driver"

log.info("shutdown: active={} queued={} completed={}",
    pool.getActiveCount(), pool.getQueue().size(), pool.getCompletedTaskCount());

jcmd <pid> Thread.print | grep -B 1 "os_prio" | grep -v daemon | head -30

log.info("iteration interrupted={}", Thread.currentThread().isInterrupted());

kill -TERM <pid>; time wait <pid>

The fix

Shut down in the right order: stop accepting external work, stop producers, then `shutdown()` the pool, `awaitTermination` with a bounded timeout, `shutdownNow()` if that expires, and await once more before giving up. Skipping the producer step is what generates the rejection storm; skipping the second await is what leaves you unsure whether force actually worked.

Give every outbound call a timeout. This is the fix for the uninterruptible-I/O case and there is no substitute: a connect timeout, a read timeout, a JDBC `queryTimeout` or statement timeout, and a client-level deadline. A task that cannot block forever cannot hold shutdown open forever, which bounds your termination time by construction.

Make tasks cancellation-aware. Check `Thread.currentThread().isInterrupted()` at the top of each loop iteration, and on catching `InterruptedException` either exit the loop or restore the flag with `Thread.currentThread().interrupt()` before returning. A task that neither exits nor restores the flag is unstoppable.

Close the resource to unblock what interruption cannot reach. Keep a reference to the socket or connection the task is using and close it during forced shutdown: the blocked read then fails with an `IOException` you can handle. This is the only reliable way to stop legacy blocking I/O.

For scheduled executors, cancel periodic tasks explicitly before shutdown and set `setRemoveOnCancelPolicy(true)` so cancelled tasks leave the queue rather than accumulating in it. Relying on shutdown to reap a fixed-rate task is fragile.

Make pool threads daemon threads *only* as a backstop for exit, never as a substitute for shutdown. Daemon threads are killed abruptly at JVM exit with no cleanup, so work in flight is lost silently — acceptable for a metrics reporter, unacceptable for anything writing data.

Match the timeouts to the platform’s grace period. If the orchestrator allows 30 seconds, budget graceful drain plus forced shutdown plus final await inside that, and derive the numbers from your slowest legitimate task rather than choosing them by feel. Two 30-second awaits in a 30-second grace period guarantee a SIGKILL.

// Hangs: producers still submitting, no force, no second await
pool.shutdown();
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);

// Reliable shutdown inside a bounded grace period
acceptingRequests = false;          // 1. stop external intake
producers.stop();                    // 2. stop anything that submits
pool.shutdown();                     // 3. no new tasks; drain what is queued
try {
    if (!pool.awaitTermination(15, TimeUnit.SECONDS)) {
        List<Runnable> dropped = pool.shutdownNow();   // 4. interrupt + drain
        log.warn("forcing shutdown, {} tasks dropped", dropped.size());
        closeOpenSockets();                            // 5. unblock uninterruptible I/O
        if (!pool.awaitTermination(10, TimeUnit.SECONDS)) {
            log.error("executor did not terminate; workers may be stuck in native I/O");
        }
    }
} catch (InterruptedException e) {
    pool.shutdownNow();
    Thread.currentThread().interrupt();   // never swallow it
}

// A cancellation-aware task loop
while (!Thread.currentThread().isInterrupted()) {
    try {
        Item item = queue.poll(1, TimeUnit.SECONDS);
        if (item != null) process(item);
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();   // restore, then exit
        break;
    }
}

How to stop it coming back

• Own every executor in a lifecycle component with an explicit close, and fail the build if a pool is created without one
• Set connect, read, and statement timeouts on every client library; treat an untimed outbound call as a shutdown defect as well as a latency one
• Include a SIGTERM-under-load test in CI that asserts exit well inside the production grace period
• Derive shutdown timeouts from the orchestrator’s grace period, and log when a phase exceeds its budget
• Use daemon threads only for work that is safe to lose abruptly, and document that choice at the thread factory
• Alarm on pods being SIGKILLed after the grace period — it is usually read as a deploy quirk when it is an unterminated pool

FAQ

Related

Other errors engineers hit next to this one

• ERROR: deadlock detected
• ERROR: canceling statement due to statement timeout
• ERROR: could not serialize access due to concurrent update
• ERROR: current transaction is aborted, commands ignored until end of transaction block
• ERROR: duplicate key value violates unique constraint
• Deep OFFSET pagination getting slower every page
• ERROR: canceling statement due to lock timeout (ALTER TABLE)
• ERROR: canceling statement due to conflict with recovery