Livelock and lock convoy on a hot shared registry

What this error actually means

A lock convoy is not a deadlock. Every thread eventually acquires the lock, does its work, and releases it — the system makes progress, so the JVM reports no cycle and nothing is stuck forever. What collapses is throughput, because the critical section has become a single-lane bridge that every request must cross one at a time, and the cost of queuing for it exceeds the cost of the work inside it.

The reason it gets *worse* with more threads is the part that surprises people. When a monitor is contended, a thread that cannot acquire it is parked by the OS and must be woken and rescheduled when the lock is released. That wake-up is not free and it is not instant: the releasing thread hands over, the woken thread has to be scheduled onto a CPU, and its cache lines for the guarded data are cold because another core just touched them. Add threads and you add context switches and cache-line transfers, not throughput. The `cs/s` figure of 400,000 with 70% idle CPU in the output above is the signature — the machine is busy scheduling, not working.

A convoy has a self-sustaining quality that makes it sticky. Once a queue forms on the monitor, every release hands the lock to a thread that then takes time to warm up, extending the hold time slightly, which lengthens the queue further. Throughput settles at a level far below what the same code achieves with two threads, and it stays there until load drops. This is why a service can be fast at 10 requests per second, fast at 100, and fall off a cliff at 250.

Livelock is a different failure that people reach for the same word for, and the distinction is diagnostic. In a convoy, threads are `BLOCKED` and idle, CPU is low, and progress is slow. In livelock, threads are `RUNNABLE` and burning CPU, repeatedly attempting and abandoning the same operation — a `tryLock`-and-back-off pair that keeps colliding, or a CAS retry loop where contention is so high that almost every attempt fails and retries. CPU near 100% with no completed work is livelock; CPU idle with a long monitor queue is a convoy.

How to diagnose it

jcmd <pid> Thread.print | grep "waiting to lock" | sort | uniq -c | sort -rn | head

vmstat 1 5        # cs column, and id for idle
pidstat -w -p <pid> 1 5   # cswch/s and nvcswch/s per thread

jcmd <pid> JFR.start name=conv settings=profile duration=60s filename=/tmp/conv.jfr
jfr summary /tmp/conv.jfr | grep -i monitor
jfr print --events JavaMonitorEnter /tmp/conv.jfr | head -40

jcmd <pid> Thread.print | grep -c "Thread.State: BLOCKED"
jcmd <pid> Thread.print | grep -c "Thread.State: RUNNABLE"

The fix

Get readers out of the lock entirely. For a structure that is read constantly and written rarely, hold the state in an immutable snapshot behind a `volatile` reference: readers do a plain volatile read with no lock at all, and writers build a new snapshot and publish it with one assignment. Contention goes to zero on the read path, which is where all your traffic is. `CopyOnWriteArrayList` and `ConcurrentHashMap` are the off-the-shelf versions of this idea.

Shrink the critical section to the smallest possible mutation. Move logging, metrics, serialisation, allocation, and any validation outside it. Hold time is the variable that controls queue length, so halving it roughly halves the queue — and moving a single log statement out of a hot lock has resolved more of these incidents than any amount of tuning.

Match granularity to the access pattern. If keys are independent, use `ConcurrentHashMap` with per-key atomic operations, or stripe an array of locks by key hash so only colliding keys contend. One lock for a structure with a thousand independent keys makes contention a function of total traffic instead of per-key traffic.

Do not use a fair lock on a hot path. Fairness forces a park-and-wake handoff per acquisition and can cost an order of magnitude in throughput. Leave `ReentrantLock` unfair (the default) and address worst-case latency with a `tryLock` timeout and a queue-length metric instead.

For the CAS-livelock variant, reduce the contention rather than the retries: `LongAdder` instead of a shared `AtomicLong`, striping instead of one hot cell, or batching updates so a thread makes one attempt per batch. Where a retry loop is unavoidable, add randomised exponential back-off — deterministic back-off is what lets two threads collide forever.

For mutual `tryLock` back-off, fix the underlying lock ordering so back-off is not needed, then keep a randomised delay as a safety net. A system whose liveness depends on threads happening not to collide is not designed, it is lucky.

Reducing the thread pool size is a legitimate mitigation while you fix the structure, and it is worth stating plainly because it is counter-intuitive: fewer threads on a contended lock means less scheduling overhead and higher throughput. Treat it as a way to buy time during an incident, not as the fix — the serial section is still there.

If there is no contended monitor and scaling is still negative, suspect false sharing and pad or separate the hot fields. `@Contended` (with `-XX:-RestrictContended`) or manual padding is the tool; `LongAdder` already does this internally, which is part of why it scales.

// Convoy: every request serialises on one monitor to read a rarely-changing map
public synchronized Route lookup(String path) {
    log.debug("lookup {}", path);          // logging inside the lock
    return routes.get(path);
}
public synchronized void update(Map<String, Route> next) {
    routes.clear();
    routes.putAll(next);
}

// Lock-free reads: immutable snapshot published through a volatile reference
private volatile Map<String, Route> routes = Map.of();

public Route lookup(String path) {
    return routes.get(path);               // plain volatile read, no lock
}
public void update(Map<String, Route> next) {
    routes = Map.copyOf(next);             // one publishing write
}

// Independent keys: stripe so only colliding keys contend
private final Object[] stripes = IntStream.range(0, 64)
        .mapToObj(i -> new Object()).toArray();

void touch(String key) {
    synchronized (stripes[Math.floorMod(key.hashCode(), stripes.length)]) {
        counters.merge(key, 1L, Long::sum);
    }
}

// CAS livelock: one hot cell retried by 64 threads -> striped adder instead
private final LongAdder hits = new LongAdder();
void record() { hits.increment(); }

How to stop it coming back

• Measure throughput against thread count as a routine benchmark; negative scaling is the only reliable early warning for contention
• Keep critical sections free of logging, metrics, allocation, serialisation, and every form of I/O — enforce it in review
• Default to immutable snapshots behind a volatile reference for read-mostly shared state, rather than a lock
• Alarm on blocked-thread counts and monitor-wait time from JFR, not only on latency, so contention is attributable before it is an incident
• Use unfair locks on hot paths and manage tail latency with timeouts and queue metrics instead of fairness
• Prefer `LongAdder` and striped structures over single hot atomics anywhere the update rate is high
• Add randomised back-off to any retry loop so two participants cannot synchronise into a livelock

FAQ

Related

Other errors engineers hit next to this one

• Liveness probe failed during startup
• CreateContainerConfigError — missing Secret or ConfigMap
• Evicted — node disk pressure
• runAsNonRoot and image will run as root
• JVM container killed despite available Java heap
• exec format error — container architecture mismatch
• no space left on device — container layers
• SIGTERM misses the application — shutdown ends in SIGKILL