ReadWriteLock writer starvation

What this error actually means

A write lock requires that *no* reader holds the lock. With a high enough read arrival rate, that condition can simply never be true: reader A is still inside when reader B arrives, B is still inside when C arrives, and the read count never reaches zero. The writer is queued, correctly, for a moment that never comes.

Whether new readers are allowed to overtake a queued writer depends on the lock’s policy, and the default is the permissive one. A non-fair `ReentrantReadWriteLock` lets an arriving reader acquire immediately if the lock is currently read-held, even with a writer waiting — barging is what makes the non-fair lock fast, and it is exactly what starves the writer. The documentation is explicit that non-fair mode makes no guarantees about ordering.

The practical shape is a background writer — a configuration refresher, a cache invalidator, a compaction task — that stops making progress under load and resumes when traffic drops. Nothing errors. The system serves stale data indefinitely, and the incident is usually discovered as "the config change never took effect on three instances", not as a concurrency problem.

Sitting next to this is a second, deterministic trap: the read lock cannot be upgraded. Calling `writeLock().lock()` while holding the read lock blocks forever, because the writer waits for all readers to leave and the caller is one of the readers. It self-deadlocks on the first execution, and the JVM does not report it as a deadlock because there is no cycle between two threads — one thread is waiting for itself. Downgrading is legal and supported: acquire the write lock, then acquire the read lock, then release the write lock.

How to diagnose it

Metrics.gauge("rw.readers", rw, ReentrantReadWriteLock::getReadLockCount);
Metrics.gauge("rw.queuedWriters", rw, ReentrantReadWriteLock::getQueuedWriterThreads);
log.info("readers={} writeLocked={} queuedWriters={}",
    rw.getReadLockCount(), rw.isWriteLocked(), rw.getQueueLength());

for i in 1 2 3; do jcmd <pid> Thread.print | grep -A 5 "config-refresher"; sleep 5; done

long t0 = System.nanoTime();
rw.writeLock().lock();
writerWait.record(System.nanoTime() - t0, TimeUnit.NANOSECONDS);

rw.readLock().lock();
assertFalse(rw.writeLock().tryLock(1, TimeUnit.SECONDS), "upgrade must not succeed");

The fix

Construct the lock in fair mode when a writer must make progress: `new ReentrantReadWriteLock(true)`. Fair mode queues arriving readers behind a waiting writer, so the read count drains and the writer acquires in bounded time. Read throughput drops measurably because barging is no longer allowed — that is the price, and for a lock protecting state that must be updatable it is usually the right price.

Better in most cases: remove the writer’s need for exclusivity. Hold the state as an immutable snapshot behind a `volatile` reference — readers take no lock at all and the writer publishes a new snapshot with a single assignment. There is no read count, no queue, and no starvation to reason about. This is the fix to prefer whenever the state can be copied cheaply enough, and for configuration and routing tables it almost always can.

Never attempt to upgrade. Release the read lock, acquire the write lock, then **re-validate the condition you read** before acting on it, because the state can have changed in the gap. Skipping the re-validation swaps a hang for a lost update, which is the more common mistake once people learn upgrade is impossible.

Use `StampedLock` where reads are short and you want optimistic reading: `tryOptimisticRead` plus `validate` lets readers proceed with no lock acquisition at all in the common case, and `tryConvertToWriteLock` supports the conversion that `ReentrantReadWriteLock` cannot. It is not reentrant and does not support conditions, so it is a targeted tool rather than a drop-in replacement — using it reentrantly will self-deadlock.

Shorten read critical sections aggressively. Copy what you need out under the read lock and process outside it. Shorter reads make zero-reader windows common, which reduces starvation pressure even in non-fair mode and helps every other contention property of the lock.

If reads dominate by orders of magnitude and sections are tiny, drop the read-write lock for a `ConcurrentHashMap` or a plain lock. A read-write lock has more bookkeeping than a simple monitor, so for very short sections it can be slower *and* expose you to starvation.

As a diagnostic backstop, have the writer use `tryLock(timeout)` and log or alert when it fails. Starvation that produces a metric is a bug you find in an afternoon; starvation that produces silence is stale data nobody notices for a week.

// Starves the writer: non-fair default lets arriving readers barge past it
private final ReentrantReadWriteLock rw = new ReentrantReadWriteLock();

Config get(String k) {
    rw.readLock().lock();
    try { return serialise(config.get(k)); }   // long read section
    finally { rw.readLock().unlock(); }
}

// Self-deadlocks on the first call: a read hold can never be upgraded
rw.readLock().lock();
try {
    if (config.isStale()) {
        rw.writeLock().lock();                 // waits for itself forever
    }
} finally { rw.readLock().unlock(); }

// Fix 1: fair mode, and a short read section
private final ReentrantReadWriteLock rw = new ReentrantReadWriteLock(true);

Config get(String k) {
    Config c;
    rw.readLock().lock();
    try { c = config.get(k); } finally { rw.readLock().unlock(); }
    return serialise(c);                       // work outside the lock
}

// Fix 2 (preferred): no lock on the read path at all
private volatile Map<String, Config> config = Map.of();

Config get(String k)            { return config.get(k); }
void publish(Map<String, Config> next) { config = Map.copyOf(next); }

// Correct "upgrade": release, acquire, RE-VALIDATE
rw.readLock().lock();
boolean stale;
try { stale = config.isStale(); } finally { rw.readLock().unlock(); }
if (stale) {
    rw.writeLock().lock();
    try {
        if (config.isStale()) reload();        // re-check: state may have changed
    } finally { rw.writeLock().unlock(); }
}

// Legal direction: downgrade write -> read without releasing exclusivity
rw.writeLock().lock();
try {
    reload();
    rw.readLock().lock();                      // acquire read while holding write
} finally { rw.writeLock().unlock(); }         // now read-held only
try { use(config); } finally { rw.readLock().unlock(); }

How to stop it coming back

• Choose fairness deliberately at construction and record why in a comment beside the field; the default is a throughput choice with a liveness cost
• Prefer immutable snapshots behind a volatile reference for read-mostly state, so there is no writer to starve
• Instrument writer wait time and alarm on it — starvation is silent by nature and needs a metric to be visible
• Ban read-to-write upgrade with a review rule and a deterministic unit test using `tryLock`
• Keep read sections to copying data out, never processing it, so zero-reader windows occur naturally
• Never take a read lock reentrantly across a call boundary where the inner method might need to write
• Give background writers a `tryLock` timeout so a starved refresh becomes an alert rather than stale data

FAQ

Related

Other errors engineers hit next to this one

• runAsNonRoot and image will run as root
• JVM container killed despite available Java heap
• exec format error — container architecture mismatch
• no space left on device — container layers
• SIGTERM misses the application — shutdown ends in SIGKILL
• Redis OOM command not allowed above maxmemory
• MISCONF Redis is configured to save RDB snapshots
• READONLY You can’t write against a read only replica