Distributed systems
Distributed locking: interview questions and how to answer them
A distributed lock is a lease, not a mutex: it can expire while its holder still believes it holds the lock, which is why correctness requires fencing at the resource rather than trust at the client.
Written and reviewed by Sahil Srivastav
What it actually is
A distributed lock lets processes on different machines agree that only one of them may act on a resource at a time. Unlike an in-process mutex, the lock service cannot see whether the holder is alive, running slowly, or stopped in a garbage-collection pause, so it can only grant the lock for a bounded period. That bound — the lease — is the source of every interesting failure in the area.
This creates two very different goals that are routinely conflated. An efficiency lock exists to avoid duplicated work: if it occasionally allows two holders, you waste effort and nothing breaks. A correctness lock exists to protect an invariant: two holders mean corrupted data. Efficiency locks can be built on a single Redis key; correctness locks cannot be built on timing assumptions at all.
The reason is that no client can prove it still holds a lease. Between the check and the write, the process may have been descheduled for longer than the TTL — a stop-the-world pause of several seconds is entirely normal under memory pressure — and by the time the write lands, someone else legitimately owns the lock. The lock service did nothing wrong; the design was unsound.
Why it matters in production
Because the pattern shows up wherever people avoid a database constraint. Reserving inventory, allocating a seat, running a nightly job once, sending one notification per event: each is often implemented as “take a lock, check, write, release”, and each fails under exactly the conditions a lock was supposed to handle. A uniqueness constraint or a conditional update usually does the same job with no lease at all.
It matters operationally too. Locks with generous TTLs turn a single crashed worker into a multi-minute stall for everything queued behind it; locks with short TTLs get stolen mid-operation. There is no TTL that is simultaneously safe against slow holders and responsive after a crash, which is why a heartbeat-extended lease plus fencing is the standard answer.
And it is a favourite interview topic precisely because the naive answer sounds complete. A candidate who says “SETNX with an expiry” has described a real mechanism, and the follow-up — “what if the holder pauses past the expiry?” — separates people who have operated one from people who have read about one.
How it works
Acquire atomically with an owner token
`SET lock:key <random-uuid> NX PX 30000` claims the lock and records who holds it in one round trip. The random value matters: release must be conditional on it, or a process whose lease expired will delete the lock a different holder now owns. Release is therefore a compare-and-delete, which in Redis means a small Lua script rather than `GET` followed by `DEL`.
Why the TTL cannot be tuned away
With no TTL, a crashed holder blocks the resource forever. With a TTL, a live-but-paused holder loses the lock without knowing. You are choosing between liveness and safety, and no value satisfies both — which is the actual content of the “distributed locks are hard” claim. Heartbeat extension narrows the window but cannot close it, because the heartbeat thread can be paused too.
Fencing tokens close the gap
The lock service issues a monotonically increasing token with each grant. Every write to the protected resource carries the token, and the resource rejects any write whose token is lower than the highest it has already seen. A delayed write from an expired holder then arrives with a stale token and is refused. This is the only mechanism that makes a lease-based lock actually safe, and it requires cooperation from the storage layer.
Why Redlock is disputed
Redlock acquires a majority of independent Redis instances to survive node loss. Kleppmann’s objection is that it is still a timing-based algorithm: it assumes bounded clock drift and bounded process pauses, and neither holds on real machines, so it does not provide safety without fencing. Antirez’s reply is that it is a reasonable efficiency lock. The defensible interview position is that Redlock adds failover, not correctness — for correctness you need fencing or a consensus-backed lock service.
The lock you did not need
Most “distributed lock” requirements are single-row invariants in disguise. `UPDATE seats SET holder = $1 WHERE id = $2 AND holder IS NULL` enforces mutual exclusion inside the database, atomically, with no lease and no clock. A unique index on `(event_id, consumer)` does the same for once-only processing. Reaching for a lock when the store can express the invariant adds a failure mode for nothing.
Implementing it
Decide out loud whether the lock is for efficiency or correctness. If correctness, either move the invariant into the storage engine as a constraint or conditional update, or obtain a fencing token and enforce it at the resource. A Redis key with a TTL and no token is an efficiency lock whatever the comment above it says.
Always release conditionally on the owner token, in one atomic operation. A plain `DEL` on a lock you may no longer hold is one of the fastest ways to turn a mild timing bug into concurrent execution.
Keep the critical section short and the TTL a small multiple of its realistic p99, then extend by heartbeat for long jobs. Long-running work inside a lease is what makes expiry-under-load the common case rather than the rare one.
For leader-style “run this job once” requirements, use a lock service with a real consensus core — ZooKeeper ephemeral nodes, etcd leases, Consul sessions — and still make the job idempotent, because a lease can lapse in the middle of one.
-- Prefer the constraint to the lock where the store can express it:
UPDATE seats
SET holder_id = $1, held_until = now() + interval '10 minutes'
WHERE seat_id = $2
AND (holder_id IS NULL OR held_until < now());
-- 0 rows affected => someone else holds it. No lease, no clock skew,
-- no fencing token, and no window between check and write.
-- If you must use a lease, release must be conditional (Redis Lua):
-- if redis.call("GET", KEYS[1]) == ARGV[1]
-- then return redis.call("DEL", KEYS[1]) else return 0 endInterview questions and how to answer them
Implement a distributed lock with Redis. Now tell me how it fails.
`SET key <uuid> NX PX 30000` to acquire, and a Lua compare-and-delete to release. It fails when the holder is paused — a long GC, a descheduled container, a blocked syscall — past the TTL. The lock expires, a second holder acquires it legitimately, and then the first holder resumes and writes as though it still owned the resource. Nothing in the client can detect this, so the fix has to be at the resource: a fencing token the storage layer enforces.
What is a fencing token and what problem does it solve?
A monotonically increasing number issued with each lock grant and attached to every write. The resource remembers the highest token it has accepted and rejects anything lower, so a write from a holder whose lease has expired is refused rather than applied. It converts a timing assumption into a verifiable check, which is why it is the only sound answer to the paused-holder problem.
When would you deliberately avoid a distributed lock?
Whenever the invariant fits in one storage operation. Reserving a seat, claiming an idempotency key, transitioning an order state — all are conditional updates or unique constraints, enforced by the database with no lease and no clock. A lock is warranted when the critical section spans systems the database cannot see, and even then I would make the work idempotent so a lapsed lease is survivable.
How do you choose the TTL?
By accepting that no single value is correct and designing around that. Start from the p99 of the critical section, add headroom, then extend the lease by heartbeat for anything long-running so a crash is detected in seconds rather than minutes. Crucially, do not treat the TTL as a safety mechanism: it bounds unavailability after a crash, it does not prevent two holders.
Redis versus ZooKeeper or etcd for locking — how do you choose?
Redis is fast and cheap and gives an efficiency lock; failover can lose the key because replication is asynchronous, so a lock can be held twice across a promotion. ZooKeeper and etcd sequence lock acquisition through a consensus log, so a grant survives leader change, and they expose session or lease ids usable as fencing tokens. Pay for consensus when two holders would corrupt data.
Two workers both think they hold the lock. What is your debugging path?
First check whether the lease expired during the operation: compare the critical-section duration distribution against the TTL, and look for GC or scheduling pauses at the timestamps involved. Then check release semantics — an unconditional delete produces exactly this symptom. Then check for a failover in the lock store between the two acquisitions. Finally, confirm whether the resource has any fencing at all; if not, the design permits the symptom regardless of cause.
Answers that lose the round
- Describing a distributed lock as a mutex and never mentioning the lease, which is where all the difficulty lives
- Releasing with an unconditional `DEL`, so a process whose lease expired deletes another holder’s lock
- Setting the key and the expiry as two commands, leaving a window where a crash leaves a permanent lock
- Claiming Redlock gives correctness — it gives failover; safety still needs fencing tokens at the resource
- Relying on wall-clock comparisons across machines to decide whether a lease is still valid
- Holding a lock across a network call or a long batch, making expiry-during-work the normal case
- Using a lock where a unique constraint or a guarded `UPDATE` would enforce the same invariant atomically
FAQ
Is a database row lock a distributed lock?
It is a lock held inside a system that also holds the data, which makes it far stronger: the lock and the write are in the same transaction, so there is no window and no lease. `SELECT ... FOR UPDATE` blocks other writers for the duration of the transaction and is released atomically with commit or rollback. The trade-off is that it only protects resources the database owns.
Do I need a lock to make a scheduled job run once?
You need either leader election or an atomic claim on the job row — `UPDATE jobs SET state = 'RUNNING', owner = $1 WHERE id = $2 AND state = 'PENDING'` — plus an idempotent job body. The claim alone is enough to prevent two starts; the idempotency covers the case where the owner dies mid-run and the row is reclaimed.
Can I extend a lease safely?
You can extend it conditionally on still owning it, which is the right way to handle long jobs, but extension does not make the lock safe. The heartbeat thread lives in the same process as the worker, so the pause that stops the worker stops the heartbeat too. Extension reduces how often expiry happens; fencing is what makes expiry harmless.