jonnify·redis /redis-patterns/coordination/redlock

R2.03 · Coordination — module hub · a contrast

The Redlock algorithm.

Achieve fault-tolerant distributed locking by acquiring locks on a majority (N/2+1) of N independent Redis instances, tolerating node failures without losing lock consistency. Taught as a contrast — the majority-of-N lock weighed against EchoMQ’s single-Valkey lease.

Grounding (contrast): echo/apps/echo_mq · the @claim script (jobs.ex:126) — EchoMQ does not implement Redlock; it keeps one Valkey, reads one server clock with redis.call('TIME'), and fences with attempts.

The majority decides — toggle each of five independent masters. Select a master to flip it OK or down; the readout reports whether the count clears the majority line.
M1OK M2OK M3OK M4down M5down majority line: 3 of 5 majority reached
readout · Three of five masters OK — a majority. The lock is held on a majority and survives the two down masters.

Static verdicts — 3+/5 OK → majority reached, held · 2/5 OK → not a majority, released on all 5 · the majority line for N=5 is floor(5/2)+1 = 3.

§1

The problem with single-instance locks

redlock.md.txt · single instance

A single Redis lock with SET resource <token> NX PX 30000 holds until the one Redis server fails. With master–replica replication, a race opens at failover. Client A acquires the lock on the master. The master crashes before the write replicates to the replica. The replica is promoted to master. Client B then acquires the same lock — the new master carries no record of A’s lock.

Two clients are now both granted the same lock. The mutual-exclusion guarantee is violated: two holders at once, which is exactly what a lock must prevent. The acquire is safe on one node; the gap opens only across a master–replica handoff. This is the weakness Redlock is built to remove.

A single-instance lock is correct on one node — until that node fails mid-acquire, and a failover hands the same lock to a second holder.

§2

The Redlock solution

redlock.md.txt · the algorithm

Redlock uses N independent Redis masters — typically 5 — with no replication between them. A lock counts as acquired only when a client holds it on a majority of instances: N/2+1, so at least 3 of 5. The acquire is time-checked end to end.

# 1. read clock T1
T1 = now_ms()
# 2. on each of N instances, with a 5–50 ms per-instance timeout:
SET resource_name <random_value> NX PX 30000
# 3. read clock T2 → elapsed = T2 − T1
# 4. ACQUIRED only if (OK count ≥ N/2+1) AND (elapsed < TTL)
# 5. validity = TTL − elapsed − clock_drift_allowance
# 6. else → release on ALL instances at once, wait a random delay, retry

Release runs the same token-checked Lua script on every instance — GET == token then DEL — so a delete only lands on a lock this client still owns. Always attempt the release on all instances, even those where the acquire appeared to fail; a network blip can produce a false negative. The random retry delay desynchronizes competing clients and reduces contention.

A lock is held only when the votes clear the majority line and the acquire fits inside the lease; miss either, and the lock is released on all five.

§3

Safety and liveness

redlock.md.txt · properties

Redlock targets three properties.

  • Safety — mutual exclusion. At most one client holds the lock at a time, as long as the client finishes its work within the validity time. If the lease expires mid-work, a second client may acquire it.
  • Liveness — deadlock freedom. A lock eventually frees even when the holder crashes, because the TTL releases it automatically — the same mortality the single-instance PX gives, now across N nodes.
  • Liveness — fault tolerance. While a majority of instances are reachable, clients can acquire and release locks. This is the property a single-instance lock cannot offer, and the whole reason to run more than one.

Mutual exclusion while the lease covers the work, no deadlock because the TTL frees a crashed holder, and progress while a majority survives.

§4

Clock drift and timing

redlock.md.txt · clock drift

Redlock assumes clocks on different machines advance at approximately the same rate. It does not require synchronized clocks — only that a local clock does not drift significantly relative to the TTL. The algorithm is immune to network delay during the acquire because it re-checks the time at T2: if too much time elapsed, the lock is treated as invalid even when a majority responded OK. The vulnerability is a clock jump — a manual time change or an aggressive NTP correction — which can move a clock past the lease and break the safety argument. The drift allowance, typically 1–2% of the TTL, is the safety margin subtracted from the usable validity:

validity = TTL − elapsed − clock_drift_allowance   # immune to network delay; vulnerable to clock jumps

The time-check at T2 neutralizes a slow network; a clock that jumps is the hazard Redlock cannot re-check away.

§5

The fencing problem

redlock.md.txt · fencing

Automatic expiry creates a hazard shared by every distributed lock: client A acquires the lock; A pauses (GC, a context switch, a scheduler stall); the lock expires; client B acquires it; A resumes and proceeds as a holder; two clients now operate on the shared resource at once.

The stronger fix is a fencing token: on acquire, also obtain a monotonically increasing token; every operation on the resource carries the token; the resource rejects any operation whose token is older than the highest it has seen. A paused, stale writer is fenced out at the resource, not merely warned at the lock. Redlock’s random value can serve as a check-and-set token, but it carries no ordering, so it does not fence on its own.

A lease can expire under a paused holder; the only guarantee that survives the pause is a token the resource itself enforces.

§6

When to use Redlock

redlock.md.txt · when to use

Redlock fits work where approximate mutual exclusion is enough and fault tolerance matters: coordinating access to an external resource with no built-in concurrency control, preventing duplicate job execution in a distributed queue, approximate distributed rate limiting, and leader election among application instances.

Reach for an alternative when the requirements harden — absolute correctness, a linearizable resource, a case where one Redis node is enough, or a setting where five independent instances cost more than the guarantee is worth.

The requirementReach for
Approximate mutual exclusion, fault tolerance mattersRedlock — majority of N
Absolute correctness, a lost lock corrupts dataA consensus system — etcd, ZooKeeper (Paxos/Raft)
A linearizable storage system is availableFencing at the storage layer
One Redis node’s reliability is sufficientA single-instance lease (R2.02)
Five independent instances are not justifiedA single-instance lease + idempotent work

Redlock sits between a single-instance lease and a consensus system: more fault-tolerant than one node, weaker than Paxos — choose it when a lost lock is a recoverable retry, not corruption.

§7

Implementation guidelines

redlock.md.txt · guidelines

The algorithm’s safety rests on a handful of concrete settings.

KnobValueWhy
Instances5 mastersTolerates up to 2 failures while keeping a majority.
Independenceno replicationFully independent, ideally in different AZs — a replica failover is the single-instance race Redlock removes.
TTL10–30 sLonger than the operation, short enough that a failed client does not block the resource for long.
Per-instance timeout5–50 msLong enough for a healthy instance to respond, short enough to skip a failed one.
Random value≥ 20 bytesFrom a cryptographically secure source (/dev/urandom), unique per client per attempt.
Drift allowance1–2%Subtracted from the validity for clock drift between machines.

Five independent masters, a TTL that bounds the work, a tight per-instance timeout, and a long random value — the settings that make the vote and the time-check sound.

§8

The debate

redlock.md.txt · the debate

Martin Kleppmann published a critique arguing Redlock is unsafe for correctness-critical work, primarily because of its timing assumptions, and that a fencing token from the protected resource is the real fix. Salvatore Sanfilippo (antirez, the Redis creator) replied defending the algorithm’s practical safety, arguing the timing checks are sufficient under real conditions.

The disagreement centres on whether those timing checks are enough to prevent a safety violation in practice. Both agree on one thing: a fencing token is the stronger guarantee where the resource supports one. Where a lock violation would corrupt data, reach for a consensus system or ensure the resource layer fences.

Kleppmann and Sanfilippo disagree on whether the timing checks suffice — and agree that a fencing token at the resource is stronger than any lock.

§9

Summary — single instance vs Redlock

redlock.md.txt · the trade

The whole trade collapses to a four-row table: Redlock buys fault tolerance and consistency-through-failover with the complexity of five instances, while a single instance stays simple and loses the lock when its one node fails.

AspectSingle instanceRedlock
Fault toleranceNone — lost when the one node failsTolerates a minority of node failures
ComplexityLow — one RedisHigher — 5 independent instances
ConsistencyLost on failoverMaintained while a majority survives
Use caseNon-critical coordinationImportant distributed coordination

Redlock trades operational complexity for fault tolerance — not a consensus algorithm, but sufficient locking with reasonable safety for many applications.

§10

The contrast — EchoMQ keeps one Valkey

echo_mq/jobs.ex:126 · @claim

EchoMQ does not implement Redlock. Its “lock” is the claim lease, and it lives on one Valkey, reached through one EchoMQ.Connector. The @claim script (echo/apps/echo_mq/lib/echo_mq/jobs.ex:126) is one atomic Lua transition: it pops the next id from pending with ZPOPMIN, increments the fencing token with HINCRBY <job> attempts 1, marks the row active, reads one server clock with redis.call('TIME'), and stamps the lease with ZADD active now + lease_ms <id>. No cross-node clock is compared, because there is only one clock; no votes are counted, because there is only one master.

# @claim (jobs.ex:126) — the lease on one server clock, attempts as the fencing token
local popped = redis.call('ZPOPMIN', KEYS[1])
if #popped == 0 then return {} end
local id = popped[1]
local jk = ARGV[1] .. id
local att = redis.call('HINCRBY', jk, 'attempts', 1)   # the monotone fence
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')                            # one server clock
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)
return {id, redis.call('HGET', jk, 'payload'), att}

The pause that breaks Redlock is handled without a fencing token at the resource and without five masters. The attempts counter, returned to the worker by @claim, is the fence: @complete and @retry refuse a stale token with EMQSTALE … token mismatch, so a worker whose lease was reaped and re-claimed by another worker carries the old attempts and is rejected at the write. The reclaim itself is Jobs.reap (consumer.ex:93), which on the same server clock returns every expired lease in active to pending — crash recovery, not a vote. A lost lease is a re-claim by the next worker; because the work is idempotent, that re-run is safe.

Notes on ValkeyA Lua script runs atomically, so the read-clock-and-stamp of @claim cannot interleave with another worker, and TIME reads the server’s own clock — one clock, no cross-node comparison — valkey.io/topics/programmability.

the pattern

Redlock answers “keep a lock when a node fails” by acquiring on a majority of N independent masters and counting votes — fault tolerance bought with five instances, a clock assumption, and the validity = TTL − elapsed − drift margin.

its EchoMQ application

EchoMQ keeps one Valkey: @claim reads one server clock with redis.call('TIME') and writes the lease with ZADD active now + lease_ms <id>; attempts (HINCRBY) is the fence, refused stale with EMQSTALE, and Jobs.reap returns a lapsed lease to pending. No majority, no Redlock.

Redlock’s five-instance cost buys fault tolerance; a job queue makes its work idempotent and reads one server clock, so it keeps one Valkey and accepts the rare recovered re-claim.

a door, not a depth · the EchoMQ course

The single-Valkey lease, on one server clock

This module teaches Redlock as the contrast and cites EchoMQ’s single-Valkey lease as the road not taken — the @claim transition that reads redis.call('TIME') once, the attempts fence refused stale with EMQSTALE, and the Jobs.reap recovery that makes a lost lease a recoverable re-claim. The committed keyspace record:

content/bcs3.1.md · the v2 keyspace, the version fencefrozen
emq:{orders}:pending | emq:{orders}:job:ORD0NgWEfAEJfs | {emq}:version | {emq}:locks   ← 17 bytes before the payload
GET {emq}:version              echomq:3.0.0   ← the version fence, through the fenced connector

The worker-side lock plane and the full v2 script bundle (EVALSHA/NOSCRIPT dispatch, the version-fence internals) are the subject of the dedicated EchoMQ course. Where a cache fact applies, open The Branded Component System.

§11

The three dives

majority → clock → the contrast

Each dive takes one part of the contrast, following the source in order: the majority mechanic that gives the fault tolerance, the timing argument the whole debate turns on, and the contrast resolved — when a single-Valkey lease is enough.

After R2.03, the Coordination chapter continues with cross-shard consistency and hash-tag colocation, closing with a workshop that makes an codemojex guess submission atomic across runtimes. Return to the Coordination chapter to continue.

§12

References

sources · related

Sources

Related in this course