R2.03.3 · The Redlock algorithm — dive 3
When single-instance is enough.
Redlock works. The question this dive resolves is whether a workload needs what it buys — because the five-instance cost is real.
Grounding (contrast): echo/apps/echo_mq/jobs.ex:126 · consumer.ex:57,93 — EchoMQ
deliberately chose the single-Valkey lease: @claim against one server clock, the
attempts fence, the Consumer’s :lease_ms default 30_000, and
Jobs.reap.
Static verdicts — recoverable + one Valkey → single-instance lease · recoverable + five → single-instance suffices, Redlock over-provisions · catastrophic + one Valkey → consensus · catastrophic + five → Redlock + fence.
When to use Redlock
redlock.md.txt · when to useRedlock is the right tool for a specific shape of problem: coordination across machines where a lost lock is costly enough to justify fault tolerance, but not so catastrophic that only a consensus system will do. The source names four fits and four signals to reach elsewhere.
Redlock is appropriate for coordinating access to an external resource that has no concurrency control of its own; for preventing duplicate job execution in a distributed task queue; for distributed rate limiting where approximate correctness is acceptable; and for leader election among application instances. Each shares a trait: the work is coordination, the consequence of a brief overlap is tolerable or recoverable, and the operational cost of five masters is worth paying for fault tolerance a single instance cannot give.
Reach for an alternative when the shape is different. Use a consensus system — etcd or ZooKeeper — when absolute correctness is required and a lock violation cannot be tolerated. Use fencing at a linearizable resource when the storage layer can enforce a monotonic token, a stronger guarantee than any lock. Stay with single-instance Redis when its reliability is sufficient. And avoid Redlock when the operational complexity of five independent masters is not justified by the failure it prevents. That last line is the hinge: Redlock’s price is not the algorithm — it is the infrastructure.
Static verdicts — job queue / rate limit → single-instance sufficient (short) · leader election → Redlock-class fits (medium) · money transfer → a fence or consensus (long, corrupts a balance).
The bar is the guarantee the work demands, not the lock on offer. Three of the four workloads are short; the money transfer is long, because a lost lock corrupts a balance and only a fence or a consensus system holds.
The decision
redlock.md.txt · the decisionThe whole module collapses to one decision, and it is about the consequence of a lost lock, not the lock itself. A lock can always be lost — a failover, a clock jump, a process pause past the lease. What separates the right answer is what happens next.
If a lost lock causes catastrophic corruption — two writers mutate a balance, a transfer is applied twice with no record, an external system is driven into an inconsistent state it cannot recover from — then the lock is a correctness boundary, and a lock that can fail under timing assumptions is the wrong boundary. The answer is a consensus system that does not rest on bounded clocks, or a resource that enforces a fencing token, so a stale holder is rejected where the data changes.
If a lost lock causes a recoverable retry — the same work runs twice, and running it twice is safe because the operation is idempotent and a recovery path reclaims the orphaned unit — then the lock is an optimization, not a correctness boundary. Here a single-instance lease is sufficient, and the five-master cost buys a guarantee the workload does not need. A job queue lands squarely on the recoverable side, and that is the contrast EchoMQ makes concrete.
The axis is the consequence of a lost lock: catastrophic corruption needs more than a lock; a recoverable retry needs less. A job queue reads the recoverable cell.
The summary table
redlock.md.txt · the tradeThe source closes on a four-row comparison. It is the trade in one frame: Redlock buys fault tolerance with complexity, and a single instance keeps it simple at the cost of surviving a node failure.
| Aspect | Single instance | Redlock |
|---|---|---|
| Fault tolerance | None — a node failure loses the lock | Tolerates a minority of node failures |
| Complexity | Low — one Redis, one SET … NX PX | Higher — five independent masters, the acquire-and-time-check protocol |
| Consistency | Lost on failover (the master-replica race) | Maintained while a majority survives |
| Use case | Non-critical coordination, or recoverable work | Important distributed coordination |
The trade has no universal winner. Redlock buys fault tolerance with five masters; a single instance keeps one and accepts that a failover drops the lock. The workload picks the axis.
The contrast, resolved — EchoMQ keeps one Valkey
echo_mq/jobs.ex:126 · consumer.ex:57,93EchoMQ takes the single-instance side, and the choice is deliberate, not a compromise. A job queue does not
need absolute mutual exclusion; it needs one active worker per job in the common case and a safe recovery when
that breaks. The lock is the claim lease on one Valkey. The @claim script
(echo/apps/echo_mq/lib/echo_mq/jobs.ex:126, run by Jobs.claim/3 at jobs.ex:283)
reads one server clock and stamps the lease — no majority of N and no second master.
# @claim (jobs.ex:126) — the single-Valkey lease, on one server clock, with the attempts fence local att = redis.call('HINCRBY', jk, 'attempts', 1) # the monotone fence, returned to the worker redis.call('HSET', jk, 'state', 'active') local t = redis.call('TIME') # one clock — no five masters, no vote local now = t[1] * 1000 + math.floor(t[2] / 1000) redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)
The reason this is safe is the recovery path, not the lock. The EchoMQ.Consumer holds the lease
for :lease_ms (default 30_000, echo/apps/echo_mq/lib/echo_mq/consumer.ex:57);
on each beat it calls Jobs.reap (consumer.ex:93), which returns every expired lease
in active to pending on the same server clock. The next worker re-claims the job and
increments attempts; when the original worker resumes, @complete/@retry
refuse its stale token with EMQSTALE … token mismatch. Because the handler is idempotent, the
re-run produces the same result. A lost lease becomes a double-run the queue recovers from, not two writers
corrupting a resource — exactly the move Redlock’s critics prescribe, applied where the data
changes instead of at the lock.
Notes on ValkeyTIME reads the server’s own clock,
and a Lua script runs atomically, so the lease, the fence, and the reclaim each land as one transition —
one Valkey, no majority —
valkey.io/topics/programmability.
Redlock buys fault tolerance with five independent masters — the right call when a lost lock corrupts data and no cheaper guarantee holds the line.
@claim holds the lease on one Valkey against one server clock, with
attempts (HINCRBY) the fence refused stale with EMQSTALE; the
EchoMQ.Consumer holds it for :lease_ms (30_000) and
Jobs.reap reclaims a lapsed lease as a safe idempotent re-claim — so the queue keeps one
Valkey instead of five.
Reach for Redlock when a lost lock corrupts; a queue makes the work idempotent, reads one server clock, and keeps one Valkey. The five-master cost buys a guarantee a recoverable workload does not need.
The single-Valkey lease, the version fence
EchoMQ does not implement Redlock — this dive cites the single-Valkey lease as the
road not taken: @claim on one Valkey, the attempts fence refused stale with
EMQSTALE, the Consumer’s :lease_ms, and Jobs.reap reclaiming a
lapsed lease. The committed keyspace record:
emq:{orders}:pending | emq:{orders}:job:ORD0NgWEfAEJfs | {emq}:version | {emq}:locks ← 17 bytes before the payload
GET {emq}:version echomq:3.0.0 ← the version fence, through the fenced connector
The worker-side lock plane and the full v2 script bundle are the subject of the dedicated EchoMQ course; where a cache fact applies, open The Branded Component System.
Recap — the contrast resolved
back → R2.03Redlock is a real tool with a real price: five independent masters, no replication, the
acquire-and-time-check protocol. The source names where it fits — external resources, duplicate-job
prevention, approximate rate limiting, leader election — and where to reach past it: a consensus system
when correctness is absolute, a fenced resource when the storage layer supports a token, a single instance
when its reliability is sufficient. The whole module reduces to one axis: a lost lock that causes catastrophic
corruption needs more than a lock; a lost lock that causes a recoverable retry needs less. EchoMQ reads the
recoverable cell — @claim on one Valkey, the attempts fence, and
Jobs.reap reclaiming a lapsed lease, with an idempotent handler that makes a re-run safe —
and keeps one Valkey.
References
sources · relatedSources
- Redis — Distributed Locks (and Redlock) — the canonical algorithm and the when-to-use guidance: the four fits, the four signals to reach for a consensus system, and the operational cost of five independent masters.
- Sanfilippo, S. — Is Redlock safe? — the Redis creator’s defence: for the coordination tasks Redlock targets, the timing checks are sufficient in practice.
- Kleppmann, M. — How to do distributed locking — the critique that draws the line: when a lock violation corrupts data, fence at the resource or use consensus.
- Valkey — SET —
SET resource <token> NX PX <ttl>, the single-instance lease that is the contrast Redlock weighs against. - Valkey — Programmability — atomic Lua execution: why EchoMQ’s
@claimis one transition over the single-Valkey lease.
Related in this course
- R2.03 · The Redlock algorithmthe module hub
- R2.02 · Distributed lockingthe single-Valkey lease that is the contrast
- R2 · Coordination & Consistencythe chapter
- /echomq/protocolthe EchoMQ protocol the claim lease lives in
- /elixir · CQRSthe functional-Elixir & OTP craft behind the echo umbrella