jonnify·redis /redis-patterns/coordination/distributed-locking

R2.02 · Coordination — module hub

Distributed locking.

Implement mutual exclusion across distributed processes using SET key value NX PX timeout for atomic lock acquisition with automatic expiration. Two workers reach for one resource; only one may hold it, and a crashed holder must not deadlock the rest.

Grounding: echo/apps/echo_mq · Jobs.claim/3 · Jobs.reap/2 · the @claim script — the honest applied form: EchoMQ has no SET NX PX mutex; its lock is the claim lease and the fence is attempts.

Four workers race one lock — with PX the lease frees a crash, without it the crash deadlocks. Hover a worker to light its path and read the result.
worker A worker B worker C worker D resource:lockheld by A OK nil nil nil
readout · Four workers run SET resource:lock <token> NX PX 30000 at once. The key is absent for exactly one; it writes its token and gets OK, the rest get nil. Hover a worker to read its outcome.

Static verdicts — worker A → SET … NX finds the key absent, writes, OK · B, C, D → the key is now present, nil · with PX, A's crash frees the lock on lease expiry · without PX, A's crash holds it forever, a deadlock.

§1

The basic lock

distributed-locking.md.txt · The Basic Lock

A Valkey lock is the SET command with two options and a token. It sets the key only if it does not exist, attaches an expiry, and stores a value that names this acquisition. If the command returns OK, the lock was acquired; if it returns nil, another client already holds it.

SET resource:lock <token> NX PX 30000   # OK = acquired · nil = held by another client

The whole acquire is one command. Nothing runs between the existence check and the write, so two clients cannot both walk away believing they hold the lock. The figure above races a fixed set of workers at the same lock and reports who wins.

One command admits one holder; the rest get nil. With PX a crashed holder releases on its own — without it, the crash is a deadlock.

§2

Why NX, PX, and a token

distributed-locking.md.txt · Why These Options Matter
  • NX (not exists) admits exactly one holder. If two clients run the command at the same instant, only one receives OK; the other receives nil and knows the lock is taken.
  • PX (expiry) makes the lock mortal. If the holder crashes without releasing, the lease expires on its own. A lock with no expiry is held until a process clears it by hand, so a crash becomes a deadlock.
  • The token — a random value, typically a UUID — names this acquisition. It is what makes release safe: a holder deletes the lock only when the stored token still matches its own.

Three pieces, three guarantees: one holder, a mortal lock, and an owner the release can check.

§3

Releasing the lock safely

distributed-locking.md.txt · Safe Lock Release

Never release a lock with a bare DEL. A blind delete risks removing a different client's lock. The hazard runs in five steps: client A acquires with token abc; A runs long and the lease expires; client B acquires with token xyz; A finishes and runs DEL; A has now deleted B's lock. The safe release is a Lua script that checks the token before deleting, so the check and the delete are one atomic unit.

if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
else
  return 0
end

Run it with EVAL <script> 1 resource:lock abc-token-123. The lock is deleted only when the token matches, so a late delete from a stale holder becomes a no-op rather than a wrong delete. The second dive builds this release and the fencing argument that follows from it.

Notes on ValkeyA Lua script runs to completion with no command interleaved, so the GET-then-DEL check is indivisible — valkey.io/topics/programmability.

§4

Lifecycle & extending the lease

distributed-locking.md.txt · Lifecycle · Extending

The full sequence is five steps: generate a unique token; acquire with SET resource:lock <token> NX PX 30000; check the result (OK → proceed, nil → another client holds it); do the work; release with the token-checked script. When acquisition fails, a client has three options — retry with backoff, fail immediately, or block and poll until the lock frees.

For work that may run longer than the TTL, the holder extends the lock before it expires — and only if it still owns the lock. The script resets the TTL only when the stored token still matches, so a holder that has already lost the lock cannot extend a lease that is no longer its own. The shape is the safe release with pexpire in place of del. The third dive scales this renewal to many jobs at once.

Generate, acquire, check, work, release — and for a long job, renew the lease while the token still matches.

§5

Limitations & common mistakes

distributed-locking.md.txt · Limitations · Mistakes

Single-instance locking has one structural limitation. If the master fails after a client acquires a lock but before the lock replicates, the lock can be lost during failover — and two clients can then both hold it at once. Applications that need a stronger guarantee reach for the Redlock algorithm (locks from several independent instances), a consensus system such as etcd or ZooKeeper, or a design that tolerates the rare violation by making the protected operation idempotent. Redlock is the contrast taught in the next module; this one stays on the single-instance lock and points forward.

Three mistakes recur, and each turns the lock into a deadlock or a wrong delete: no expiry (SET lock token NX holds forever on a crash); a bare DEL for release (may delete another client's lock); and a non-atomic acquire — separate SETNX then EXPIRE is a race, where a crash between the two leaves a lock with no expiry, a deadlock. Always use the combined SET with NX and PX.

SETNX lock token    # if the process crashes here...
EXPIRE lock 30      # ...the lock never gets an expiry — deadlock

One command, or a race: the atomic SET … NX PX is the only acquire that cannot crash into a deadlock.

§6

When to use — and the claim lease

echo_mq/jobs.ex:283 · claim/3

Use a lock when coordinating access to an external resource, preventing duplicate job execution, electing a leader, or serializing updates to shared state. Avoid it when contention would be severe, when absolute correctness is required (reach for consensus), or when the protected resource already handles concurrent access safely.

EchoMQ does not implement a SET NX PX mutex at all. Its mutual exclusion is the claim lease: a job leased to exactly one worker for lease_ms, on the server clock, with the attempt count as the fence. EchoMQ.Jobs.claim/3 runs the @claim script, which ZPOPMINs the oldest pending id and in the same atomic step does two locking moves — it increments the fence and leases the job into the active set scored on the server's own clock.

# echo/apps/echo_mq/lib/echo_mq/jobs.ex — the @claim script (jobs.ex:126), leasing in one atomic step.
local att = redis.call('HINCRBY', jk, 'attempts', 1)        # the fencing token, +1 per grant
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)   # KEYS[2] = emq:{q}:active, scored now + lease_ms

Notes on ValkeyThe lease deadline is read inside the script with TIME, not the client's clock, so a worker's clock drift cannot misprice a lease — valkey.io/commands/zadd.

the consumer · the claim lease, against live Valkey

The lease, the fence, and the reaper

The lock is the lease-scored active set, not a string mutex — ZADD writes the deadline (valkey.io/commands/zadd) and the emq:{q}:active hashtag keeps it on one slot. EchoMQ.Consumer runs Jobs.reap/2 on its beat to return an expired lease to pending; the committed manuscript line names the keyspace:

content/bcs3.1.md · the v2 keyspace, the lease set among the per-queue keysfrozen
emq:{orders}:pending          ← score 0, mint order
emq:{orders}:job:ORD0NgWEfAEJfs   ← the three-field row: state, attempts, payload
{emq}:version | {emq}:locks      ← 17 bytes before the payload
§7

The three dives

acquire → fence → renew

Each dive takes one part of the lock, following the source in order: the atomic acquire, the fence that makes a stale writer harmless, and the renewal that keeps a long lease alive. Read them in order — the acquire first, the attempts fence second, the reaper-backed renewal third.

R2.02.1built

SET NX PX

The atomic acquire: NX admits one holder, PX keeps the lock mortal, the random token enables safe release — against the SETNX+EXPIRE race. The applied form is the lease-scored active set, no mutex.
R2.02.2built

Fencing tokens

Why the token makes release safe, then the monotonic fencing token a resource uses to reject a stale writer. The real one: EchoMQ's attempts, refused at @complete / @retry with EMQSTALE.
R2.02.3built

Lease renewal

The generic one-timer renewal, then the honest applied form: the consumer holds the lease for :lease_ms and Jobs.reap returns an expired lease to pending on the server clock.
the pattern

A lock makes one holder own a resource at a time: SET … NX PX admits one holder and bounds the hold, the token makes release safe, and a token-checked PEXPIRE keeps a long lease alive.

the claim lease in EchoMQ

@claim leases the job into emq:{q}:active scored now + lease_ms on the server clock; attempts (HINCRBY) is the fence, refused at @complete with EMQSTALE; Jobs.reap recovers an expired lease.

A one-command lock with a mortal lease; EchoMQ makes it a server-clock claim lease, fenced by a monotone attempts and recovered by a reaper — no mutex needed.

After R2.02, the Coordination chapter continues with the Redlock contrast, cross-shard consistency, and hash-tag colocation, closing with a workshop that makes a codemojex guess submission atomic across runtimes. Return to the Coordination chapter to continue, or open the Overview for where Valkey sits under codemojex.

§8

References

sources · related

Sources

Related in this course