R2.02.2 · Distributed locking — dive 2

The token makes release safe.

A blind DEL deletes whatever lock is there now. After a lapsed lease, that is the successor's lock, not yours. The fix conditions the act on a token; the deeper fence pushes the same check into the protected resource.

Grounding: the real fence — echo/apps/echo_mq/jobs.ex:131, attempts via HINCRBY in @claim; @complete (jobs.ex:139) / @retry (jobs.ex:173) refuse a stale token with EMQSTALE.

The lost-lock deletion — blind DEL vs a token check. Hover a release to light its path and read what the resource holds at the end.

readout · A stalled past its lease, B took the same key. Hover blind DEL to watch A delete B's lock, then the token-checked release to watch A's release no-op.

Static verdicts — blind DEL → A deletes B's lock, two clients now proceed, exclusion broken · token-checked release → A reads "xyz" ≠ "abc", returns 0, B's lock stands.

§1

Never DEL a lock blindly

distributed-locking.md.txt · Releasing the Lock

The release looks harmless: the work is done, so the holder calls DEL resource:lock. The hazard is that the key under resource:lock may no longer hold the lock the caller took. A lease expires on its own — that expiry is what keeps a crashed holder from deadlocking the key. Once the lease lapses, another client can take the same key. A late DEL then deletes that client's lock.

# DANGEROUS: deletes whatever lock is under the key right now
DEL resource:lock

The scenario the source spells out, step by step:

1. A acquires: Client A takes the lock with token abc.
2. A stalls: Client A takes too long; the lease expires on its own.
3. B acquires: Client B takes the same key with token xyz.
4. A releases: Client A finishes and calls DEL.
5. wrong lock: The DEL removes client B's lock, not A's.

Now two clients hold the lock at once. The fault is not the expiry; the fault is releasing without checking whose lock is under the key.

§2

The token-checked release

distributed-locking.md.txt · Safe Lock Release

The fix conditions the delete on the token: read the key, and delete it only if the stored value is still the token the caller acquired with. The read and the delete must be one atomic step. Between a bare GET and a separate DEL, the lease could expire and another client could take the key. A Lua script runs to completion with no command interleaved.

# safe release: DEL only if the stored token is still ours
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

Called with EVAL <script> 1 resource:lock abc-token-123. In the A/B scenario, A's release reads xyz under the key, finds it is not abc, and returns 0 — a no-op. B's lock stands. The token is the whole defence: without it, release has no way to tell its own lock from a successor's.

Notes on ValkeyA Lua script is atomic: its read-compare-delete cannot interleave with another writer — valkey.io/topics/programmability.

A lock without an owner token is a delete waiting to hit the wrong client.

§3

Fencing a stale writer

Kleppmann · Sanfilippo

The token-checked release closes the in-the-store hazard, but a longer argument remains. Suppose a holder is paused — a long garbage-collection pause, a scheduler stall, a network partition — past the end of its lease. While it is paused, the lease expires, another worker takes the lock, and that worker starts writing to the protected resource. Then the first holder resumes. It still carries its lock value. It writes.

A lock alone cannot stop that write, because from the resource's side the late write arrives like any other. Martin Kleppmann's argument is that the resource itself must reject the stale writer, and the mechanism is a fencing token: a number that increases every time the lock is granted. Each writer presents its token with every write; the resource keeps the highest token it has accepted and rejects any write carrying a lower one.

# grant 1 -> token 33; grant 2 -> token 34; ...
# the resource accepts a write only if its token >= the highest seen
write(payload, fencing_token)   # rejected when fencing_token < max_seen

The paused holder acquired with token N. While it was paused, the next holder acquired with token N+1 and wrote. The resource has now seen N+1. When the paused holder resumes and writes with token N, the resource compares N against the N+1 it has already accepted and rejects the write. The lease being a wall-clock guess no longer decides correctness. Hover a writer below to run the check against a resource that has seen token 34.

Run a fenced write — the resource has seen token 34. Hover a writer to light its path and read whether the write is accepted.

readout · The resource has accepted token 34. Hover a writer to read whether its token clears the fence.

Static verdicts — 33 (paused) → 33 < 34, rejected · 34 (current) → 34 ≥ 34, accepted · 35 (next) → 35 > 34, accepted, max seen becomes 35.

Salvatore Sanfilippo, Redis's author, replied to that critique in Is Redlock safe?. The honest summary is that a monotonic fencing counter and an owner token solve overlapping but not identical problems — the owner token stops the wrong client's lock being deleted; the fencing token stops a holder whose lease has already lapsed.

A fencing token moves correctness into the resource: it rejects anything older than what it has already accepted, so a wall-clock lease no longer has to be exact.

§4

The applied form — `attempts` is the fence

echo_mq/jobs.ex:131,139 · @claim · @complete

EchoMQ does not run a separate string lock with an owner token. Its fence is the monotone attempt count built into every claim, and it is precisely the fencing-token idea Kleppmann describes — a number that increases each time the job is granted, presented back on every transition. EchoMQ.Jobs.claim/3's @claim script increments it as it leases the job; the worker now holds att and must present it back.

# echo/apps/echo_mq/lib/echo_mq/jobs.ex:131 — the fence is set in @claim.
local att = redis.call('HINCRBY', jk, 'attempts', 1)   # the fence: +1 per grant
redis.call('HSET', jk, 'state', 'active')
# ... ZADD the job into emq:{q}:active scored now + lease_ms ...
return {id, redis.call('HGET', jk, 'payload'), att}     # the worker carries att, presents it back

Those scripts fence on it. @complete reads the stored attempts and refuses a mismatch:

# echo/apps/echo_mq/lib/echo_mq/jobs.ex:139 — @complete refuses a stale token.
local att = redis.call('HGET', KEYS[2], 'attempts')
if not att then return 0 end
if att ~= ARGV[2] then
  return redis.error_reply('EMQSTALE complete token mismatch')
end

@retry (jobs.ex:173) carries the same EMQSTALE retry token mismatch fence. The chain is the whole fence: a worker whose lease was reaped and re-claimed by another worker carries a stale att, because the re-claim ran HINCRBY attempts 1 again and bumped the stored value past the paused worker's copy. When the paused worker resumes and calls complete with its old att, the stored attempts is higher, the comparison fails, and the transition is refused — exactly Kleppmann's “reject the lower token.” EchoMQ.Jobs.complete/4 surfaces this as {:error, :stale}.

Notes on ValkeyHINCRBY increments a hash field atomically and returns the new value, so the attempt count is a monotone counter with no read-then-write race — valkey.io/commands/hincrby.

the pattern

Owner-token release: never DEL blindly; delete only if the value under the key is still your token. A monotone fencing token extends the check into the resource, which rejects any write below the highest it has accepted.

attempts in EchoMQ

attempts (HINCRBY in @claim) is the monotone fence; the worker carries it and presents it back, and @complete / @retry refuse a stale token with EMQSTALE — a re-claimed job fences out the lapsed worker.

door · the EchoMQ course

The worker-side lock plane

This dive cites the attempts fence as proof the pattern is real. The worker-side lock-subkey plane — emq:{q}:job:<id>:lock, the §6 subkey remove_job already refuses to clear with the EMQLOCK class — is taught in the dedicated EchoMQ course. Keeping the lease fresh so a long job is not reaped mid-work is the next dive, R2.02.3 · Lease renewal.

§5

Recap — the fence is a number

next → R2.02.3

A blind DEL deletes whatever lock is under the key, which after a lapsed lease is a successor's lock. The token-checked release deletes only your own. The fencing-token argument pushes the same idea into the resource so a holder paused past its lease cannot write through it. EchoMQ's fence is attempts: HINCRBY'd once per claim, carried by the worker, and checked at @complete / @retry, which refuse a stale token with EMQSTALE. A reaped and re-claimed job bumps the stored count, so the paused worker's old count fences it out. The next dive keeps the lease fresh so a long job does not get reaped mid-work.

§6

References

sources · related

Sources

Redis — Distributed Locks — single-instance acquisition, owner-token release, and the failover caveat.
Valkey — Programmability — why a GET-then-DEL, or an HINCRBY-then-ZADD, must be one atomic script.
Valkey — HINCRBY — the atomic hash increment behind the monotone attempt count.
Sanfilippo, S. — Is Redlock safe? — the reply to the fencing critique, weighing the lock service against the resource.

Related in this course

R2.02 · Distributed lockingthe module hub
R2.02.1 · SET NX PXwhere the lease and the fence are taken
R2.02.3 · Lease renewalthe next dive
/echomq/protocolthe EchoMQ protocol · the worker-side lock plane