R2.02.3 · Distributed locking — dive 3

Keep the lease fresh.

A 30-second lease and a long job will lapse mid-work. The generic fix extends the lock before it expires — but only while the token is still yours: if GET == token then PEXPIRE.

Grounding: the honest applied form — echo/apps/echo_mq/consumer.ex, the loop runs Jobs.reap/2 (jobs.ex:357) on its beat to return an expired lease to pending; :lease_ms default 30_000 (consumer.ex:57).

The renewal heartbeat — a 75 s job under a 30 s lease. Hover a mode to light its timeline and read whether the job survives.

readout · A 75-second job runs under a 30-second lease. Hover “no renewal” to watch the lease lapse, then “renew at TTL/2” to watch it held to the end.

Static verdicts — no renewal → the lease ends at 30 s, lapses, the job is reclaimed and may double-run · renew at TTL/2 → the lease is pushed out every 15 s, held past 75 s, the job finishes once.

§1

Extend before the lease expires

distributed-locking.md.txt · Extending Lock Duration

The lease is a deadline, and a long job can outrun it. A 30-second PX set at acquire is a guess at how long the work will take; for work that may take longer, the holder extends the lease while it is still held. The extension resets the TTL — but it must be conditional. Extend the key only if the value under it is still the token the holder acquired with.

# extend: reset the TTL only if the stored token is still ours
if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("pexpire", KEYS[1], ARGV[2])
else
  return 0
end

The shape is the token-checked release with pexpire in place of del: the same GET == token admission, a different verb. If the token still matches, the TTL is pushed out; if it does not — the lease lapsed and another client took the key — the script returns 0 and the holder has lost the lock. A holder that renews on a fixed cadence shorter than the lease keeps the lock for as long as it still owns it, and stops the instant the lock is gone.

PEXPIRE key ms: Reset a key's time to live, in milliseconds. The extend swaps it in for DEL: the lock survives instead of being removed.
GET == token: The owner check. The same guard that makes release safe, reused: extend only what is still yours.
cadence: How often the holder renews. Renewing at half the lease leaves a missed tick the full lease's grace before the lock expires.
returns 0: The token no longer matches — the lease lapsed and a successor took the key. The renewal is a no-op, and the 0 tells the holder the lock is gone.

This extend is the R2.02.2 token-checked release with one verb changed. The acquire that takes the lease is R2.02.1 · SET NX PX.

§2

The lifecycle — renewal sits inside the work

distributed-locking.md.txt · Lock Lifecycle

The lifecycle the source spells out is: generate a token, acquire (SET … NX PX), check the result, do the work, release with the token-checked script. Renewal sits inside step four. For work that may outrun the lease, the holder renews on a cadence for the whole duration of the work, then releases.

Renewal is what makes a short lease safe for a long job. A short lease bounds the damage when a holder crashes — a crashed holder's lock expires in seconds, not minutes. Renewal lets a live holder keep that short, safe lease as long as it needs it. The two together give a lock that is both mortal (it cannot deadlock the key) and durable (it does not expire under a holder still doing the work).

short lease

A crashed holder's lock expires fast, so the resource is freed in seconds — the safety the PX on acquire buys.

renewal

A live holder pushes the lease out on a cadence, so a long job keeps its lock — the durability the PEXPIRE extend buys.

A short lease bounds a crash; renewal extends a live holder. The lock is mortal and durable at once — neither alone is enough.

§3

One timer per job does not scale

distributed-locking.md.txt · the renewal layer

The straightforward implementation is one renewal timer per held lock: a job acquires, schedules a timer at half its lease, the timer fires a single-key extend, reschedules, and cancels on release. For one job that is fine. For a worker running many jobs at once it is not. K jobs means K timers, K wake-ups per renewal cycle, and K separate round-trips each cycle. The work each call does is small — a GET, a compare, a PEXPIRE — but the count grows with the number of held jobs, and the renewal layer becomes load that scales with concurrency rather than with the work. Hover the two designs below to compare the round-trips each renewal tick costs.

Per-job timers vs one server-side sweep — round-trips per renewal tick. Hover a design to light it and read the cost for a worker holding K jobs.

readout · A worker holds K = 6 jobs. Hover “K per-job timers” for the per-job cost, then “one server-side sweep” for the bounded one.

Static verdicts — K per-job timers → 6 extend round-trips per tick, scaling with held jobs · one server-side sweep → 1 bounded sweep per tick, scaling with ticks not jobs.

Per-job timers make the renewal traffic scale with the held-job count. A single server-side sweep makes it scale with the number of ticks instead.

§4

The applied form — the consumer's lease and the reaper

echo_mq/consumer.ex:91 · jobs.ex:357

EchoMQ does not renew a worker-held lock on a holder-side timer in the shipped claim path. The honest applied form is simpler and lives on the server clock: the consumer holds the lease for :lease_ms, and a single loop — the reaper — returns any expired lease to pending. Crash recovery is the server clock, not a holder heartbeat. EchoMQ.Consumer is one supervised loop that beats on a cadence and runs the reaper first on every beat.

# echo/apps/echo_mq/lib/echo_mq/consumer.ex:91 — the loop runs the reaper each beat.
defp loop(s) do
  check_control()
  {:ok, _} = Jobs.reap(s.conn, s.queue)              # return expired leases to pending
  {:ok, _} = Jobs.promote(s.conn, s.queue, s.pump_batch)
  drain(s)                                            # claim and run ready jobs
  park(s)                                             # BLPOP on the wake key until a wake or the beat elapses
  loop(s)
end

Jobs.reap/2 (jobs.ex:357) runs the @reap script, which reads the server clock and returns every job in emq:{q}:active whose lease score is past now back to emq:{q}:pending:

# echo/apps/echo_mq/lib/echo_mq/jobs.ex:245 — @reap returns expired leases on the server clock.
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
local exp = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, 100)   # KEYS[1] = emq:{q}:active
for _, id in ipairs(exp) do
  redis.call('ZREM', KEYS[1], id)
  # ... ZADD the id back into emq:{q}:pending at score 0, HSET state = pending ...
end
return #exp

The default :lease_ms is 30_000 (consumer.ex:57). A crashed worker's lease is never renewed, so its deadline passes; the next reaper beat returns its job to pending; another worker re-leases it by re-claiming. The PEXPIRE-extend of the generic pattern becomes a server-side reaper sweep — no holder-side timer, no per-job round-trip, one bounded ZRANGEBYSCORE per beat. The re-claim runs HINCRBY attempts 1 again, so the crashed worker's stale attempt count fences it out if it ever resumes.

Notes on ValkeyZRANGEBYSCORE with a LIMIT reads a bounded slice of a sorted set by score, so the reaper sweeps at most a fixed batch of expired leases per beat — valkey.io/commands/zrangebyscore.

the consumer · the reaper, against live Valkey

The lease set, recovered on the server clock

The lease is a score in the active sorted set, and the reaper returns it on the server's own TIME (valkey.io/commands/zrangebyscore); the emq:{q}:active hashtag keeps it on one slot. The committed manuscript line names the keyspace:

content/bcs3.1.md · the v2 keyspace, the lease set among the per-queue keysfrozen

emq:{orders}:pending          ← where the reaper returns an expired lease
emq:{orders}:job:ORD0NgWEfAEJfs   ← the three-field row: state, attempts, payload
{emq}:version | {emq}:locks      ← 17 bytes before the payload

The worker-side lock-subkey renewal plane — the emq:{q}:job:<id>:lock subkey a long-running worker holds and renews, which remove_job refuses to clear with the EMQLOCK class — is taught in the dedicated EchoMQ course.

§5

Recap — the reaper renews on the server clock

back → R2.02

A lock with a short lease will lapse under a long job. The generic fix is conditional renewal: if GET == token then PEXPIRE. One timer per job does not scale. EchoMQ's answer is the reaper: the consumer holds the lease for :lease_ms, and on every beat Jobs.reap returns any active job whose server-clock deadline has passed back to pending, where another worker re-leases it by re-claiming. That closes R2.02: lease one holder via the claim, fence a stale worker on attempts, and recover an expired lease with the reaper on the server clock.

the pattern

Extend the lease before it lapses: if GET == token then PEXPIRE, renew only while you still own the lock — and at scale, one timer rather than one per job.

the reaper in EchoMQ

EchoMQ.Consumer holds the lease for :lease_ms and runs Jobs.reap on its beat; the @reap script returns any expired lease to pending on the server clock — one bounded sweep, no holder-side timer.

§6

References

sources · related

Sources

Redis — Distributed Locks — single-instance lock acquisition, owner-token release, and lock extension.
Valkey — PEXPIRE — reset a key's time to live in milliseconds; the verb the generic extend swaps in for DEL.
Valkey — ZRANGEBYSCORE — the bounded range read the reaper uses to find leases past the server clock.
Valkey — Programmability — why the reaper's read-and-move runs as one atomic script.

Related in this course

R2.02 · Distributed lockingthe module hub
R2.02.1 · SET NX PXthe claim lease the reaper recovers
R2.02.2 · Fencing tokenswhy a re-claimed job fences the lapsed worker
/echomq/protocolthe EchoMQ protocol · the worker-side lock renewal plane