R3.03.1 · Stalled recovery — dive

Lock-expiry detection — the lease as the death signal.

Before the reaper can recover anything it has to answer one question: which in-flight job has no live worker? EchoMQ answers it with a lease — a deadline a live worker keeps in the future and a dead worker lets pass.

Grounding: echo/apps/echo_mq/lib/echo_mq/jobs.ex · EchoMQ.Jobs.claim/3, the @claim lease mint · EchoMQ.Jobs.reap/2 — the active ZSET scored by now + lease_ms, on the server clock.

Three workers, one clock. Select a worker to see where its lease deadline sits relative to now. A deadline ahead of now is a live worker; behind it, the lease lapsed.

readout · Select a worker. A lease deadline ahead of now means a live worker; a deadline behind now means the lease lapsed and the sweep recovers the job.

§1

The lease, minted at claim

jobs.ex · @claim

When EchoMQ.Jobs.claim/3 pops the oldest pending job, it does more than hand the worker a payload — it stamps a deadline. The @claim script reads the server clock once, computes now, and scores the id on the active set at now + lease_ms (the consumer's default :lease_ms is 30_000). The same HINCRBY attempts 1 that fences the job is the worker's proof it holds the lease.

local popped = redis.call('ZPOPMIN', KEYS[1])          -- pending: oldest first
if #popped == 0 then return {} end
local id = popped[1]
local jk = ARGV[1] .. id
local att = redis.call('HINCRBY', jk, 'attempts', 1)   -- the fencing token
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')                           -- the server clock
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)  -- active, scored by lease deadline
return {id, redis.call('HGET', jk, 'payload'), att}

The score is the death signal. A worker that finishes calls complete and the id leaves active. A worker still working extends its lease — it re-claims or re-scores the deadline forward. A worker that dies does neither, and its score stays a fixed instant that the clock walks past.

A claim is more than a hand-off; it is a deadline. The lease deadline, written into the score, is the only signal the recovery sweep needs.

§2

Detect by reading the clock

jobs.ex · @reap

Because the deadline lives in the score, detection is one range read: ZRANGEBYSCORE active -inf now, every member at or below the current server clock. There is no per-job key to probe, no EXISTS round trip per id — the in-flight set is already a min-heap on the deadline, so the expired members are the cheap prefix. EchoMQ.Jobs.reap/2 does exactly this and returns each expired id to pending once.

local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
local exp = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, 100)
for _, id in ipairs(exp) do
  redis.call('ZREM', KEYS[1], id)
  redis.call('ZADD', KEYS[2], 0, id)        -- back to pending
  redis.call('HSET', ARGV[1] .. 'job:' .. id, 'state', 'pending')
end

Two ways to ask “is the worker alive?” Select a detector to read what it costs per in-flight job. The lease is one range read; a per-job lock is a round trip each.

readout · Select a detector to compare its cost. The lease score is the in-flight set already sorted by death time.

The clock is the server's, read inside the script. A caller's clock could be skewed, and a lease that expired on a fast client but not on the server would recover a job whose worker is still alive. Reading TIME inside the eval means the deadline that was written and the clock that reads it are the same clock — the master invariant for every leased transition on the bus.

Notes on ValkeyZRANGEBYSCORE active -inf now returns every member whose score is at or below the server clock — the expired prefix of a set sorted by deadline — valkey.io/commands/zrangebyscore.

§3

Live, slow, dead — three scores

consumer.ex · :lease_ms 30_000

The lease distinguishes the three states a worker can be in without a separate health channel.

Live and fast — completes before the deadline; the id leaves active, never seen by a sweep.
Live but slow — extends the lease before it lapses; the deadline moves forward, so the slow worker is never reclaimed out from under itself.
Dead — neither completes nor extends; the deadline passes, and the next sweep returns the job to pending.

The lease length is the only tuning knob: too short and a slow-but-alive worker is reclaimed mid-job (a needless redelivery); too long and a dead worker's job waits the full lease before recovery. The default 30_000 ms is the consumer's; a workload with long handlers raises it, and a workload that needs fast recovery lowers it and extends the lease on a heartbeat.

the pattern

A separate reaper finds a stalled job by asking whether its worker is still alive — the source measures that as “older than a timeout.”

its EchoMQ application

The answer is a lease deadline on the active ZSET, scored by now + lease_ms from the server clock; ZRANGEBYSCORE active -inf now reads every member whose lease lapsed — no per-job probe, one range read.

Detection is not a health check on each worker; it is one range read over a set already sorted by death time.

/echomq/queue

The lease and the fetch loop

The Queue pillar teaches the claim, the lease mint, and the consumer's reap beat in depth — the engine behind this one range read.

R4 · Time, Delay & Priority

The sorted set as a clock

The same scored structure the lease uses — a member's score is a future instant, and a range read finds everything due.

§4

References

sources · related

Sources

Valkey — Sorted Sets — the active set is a ZSET scored by lease deadline; the expired prefix is -inf to now.
Valkey — ZRANGEBYSCORE — read every member whose score is at or below the server clock.
Valkey — SET — SET key value PX <ms>, the expiry primitive a per-job lock would use; the lease puts the same idea in a score.
Redis — EVAL / scripting — reading TIME inside the script binds the deadline and the clock that reads it.

Related in this course

R3.03 · Stalled recoverythe module hub
R3.03.2 · Two-phase mark/recoverthe stall count, once a job is detected
R3.03.3 · Atomic vs non-atomicwhy the detect-and-move is one step
R4 · Time, Delay & Prioritythe sorted set as a clock
/echomq/queuethe lease and the worker fetch loop in depth