jonnify·redis /redis-patterns/queues/the-reliable-queue

R3 · dive 1

The reliable queue.

A reliable queue loses no job. The naive loop — claim, process, done — drops a job the instant a worker dies between the claim and the finish.

Three guarantees build a queue that survives a crash, and they build in order: claim a job into a recoverable in-flight state, so a worker that dies leaves it reclaimable, not lost; accept that redelivery means at-least-once, so the consumer must be idempotent; and reclaim a job whose worker died — its lease expired — back to pending. Each guarantee is EchoMQ's real worker path, grounded in echo/apps/echo_mq. Toggle the worker below to crash mid-job, and watch the job stay leased in emq:{q}:active until the reap returns it to emq:{q}:pending.

The recoverable claim — finish completes the job, a crash leaves it leased. Hover or select a step to light its path and read the move it runs.
pendingemq:{q}:pending activeunder a lease donerow retired JOB ZPOPMIN complete/5 crash → leased reap → pending
readout · Select a step. The job is claimed pending→active under a lease; finish completes it; a crash leaves it leased; reap returns an expired lease to pending.
§1

The recoverable claim

echo_mq/jobs.ex · @claim

A pop-and-process loop is the leak. If a job leaves the queue the moment it is taken — before any work has happened — and the worker then crashes (a deploy, an OOM kill, a power loss), the job is neither in Valkey nor finished. It is lost, with nothing to recover it from.

The fix is to never let a job leave a recoverable place until it is done. EchoMQ does not pop a job out: it claims it from the pending sorted set into the active set under a lease. EchoMQ.Jobs.claim/3 runs one inline Lua script — ZPOPMIN emq:{q}:pending takes the oldest id, HINCRBY bumps the row's attempts, HSET marks the row active, and ZADD emq:{q}:active parks the id scored by its lease deadline, where the deadline is read from the server's TIME. The job is in exactly one set at every instant; a worker that dies leaves its job leased in the active set, recoverable.

-- @claim (inline EchoMQ.Script.new/2) — the leased, recoverable claim
local popped = redis.call('ZPOPMIN', KEYS[1])         -- KEYS[1] = emq:{q}:pending
if #popped == 0 then return {} end
local id = popped[1]
local att = redis.call('HINCRBY', jk, 'attempts', 1)  -- the fencing token
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')                          -- the server clock, not the caller's
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)  -- KEYS[2] = emq:{q}:active, lease deadline
return {id, redis.call('HGET', jk, 'payload'), att}

A job is never popped out into nowhere — it is claimed into the active set under a lease, so a crash leaves it leased and reclaimable, not lost.

§2

At-least-once and idempotency

echo_mq/jobs.ex · @complete

The recoverable claim makes redelivery possible — that is the whole point. It also makes redelivery inevitable in one case: a worker that finishes the work, then crashes before it completes the job. On recovery the lease has expired, so the job is reclaimed and the work runs again.

This is the honest cost of never losing a job. Exactly-once delivery is a lie — you cannot both guarantee a job is never lost and guarantee it is never delivered twice, because the completion itself can be lost. A reliable queue offers at-least-once: a job is delivered one or more times, never zero. The responsibility moves to the consumer, which must be idempotent.

The stable key for that is the job's identity. Every EchoMQ job is a branded JOB id, gated at the key builder (EchoMQ.Keyspace.job_key/2). Key the effect on that id and a redelivery is a no-op. EchoMQ also fences completion itself: EchoMQ.Jobs.complete/5 carries the attempts token the claim minted, and a completion whose token no longer matches is refused, so a stale worker cannot complete a job another worker now holds. Slide the delivery count below and watch the id-keyed effect stay at one.

At-least-once vs effect-once — N deliveries, the JOB-id key, how many effects. Select the consumer kind.
deliver 1 effect deliver 2 effect deliver 3 effect 3 deliveries → 1 effect

Exactly-once delivery is a lie; a reliable queue is at-least-once. Keying the effect on the branded JOB id makes the effect happen once across every redelivery.

§3

Stalled recovery

echo_mq/jobs.ex · @reap · stalled.ex

A job in the active set whose worker has died must come back. The death signal is a lease that expired: the active set is scored by each job's lease deadline, so any member whose score is in the past is held by no live worker. EchoMQ.Jobs.reap/2 is the recovery — one inline Lua sweep over emq:{q}:active for members scored at or before now (server TIME), each ZADDed back to emq:{q}:pending and re-marked pending. The job re-enters the normal pickup path and is delivered again — at-least-once in action.

The sweep must itself be careful. Done as two steps — read the expired set, then move each member — two recovery processes can both see the same stalled job and both move it. EchoMQ runs detect-and-move as one Lua script over the active and pending sets, so each stalled member is reclaimed once per sweep. A non-atomic check-then-move is the cautionary contrast, correct only until two sweeps overlap.

Above the bare reaper sits EchoMQ.Stalled.check/3, a count-thresholded sweep: each pass increments a per-job stalled field, recovers a job below the :max_stalled threshold, and dead-letters one at or above it — so a job that repeatedly stalls is not recovered forever, but lands on the durable floor.

Notes on ValkeyZRANGEBYSCORE active -inf now selects every member whose lease deadline is at or before the server's now, in one pass — the reap reads the same TIME the claim wrote, so a lease is never judged expired against a clock it never saw — valkey.io/commands/zrangebyscore.

§4

In EchoMQ — the worker path that loses no job

echo/apps/echo_mq

EchoMQ's worker loop is all three guarantees in real code. The recoverable claim is EchoMQ.Jobs.claim/3 — ZPOPMIN from emq:{q}:pending into emq:{q}:active under a server-clock lease. The idempotency key is the branded JOB id, fenced by the attempts token on EchoMQ.Jobs.complete/5. Stalled recovery is EchoMQ.Jobs.reap/2 and the count-thresholded EchoMQ.Stalled.check/3. The EchoMQ.Consumer loop drives them on a beat: reap, then drain.

the pattern

A reliable queue claims a job into a recoverable in-flight state under a lease, redelivers it at-least-once to an idempotent consumer, and reclaims a crashed job whose lease expired.

its EchoMQ application

EchoMQ.Jobs.claim/3's ZPOPMIN + server-clock lease, the branded JOB id fenced by the attempts token on complete/5, and EchoMQ.Jobs.reap/2 with EchoMQ.Stalled.check/3 — the real Elixir worker path.

The naive pop-and-process loses a job on a crash; EchoMQ's leased claim, its JOB-id-keyed idempotent effect, and its reap sweep are the three guarantees that make the queue lose none.

→ EchoMQ · /echomq/queue

The state machine in depth

The full worker fetch loop — the lease cadence, the metronome that fans readiness across a worker pool, fair lanes, and the schedule set — is the dedicated EchoMQ Queue pillar. This dive teaches the reliable-queue pattern; that course teaches the engine that runs it.

/echomq/queue · the EchoMQ Queue pillar

§5

References

sources · related

Sources

  • Valkey — ZPOPMIN — the pop that claims the oldest pending job from the score-0 mint-ordered set, the heart of the claim script.
  • Valkey — HINCRBY — increments the row's attempts, the lease fencing token that makes a stale completion a no-op.
  • Valkey — TIME — the server clock the lease deadline and reap both read, never the caller's.
  • Redis — Scripting with Lua — how the multi-key claim and the reap sweep each run as one atomic script.

Related in this course