jonnify·redis /redis-patterns/queues/stalled-recovery

R3.03 · Reliable Queues — module hub

Stalled recovery — reclaim a dead worker's job.

Guarantee at-least-once message delivery using LMOVE to atomically transfer messages to a processing list, enabling recovery if consumers crash before completing work. A worker that dies leaves a job in flight with no one to finish it; the sweep returns it to be claimed again.

Grounding: echo/apps/echo_mq · EchoMQ.Stalled.check/3 · EchoMQ.Jobs.reap/2 — the active lease ZSET scored by the server clock, the count-thresholded sweep, the morgue. Consumer: Codemojex.CommandWorker.

One sweep over the in-flight set. Select a job to read its lease state and where the sweep sends it. The lease deadline is the death signal — expired means no live worker.
pending active · lease dead JOB-a live JOB-b exp JOB-c slow JOB-d exp3 recover dead-letter
readout · Select a job. The sweep reads ZRANGEBYSCORE active -inf now — a lease in the past means recover; a job at the stall threshold is dead-lettered; a live lease stays in flight.
§1

Recovery from failures

reliable-queue.md.txt · echo_mq

A separate monitor process — sometimes called a reaper — periodically scans the in-flight set for stalled jobs. The source describes the list-based form: check each processing queue, and if a message has been there longer than a timeout (say ten minutes), assume the worker died and move it back to the main queue with LMOVE processing work_queue RIGHT RIGHT. This guarantees at-least-once delivery — a message may be processed more than once, but it is never lost.

EchoMQ answers the same question — which in-flight job has no live worker? — with a sharper signal than a wall-clock age. When a worker claims a job, EchoMQ.Jobs.claim/3 puts its id on the active set scored by its lease deadline (now + lease_ms, read from the server clock with TIME). A worker that finishes calls complete; a worker still working extends its lease. A worker that dies does neither: the deadline passes, and the job sits in active with a score in the past. The sweep does not measure how long a message has waited — it reads every member whose lease deadline is at or before the clock.

ZRANGEBYSCORE emq:{cm.bot.commands}:active -inf <now>   # every expired lease, server clock
ZREM  emq:{cm.bot.commands}:active   CMD0N…            # claim it out of the in-flight set
ZADD  emq:{cm.bot.commands}:pending  0  CMD0N…        # return it to be claimed again

Detection is one range read over a set already sorted by death time — not a health check on each worker. Lease expired means no live worker; recover.

§2

Two layers on one signal

jobs.ex · stalled.ex

Two layers ride that one lease signal — an unconditional reaper for crash recovery, and a count-thresholded sweep on top so a job that keeps stalling is not recovered forever.

  • The reaper — crash recovery. EchoMQ.Jobs.reap/2 runs one server-side scan that returns every expired-lease id from active to pending once, with no count. EchoMQ.Consumer runs it on every beat, so a crashed worker's job is back in line within one cadence. This is at-least-once delivery in one verb.
  • The sweep — stall-count recovery. EchoMQ.Stalled.check/3 rides on top: each pass increments a per-job stalled field (HINCRBY), recovers a job whose count is below max_stalled, and dead-letters one at or above it — HSET state dead, ZADD emq:{q}:dead. A repeatedly-stalling job reaches the morgue and stops poisoning the queue.

Both run inside one Lua script, one EVALSHA. The read (which leases expired), the decide (recover or dead-letter), and the write (move the id, bump the field) happen on the server's single thread with no interleaving — so each expired lease is reclaimed exactly once per sweep, and the stall counter is incremented in-script, never by a client that could lose the update.

echo/apps/echo_mq/lib/echo_mq/stalled.ex · @sweep_stalled, the count-thresholded branchverbatim
local st = redis.call('HINCRBY', jk, 'stalled', 1)   ← this job's stall count, this pass
if st >= maxst then
  redis.call('HSET', jk, 'state', 'dead')
  redis.call('ZADD', KEYS[3], 0, id)                 ← emq:{q}:dead · dead-lettered
else
  redis.call('ZADD', KEYS[2], 0, id)                 ← emq:{q}:pending · recovered
  redis.call('HSET', jk, 'state', 'pending')
end

Notes on ValkeyThe sweep reads TIME inside the script, so the deadline that was written and the clock that reads it are one clock — a caller's clock could be skewed and recover a job whose worker is still alive — valkey.io/topics/sorted-sets.

§3

The three dives

detect → bound → one step

The arc is detect → recover safely → why one step. Read them in order — the lease is the detector, the stall count bounds the recovery, and the atomic move keeps two sweeps from colliding.

built ready to read
R3.03.1built

Lock-expiry detection

The lease deadline on the active ZSET as the death signal: ZRANGEBYSCORE active -inf now reads every member whose lease lapsed — no per-job probe.
R3.03.2built

Two-phase mark/recover

The per-job stalled count and the max_stalled threshold: recover below it into pending, dead-letter at it into dead.
R3.03.3built

Atomic vs non-atomic

One EVALSHA detect-and-move versus an app-side round-trip loop — the double-recovery window the atomic form closes.
the pattern

A separate sweep reclaims a job whose worker died, and the detect-and-move must be one indivisible step or two overlapping sweeps redeliver it. The source measures liveness as a wall-clock timeout.

its EchoMQ application

EchoMQ.Stalled is that one step: ZRANGEBYSCORE active -inf now + ZREM + ZADD + HINCRBY inside one EVALSHA, on the server clock; below max_stalled recover, at it dead-letter to emq:{q}:dead.

Three dives, one sweep: the lease detects the death, the stall count bounds the recovery, and the atomic move keeps two sweeps from reclaiming the same job twice.

§4

The pattern, applied

echo/apps/codemojex · EchoStore

A CMD command job that Codemojex.CommandWorker claimed and never completed — its handler crashed mid-dispatch — has a lease deadline in the past. The next sweep returns it to the cm.bot.commands queue's pending, and another worker drains it. A job that exhausts its stall budget does not vanish: at max_stalled it lands in emq:{q}:dead, and when the bus trims its stream history, the durable floor keeps it.

/echomq/queue

The lease and the reaper, in depth

The Queue pillar teaches the emq:{q}: state machine end to end: the lease mint, the reaper beat, the consumer fetch loop, and the inline Lua the sweep is one of.

 /echo-persistence

Where a dead-lettered job lands

A job in emq:{q}:dead is durable. EchoStore.StreamArchive folds trimmed stream segments into the Graft floor on CubDB and on to Tigris — the durability dial taught in full.

 /bcs

EchoMQ in the manuscript

Part B3 builds the bus the sweep runs on: the emq:{q}: keyspace, jobs and lanes, the Stream Tier that the archive folds into the floor.

Notes on ValkeyA grouped job recovers into its fair lane, emq:{q}:g:<group>:pending, mirroring the reaper's group branch — lane fairness survives a recovery — valkey.io/commands/hincrby.

§5

References

sources · related

Sources

  • Redis — EVALSHA — the cached atomic script: detect-and-move runs as one indivisible step on the server's single thread.
  • Redis — EVAL / scripting — why a Lua sweep cannot interleave with another sweep.
  • Valkey — Sorted Sets — the scored structure the active lease set is built on; ZRANGEBYSCORE reads every expired lease.
  • Valkey — SET — a value with a PX expiry in one command, the lease-renewal primitive a heartbeat would use.

Related in this course