R3 · Reliable Queues
Reliable queues —
lose no job.
The pivot of the course: a job is never lost — claimed under a server-clock lease, completed once that counts, reclaimed if its worker dies.
A reliable queue is not one pattern but a family — a recoverable in-flight claim,
at-least-once delivery to an idempotent consumer, recovery of a job whose worker crashed, the whole lifecycle
moved as one inline Lua script, and pickup without busy-polling. The grounding is the real EchoMQ
leased state machine (echo/apps/echo_mq) — EchoMQ.Jobs.claim/3 pops the oldest
pending JOB with ZPOPMIN, mints a lease on the server clock, and increments the
row's attempts fencing token — worked through codemojex's guess-command queue. This
chapter stands on R2's atomic moves.
Why & when
echo/apps/echo_mqA naive queue pops a job off one end and hands it to a worker. The moment that worker crashes — mid-job, before the work is acknowledged — the job is gone from Valkey and gone from memory, with nothing to reclaim it. Reach for the reliable-queue family whenever losing a job is not acceptable and a worker can crash at any instant, which is every real background-job system. Each failure mode below has one matching technique, and the chapter is the set of those answers.
A worker crashes mid-job
The job must not vanish when a worker dies — claim it under a server-clock lease, and reap it back to pending when the lease expires.
A job runs twice
At-least-once delivery means a job can arrive more than once — make the consumer idempotent and key the effect on the branded JOB id.
A multi-step transition tears
A lifecycle move touches the row, a set, and a counter and must be all-or-nothing — run the whole transition as one inline Lua script.
The worker busy-polls the queue
A worker that loops on the queue burns CPU and adds latency — park on a wake key and let Valkey wake it.
Every technique in this chapter answers one of four failures of a naive queue: a lost in-flight job, a double run, a torn transition, and a busy-poll.
The patterns
six modules · one workshopSix teaching modules, then a workshop. Each module is a hub with three dives, grounded in the real EchoMQ worker path.
Processing list
Move a job into a recoverable in-flight state: the leased EchoMQ.Jobs.claim/3, not a list pop.
At-least-once
Delivery guarantees and idempotent consumers; the branded JOB id as the idempotency key.
Stalled recovery
Reclaim a dead worker's job by lease expiry on the server clock; EchoMQ.Stalled and the durable floor.
Atomic state machine
The lifecycle as one inline Script.new/2 Lua transition; states are emq:{q}: locations.
Blocking vs polling
Stop busy-polling; the EchoMQ.Consumer parks on the wake key with BLPOP.
Workshop
A reliable codemojex guess-command queue assembling R3.01–R3.05.
How to apply
name the failure · the technique followsThe hard part is matching the reliable-queue technique to the failure you cannot allow. Name the failure, and the technique — and the real EchoMQ artifact that implements it — follows.
There is no single reliable-queue trick — only the technique that closes the failure you cannot accept, each one a real move in EchoMQ's worker loop.
The workshop
echo/apps/codemojex · EchoMQ.LanesThe chapter closes with R3.06: a reliable codemojex guess-command queue assembling
R3.01–R3.05. Codemojex.Guesses.submit/3 mints a branded JOB id and enqueues
it on the player's fair lane with EchoMQ.Lanes.enqueue/5;
Codemojex.ScoreWorker.handle/1 — wired as an EchoMQ.Consumer with
lease_ms: 10_000 — drains the lane through EchoMQ.Lanes.claim/3, scores the
guess, and completes the job with EchoMQ.Jobs.complete/5. A worker that crashes mid-score
leaves its JOB leased in the active set; the next reap returns it to its lane. A guess is never
lost, and one keyboard masher cannot starve the field because the lane is named by the player.
The state machine in depth
The EchoMQ Queue pillar teaches the leased state machine, the inline Lua, fair lanes, and the schedule set behind the emq:{q}: keyspace, rung by rung.
The durable floor
A job that exhausts its retries, or whose completion is archived, reaches the durable tier — EchoStore.StreamArchive folding into the Graft floor, the durability dial taught end to end.
Notes on Valkey. EchoMQ.Jobs.claim/3 reads the lease deadline from
TIME — the server's clock, not the caller's — so a slow client never claims a longer
lease than it earned, and reap is measured on the same clock that minted it.
valkey.io/commands/time
Up next
R4–R8R3 is the spine the rest of the course hangs from. Each later chapter is a different surface over the same jobs — time and priority first, then the durable event log and flow control.
Time, Delay & Priority
The sorted set as a clock.
Streams & Events
The durable log.
Flow Control
Staying stable under load.
Data Modeling
How data lives in RAM.
Production & Ops
Running the tier at scale.
References
sources · relatedSources
- Valkey — ZPOPMINthe pop that claims the oldest pending job from the score-0 mint-ordered set, the engine the connector is gated against
- Valkey — HINCRBYincrements the row's attempts field, the lease fencing token that makes a stale completion a no-op
- Redis — Scripting with Luathe atomic lifecycle transition via EVAL / EVALSHA that every EchoMQ script runs
- Redis — Patternsthe canonical write-ups of the reliable work-queue access pattern
Related
- R2 — Coordinationthe atomic moves and lock lease R3 builds on
- EchoMQ — the Queue pillarthe state machine, lanes, and schedule set in depth
- The persistence floorwhere a dead-lettered or archived job lands durably
- The Branded Component SystemPart III builds the EchoMQ bus these patterns ground in