R3.01 · Reliable Queues — module hub
Processing list.
Guarantee at-least-once message delivery by atomically transferring a job from the waiting queue into a processing list, so recovery is possible if a consumer crashes before completing the work.
A naive queue pops a job off with RPOP and is one crash away from losing it:
between the pop and the finish the job exists nowhere. This module is the move that prevents it — take the
job and record it as in flight in one indivisible step. The grounding is the real EchoMQ worker path
(echo/apps/echo_mq): the leased claim, EchoMQ.Jobs.claim/3 over the
emq:{queue}:pending and emq:{queue}:active sorted sets. The three dives take the move
apart — the list as a queue, the atomic transfer command, the in-flight list as the recovery ledger.
The problem with simple queues
reliable-queue.md.txt · the gapA naive queue runs four steps. A producer pushes a job with LPUSH queue "message". A consumer
takes it with RPOP queue, which removes it from Redis. The consumer then crashes
during processing. The job is gone permanently — not in the queue, not finished, with nothing to recover
it from.
The flaw is the RPOP. It removes the job before any work has run, so between the pop and the
finish there is a window in which the job is in neither place. A deploy, an out-of-memory kill, or a power loss
in that window drops the job. The window is small, but a busy queue runs the move millions of times, and small
windows fire.
A simple queue loses a job on a crash because RPOP removes it before the work is done;
the cure is to move the job, not remove it.
The solution: atomic transfer
reliable-queue.md.txt · move, do not removeInstead of removing the job, atomically move it into a "processing" list. The job stays in Redis until it is
explicitly removed after successful processing. The transfer is one command —
RPOPLPUSH, or the modern LMOVE — so there is no instant at which the job is in
neither place.
A crashed worker therefore leaves the job parked in the processing list, where it can be found and returned. The naive form removes the job and bets the worker survives; the atomic form moves the job and keeps it recoverable whether the worker survives or not. That single change — move, do not remove — is the whole pattern.
Move the job into a processing list rather than removing it from the queue; it stays in Redis, recoverable, until the work is acknowledged.
How it works
reliable-queue.md.txt · the move + the ackThe consumer uses LMOVE (or the older RPOPLPUSH) to atomically transfer a job from
the main queue to a processing queue. The one command removes the job from the right of
work_queue, adds it to the left of processing:worker1, and does both atomically
— no job loss is possible.
LMOVE work_queue processing:worker1 RIGHT LEFT # atomic dequeue into the processing list # RPOPLPUSH src dst == LMOVE src dst RIGHT LEFT
The older RPOPLPUSH source dest is the same move in fixed directions. After successful
processing, remove the job from the processing queue with LREM processing:worker1 1 "message". For
consumption without polling, the blocking form BLMOVE work_queue processing:worker1 RIGHT LEFT 30
waits up to 30 seconds for a job rather than returning immediately on an empty queue. The second dive walks
RPOPLPUSH vs LMOVE and the blocking variant.
LREM processing:worker1 1 "message" # the ack, after the work is truly done
Notes on ValkeyZPOPMIN pops the lowest-scored member of a
sorted set and returns it atomically — the take inside EchoMQ's leased claim, the sorted-set form of
popping the oldest job — valkey.io/commands/zpopmin.
One command — LMOVE or RPOPLPUSH — does the pop and the push
atomically; the ack is a separate LREM only after the work is done.
The processing lifecycle
reliable-queue.md.txt · dequeue · process · ackA job moves through three steps. Dequeue:
LMOVE work_queue processing:worker1 RIGHT LEFT. Process: the application handles
the job. Acknowledge: LREM processing:worker1 1 "message". If the worker crashes
between the dequeue and the ack, the job remains in processing:worker1. That parked job is the
seam reliability rests on: still in Redis, attributable to a worker, returnable for another attempt. Delivery
is therefore at-least-once — delivered one or more times, never zero — which R3.02
makes the consumer idempotent for, and R3.03 reclaims a stalled job with. This module stops at the move.
| Step | Command |
|---|---|
| Dequeue | LMOVE work_queue processing:worker1 RIGHT LEFT |
| Process | the application handles the job |
| Acknowledge | LREM processing:worker1 1 "message" |
Dequeue, process, acknowledge — a crash between the dequeue and the ack leaves the job in the processing list, recoverable, which is what makes delivery at-least-once.
Per-worker processing queues
reliable-queue.md.txt · the recovery ledgerEach worker should have its own processing queue — processing:worker1,
processing:worker2. A per-worker list lets a recovery monitor attribute a parked job to the worker
that holds it, and return it cleanly. The processing list is not only a buffer; it is the
recovery ledger, a list of exactly the jobs currently in flight per worker. EchoMQ records the
lease deadline as the in-flight set's score instead, so the score is the recovery clock and no per-worker list
is needed — the third dive reads that ledger.
Give the in-flight place a way to name when each job is due back — a per-worker list, or a lease score — so a recovery monitor can return a parked job; the in-flight place is the recovery ledger.
The three dives
primitive → command → ledgerEach dive takes one layer of the move, in source order: the list as a queue underneath everything, the atomic transfer command that closes the gap, and the in-flight list read as the recovery ledger. Read them in order — the primitive first, the command second, the ledger third.
List as wait + active
The LIST as a queue: LPUSH at one end, RPOP at the other, FIFO. Two named
places — emq:{queue}:pending and emq:{queue}:active — and a job whose
state is which holds it.
LMOVE / RPOPLPUSH
The atomic transfer command: RPOP then LPUSH (a gap that loses a job) vs
RPOPLPUSH / LMOVE (one step); the blocking BLMOVE against a busy poll.
The in-flight list
active as the recovery ledger: a parked job whose worker died is still there; the
lease is the recovery clock; the LREM ack only after the work is done.
Move a job into a processing list in one atomic step, so it is in exactly one place at every instant and a crashed worker leaves it parked, recoverable, never lost.
EchoMQ.Jobs.claim/3 runs one inline @claim script —
ZPOPMIN emq:{queue}:pending, HINCRBY the attempts token,
HSET state active, ZADD emq:{queue}:active at the lease deadline — the
pop-and-record move with no gap, the real worker path.
Redis offers one move that never loses a job; EchoMQ's queue chose it — the leased claim — and proves the pattern is real.
A door, not a depth — the EchoMQ Queue pillar
This module cites one excerpt of EchoMQ's protocol — the leased @claim script inside
EchoMQ.Jobs.claim/3 — as proof the move ships. The full worker fetch loop, the eight inline
scripts of the leased state machine, the attempts fencing token, and the consumer's parked
BLPOP wake are the subject of the dedicated EchoMQ course. From here, open onto
the Queue pillar. This module teaches the move; that course teaches the engine that
runs it.
After R3.01, the Reliable Queues chapter continues with at-least-once delivery and an idempotent consumer, then stalled recovery, closing with the EchoMQ worker-path workshop. Return to the Reliable Queues chapter to continue.
References
sources · relatedSources
- Redis — RPOPLPUSH — the original atomic move that parks a job in the in-flight list, the reliable-queue move at the command level.
- Redis — LMOVE — the modern successor (6.2+) with explicit
RIGHT/LEFTdirections, recommended for new code. - Valkey — ZPOPMIN — pop the lowest-scored member of a sorted set, the take inside EchoMQ's leased claim on the engine the connector is gated against.
- Sanfilippo, S. — RPOPLPUSH, the reliable queue pattern — the Redis creator's design note on the reliable-queue move.
Related in this course
- R3.01.1 · List as wait + activethe list as a queue
- R3.01.2 · LMOVE / RPOPLPUSHthe atomic transfer command
- R3.01.3 · The in-flight listthe recovery ledger
- R3 · The reliable queuethe family in one place
- R2.01 · Atomic updatesthe atomic move this rests on
- /echomq · Queuethe leased state machine in depth