Never lose a job to a crash: list-based reliable queues.

A queue built on a single list is one command from losing work. The producer pushes a job with LPUSH; the consumer takes one with RPOP. The job leaves the list the moment RPOP returns it. If the worker then crashes — a deploy, an out-of-memory kill, a hardware fault — before the job’s effect is durable, the job is gone. Nothing on the server records that it was ever in flight.

The fix is to make claiming a job a move, not a removal. The job does not leave the system when a worker takes it; it moves to a second list — a processing list — where it stays visible until the worker confirms the job is done. A crash now leaves a trace: the job sits on the processing list, and a recovery sweep can find it and put it back.

The claim is one server-side operation, so there is no window in which the job belongs to neither list. The original Redis primitive for this is RPOPLPUSH: it pops the tail of the source list and pushes it onto the head of the destination list, returning the element, all in one step.

For a worker that should wait when the queue is empty rather than spin, the blocking form BRPOPLPUSH blocks until an element arrives or a timeout elapses:

RPOPLPUSH and BRPOPLPUSH were deprecated in Redis 6.2.0 in favour of LMOVE and BLMOVE, which take an explicit direction on each end (LEFT or RIGHT) instead of fixing the ends. LMOVE queue:work queue:processing RIGHT LEFT is the modern spelling of RPOPLPUSH — pop the right (tail) of the work list, push to the left (head) of processing. Teach LMOVE / BLMOVE as today’s primitive; RPOPLPUSH is the original.

A worker that finishes a job removes it from the processing list with LREM. LREM deletes matching elements by value; with a count of 1 it removes the first occurrence of the job from the head:

That is the only path off the processing list under normal operation. If the worker never reaches the LREM — it crashed mid-job — the job stays on the processing list. A separate recovery worker scans the processing list, finds entries that have sat there longer than a job should take, and moves them back onto the work list with another LMOVE (processing → work). The crashed job is re-queued, and a healthy worker claims it again.

The cost of the pattern is the choice between exactly-once and at-least-once. A worker that does its work and then crashes before the LREM will have its job re-queued and run a second time. Reliable-queue work is therefore at-least-once; the job handler must be idempotent. That is the trade the pattern makes — it would rather run a job twice than lose it.

Pinterest’s asynchronous job system, PinLater, names the same contract at production scale. PinLater was a Thrift service whose three core actions were enqueue, dequeue, and ACK. Its reliability mechanism, in its own words, was “explicit acks and automatically retries with configurable delay.” A worker replied with a positive or negative ACK depending on whether execution succeeded, and a job that was not positively acknowledged was retried after a configurable delay — the recovery worker, generalized into the service.

PinLater ran “more than 500 job queues processing north of six million jobs per minute” across more than ten clusters on EC2. The explicit-ack contract is what let it run at that rate without silently dropping work: a job is not done until a worker says it is done.

PinLater offered two storage backends, MySQL and Redis, and used Redis as the high-throughput one. The ack-and-retry guarantee held regardless of the backend: the list-in-RAM form is fast; a durable form trades some speed for a record that survives a restart. That tension is the next dive’s punch line.

The course’s worked bus is the Valkey-native echo_mq — the owned protocol over the braced emq:{q}: keyspace that the codemojex game’s queue runs on. It solves the identical problem — no lost job, bounded retries — but it does not use a processing list. It uses a server-clock lease plus an attempts counter carried in the job’s own hash.

A job is an entity: its row is a hash at the job key, written when the job is enqueued. The enqueue script sets three fields on that hash — state, attempts, and payload — in one atomic step:

When a worker claims the job, the claim script reads the server clock and stamps a lease deadline, moves the job from pending to active, and increments attempts — the fencing counter:

The lease is the processing list’s job, done by a clock rather than a second list. If the worker finishes, it calls complete and the job’s row is cleared. If the worker crashes, it never calls complete; the lease deadline passes; a sweep finds the expired lease and the job is re-claimed — and because the claim does HINCRBY 'attempts' 1, the retry count is already there to bound the retries. The reader never scans a processing list for stuck items; the deadline on the lease is the staleness signal, read from the same TIME clock every claim, promote, and reap uses.

The owned wire still knows the list family — EchoWire.Cmd.rpoplpush/2 (“Open an RPOPLPUSH builder”) and blmove (the blocking move) are both real builders in echo_wire/lib/echo_wire/cmd.ex — but the bus’s crash-safety is the lease, not a processing-list move.

The gap is the instant between a naive pop and a durable result; a crash in that instant loses the job. The list-based reliable queue closes it by making the claim an atomic move onto a processing list (LMOVE / BLMOVE), taking the job off only on a positive ack (LREM), and re-queuing strays with a recovery worker. PinLater names the same contract — explicit acks and configurable retry — at six million jobs per minute. The bus reaches the same goal with a server-clock lease and an attempts counter instead of a processing list. The next dive turns to scale: how “add a shard” becomes a configuration change, not a data migration, and where the in-RAM record gets made durable.