jonnify·redis /redis-patterns/queues/processing-list/lmove-rpoplpush

R3.01.2 · dive 2

LMOVE / RPOPLPUSH.

Pop, then push, is two commands with a gap in the middle. One command — RPOPLPUSH, or the modern LMOVE — closes the gap.

Moving a job from the waiting queue to the in-flight place is one logical step, so it must be one Redis command. Done as two — RPOP, then LPUSH — it has a window in the middle where the job is in neither place, and a crash there drops it. RPOPLPUSH and its successor LMOVE do the pop and the push as one atomic command. Select the move between two steps and one, then read which form survives a crash at the gap.

Two steps, a gap One command Block, not poll In EchoMQ References
Two steps vs one — crash at the gap, job 7. Select the form to light the job's resting place after a crash.
wait— the gap— active— RPOP LPUSH crash in the gap — job 7 is in neither place
§1

Two steps leave a gap

reliable-queue.md.txt · the gap

The obvious way to take a job is to pop it from the waiting list and push it to the in-flight list. Between the RPOP and the LPUSH, the job exists nowhere: it has left wait and has not yet reached active. A crash in that window — a deploy, an out-of-memory kill, a dropped connection — drops the job with nothing to recover it from.

# two commands: a gap in the middle
job = RPOP  wait        # job leaves wait — it is now in neither place
# ---- if the worker dies HERE, the job is gone ----
LPUSH active job        # job lands in active

The window is small, but a busy queue runs the move millions of times, and small windows fire. Two commands cannot be made atomic by running them quickly; the only fix is to make the move a single command.

Pop-then-push is two commands with a gap in which the job is in neither place; a crash there loses it, and running faster cannot close the gap.

§2

One command closes it

reliable-queue.md.txt · the atomic transfer

RPOPLPUSH source destination pops from the tail of source and pushes to the head of destination as a single atomic command. The job leaves wait and lands in active with no instant in between in which it is in neither place.

job = RPOPLPUSH wait active   # atomic: pop tail of wait, push head of active
process(job)                  # a crash here leaves the job parked in active
LREM active 1 job             # remove only after the work is truly done

LMOVE source destination RIGHT LEFT (Redis 6.2+) is the same move with explicit directions. RPOPLPUSH src dst is exactly LMOVE src dst RIGHT LEFT — pop the right (tail), push the left (head). LMOVE generalises it to any pair of ends and is the recommended form for new code; RPOPLPUSH remains for compatibility. Either way the guarantee is identical: the job is in exactly one place at every instant.

RPOPLPUSH src dst            ==  LMOVE src dst RIGHT LEFT
LMOVE src dst LEFT  RIGHT   # pop the head of src, push the tail of dst
RPOPLPUSH
Pop the tail of source, push the head of destination, atomically. The original move, fixed directions.
LMOVE
The Redis 6.2+ successor with explicit RIGHT/LEFT ends; RPOPLPUSH src dst == LMOVE src dst RIGHT LEFT. The recommended form for new code.

One command — RPOPLPUSH, or LMOVE — does the pop and the push atomically, so there is no instant at which the job exists nowhere.

§3

The blocking variant — block, do not poll

reliable-queue.md.txt · consumer.ex

A worker with nothing to do can poll: run the take in a loop, get nil on an empty queue, sleep, try again. Polling wastes round trips when the queue is idle and adds latency between a job arriving and a worker taking it. The blocking variants remove the loop. BLMOVE source destination RIGHT LEFT timeout (and the older BRPOPLPUSH source destination timeout) do the same atomic move, but on an empty source they block the connection for up to timeout seconds, waking the instant a job is pushed.

BRPOPLPUSH wait active 30              # block up to 30s; wake the moment a job arrives
BLMOVE     wait active RIGHT LEFT 30   # the modern form, same blocking move

A timeout of 0 blocks indefinitely. The block is server-side: no round trips while waiting, and the worker takes the job with no polling delay. The move is still atomic — blocking changes only when the command returns, not what it does. Slide the queue's idle gap and compare the round trips a poll loop makes against the single blocked call.

Poll vs block — round trips over an idle gap. Select the form and slide the idle seconds; the readout reports the wasted round trips.
4
round trips over the idle gap 4 wasted round trips poll · one take per second, each returns nil

A poll loop makes one wasted round trip per idle second; a blocking BLMOVE makes one call and waits, the same atomic move with no polling delay.

§4

In EchoMQ — the move is one script

echo_mq/jobs.ex · consumer.ex

EchoMQ's in-flight move is one indivisible step, not two, in real code in echo/apps/echo_mq. EchoMQ.Jobs.claim/3 runs one inline EchoMQ.Script.new(:claim, …) script: ZPOPMIN emq:{queue}:pending takes the oldest id, HINCRBY increments the row's attempts (the fencing token), HSET marks the row state = active, and ZADD emq:{queue}:active records the id at the lease deadline — all inside one EVALSHA, so there is no instant at which the id is in neither set. The script is loaded once and run EVALSHA-first; a NOSCRIPT reply re-loads it.

-- the @claim script (echo/apps/echo_mq/lib/echo_mq/jobs.ex) — pop-and-record, one EVAL
local popped = redis.call('ZPOPMIN', KEYS[1])            -- KEYS[1] = emq:{queue}:pending
if #popped == 0 then return {} end
local id  = popped[1]
local att = redis.call('HINCRBY', jk, 'attempts', 1)     -- the fencing token
redis.call('HSET', jk, 'state', 'active')
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)   -- KEYS[2] = emq:{queue}:active, at the lease deadline

EchoMQ runs the move inside one script rather than a bare BLMOVE because the take also has to mint the lease and the fencing token — work that has to be atomic with the pop-and-record, so it goes in one script. The consumer follows the blocking-variant discipline: EchoMQ.Consumer parks on BLPOP emq:{queue}:wake and a producer pushes the wake token on enqueue, so an idle worker costs the wire nothing and wakes on arrival. The bare RPOPLPUSH and BLMOVE are the same closing-the-gap move at the command level; the @claim script is that move plus the lease, all or nothing.

Notes on ValkeyZPOPMIN pops the lowest-scored member of a sorted set atomically — the take inside EchoMQ's @claim script, the sorted-set form of the atomic transfer — valkey.io/commands/zpopmin.

the pattern

The move from waiting to in-flight is one logical step and must be one Redis command; RPOPLPUSH / LMOVE closes the gap a pop-then-push leaves.

its EchoMQ application

EchoMQ.Jobs.claim/3 runs one inline @claim script — ZPOPMIN the pending set, then ZADD the active set — one EVALSHA, no gap, and the consumer parks on BLPOP instead of polling.

Two commands leave a gap; one command closes it. EchoMQ's move is the single ZPOPMIN inside the @claim script, run as one EVALSHA.

/echomq/queue

A door, not a depth

Why the move lives in a Lua script and not a bare BLMOVE — the lease it mints, the fencing token it carries, the EVALSHA-first dispatch, and the parked BLPOP wake the consumer rides — is the dedicated EchoMQ course. From here, this dive opens onto the Queue pillar — the as-built claim and its EVAL path. This dive teaches the atomic move; that course teaches the whole script.

§5

References

sources · related

Sources

  • Redis — RPOPLPUSH — the original atomic move: pop the tail of source, push the head of destination, in one command.
  • Redis — LMOVE — the modern successor (Redis 6.2+), explicit RIGHT/LEFT, recommended for new code.
  • Redis — BLMOVE — the blocking variant: the same atomic move, block on an empty source instead of polling.
  • Valkey — ZPOPMIN — pop the lowest-scored member of a sorted set atomically, the take inside EchoMQ's leased claim.

Related in this course