jonnify·redis /redis-patterns/queues/processing-list/list-wait-active

R3.01.1 · dive 1

List as wait + active.

A queue is a list used one way: push at one end, pop at the other. A reliable queue is two such places — and a job that moves between them.

Everything in a reliable queue rests on an ordered collection a worker pops one end of. LPUSH at the head, RPOP at the tail, FIFO with no extra structure. A reliable queue names two such places — a waiting one and an in-flight one — and a job lives in exactly one; its membership is its state. Push jobs into the waiting collection and pop them in FIFO order; the readout shows the contents, head, and tail.

A list is a queue Two named places The move between In EchoMQ References
LPUSH head, RPOP tail — FIFO over jobs 5, 6, 7. Push and pop; the readout reports the contents, head, and tail.
head · LPUSH tail · RPOP — — — — tail (rightmost) is the oldest — RPOP returns it next
§1

A list is a queue

reliable-queue.md.txt · the primitive

A Redis LIST is an ordered sequence of strings with cheap push and pop at both ends. Treat one end as the head and the other as the tail, push at the head and pop at the tail, and the list becomes a first-in-first-out queue. LPUSH key value prepends to the head; RPOP key removes from the tail. A job pushed first is popped first.

LPUSH work 5      # head: [5]
LPUSH work 6      # head: [6, 5]
LPUSH work 7      # head: [7, 6, 5]
RPOP  work        # -> 5   (the oldest, popped from the tail)
RPOP  work        # -> 6

The list holds the order for free. The producer pushes at the head; the consumer pops at the tail; the list keeps the sequence between them. A queue is not a separate data structure — it is a list used with a push at one end and a pop at the other.

A queue is a list used one way: push at the head, pop at the tail, and the order out matches the order in — FIFO, with no extra structure.

§2

wait and active are two named places

echo_mq/keyspace.ex · queue_key/2

A reliable queue does not use one collection — it uses two. The first is the waiting place: jobs a producer has enqueued and no worker has yet taken. The second is the active (in-flight) place: jobs a worker has taken and is processing. A job lives in exactly one of them at a time, and its membership names its state. A job in the waiting set is waiting; a job in the active set is being worked.

The two places are named keys. EchoMQ names them with a queue prefix: emq:{queue}:pending is the waiting set and emq:{queue}:active is the in-flight set, where {queue} stands in for the queue name. The job does not change — it is the same branded JOB id — but which set holds it changes, and that change is the state transition. Reading a job's state is asking which set its id is in.

emq:{queue}:pending  (sorted set)   # jobs enqueued, not yet taken — scored at mint order
emq:{queue}:active   (sorted set)   # jobs taken by a worker — scored at the lease deadline
waiting set
The set a producer admits into — EchoMQ's emq:{queue}:pending. A job here is waiting for a worker.
in-flight set
The set a worker moves a job into — EchoMQ's emq:{queue}:active. A job here is being worked, and a crash leaves it here, recoverable.
state = location
A job's state is which set holds its id. The transition is the move between the two.

A reliable queue is two named places, a waiting one and an in-flight one; a job lives in exactly one, and its location is its state.

§3

The move between the two

echo_mq/jobs.ex · the gap to close

Taking a job for work means moving its id out of pending and into active. Doing that as two separate steps — pop from pending, then record on active — opens a gap: between the two, the id is in neither set, and a crash there drops it. The next dive is exactly that gap and the single command that closes it. For now, hold the shape: two named places, and a job that belongs to whichever one currently holds it.

The classic LIST form gives each worker its own in-flight list (processing:worker1, processing:worker2), so a recovery monitor can attribute a parked job to the worker that holds it. EchoMQ records the lease deadline as the active set's score instead, so the recovery clock is the score rather than a per-worker list. The third dive reads that ledger.

A job's state is its set — job 7 in pending, then in active. Select a state to light the set its id is in.
pending set7 active set— ZPOPMIN → ZADD job 7 is in pending · its state is waiting

The move that takes a job for work is a move between two named places; doing it in two steps opens a gap, and closing that gap is the next dive.

§4

In EchoMQ — the two named sets

echo_mq/keyspace.ex · jobs.ex

EchoMQ names the two places in real code, in echo/apps/echo_mq. EchoMQ.Keyspace.queue_key(queue, "pending") builds emq:{queue}:pending and EchoMQ.Keyspace.queue_key(queue, "active") builds emq:{queue}:active. A new job is admitted to the waiting set by EchoMQ.Jobs.enqueue/4; a worker that takes a job moves its id into the active set. The move itself is EchoMQ.Jobs.claim/3's leased claim — ZPOPMIN emq:{queue}:pending then ZADD emq:{queue}:active scored at the lease deadline — the id leaves the pending set and lands in the active set in one indivisible script.

# EchoMQ.Keyspace.queue_key/2 — the two named places (real)
emq:{queue}:pending   # EchoMQ.Keyspace.queue_key(queue, "pending")
emq:{queue}:active    # EchoMQ.Keyspace.queue_key(queue, "active")
# the claim moves a JOB id: ZPOPMIN pending -> ZADD active <now + lease_ms>

{queue} is the queue name carried inside the braces so every key of one queue lands on one cluster slot. The branded JOB id is gated at the key builder — EchoMQ.Keyspace.job_key/2 raises on an ill-formed id, so a malformed id never reaches the wire.

Notes on ValkeyZADD key score member sets a member's score in a sorted set; EchoMQ scores the active member at the lease deadline, so the same score doubles as the recovery clock — valkey.io/commands/zadd.

the pattern

A reliable queue is two named places — a waiting place and an in-flight place — and a job whose state is which of them holds it.

its EchoMQ application

EchoMQ.Keyspace.queue_key/2 builds emq:{queue}:pending and emq:{queue}:active; the move between them is EchoMQ.Jobs.claim/3's ZPOPMIN-then-ZADD leased claim.

A reliable queue is two named places and a move between them; EchoMQ's pending and active sets are exactly those two places.

/echomq/queue

A door, not a depth

The full key layout — the schedule set, the dead set, the metrics hash, and the fair lanes the worker path touches alongside pending and active — is the dedicated EchoMQ course. From here, this dive opens onto the Queue pillar — the as-built library and its full set of keys. This dive teaches the two-place primitive; that course teaches the whole layout.

§5

References

sources · related

Sources

  • Redis — Lists — the list type and its push/pop ends, the list-as-queue primitive everything rests on.
  • Redis — LPUSH — prepend a job to the head of a list.
  • Redis — RPOP — remove and return a job from the tail of a list.
  • Valkey — ZPOPMIN — pop the lowest-scored member of a sorted set, the move EchoMQ uses to take the oldest pending job.

Related in this course