The Queue · module 01
The lifecycle
A job has a life, and the whole of it is four sorted sets and one row.
A producer enqueues a job; a worker claims it, runs it, and completes it; a failure retries after a
delay or dies into a morgue; a lapsed lease returns the job to the queue when a client sweeps. There is no separate state table, no
lock service, no second index — a job moves between four sorted sets (pending,
active, schedule, dead) and its own row
carries a state field that says where it is. This module reads that machine: the four
sets, the claim and its lease, and the token-fenced completion and recovery.
Four sets, one row
Every transition moves a job between two of the four sets and rewrites its row's state
(completion deletes the row instead). Pick a transition and read which set a job leaves, which it enters, what
the row says afterward, and which native command, if any, runs the same transition. The map below is the real
machine: the five transitions of EchoMQ.Jobs over a fixed example job.
The state machine · pick a transition
pending → active · row state active
ZPOPMIN pending, skip a ghost (no row), HINCRBY attempts, HSET state active, ZADD active at now + lease — the oldest pending id becomes active and gets a lease deadline
native twin: EMQ.CLAIM — the same pop, token and lease, with the pause read folded in and lease_epoch stamped
Each transition runs as an inline Lua script that EchoMQ.Connector
evaluates (wire 4), and most of them also exist as native commands in the trafikey module
(wire 6): the claim, the lease extension, completion, the retry-or-dead verdict, reprocess and the counted stall
sweep each have an EMQ.* twin, while the reaper runs only as a script. No sweep runs on
its own on either executor: a lapsed lease returns to pending only when a client process calls the sweep.
The native module teaches the mechanism.
The lifecycle is not a status column you update — it is a set a job is a member of. Move the id, and the state has changed.
The dives
Three dives read the machine in depth, each grounded in real echo/apps/echo_mq code:
the four sets and why the members are the ids themselves, the claim and its lease-and-token, and the token-fenced
completion plus the recovery transitions.
The four sets
pending / active / schedule / dead are sorted sets whose members are the branded ids themselves — so byte order is mint order and the queue needs no second index. browse/3 and pending_size/2.
Claim & the lease
The @claim2 transition in two beats: ZPOPMIN, skip a ghost, HINCRBY attempts, set active, ZADD at now + lease. The active-set score IS the lease deadline; attempts IS the fencing token. Its native twin is EMQ.CLAIM.
Completion & recovery
The token-fenced terminal (@complete) and the recovery transitions: @retry parks or dead-letters, @reap returns a lapsed lease when a client sweeps, reprocess_job/3 re-enters a dead job. All but the reaper have native twins.
Pattern & implementation
Where Redis Patterns Applied teaches the reliable queue — a job's state is the collection it lives in, and every move is atomic so a crash loses nothing — this module is where the pattern lands as one shipped state machine.
A reliable queue moves a job between named locations atomically and recovers a crashed worker's job from a visibility timeout. States as locations and The reliable queue teach the move.
EchoMQ.Jobs runs @claim2 / @complete
/ @retry / @reap — four sorted sets and one row, every
transition one atomic script on one slot; trafikey runs the same transitions, the
reaper aside, as native commands.
Take the dives in order, or jump to completion-and-recovery if you already know the four sets and the claim. The next module — Jobs, lanes & the consumer — reads the producer, the fair worker, and the loop that owns the rhythm.
References
Sources
- Valkey — ZADD — the score-carrying set insertion every transition uses.
- Valkey — ZPOPMIN — the atomic pop the claim transition begins with.
- Valkey — ZRANGEBYSCORE — the due-window read promote and reap run.
- Redis — EVALSHA — the load-once dispatch each transition runs by SHA.
Related in this course
- The Queue — the chapter this module belongs to.
- The owned keyspace — where the four set keys are built.
- The native module — the commands that run these transitions on wire 6.
- redis-patterns · Reliable Queues — the pattern this machine implements.
- redis-patterns · States as locations — state is the set you live in.