The Queue · module 02
Jobs, lanes & the consumer
The lifecycle is the machine. This module is the people who run it — the producer, the fair worker, and the loop that owns the rhythm.
A job enters the machine through one verb and leaves through another, but between them stand three roles. The producer admits work with one idempotent script. The fair worker claims in turn off a constructed ring, so no identity starves the rest. The loop owns the cadence — a supervised process that parks rather than polls, costing the wire nothing while it waits. Pick a role below to read the real surface it owns, then take the three dives.
Three roles over one wire
A job's working life runs through three hands: the producer that admits it, the worker that claims it fairly,
and the loop that sets when to look. Each is a real surface in echo/apps/echo_mq.
Pick a role to read the verb it owns, what it touches, the verdict it returns, and its native twin, if any.
The three roles · pick one
touches: the job row emq:{q}:job:<id>, the pending set emq:{q}:pending
does: run @enqueue: gate the JOB-namespaced id, refuse a duplicate, HSET the three-field row, ZADD the pending entry — one atomic step
verdict: {:ok, :enqueued} | {:ok, :duplicate} | {:error, :kind}
native twin: EMQ.ENQUEUE, the same keys and replies, on a node that loads the trafikey module.
Three roles, three verbs, one wire: admit once, claim in turn, and look only when there is reason to.
The dives
Three dives read the roles in depth, each grounded in real echo/apps/echo_mq code:
the idempotent enqueue and the flat claim, the rotating ring that builds fairness, and the supervised loop that
parks on the wake key.
Enqueue & claim
The @enqueue two-beat — the EMQKIND kind-gate, EXISTS idempotency, the three-field row, the pending entry — and the flat @claim2 that pops, skips a ghost, leases, and returns. Both have native twins.
Fair lanes & the ring
Per-group lanes and a rotating ring. @gclaim2 does LMOVE LEFT RIGHT to rotate one step, then serves that lane's head — fairness constructed, not hashed. Lanes run only as scripts.
The consumer loop
The supervised recover → promote → drain → park beat. BLPOP the wake key — park, don't poll — and a raising handler becomes a retry after a fixed delay, not a crash.
Two executors serve these transitions. Enqueue and the flat claim have native twins,
EMQ.ENQUEUE and EMQ.CLAIM, which
EchoMQ.Jobs sends through a wire: 6 connector to a node that
loads the trafikey module, as a raw client can. The lanes run only as scripts, so a
wire: 6 connector refuses them with {:wire6_unsupported, name},
and the consumer loop, which drains through the lanes, has no wire 6 path. The native module teaches
the mechanism.
Pattern & implementation
Where Redis Patterns Applied teaches the move — a reliable queue that admits work once and hands it to exactly one consumer — this module is where the move lands, with fairness and a parking loop built on top.
A reliable queue admits each item once and delivers it to exactly one worker, which claims, runs, and acknowledges — blocking for the next item rather than busy-polling. Reliable queues teaches it.
EchoMQ.Jobs.enqueue/5 admits idempotently; EchoMQ.Lanes.claim/3
serves identities in turn off a rotating ring; EchoMQ.Consumer runs the loop and
parks on the wake key with BLPOP.
Take the dives in order, or jump to the consumer loop if you already know enqueue and claim. The next module — Batches — reads how the same per-item contract is admitted and extended in bulk, in one wire flush.
References
Sources
- Valkey — LMOVE — the atomic rotate the ring is turned with, one step per claim.
- Valkey — ZPOPMIN — popping the oldest pending id the claim leases.
- Valkey — BLPOP — the blocking pop the consumer parks on instead of polling.
- Redis — EVALSHA — load-once, run-by-SHA dispatch of each declared-keys script.
Related in this course
- The Queue — the pillar this module belongs to.
- Enqueue & claim — the producer and the flat worker.
- The native module — the wire-6 commands, EMQ.ENQUEUE and EMQ.CLAIM among them.
- redis-patterns · Reliable queues — the reliable-queue / consumer pattern.