R3.06 · Reliable Queues — the chapter workshop
A reliable guess queue.
The chapter capstone: take the five reliable-queue patterns and run them over one real consumer
job — codemojex’s guess job, minted by Codemojex.Guesses.submit/3 and scored by
Codemojex.ScoreWorker.
Grounding: echo/apps/echo_mq (EchoMQ.Jobs, EchoMQ.Lanes, EchoMQ.Consumer, EchoMQ.Stalled, EchoMQ.Keyspace) · echo/apps/codemojex (Codemojex.Guesses, Codemojex.ScoreWorker) — every surface verified on disk. The engine is Valkey.
Static verdicts — step → claim (lease) → score → complete, scored stays 1 · crash → lease expires → stalled sweep → re-claim → re-run, scored stays 1 (the branded JOB id dedups).
The scenario — a guess as a job
codemojex/game.excodemojex is a competition to crack a six-emoji code. A player submits a guess; the game scores it against
the secret; the score lands on the leaderboard. The host never scores — a single consumer is the
authority. That split is already a queue. Codemojex.Guesses.submit/3 validates the guess against
the round’s keyboard, overlays the player’s locked positions, charges the right currency through the
wallet, then mints a branded JOB and enqueues it on the player’s lane.
The lane is named by the player’s PLR, so the bus rotates service across players and one
masher cannot starve the field. Codemojex.ScoreWorker.handle/1 is the consumer the bus drains
through EchoMQ.Lanes.claim/3: it reads the secret, scores with the pure linear engine, writes a
GES guess, counts the attempt, and records the result.
Reliable means two things at the same time. A worker crash never drops a guess — the guess is scored, eventually, even if the first worker dies mid-score. And a worker crash never scores one guess twice — the leaderboard counts each guess once, even when the job is redelivered. The whole point of R3 was to build a queue that holds both at once.
job = EchoData.BrandedId.generate!("JOB")
payload = :erlang.term_to_binary({:guess, game, player, guess})
EchoMQ.Lanes.enqueue(Bus.conn(), "cm", player, job, payload) ← the player's lane
A guess is a job: enqueued on the player’s lane, claimed by the scoring consumer, completed once. Reliable means lose no guess and score no guess twice — even across a worker crash.
The assembled path — R3.01 through R3.05
echo_mq/jobs.ex · lanes.ex · consumer.exFive patterns assemble into one reliable loop. Each prevents a different failure of the naive form — pop a job, process it, done — and they build in order: the claim leaves the job recoverable, that makes redelivery the at-least-once price, the stalled sweep reclaims a dead worker’s job, the atomic finish keeps the completion indivisible, and the blocking pickup keeps the consumer idle-cheap between guesses.
| Pattern | The move over the guess job | The failure it prevents |
|---|---|---|
| R3.01 · processing list | EchoMQ.Lanes.claim/3: rotate the ring, ZPOPMIN the lane → HSET active → ZADD lease | a lost job — a crash leaves it recoverable in active |
| R3.02 · at-least-once | a reaped lease redelivers; delivery is at-least-once, the branded id pays | a dropped guess — the job runs again, not zero times |
| R3.03 · stalled recovery | EchoMQ.Stalled.check/2: expired lease → pending, past max_stalled → dead | a stuck job — a dead worker’s job returns or dead-letters |
| R3.04 · atomic finish | EchoMQ.Jobs.complete/5: one EVALSHA, token-fenced | a torn finish — no half-completed state |
| R3.05 · blocking pickup | EchoMQ.Consumer parks on emq:{cm}:wake (BLPOP) | a busy-poll — the consumer sleeps until woken |
The claim is the foundation: the guess job moves from its player lane to active in one atomic
step — LMOVE rotates the ring of serviceable lanes, ZPOPMIN pops the head of the
rotated lane, HSET state active sets the row, and ZADD places the job on the active
set at a lease deadline read from the server clock (TIME). So it is in exactly one set at every
instant, and a crash leaves it recoverable. That in-flight move is what makes
redelivery possible — which is the point — and it is also what makes delivery
at-least-once: a worker that scores a guess, then dies before it acknowledges, leaves the job
in active to be reclaimed. The stalled sweep returns that job; the atomic finish keeps the
completion indivisible; the blocking pickup keeps the consumer idle-cheap. Assembled, they are a queue that
loses no guess.
# The assembled path, over a codemojex guess job (real today)
submit → Codemojex.Guesses.submit/3 mints a branded JOB, EchoMQ.Lanes.enqueue on the player lane
worker → EchoMQ.Consumer parks on emq:{cm}:wake (BLPOP); an enqueue pushes a wake # R3.05
claim → @gclaim: LMOVE ring, ZPOPMIN the lane, HSET state active, ZADD active (lease, server TIME) # R3.01
process → Codemojex.ScoreWorker.handle/1 scores, writes one GES guess, records the result
finish → complete/5: one EVALSHA, token-fenced (attempts), ZREM active + DEL row # R3.04
crash? → lease expires → reap/2 / Stalled.check/2 returns it to pending; re-claimed # R3.02 + R3.03
→ past max_stalled: dead-letter onto emq:{cm}:dead → the durable floor (the archive)
Each pattern guards one failure: a lost job, a dropped guess, a stuck job, a torn finish, a busy-poll. Assembled in order, they are a queue that never loses a guess.
Exactly-once-in-effect — the branded id is the key
echo_mq/jobs.ex · @enqueueThe assembled queue guarantees at-least-once delivery, so a guess job can run twice. That is safe because the
branded JOB id is the idempotency key, and the bus dedups on it by construction.
EchoMQ.Jobs’ enqueue script is one idempotent step: a second enqueue of the same branded id
finds the row present (EXISTS KEYS[1]) and returns 0, which the host reads as
{:ok, :duplicate}. The same guess admitted twice is admitted once.
if redis.call('EXISTS', KEYS[1]) == 1 then
return 0 ← the JOB id is already enqueued
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1]) ← KEYS[1] = emq:{cm}:job:<JOB id>
return 1
The finish is fenced too. EchoMQ.Jobs.complete/5 refuses a stale attempts token
({:error, :stale}), so a worker whose lease was reaped and re-claimed by another worker cannot
complete a job it no longer owns. The id orders the queue (mint order is byte order is sort order), names the
row, and dedups admission — one 14-byte value doing the work of an index and a guard. So a guess that is
redelivered after a crash is scored once, not twice: the first run writes the GES guess; a
redelivered job re-claims and re-runs against the same secret, and the consumer is the authority that holds the
count.
The interactive below makes the proof concrete. Submit the same guess job any number of times; the scored count stays at one. The first delivery records the guess; every later delivery is refused at admission or runs as a no-op against the dedup. At-least-once delivery over a branded-id-deduped effect is exactly-once-in-effect.
A reliable queue — the in-flight claim (R3.01), at-least-once delivery (R3.02), stalled recovery (R3.03), the atomic finish (R3.04), the blocking pickup (R3.05) — loses no job, but delivers each one or more times.
The consumer is Codemojex.ScoreWorker; the producer mints a branded JOB whose id
is the idempotency key EchoMQ.Jobs.enqueue dedups, and complete/5 is token-fenced
— so the assembled queue is exactly-once-in-effect.
A reliable queue plus a branded-id-deduped, token-fenced consumer is the whole reliable design: lose no work, score nothing twice. Exactly-once delivery is a lie; exactly-once-in-effect is the achievable truth.
Grounded in the real surfaces
echo_mq/* · codemojex/game.exEvery primitive in the assembled path is real EchoMQ code under echo/apps/echo_mq, and the
consumer is real codemojex code under echo/apps/codemojex. The states are locations in the braced
emq:{cm}: keyspace, and the branded JOB id is gated at the key builder
(EchoMQ.Keyspace.job_key/2 raises on an ill-formed id).
| Guarantee | The EchoMQ surface (Elixir, verified) | The location |
|---|---|---|
| in-flight move | EchoMQ.Lanes.claim/3 → @gclaim | the player lane → emq:{cm}:active |
| stalled recovery | EchoMQ.Stalled.check/2, EchoMQ.Jobs.reap/2 | emq:{cm}:active → pending / dead |
| atomic finish | EchoMQ.Jobs.complete/5 → @complete | emq:{cm}:active → row DEL |
| blocking pickup | EchoMQ.Consumer (BLPOP, :lease_ms 30000) | emq:{cm}:wake |
| idempotency key | the branded JOB id; enqueue refuses a duplicate | emq:{cm}:job:<JOB id> |
Notes on ValkeyThe claim is one atomic move: ZPOPMIN on the
pending set and ZADD onto the active set at a lease deadline read from the server’s own clock
(TIME), so a job is in exactly one set and the lease is timed on one clock — never the
caller’s — via valkey.io/commands/zpopmin.
A dead-lettered guess lands on the floor
A guess that stalls past max_stalled is dead-lettered onto emq:{cm}:dead, and a
trimmed stream is not discarded. EchoStore.StreamArchive folds trimmed segments into the native
EchoStore.Graft engine’s CubDB and on to Tigris — deep history without resident
memory. Durability is a dial the system turns; the enqueue and claim touch only the bus.
A store-side fold of trimmed EchoMQ.Stream segments into the native EchoStore.Graft engine's CubDB (and, with remote_cfg, on to Tigris), readable beside the live tail through a merge-read split on an engine-derived watermark W.
The engine that runs these scripts
The claim, the lease, the fencing token, the fair lanes — the Queue pillar teaches the EchoMQ state machine in depth, rung by rung.
/echo-persistenceWhere a dead-lettered job lands
The durability dial — ETS head, Valkey, the CubDB/Fjall page tier, Tigris remote. The archive a reaped or dead-lettered job folds into, built and gated.
back to the chapterR3 · Reliable Queues
The five patterns this workshop assembles — the processing list, at-least-once, stalled recovery, the atomic state machine, blocking versus polling.
References
sources · relatedSources
- Valkey — RPOPLPUSH — the atomic in-flight move family; the claim is its sorted-set form (
ZPOPMINthenZADD), one job in exactly one set. - Valkey — EVALSHA — the atomic finish:
complete/5runs as one cached script, so no second command can tear away. - Valkey — BLPOP — the consumer blocks on the wake key between jobs; park, do not poll.
- Valkey — SET —
SET … PX, value and expiry in one atomic command, the lease idiom read back on the server clock. - Redis — Documentation — sorted sets, hashes, and scripting: the command families the reliable-queue path is built from.
Related in this course
- R3 · Reliable Queuesthe chapter this workshop closes
- R3.01 · The processing listthe in-flight move (pending → active under a lease)
- R3.02 · At-least-oncethe delivery guarantee the branded id pays for
- R3.03 · Stalled recoverythe sweep that returns a dead worker’s job
- R3.04 · Atomic state machinethe one-script finish
- R3.05 · Blocking vs pollingthe blocking pickup on the wake key