jonnify·redis /redis-patterns/queues/workshop

R3.06 · Reliable Queues — the chapter workshop

A reliable guess queue.

The chapter capstone: take the five reliable-queue patterns and run them over one real consumer job — codemojex’s guess job, minted by Codemojex.Guesses.submit/3 and scored by Codemojex.ScoreWorker.

Grounding: echo/apps/echo_mq (EchoMQ.Jobs, EchoMQ.Lanes, EchoMQ.Consumer, EchoMQ.Stalled, EchoMQ.Keyspace) · echo/apps/codemojex (Codemojex.Guesses, Codemojex.ScoreWorker) — every surface verified on disk. The engine is Valkey.

The guess job through the reliable loop. Step it through, or inject a mid-score crash, and read the scored count — one, even when the job runs twice.
stage: queued — the job is in pending, the consumer is parked emq:{cm} player lane JOB g_1 emq:{cm}:active · lease JOB g_1 emq:{cm}:completed JOB g_1 action: push a wake on emq:{cm}:wake — wake the consumer return queued — press step to pick it up scored (effect) 0 job runs so far: 0 — the effect count is what matters, not the delivery count effect: nothing yet — the loop has not claimed the job
readout · Queued: the guess job JOB g_1 is in pending, the consumer is parked on the wake key. Press step to run the loop, or crash to inject a worker death.

Static verdicts — step → claim (lease) → score → complete, scored stays 1 · crash → lease expires → stalled sweep → re-claim → re-run, scored stays 1 (the branded JOB id dedups).

§1

The scenario — a guess as a job

codemojex/game.ex

codemojex is a competition to crack a six-emoji code. A player submits a guess; the game scores it against the secret; the score lands on the leaderboard. The host never scores — a single consumer is the authority. That split is already a queue. Codemojex.Guesses.submit/3 validates the guess against the round’s keyboard, overlays the player’s locked positions, charges the right currency through the wallet, then mints a branded JOB and enqueues it on the player’s lane.

The lane is named by the player’s PLR, so the bus rotates service across players and one masher cannot starve the field. Codemojex.ScoreWorker.handle/1 is the consumer the bus drains through EchoMQ.Lanes.claim/3: it reads the secret, scores with the pure linear engine, writes a GES guess, counts the attempt, and records the result.

Reliable means two things at the same time. A worker crash never drops a guess — the guess is scored, eventually, even if the first worker dies mid-score. And a worker crash never scores one guess twice — the leaderboard counts each guess once, even when the job is redelivered. The whole point of R3 was to build a queue that holds both at once.

codemojex/game.ex · Codemojex.Guesses.submit/3 — the producerverbatim
job = EchoData.BrandedId.generate!("JOB")
payload = :erlang.term_to_binary({:guess, game, player, guess})
EchoMQ.Lanes.enqueue(Bus.conn(), "cm", player, job, payload)   ← the player's lane

A guess is a job: enqueued on the player’s lane, claimed by the scoring consumer, completed once. Reliable means lose no guess and score no guess twice — even across a worker crash.

§2

The assembled path — R3.01 through R3.05

echo_mq/jobs.ex · lanes.ex · consumer.ex

Five patterns assemble into one reliable loop. Each prevents a different failure of the naive form — pop a job, process it, done — and they build in order: the claim leaves the job recoverable, that makes redelivery the at-least-once price, the stalled sweep reclaims a dead worker’s job, the atomic finish keeps the completion indivisible, and the blocking pickup keeps the consumer idle-cheap between guesses.

PatternThe move over the guess jobThe failure it prevents
R3.01 · processing listEchoMQ.Lanes.claim/3: rotate the ring, ZPOPMIN the lane → HSET active → ZADD leasea lost job — a crash leaves it recoverable in active
R3.02 · at-least-oncea reaped lease redelivers; delivery is at-least-once, the branded id paysa dropped guess — the job runs again, not zero times
R3.03 · stalled recoveryEchoMQ.Stalled.check/2: expired lease → pending, past max_stalled → deada stuck job — a dead worker’s job returns or dead-letters
R3.04 · atomic finishEchoMQ.Jobs.complete/5: one EVALSHA, token-fenceda torn finish — no half-completed state
R3.05 · blocking pickupEchoMQ.Consumer parks on emq:{cm}:wake (BLPOP)a busy-poll — the consumer sleeps until woken

The claim is the foundation: the guess job moves from its player lane to active in one atomic step — LMOVE rotates the ring of serviceable lanes, ZPOPMIN pops the head of the rotated lane, HSET state active sets the row, and ZADD places the job on the active set at a lease deadline read from the server clock (TIME). So it is in exactly one set at every instant, and a crash leaves it recoverable. That in-flight move is what makes redelivery possible — which is the point — and it is also what makes delivery at-least-once: a worker that scores a guess, then dies before it acknowledges, leaves the job in active to be reclaimed. The stalled sweep returns that job; the atomic finish keeps the completion indivisible; the blocking pickup keeps the consumer idle-cheap. Assembled, they are a queue that loses no guess.

# The assembled path, over a codemojex guess job (real today)
submit  → Codemojex.Guesses.submit/3 mints a branded JOB, EchoMQ.Lanes.enqueue on the player lane
worker  → EchoMQ.Consumer parks on emq:{cm}:wake (BLPOP); an enqueue pushes a wake          # R3.05
claim   → @gclaim: LMOVE ring, ZPOPMIN the lane, HSET state active, ZADD active (lease, server TIME)  # R3.01
process → Codemojex.ScoreWorker.handle/1 scores, writes one GES guess, records the result
finish  → complete/5: one EVALSHA, token-fenced (attempts), ZREM active + DEL row            # R3.04
crash?  → lease expires → reap/2 / Stalled.check/2 returns it to pending; re-claimed         # R3.02 + R3.03
        → past max_stalled: dead-letter onto emq:{cm}:dead → the durable floor (the archive)

Each pattern guards one failure: a lost job, a dropped guess, a stuck job, a torn finish, a busy-poll. Assembled in order, they are a queue that never loses a guess.

§3

Exactly-once-in-effect — the branded id is the key

echo_mq/jobs.ex · @enqueue

The assembled queue guarantees at-least-once delivery, so a guess job can run twice. That is safe because the branded JOB id is the idempotency key, and the bus dedups on it by construction. EchoMQ.Jobs’ enqueue script is one idempotent step: a second enqueue of the same branded id finds the row present (EXISTS KEYS[1]) and returns 0, which the host reads as {:ok, :duplicate}. The same guess admitted twice is admitted once.

echo_mq/jobs.ex · @enqueue — one idempotent stepverbatim
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0                                       ← the JOB id is already enqueued
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])         ← KEYS[1] = emq:{cm}:job:<JOB id>
return 1

The finish is fenced too. EchoMQ.Jobs.complete/5 refuses a stale attempts token ({:error, :stale}), so a worker whose lease was reaped and re-claimed by another worker cannot complete a job it no longer owns. The id orders the queue (mint order is byte order is sort order), names the row, and dedups admission — one 14-byte value doing the work of an index and a guard. So a guess that is redelivered after a crash is scored once, not twice: the first run writes the GES guess; a redelivered job re-claims and re-runs against the same secret, and the consumer is the authority that holds the count.

The interactive below makes the proof concrete. Submit the same guess job any number of times; the scored count stays at one. The first delivery records the guess; every later delivery is refused at admission or runs as a no-op against the dedup. At-least-once delivery over a branded-id-deduped effect is exactly-once-in-effect.

The idempotency proof. Submit the same guess job repeatedly; deliveries climb, but the scored count stays one.
EchoMQ.Jobs.enqueue over the same branded JOB id — the effect records once deliveries (at-least-once) 0 scored (effect) 0 press submit — the first delivery enqueues and scores
readout · Start: zero deliveries, zero scored. Press submit repeatedly — deliveries climb, but the scored count stays one after the first.
the chapter, assembled

A reliable queue — the in-flight claim (R3.01), at-least-once delivery (R3.02), stalled recovery (R3.03), the atomic finish (R3.04), the blocking pickup (R3.05) — loses no job, but delivers each one or more times.

its codemojex application

The consumer is Codemojex.ScoreWorker; the producer mints a branded JOB whose id is the idempotency key EchoMQ.Jobs.enqueue dedups, and complete/5 is token-fenced — so the assembled queue is exactly-once-in-effect.

A reliable queue plus a branded-id-deduped, token-fenced consumer is the whole reliable design: lose no work, score nothing twice. Exactly-once delivery is a lie; exactly-once-in-effect is the achievable truth.

§4

Grounded in the real surfaces

echo_mq/* · codemojex/game.ex

Every primitive in the assembled path is real EchoMQ code under echo/apps/echo_mq, and the consumer is real codemojex code under echo/apps/codemojex. The states are locations in the braced emq:{cm}: keyspace, and the branded JOB id is gated at the key builder (EchoMQ.Keyspace.job_key/2 raises on an ill-formed id).

GuaranteeThe EchoMQ surface (Elixir, verified)The location
in-flight moveEchoMQ.Lanes.claim/3 → @gclaimthe player lane → emq:{cm}:active
stalled recoveryEchoMQ.Stalled.check/2, EchoMQ.Jobs.reap/2emq:{cm}:active → pending / dead
atomic finishEchoMQ.Jobs.complete/5 → @completeemq:{cm}:active → row DEL
blocking pickupEchoMQ.Consumer (BLPOP, :lease_ms 30000)emq:{cm}:wake
idempotency keythe branded JOB id; enqueue refuses a duplicateemq:{cm}:job:<JOB id>

Notes on ValkeyThe claim is one atomic move: ZPOPMIN on the pending set and ZADD onto the active set at a lease deadline read from the server’s own clock (TIME), so a job is in exactly one set and the lease is timed on one clock — never the caller’s — via valkey.io/commands/zpopmin.

the application · what happens at the durability frontier

A dead-lettered guess lands on the floor

A guess that stalls past max_stalled is dead-lettered onto emq:{cm}:dead, and a trimmed stream is not discarded. EchoStore.StreamArchive folds trimmed segments into the native EchoStore.Graft engine’s CubDB and on to Tigris — deep history without resident memory. Durability is a dial the system turns; the enqueue and claim touch only the bus.

echo_store/stream_archive.ex · the durable foldverbatim
A store-side fold of trimmed EchoMQ.Stream segments into the native
EchoStore.Graft engine's CubDB (and, with remote_cfg, on to Tigris),
readable beside the live tail through a merge-read split on an
engine-derived watermark W.
§5

References

sources · related

Sources

  • Valkey — RPOPLPUSH — the atomic in-flight move family; the claim is its sorted-set form (ZPOPMIN then ZADD), one job in exactly one set.
  • Valkey — EVALSHA — the atomic finish: complete/5 runs as one cached script, so no second command can tear away.
  • Valkey — BLPOP — the consumer blocks on the wake key between jobs; park, do not poll.
  • Valkey — SET — SET … PX, value and expiry in one atomic command, the lease idiom read back on the server clock.
  • Redis — Documentation — sorted sets, hashes, and scripting: the command families the reliable-queue path is built from.

Related in this course