R3 · dive 3

The road ahead.

The reliable queue is the spine. Each later chapter adds one capability on top of it — and on the far side wait the two systems that put the patterns to work.

R3 builds a queue that survives a crash. R4 through R8 each add one capability to that queue: a clock, a memory, restraint, frugality, endurance — and each is one more Valkey structure used one more way over EchoMQ's real worker path. Then the patterns hand off: the EchoMQ Queue pillar teaches the protocol in depth, and the persistence floor holds a job that lands for keeps. Walk the arc below, then open the doors. The architecture law beneath it all is /bcs.

The course arc — the reliable queue is the spine, each chapter adds one capability. Hover or select a chapter to light its path and read what it adds.

§1

The arc — one queue, one sense at a time

echo/apps/echo_mq

R3 builds a queue that is correct under a crash. A JOB moves from emq:{q}:pending into emq:{q}:active under a server-clock lease and on to completion; a worker that dies leaves its job leased and recoverable, not lost. That correct queue is the floor. Every chapter after it adds exactly one capability, and each is one more Valkey structure put to one more use.

R4 · Time, Delay & Priority turns the sorted set into a clock. EchoMQ already schedules: EchoMQ.Jobs.enqueue_at and enqueue_in park a job on emq:{q}:schedule at a server-clock run-at, and EchoMQ.Jobs.promote/3 releases it to pending once due. The queue learns when.

R5 · Streams & Events adds the durable log. EchoMQ.Events records lifecycle transitions on an ordered, replayable stream, so the queue learns to remember and replay.

R6 · Flow Control & Scale keeps the tier stable under load — fair lanes (EchoMQ.Lanes), per-group limits, and bulk batches. The queue learns restraint.

R7 · Data Modeling & Memory attends to how the job data lives in RAM — compact encodings and capped structures. The queue learns to spend memory well.

R8 · Production & Operations runs the tier at scale: pooling, failover, and operating everything above. The queue learns to survive production.

Read the arc as one queue gaining one sense at a time: a clock, a memory, restraint, frugality, endurance. The reliable queue is the constant; each chapter is a verb it learns.

§2

The durable floor — where a job lands for keeps

echo_store/stream_archive.ex · graft.ex

A job that exhausts its retries, or whose completion is archived, reaches the persistence floor beneath the volatile tiers. EchoMQ.Stalled.check/3 dead-letters a repeatedly-stalled job; EchoStore.StreamArchive folds a completed stream into the durable page engine EchoStore.Graft (CubDB → Tigris remote behind a create-only commit fence).

Durability is a dial a system turns — hold nothing, a bounded in-heap window with a checkpoint, or commit-per-record replicated off-box — and the enqueue hot path touches only a small, mostly-idle outbox, never a database on the path of every claim. The comparison is Oban, which keeps jobs in the same PostgreSQL as the data so a job and a row commit in one transaction; Echo separates the bus from the store and buys an in-memory hot path plus the dial, giving up that one-transaction coupling. The durable floor is taught end to end in /echo-persistence.

Notes on Valkeythe bus and the store stay separate engines — the enqueue path writes only the in-memory emq:{q}: keyspace and a small outbox, so a claim and a complete never wait on a disk write; the archive fold to the durable floor runs off the hot path. Valkey is the only engine on the queue path — valkey.io/topics/persistence.

§3

The doors beyond

/echomq/queue · /echo-persistence

R3 hands off twice. The patterns have shown the techniques — the leased state machine, the inline Lua, the fencing token, the reliable queue — and grounded each in how EchoMQ really uses it. What this course deliberately does not do is teach the protocol in full: the complete script bundle, the version-fence governance, the worker-pool metronome. That depth is the EchoMQ Queue pillar. And the durable substrate beneath a reaped or archived job is the persistence floor.

/echomq/queue

The EchoMQ Queue pillar

The leased state machine, the inline Lua script bundle, fair lanes, and the schedule set — the engine that runs the reliable queue, taught rung by rung.

/echo-persistence

The durable floor

The durability dial: EchoStore.StreamArchive folding into EchoStore.Graft (CubDB → Tigris), the outbox beside the bus, and the Oban comparison.

the patterns

R0–R8 teach the Redis techniques applied. Each pattern lands twice — the technique and its trade-offs, then the one real EchoMQ excerpt that proves it. A catalog of judgement, grounded.

the systems

The EchoMQ Queue pillar teaches the protocol in depth; the persistence floor holds what a reaped or archived job leaves behind. The patterns are the vocabulary; these are the systems that speak it.

The patterns are the vocabulary; EchoMQ and the persistence floor are the systems that speak it. Learn the patterns here, then go build with them there.

§4

References

sources · related

Sources

Redis — Sorted sets — the ZSET as a clock and a priority ladder, the structure R4 turns into delay and run-at.
Redis — Streams — the durable, replayable log R5 teaches on top of the reliable queue.
Valkey — TIME — the server clock the schedule run-at and the lease both read.
llms.txt — The /llms.txt convention — the machine-readable map format both courses publish for agents.

Related in this course

R3 · Reliable Queuesthe chapter this dive closes
R3 · States as locationsdive 2: the lifecycle as one atomic Lua move
/echomq/queuethe EchoMQ Queue pillar: the protocol in depth
/echo-persistencethe durable floor a dead-lettered or archived job reaches