← the depth behind redis-patterns R5 · Streams & Events
The Bus · pillar II
Broadcast signals
over the wire
Fan a message out to everyone, then keep an ordered, replayable log of what was said.
The Bus is the second pillar, and it is the mirror of the Queue. Where the Queue hands one job to
one worker, the Bus does the opposite: it broadcasts to everyone, and it keeps
an append-only log many readers replay at their own pace. Two surfaces, one wire. Events
(EchoMQ.Events) are fire-and-forget pub/sub — a consumer reacts to work as it happens
without polling. The stream log (EchoMQ.Stream) is a retained, replayable
log ordered by mint — consumed by groups at-least-once, read by a mint instant for
time-travel, bounded by retention, and what it trims is folded into the
durable floor. The thread through both is the branded id. It is all real code.
The two surfaces, one wire
The Bus is one wire read two ways. Events broadcast a signal to whoever is listening, now and gone. The stream is the same idea made durable: an append-only log, ordered the way the store is ordered — by the branded id — that survives, replays, and folds to disk. Pick a verb to read what it does, the Valkey command beneath it, and the guarantee it makes.
One wire, two reads · pick a verb
One wire: events say it once, the stream remembers — appended in mint order, replayed by a group, read by an instant, trimmed by policy, and what is trimmed is folded, not lost.
The modules
Five modules, each a hub and three dives, then a workshop. Each grounds in the real
echo/apps/echo_mq code — the events channel, the append-only log, consumer
groups, the mint-instant read, and retention folding into the
echo/apps/echo_store archive.
The events channel
Fire-and-forget pub/sub on emq:{q}:events: subscribe once, publish after a verdict, react without polling.
The stream log
The append-only log: the host-side EVT mint, the order theorem, the claims-only id field, the non-monotonic signal.
Consumer groups
At-least-once with idempotent handlers: the lazy group door, drain-self then reclaim dead peers, the re-claim ordering cost.
Time-travel
Read the log by a mint instant: a DateTime becomes an id range bound for backtest, audit, and debug.
Retention & the archive
Bound the log by length or age, then fold the trimmed segments into the durable Graft floor — the door to persistence.
Workshop
Publish an event, append and replay a log by instant, then watch a retention sweep fold a segment to disk.
Redis Patterns Applied
This is the depth behind the Redis Patterns Applied chapter that doors here: R5 · Streams & Events — the pub/sub events and the retained, replayable log that land in this pillar, and the stream patterns (the append-only log, consumer groups, event sourcing) this pillar's log makes concrete. There the pattern is the door; here is the wire.
The durable floor
A log that only grows is a leak, so retention is a policy — and what a stream trims is not lost.
EchoStore.StreamArchive folds the trimmed segments into the durable
EchoStore.Graft floor — CubDB's append-only B-tree, on to Tigris object storage
— at a reserved page range ordered by the branded EVT id, readable beside the
live tail through a watermark merge-read. Deep history without resident memory. That floor is taught in full in
Echo Persistence.
The rest of the system
The Bus is one reading of the shared substrate. The Queue distributes work over the same wire; the Cache serves reads in front of it; the Proof chapter shows the whole system holds. The Protocol below them all — the keyspace the stream keys are born to — is ready to read now.
The Protocol
The substrate the Bus runs on: the braced keyspace, the record, the Lua layer, the branded-id gate.
The Queue
Distribute work: the lifecycle, jobs, lanes, the consumer, batches, flows.
The Cache
Serve reads: cache-aside, single-flight, coherence on the Bus.
The Proof
The system holds: the conformance suite, telemetry, the benchmark gate.
The Queue and the Protocol are ready to read; the Bus modules, the Cache, and the Proof are on the build front, each added as its pages land.
References
Sources
- Valkey — Introduction to Streams — the append-only log and consumer groups the stream tier is built on.
- Valkey — Cluster specification — the
{q}hashtag forces a queue's keys, the stream included, onto one of 16384 slots. - Kreps — The Log — the log as the unifying abstraction beneath a stream.
- Lamport — Time, Clocks, and the Ordering of Events — the happens-before the mint order and the time-travel read order by.
Related in this course
- The Queue — distribute work; the first pillar over the same wire.
- The Protocol — the keyspace and branded-id gate the stream keys are born to.
- Overview — the chapter that frames the three pillars.
- Echo Persistence — the durable floor a trimmed stream history folds into.
- The Branded Component System · the Bus — the manuscript chapter this pillar realizes.