R5.01 · Streams & Events — module hub
Event sourcing on Streams.
Store every change as an immutable event appended to a log that is the source of truth, and treat current state as the replay of that log — not a row you overwrite, but a fold you recompute.
A row says what is true now; a log says everything that ever happened, in order. Event sourcing
keeps the log and derives the row: nothing is overwritten, every transition is appended, the present is
reconstructed by folding the log from the start. The grounding is EchoMQ's real Stream Tier
(echo/apps/echo_mq): EchoMQ.Stream.append/4 issues XADD emq:{q}:stream:<name>
and returns the branded EVT id; EchoMQ.Stream.read/6 issues XRANGE and hands
the entries back in mint order to fold. The three dives take it apart — the append, the rebuild, the cursor.
The pattern in one move
streams-event-sourcing.md · the inversionA row says what is true now. A log says everything that ever happened, in order. Event sourcing keeps the log and derives the row: nothing is overwritten, every transition is appended, and the present is reconstructed by folding the log from the start. A Redis Stream is the structure — an append-only sequence of entries, each with a time-ordered id that orders it against every other.
Three properties follow, and they are the three dives. The log is the source of truth: every state change is
appended, never updated in place. State is derived, not stored: reading the log and folding the entries
reconstructs the current state, recomputable at any time. And the entry id is the cursor: monotonic, it is the
resume position a reader saves, and — because the id carries time — a DateTime becomes a
range bound, so the log can be read as of any past instant.
Event sourcing keeps the immutable log and derives the present from it: append every change, fold the log to get the state, and let the entry id be the cursor.
How it works
streams-event-sourcing.md · XADD append · XRANGE readA Redis Stream is created the first time something is appended to it. XADD key * field value …
appends an entry, assigning a time-ordered id of the form <milliseconds>-<sequence>; the
* asks the server to mint the id, or an explicit id may be supplied. The entries form an ordered log:
a later append always carries a larger id than an earlier one.
Reading is by range. XRANGE key - + returns every entry from the first (-) to the last
(+) in id order; XRANGE key <start> <end> COUNT n reads a bounded slice. To
rebuild state, a reader starts with an empty accumulator and applies each entry in order — the same fold
every time, deterministic in the log.
XADD orders * type placed amount 4200 # append; * mints a time-ordered id -> "…000-0" XADD orders * type filled amount 4200 # append the next event -> "…001-0" XRANGE orders - + # read the whole log, oldest first, to fold
Append with XADD, read the range with XRANGE, fold the entries in id order;
the state is the fold and the log is the truth.
The events are immutable; the state is a fold
streams-event-sourcing.md · fixed history, replaceable foldAn event is a fact that happened: it is never edited and never deleted as a correction (a correction is a new event). The log grows forward only. Because the entries never change, the fold over them is reproducible — replay the same log and get the same state. A bug in how state is derived is fixed by changing the fold and replaying; the events stay as they were recorded.
Notes on ValkeyXLEN key returns the number of entries in a
stream — the size of the log a reader must fold, and the count a snapshot bounds the replay against —
valkey.io/commands/xlen.
The events are fixed history; the fold over them is reproducible and replaceable — fix the derivation and replay, and the recorded events never change.
When to use it, when to avoid it
streams-event-sourcing.md · the tradeUse event sourcing when the history matters as much as the present: an audit trail, a feed of activity, a value that must be reconstructable as of a past time, or a state machine whose transitions other readers also consume. The log is the one authority, and multiple independent readers each fold it for their own view.
Avoid it when only the latest value matters and history is noise — a plain key holds the current value far more cheaply than a log that must be folded. A log also grows without bound unless retention is applied (the subject of R5.02); an unbounded log is a leak. And a fold over a long history is work — a snapshot of the folded state, taken periodically, bounds the replay a reader must do.
Keep the log when the history is the point and many readers fold it; reach for a plain value when only the present matters and the log would only grow.
Applied — EchoMQ's Stream Tier
echo_mq/stream.ex · append/4 · read/6EchoMQ's bus carries a queue (work claimed and completed once) and, beside it, a Stream Tier: an
append-only log many readers consume at their own pace. EchoMQ.Stream is the writer.
EchoMQ.Stream.append(conn, queue, name, fields) mints an EVT-branded record id host-side,
derives the explicit XADD id from it, and issues
XADD emq:{q}:stream:<name> <xadd_id> id <branded> <fields…> —
returning {:ok, branded}, the branded id as the receipt. The log is ordered by append the way the
property store is ordered by mint: a strictly monotone writer makes the stream id, the entry order, and the mint
order one and the same.
EchoMQ.Stream.read(conn, queue, name, from, to, count) wraps XRANGE and parses each
entry back into a {branded, fields_map} tuple in mint order — the read side of the rebuild. The
stream key emq:{q}:stream:<name> carries the per-queue {q} hash tag, so every key
of one queue lands on one Valkey cluster slot.
# EchoMQ.Stream — the writer and reader of the Stream Tier (real) EchoMQ.Stream.append(conn, "orders", "events", [{"type", "placed"}, {"amount", "4200"}]) # mints an EVT id -> XADD emq:{orders}:stream:events <xadd_id> id <branded> type placed amount 4200 # -> {:ok, "EVT0Nk2…"} (the branded id is the receipt) EchoMQ.Stream.read(conn, "orders", "events") # XRANGE - + -> [{branded, fields_map}, …] in mint order
Notes on ValkeyA stream key carries the queue name inside the braces
(emq:{q}:stream:<name>), so every key of one queue hashes to one of the 16384 cluster slots and a
multi-key script stays legal —
valkey.io/topics/cluster-spec.
Append every state change as an immutable entry to a log that is the source of truth; reconstruct current state by folding the log from the start.
EchoMQ.Stream.append/4 issues XADD emq:{q}:stream:<name> and returns the branded
EVT id; EchoMQ.Stream.read/6 issues XRANGE and hands the entries back in
mint order to fold. The append order is the truth order.
Redis offers an append-only log whose id orders it; EchoMQ's Stream Tier is that log, with the branded
EVT id as the receipt and the fold over XRANGE as the rebuild.
A door, not a depth — the EchoMQ Bus pillar
This module cites two excerpts — the append in EchoMQ.Stream.append/4 and the read in
EchoMQ.Stream.read/6 — as proof the log ships. The consumer-group reader law, the polyglot
seam, retention, and the archive fold are the subject of the dedicated EchoMQ course. Open onto
the Bus pillar — the broadcast and the retained, replayable log.
The Stream Tier in the manuscript
The figures here are drawn from the Branded Component System bus chapter. As B3.3, the Stream Tier puts it: a queue is for work claimed and completed once; a stream is an append-only log many readers consume at their own pace, and the stream keys are born braced and branded like every other key on the bus.
After R5.01, the Streams & Events chapter continues with reading the log reliably — the blocking read, consumer groups, and trimming. Return to the Streams & Events chapter to continue.
The three dives
append → rebuild → cursorEach dive takes one property of the log, in order: the append that makes it the source of truth, the read that folds it back to state, and the entry id read as the cursor and the time bound. Read them in order — the write first, the read second, the cursor third.
The append-only log
XADD: the log is the source of truth, every transition appended, the entry immutable once
written; a smaller id is refused. Grounded in EchoMQ.Stream.append/4.
Replay and rebuild
XRANGE: read the log in order and fold the entries to reconstruct current state; an event
is immutable, the state is derived. Grounded in EchoMQ.Stream.read/6.
The cursor
The entry id as the resume position; read_since from a %DateTime{} —
time as a range bound, the time-travel seam. The archive watermark where durable history meets the live tail.
The three dives ground in the same writer and reader: the append, the rebuild, and the cursor. Begin with the append-only log.
References
sources · relatedSources
- Valkey — Introduction to Streams — the append-only log, entry ids, and the range read this pattern is built on.
- Valkey — XADD — append an entry to a stream, assigning a time-ordered id; the write side of event sourcing.
- Valkey — XRANGE — read a range of stream entries in id order, the rebuild read folded to state.
- Valkey — Cluster specification — the
{q}hash tag forces a queue's keys onto one of the 16384 slots. - Kreps, J. — The Log — the log as the shared abstraction beneath a stream.
Related in this course
- R5.01.1 · The append-only logthe write side, XADD
- R5.01.2 · Replay and rebuildthe read side, XRANGE and the fold
- R5.01.3 · The cursorthe entry id as resume position and time bound
- R5.02 · Stream consumer patternsreading the log reliably
- /echomq · Busthe retained, replayable log in depth
- /bcs · The Stream Tierthe manuscript bus chapter, B3.3