jonnify·redis /redis-patterns/streams-events/streams-event-sourcing

R5.01 · Streams & Events — module hub

Event sourcing on Streams.

Store every change as an immutable event appended to a log that is the source of truth, and treat current state as the replay of that log — not a row you overwrite, but a fold you recompute.

A row says what is true now; a log says everything that ever happened, in order. Event sourcing keeps the log and derives the row: nothing is overwritten, every transition is appended, the present is reconstructed by folding the log from the start. The grounding is EchoMQ's real Stream Tier (echo/apps/echo_mq): EchoMQ.Stream.append/4 issues XADD emq:{q}:stream:<name> and returns the branded EVT id; EchoMQ.Stream.read/6 issues XRANGE and hands the entries back in mint order to fold. The three dives take it apart — the append, the rebuild, the cursor.

A row overwrites; a log appends — only the log keeps the history. Select how the third change is stored, then read what is left to reconstruct from.
the log · XADD appends …000-0placed …001-0filled …002-0closed the row · SET overwrites order:7 (one value)closed append entry 3 overwrite, history lost
readout · Select a mode. The log keeps placed, filled, and closed as three entries (the whole history, replayable); the row keeps only closed (placed and filled are gone).
§1

The pattern in one move

streams-event-sourcing.md · the inversion

A row says what is true now. A log says everything that ever happened, in order. Event sourcing keeps the log and derives the row: nothing is overwritten, every transition is appended, and the present is reconstructed by folding the log from the start. A Redis Stream is the structure — an append-only sequence of entries, each with a time-ordered id that orders it against every other.

Three properties follow, and they are the three dives. The log is the source of truth: every state change is appended, never updated in place. State is derived, not stored: reading the log and folding the entries reconstructs the current state, recomputable at any time. And the entry id is the cursor: monotonic, it is the resume position a reader saves, and — because the id carries time — a DateTime becomes a range bound, so the log can be read as of any past instant.

Event sourcing keeps the immutable log and derives the present from it: append every change, fold the log to get the state, and let the entry id be the cursor.

§2

How it works

streams-event-sourcing.md · XADD append · XRANGE read

A Redis Stream is created the first time something is appended to it. XADD key * field value … appends an entry, assigning a time-ordered id of the form <milliseconds>-<sequence>; the * asks the server to mint the id, or an explicit id may be supplied. The entries form an ordered log: a later append always carries a larger id than an earlier one.

Reading is by range. XRANGE key - + returns every entry from the first (-) to the last (+) in id order; XRANGE key <start> <end> COUNT n reads a bounded slice. To rebuild state, a reader starts with an empty accumulator and applies each entry in order — the same fold every time, deterministic in the log.

XADD orders * type placed amount 4200   # append; * mints a time-ordered id -> "…000-0"
XADD orders * type filled amount 4200   # append the next event           -> "…001-0"
XRANGE orders - +                       # read the whole log, oldest first, to fold

Append with XADD, read the range with XRANGE, fold the entries in id order; the state is the fold and the log is the truth.

§3

The events are immutable; the state is a fold

streams-event-sourcing.md · fixed history, replaceable fold

An event is a fact that happened: it is never edited and never deleted as a correction (a correction is a new event). The log grows forward only. Because the entries never change, the fold over them is reproducible — replay the same log and get the same state. A bug in how state is derived is fixed by changing the fold and replaying; the events stay as they were recorded.

State is the fold of the log — apply each event in turn. Step the fold over placed, filled, closed; the accumulator is the present after the last applied event.
placed filled closed state (accumulator)none
readout · Step the fold. The accumulator starts at none; applying placed → filled → closed reconstructs the present from the events, in order.

Notes on ValkeyXLEN key returns the number of entries in a stream — the size of the log a reader must fold, and the count a snapshot bounds the replay against — valkey.io/commands/xlen.

The events are fixed history; the fold over them is reproducible and replaceable — fix the derivation and replay, and the recorded events never change.

§4

When to use it, when to avoid it

streams-event-sourcing.md · the trade

Use event sourcing when the history matters as much as the present: an audit trail, a feed of activity, a value that must be reconstructable as of a past time, or a state machine whose transitions other readers also consume. The log is the one authority, and multiple independent readers each fold it for their own view.

Avoid it when only the latest value matters and history is noise — a plain key holds the current value far more cheaply than a log that must be folded. A log also grows without bound unless retention is applied (the subject of R5.02); an unbounded log is a leak. And a fold over a long history is work — a snapshot of the folded state, taken periodically, bounds the replay a reader must do.

Keep the log when the history is the point and many readers fold it; reach for a plain value when only the present matters and the log would only grow.

§5

Applied — EchoMQ's Stream Tier

echo_mq/stream.ex · append/4 · read/6

EchoMQ's bus carries a queue (work claimed and completed once) and, beside it, a Stream Tier: an append-only log many readers consume at their own pace. EchoMQ.Stream is the writer. EchoMQ.Stream.append(conn, queue, name, fields) mints an EVT-branded record id host-side, derives the explicit XADD id from it, and issues XADD emq:{q}:stream:<name> <xadd_id> id <branded> <fields…> — returning {:ok, branded}, the branded id as the receipt. The log is ordered by append the way the property store is ordered by mint: a strictly monotone writer makes the stream id, the entry order, and the mint order one and the same.

EchoMQ.Stream.read(conn, queue, name, from, to, count) wraps XRANGE and parses each entry back into a {branded, fields_map} tuple in mint order — the read side of the rebuild. The stream key emq:{q}:stream:<name> carries the per-queue {q} hash tag, so every key of one queue lands on one Valkey cluster slot.

# EchoMQ.Stream — the writer and reader of the Stream Tier (real)
EchoMQ.Stream.append(conn, "orders", "events", [{"type", "placed"}, {"amount", "4200"}])
# mints an EVT id -> XADD emq:{orders}:stream:events <xadd_id> id <branded> type placed amount 4200
# -> {:ok, "EVT0Nk2…"}    (the branded id is the receipt)
EchoMQ.Stream.read(conn, "orders", "events")   # XRANGE - + -> [{branded, fields_map}, …] in mint order

Notes on ValkeyA stream key carries the queue name inside the braces (emq:{q}:stream:<name>), so every key of one queue hashes to one of the 16384 cluster slots and a multi-key script stays legal — valkey.io/topics/cluster-spec.

the pattern

Append every state change as an immutable entry to a log that is the source of truth; reconstruct current state by folding the log from the start.

its EchoMQ application

EchoMQ.Stream.append/4 issues XADD emq:{q}:stream:<name> and returns the branded EVT id; EchoMQ.Stream.read/6 issues XRANGE and hands the entries back in mint order to fold. The append order is the truth order.

Redis offers an append-only log whose id orders it; EchoMQ's Stream Tier is that log, with the branded EVT id as the receipt and the fold over XRANGE as the rebuild.

/echomq/bus

A door, not a depth — the EchoMQ Bus pillar

This module cites two excerpts — the append in EchoMQ.Stream.append/4 and the read in EchoMQ.Stream.read/6 — as proof the log ships. The consumer-group reader law, the polyglot seam, retention, and the archive fold are the subject of the dedicated EchoMQ course. Open onto the Bus pillar — the broadcast and the retained, replayable log.

/bcs/bus

The Stream Tier in the manuscript

The figures here are drawn from the Branded Component System bus chapter. As B3.3, the Stream Tier puts it: a queue is for work claimed and completed once; a stream is an append-only log many readers consume at their own pace, and the stream keys are born braced and branded like every other key on the bus.

After R5.01, the Streams & Events chapter continues with reading the log reliably — the blocking read, consumer groups, and trimming. Return to the Streams & Events chapter to continue.

§6

The three dives

append → rebuild → cursor

Each dive takes one property of the log, in order: the append that makes it the source of truth, the read that folds it back to state, and the entry id read as the cursor and the time bound. Read them in order — the write first, the read second, the cursor third.

The three dives ground in the same writer and reader: the append, the rebuild, and the cursor. Begin with the append-only log.

§7

References

sources · related

Sources

Related in this course