jonnify·redis /redis-patterns/streams-events

R5 · Streams & Events

The durable,
replayable log.

A row says what is true now; a Redis Stream says everything that ever happened, in order — append every event, replay to rebuild state, and let many readers fold the same log at their own pace.

A stream is not a work queue but an append-only log: a read moves a cursor instead of removing the entry, so the entry stays until each reader acknowledges it. The grounding is EchoMQ's real shipped Stream Tier (echo/apps/echo_mq) — EchoMQ.Stream.append/4 issues XADD emq:{q}:stream:<name> and returns the branded EVT id, EchoMQ.StreamConsumer carries the consumer-group reader law, and EchoMQ.StreamRetention trims the log under a named policy — worked through codemojex's activity feed. This chapter records the log that R6–R8 lean on.

Why & when The patterns How to apply Workshop References
§1

Why & when

echo/apps/echo_mq

A plain key holds the latest value and forgets the rest. The moment the history matters as much as the present — an audit trail, an activity feed, a value that must be reconstructable as of a past instant, a state machine other readers also consume — a single overwritten value is not enough. Reach for streams whenever the record of what happened must outlive the latest value, and many independent readers each need to consume that record at their own pace. Each demand below has one matching technique, and the chapter is the set of those answers.

The history is the truth

Keep the immutable log and derive the present by folding it — nothing is overwritten, every transition is appended, the state is recomputed from the start.

Many readers, one log, different speeds

A read moves a cursor, not the entry — one reader can sit at the tail while another replays from the beginning, and the entry stays until each acknowledges it.

A reader must wait, not busy-poll

A blocking read parks until the next entry arrives, so a caught-up consumer costs the engine nothing and wakes the instant there is work.

The log must stay bounded

Retention is a policy, not a default — trim the live log by length or age, and fold what is trimmed to a durable floor so deep history survives without resident memory.

The stream lifecycle · append to the log → fold to rebuild state → trim under a policy, with the archive fold to the durable floor
append XADD emq:{q}:stream time-ordered id the log append-only · immutable XRANGE · fold state derived, replayable trim · MAXLEN → archive fold to the durable floor

Every technique in this chapter answers one demand the log makes good on: the history as the truth, many readers at their own pace, a reader that waits without polling, and a log kept bounded with its deep history folded to a durable floor.

§2

The patterns

five modules · two built

Two teaching modules are built; three more are specified. Each module is a hub with its dives, grounded in the real EchoMQ Stream Tier.

built ready to read soon specified
R5.01built

Event sourcing on Streams

The append-only log is the source of truth; replay rebuilds state. Grounded in EchoMQ.Stream.append/4 and read/6.
R5.02built

Stream consumer patterns

Block, batch, trim, resume — read the log reliably. Grounded in EchoMQ.StreamConsumer and StreamRetention.
R5.03built

Pub/Sub vs Streams

Fire-and-forget against a durable, replayable log; the rule for choosing a channel a subscriber must be present for over a log a reader can resume.
R5.04built

Custom events & projections

Arbitrary domain events appended to the same stream, and windowed projections folded from them — many independent views over one log.
R5.05built

Workshop

A live codemojex activity feed off the :events stream lifecycle — every guess, round, and settlement an event, the feed a fold of the log.
§3

How to apply

name the demand · the technique follows

The hard part is matching the stream technique to the demand you have. A stream is read non-destructively, and the four moves below are how a reader stays reliable while many readers share the one log. Name the demand, and the technique — and the real EchoMQ surface that implements it — follows.

Name the demand · get the stream technique and the EchoMQ surface that becomes it
readout · Select a demand to see the technique and the real EchoMQ surface that answers it.

There is no single streams trick — only the technique that answers the demand you have, each one a real move in EchoMQ's Stream Tier.

§4

The workshop

echo/apps/codemojex · EchoMQ.Stream

The chapter closes with R5.05: a live codemojex activity feed built off the :events stream lifecycle. Each event the game produces — a guess submitted, a round opened, a settlement posted — is appended to emq:{q}:stream:events with EchoMQ.Stream.append/4, minting a branded EVT id as the receipt. The feed is a fold of that log: a reader replays the stream in mint order to render what happened, and a consumer group lets the notification side resume after a restart without re-sending what it already delivered. Retention trims the live log under EchoMQ.StreamRetention, and EchoStore.StreamArchive folds older segments to the durable floor, so the deep feed survives without holding memory.

Open the workshop → — build the feed end to end: append the domain events, fold the log to render the feed, consume reliably with a group, then bound the live log and fold the deep history to the archive.

One log, many readers — a read moves a cursor, not the entry. Select a reader to see where it sits; the tailer waits at the head for the next event, the replayer folds from the start, and trimmed entries fold to the durable floor.
emq:{q}:stream:events · append-only, read non-destructively …000-0guessed …001-0round …002-0guessed …003-0settled …004-0round trimmed → folds to the durable floor replayer · fold tailer · BLOCK
readout · Select a reader. The tailer parks at the head for the next event; the replayer folds from the start to rebuild the feed. The first entry has been trimmed and folded to the durable floor — both readers share the one log.
/echomq/bus

The retained log in depth

The EchoMQ Bus pillar teaches the broadcast and the retained, replayable log — the consumer-group reader law, the polyglot seam, retention, and the archive fold behind the emq:{q}:stream keyspace, rung by rung.

 /echo-persistence

The durable floor

A trimmed stream segment, or a stream archived for deep history, reaches the durable tier — EchoStore.StreamArchive folding into the Graft engine and out to Tigris, the durability dial taught end to end.

Notes on Valkey. A stream key carries the queue name inside the braces (emq:{q}:stream:<name>), so every key of one queue hashes to one of the 16384 cluster slots and a multi-key script stays legal — no CROSSSLOT. valkey.io/topics/cluster-spec

§5

Up next

R6–R8

R5 records the durable event log the rest of the tier can lean on. Each later chapter is a different concern over the same record — staying stable under load, how the record lives in RAM, then running the tier in production.

R6soon

Flow Control & Scale

Staying stable under load.

R7soon

Data Modeling

How data lives in RAM.

R8soon

Production & Ops

Running the tier at scale.

§6

References

sources · related

Sources

  • Valkey — Introduction to Streams — the append-only log, entry ids, consumer groups, and the range read this chapter is built on.
  • Valkey — Cluster specification — the {q} hash tag forces a queue's stream keys onto one of the 16384 slots.
  • Kreps, J. — The Log — the log as the shared abstraction beneath a stream.
  • Oban — jobs and data in one Postgres transaction; the bus separates the log from the store and buys an in-memory hot path with the durability dial.

Related in this course