Retention, and what it keeps

Bound the live log so it cannot leak — and fold what you trim to disk, so the past is kept, not lost.

An append-only log appends forever, and memory is finite, so the live log has to be bounded. Retention is how: EchoMQ.Stream.trim/4 caps it by length (MAXLEN) or by age (MINID). The trap it refuses is the one that makes retention dangerous elsewhere — coupling a safety property (bounded memory) to a liveness fact (a consumer is up). So the trim cadence lives on its own beat. And the whole point of this module: what is trimmed is not lost. Before a record leaves the live stream it is folded into the durable Graft floor — committed, kept, and readable beside the live tail. A stream bounded in memory but deep in history. This is the Bus's door to persistence. It is all real code.

A log, a trim, a fold The dives The rest of the pillar References

A log, a trim, a fold

Three phases. The live log grows as records are appended in mint order. A trim bounds it — XTRIM removes the records outside the window. But the trim is not the first thing to touch a record on its way out: a fold writes it to the durable floor first, ordered by the branded id, where it stays queryable. Step the three phases below to read what each does, the verb beneath it, and the guarantee it makes.

The live log, bounded and kept · step the phases

Retention is a policy, not a default — and the trimmed past folds to the floor, kept and queryable beside the present. Bounded in memory, deep in history.

The dives

Three dives: the trim and its safe default, the fold that keeps what the trim removes, and the durability dial the archive sits on. Each grounds in the real echo/apps/echo_mq and echo/apps/echo_store code — EchoMQ.Stream.trim/4, EchoMQ.StreamRetention, EchoStore.StreamArchive. The trim and the fold are issued direct — there is no Lua here.

built page ready to read

01built

Retention is a policy

trim/4 by length or age; the safe ~ default that never over-trims (INV4); the opt-in EchoMQ.StreamRetention driver on its own beat.

02built

Nothing is lost

StreamArchive.fold/3 into the Graft floor: the reserved 2^49 range, the watermark W, the merge-read, and fold-before-trim.

03built

The door to persistence

The durability dial in depth: hold nothing, a bounded window, or commit-and-replicate — deep history without resident memory. The door to /echo-persistence.

Redis Patterns Applied

This module is the depth behind the Redis Patterns Applied chapter that doors here: R5 · Streams & Events — the retention and the durable, replayable log that land in this pillar, made concrete on the wire as XTRIM on the live log and a fold into the page engine. There the pattern is the door; here is the floor.

The durable floor

A log that only grows is a leak, so retention is a policy — and what a stream trims is not lost. EchoStore.StreamArchive folds the trimmed segments into the durable EchoStore.Graft floor — CubDB's append-only B-tree, on to Tigris object storage — at a reserved page range ordered by the branded EVT id, readable beside the live tail through a watermark merge-read. Deep history without resident memory. That floor is taught in full in Echo Persistence, and narrated in the manuscript at The Branded Component System · the persistence floor.

The rest of the pillar

Retention closes the Bus's reading of the retained log. The stream log writes it; the consumer group reads the live tail forward; time-travel reads it by a mint instant. This module bounds it and folds what it trims to disk — and that fold is the pillar's door to the persistence floor, where the whole durability story is told.

03built

Consumer groups

The reliable forward read: at-least-once, resumable, idempotent on the branded id.

04built

Time-travel

Read the live log by a mint instant; deep history past it is the archive this module folds.

→built

Echo Persistence

The durable floor a trimmed stream history folds into — the engine, the lazy reader, the remote.

↑built

The Bus

The pillar landing: the two surfaces over one wire, and all of its modules.

The stream log, the consumer group, and time-travel are ready to read; this module is the last reading of the retained log — and the door to the persistence floor beneath the whole stack.

References

Sources

Valkey — XTRIM — the trim that bounds the live log by MAXLEN or MINID.
Valkey — Introduction to Streams — the append-only log retention bounds, and the approximate ~ trim.
Valkey — Cluster specification — the {q} hashtag co-locates a queue and its stream on one of 16384 slots.
Kreps — The Log — the commit log whose retained, replayable form the archive extends to disk.

Related in this course

Retention is a policy — trim/4, the windows, the safe default.
Nothing is lost — the fold into the Graft floor and the merge-read.
The door to persistence — the durability dial in depth.
Time-travel — reading the live log by a mint instant.
Echo Persistence — the durable floor a trimmed stream history folds into.
The Branded Component System · the persistence floor — the manuscript chapter this module's floor realizes.