Retention & the archive · dive 03

The door to persistence

Durability is a dial a system turns — and the stream's archive is one rung of the floor beneath the whole stack.

The archive is one rung of a larger structure: the persistence floor, the durable substrate beneath the volatile tiers. Above it sit the fast, droppable tiers — the in-heap window, the Valkey bus and warm L2. Beneath it sits durable, replicated storage — the native EchoStore.Graft engine on CubDB, on to Tigris object storage. Durability is not a switch; it is a dial a system turns, per surface, for the guarantee that surface needs. Turn it up where you need every record to survive the box; turn it down where the data is cheap to recompute. The Bus's stream archive is one position on that dial — and this page is the door to the whole of it. It is all real code.

The durability dial Deep history, no resident memory The comparison — Oban Pattern & implementation References

The durability dial · pick a position

The durability dial

Durability is a dial a system turns, in three positions ordered by how much each costs and how much it keeps:

Hold nothing. The data is derived and droppable — recompute it. The cheapest position; no durable write at all (the ETS head cache that fronts a read). A bounded window + a checkpoint per K. Keep a bounded slice in heap and checkpoint it to disk every K records. The middle position; bounded memory, bounded loss on a crash, cheap steady state (the stream's live tail bounded by retention, folded to the floor per beat). Commit-per-record + replicate off-box. Commit every record durably and replicate it off the machine. The most durable position; every record survives the box (the Graft engine committing pages, the streamer shipping them to Tigris).

The stream tier sits across positions 2 and 3: the live log holds a bounded window (retention), and the archive commits each trimmed record to the floor and on to the remote. The hot enqueue path touches only the small, mostly-idle outbox beside the bus — never a database on the path of every dequeue or ack. You turn the dial up where you need durability and pay for it; you turn it down where the data is cheap to recompute, and keep the hot path fast.

One dial, three positions — and each surface picks its own. The stream keeps a bounded live tail and a durable archive, so it is fast at the head and deep at the floor at once.

Deep history, no resident memory

The merge-read is what makes the dial pay off: the deep feed survives without resident memory of it. merge_read/5 serves archived ∪ live as one mint-ordered stream — the archive read from the engine's reserved page range below the watermark W, the live tail read from the stream above it. A reader queries a stream's whole past beside its present, and the past costs no live memory: it is on the floor, page-ordered by the branded id, fetched only when a read reaches for it. A stream is bounded in RAM and deep in history at the same time — the two used to be a trade; here they are not.

In the words of the manuscript: “the stream archive folds trimmed EchoMQ.Stream segments into the same floor — deep history without resident memory,” and “what a system trims is kept, not lost.”

A deep read · how far back can a reader go

read back — records into the past

The dial's payoff is concrete: the resident footprint is the live tail's size, fixed by retention, whether a read reaches back ten records or ten million. The archive's depth costs disk, not heap — and a polyglot reader finds the seam from the wire cache without a store call.

The comparison — Oban

The contrast that names the trade is Oban. Oban keeps jobs in the same Postgres as the application data, so a job and a row commit in one transaction — you cannot enqueue work and forget to do it, because the enqueue and the state change are atomic together. That coupling is real and valuable. Echo gives it up: it separates the bus from the store, so the enqueue and the data write are not one transaction. What it buys for that is an in-memory hot path (the enqueue touches the bus, not a database) and the dial (each surface picks its own durability).

State the trade beside the win: Echo does not have Oban's one-transaction coupling; it has a fast path and a per-surface dial instead. Neither is free. The right tool depends on which guarantee the workload needs — the atomic job-and-data commit, or the fast head with a tunable floor.

The trade · one transaction, or a tunable floor

Echo gives up Oban's one-transaction coupling for an in-memory hot path and a per-surface dial — a trade, stated beside the win, never a free upgrade.

Walk through the door

The whole floor — the single-writer Graft engine, the lazy partial reader at a snapshot, the portable Tigris remote, the commit-LSN cursor a replica reads from, and the stream archive that folds trimmed segments into the same engine — is taught in full next door. The Bus's archive is one rung of that floor; here is the door to the rest of it.

The persistence floor — in full

Echo Persistence is the durable floor beneath the whole stack: EchoStore.Graft, a native-BEAM build of lazy, partial, strongly-consistent, page-based replication on object storage — the single-writer engine on CubDB, the lazy reader that fetches only the pages a read touches, the portable Tigris remote behind a create-only commit fence, and the stream archive this module folds into. The manuscript narrates the same substrate at The Branded Component System · the persistence floor.

Open Echo Persistence

Pattern & implementation

the pattern (Redis Patterns Applied)

A durable log archives its history off the hot store and serves deep reads from it, so the live store stays small. Streams & Events teaches the durable, replayable log; the durability dial is the floor beneath it.

the implementation (echo_store)

EchoStore.StreamArchive folds trimmed EVT segments into the native EchoStore.Graft engine (CubDB → Tigris), and the merge-read serves archived ∪ live as one mint-ordered stream — deep history without resident memory, the durability dial turned to keep-and-replicate.

The pattern says: keep deep history off the hot path. The implementation makes that a dial — the same branded id from the bus to the floor, the guarantee tuned per surface.

Recap — one rung of the floor

The archive is one rung of the persistence floor, and durability is a dial — hold nothing, a bounded window + a checkpoint, or commit-per-record + replicate off-box. The stream tier sits across the last two: a bounded live log and an archive on the durable floor, with the merge-read serving deep history without resident memory. The comparison is Oban — Echo gives up the one-transaction coupling for an in-memory hot path and the dial. The whole floor is one door away. That completes the Bus pillar's reading of the retained log: published, appended, consumed, replayed by instant, bounded, and kept.

The door out: Echo Persistence — the durable floor in full, and The Branded Component System · the persistence floor, its narrative.

References

Where the durability dial, the durable engine, and the Oban comparison are grounded.

Sources

orbitinghail — Graft — the lazy, partial, strongly-consistent page-replication engine EchoStore.Graft builds natively on the BEAM.
Tigris — conditional object operations — the create-only commit fence that makes object storage a transactional remote at the dial's top position.
Sorin et al. — Oban — the same-Postgres job coupling Echo gives up for an in-memory hot path and the durability dial.
Helland — Life Beyond Distributed Transactions (CIDR 2007) — entities made durable and addressed by identity across a boundary, the floor's frame.

Related in this course

Retention & the archive — the module this dive belongs to.
Nothing is lost — the fold this dive opens the floor behind.
Retention is a policy — the trim the fold runs before.
The Bus — the pillar landing.
Echo Persistence — the durable floor in full.
The Branded Component System · the persistence floor — the manuscript chapter this floor realizes.