Retention & the archive · dive 02

Nothing is lost

Fold what you trim to the durable floor first — so a bounded log keeps a deep, queryable past.

XTRIM returns only a removed count — never the deleted entries. So there is no recovery after a trim: once a record is gone from the live stream, the stream cannot hand it back. That single fact makes the no-loss guarantee rest on ordering: a record leaves the live stream only after it is committed to the durable floor and the watermark has advanced past it. EchoStore.StreamArchive.fold/3 writes the about-to-trim slice into the native EchoStore.Graft engine, one page per record, at a reserved page range ordered by the branded id; merge_read/5 reads it back beside the live tail as one mint-ordered stream. Drag the watermark on the right to see the seam. It is all real code.

The fold The watermark & the merge-read Fold-before-trim Pattern & implementation References

The merge-read · drag the watermark W

watermark — W

The fold — one page per record, at a reserved range

EchoStore.StreamArchive.fold(volume_id, slice, db) folds a mint-ordered {branded, fields} slice into the native EchoStore.Graft engine's CubDB — one page per record — through the public EchoStore.Graft.VolumeServer.commit/3. The engine internals stay untouched: the archive is a new consumer of the public commit surface, not an edit to it. It returns {:ok, W} (the new frontier), :noop for an empty slice, or the engine's {:error, term} verbatim — and on any fold error the caller does not trim, the safe direction.

The Graft engine multiplexes one page axis: a business write commits its pages at low indices, a page per table row counting up from 0. The archive must never overwrite one of those, so each folded record lands in a reserved high range — @archive_base = 2^49. That sits ~563 trillion indices above the business-page floor, so a forward :arc_seq allocator counting up from 0 can never reach a business page: the archive range [2^49, ∞) is disjoint from where real commits land by construction. The disjointness is proven by the arithmetic, not enforced by a runtime check.

And because records fold in mint order — the writer mints monotone EVT ids, and EchoMQ.Stream.read/6 returns the slice in mint order (the order theorem) — the page axis @archive_base + 0, +1, … is branded-id-monotone. A forward scan reads the archive oldest-first, a reverse scan newest-first, with no second index. The order theorem that held in memory now holds on disk: the page index is the sort.

# echo_store — EchoStore.StreamArchive
# Fold a mint-ordered {branded, fields} slice into the Graft engine at the
# @archive_base range — one CubDB page per record at @archive_base + n — through
# the PUBLIC VolumeServer.commit/3, and advance the watermark W to the highest id.
def fold(volume_id, [_ | _] = slice, db) when is_binary(volume_id) do
  db = db || store_for(volume_id)
  :ok = assert_evt_slice!(slice)   # every id must be a well-formed EVT (one namespace)

  # PEEK the page-index base (read, do NOT advance) — the advance is
  # conditional-on-commit below, so a commit error leaves :arc_seq untouched.
  base_n = peek_seq(db)
  n = length(slice)

  {staged, last_branded} =
    slice
    |> Enum.with_index(base_n)
    |> Enum.reduce({%{}, nil}, fn {{branded, fields}, idx}, {pages, _prev} ->
      page = :erlang.term_to_binary({branded, fields})
      {Map.put(pages, @archive_base + idx, page), branded}
    end)

  {:ok, base} = VolumeServer.begin(volume_id)

  case VolumeServer.commit(volume_id, base, staged) do
    {:ok, _lsn} ->
      # the pages are durable: NOW advance :arc_seq and the frontier W together
      commit_seq(db, base_n, n)
      advance_frontier(db, last_branded)
      {:ok, last_branded}

    {:error, _} = err ->
      err   # on a fold error the caller does NOT trim — the safe direction
  end
end

One page per record, at a range disjoint from business pages by construction and ordered by the branded id — the order theorem reaching disk, so the archive needs no second index to read its history in order.

The watermark W — a branded id, never `head_lsn`

W is the branded EVT id of the highest-folded record, persisted under :arc_frontier and read by archive_frontier/1. It is not the engine's integer head_lsn — that is the engine's page cursor (it addresses pages), the wrong type to compare against a live-tail EVT id. W is the seam: everything with a branded id ≤ W is in the archive; everything with a branded id > W is still on the live stream.

merge_read/5 reads archived ∪ live-tail, split on W: records with branded id ≤ W come from the engine's @archive_base range; records with branded id > W come from the live stream via EchoMQ.Stream.read/6 (the XRANGE lower bound is W's xadd id, exclusive). The union is in mint order — the archive is oldest-first, the live tail follows. When W is :empty (nothing folded), the whole stream is read live; there is no archive seam.

# echo_store — EchoStore.StreamArchive
# THE MERGE-READ: archived ∪ live-tail for a stream, split on W. id ≤ W from the
# @archive_base range; id > W from the live stream. No-gap/no-overlap is a
# CONSEQUENCE of fold-before-trim + the order theorem, never a per-read check.
def merge_read(conn, queue, name, volume_id, db) do
  case archive_frontier(db) do
    :empty ->
      # nothing folded: the whole stream is the live tail
      MqStream.read(conn, queue, name)

    {:ok, w} ->
      {:ok, archived} = read_archive(volume_id, db)
      from = "(" <> w_bound(w)   # strictly above W (the `(` exclusive XRANGE prefix)

      case MqStream.read(conn, queue, name, from, "+") do
        {:ok, live} -> {:ok, archived ++ live}   # archive oldest-first, live tail follows
        {:error, _} = err -> err
      end
  end
end

The watermark is also cached on the wire by EchoMQ.Stream.put_archived/4 / get_archived/3 under emq:{q}:stream:<name>:archived — so a non-BEAM reader finds where the archive ends and the live tail begins without a store call. But it is a polyglot cache of the seam, never the source of truth: archive_frontier/1 is the truth, and the wire seam is overwritten on each fold.

Fold-before-trim — the no-loss ordering

The whole guarantee is an order of two calls. Because XTRIM returns only a count, the no-loss invariant has exactly one defense: a record leaves the live stream only after it is committed to the engine and W has advanced past it. The cycle that bounds an archived stream lives in one process, EchoStore.StreamArchive.Driver, which owns both calls so the ordering cannot be violated. On each beat, per archived stream:

The fold-then-trim cycle · step it, then force a fault

Fold first, trim second — in one process that owns both calls. If the fold fails, the cycle aborts before the trim: the slice stays on the live stream, over-retention but never loss.

This is why an archived stream is not also driven by the bare EchoMQ.StreamRetention trim-only driver: that one would trim before a fold and lose the slice. For an archived stream this driver is the retention path, trimming to the same declared windows after folding — so bounded memory and deep history coexist with no loss. And no-gap/no-overlap is a consequence of fold-before-trim plus the order theorem, never a per-read boundary check: the fold captures the slice in order before the trim removes it, so the archive ends exactly where the live tail begins.

Pattern & implementation

the pattern (Redis Patterns Applied)

A retained log needs deep history that outlives the resident window; archive the trimmed segments to durable storage and read them beside the live tail. Streams & Events teaches event sourcing over a retained log.

the implementation (echo_store)

EchoStore.StreamArchive.fold/3 folds the trimmed slice into the Graft engine at a reserved 2^49 range, branded-id-monotone, via the public VolumeServer.commit/3; the watermark W (a branded id, never head_lsn) splits the merge_read/5; and fold-before-trim is the no-loss ordering.

The pattern says: archive what you trim. The implementation makes the archive an ordered extension of the log itself — the same branded id, the same mint order, on the durable floor.

Recap — trimmed, but kept

What is trimmed is folded first. The fold lands one page per record at a reserved 2^49 range — disjoint from business pages by construction, and branded-id-monotone, the order theorem on disk. The watermark W (a branded EVT id, never the engine's head_lsn) splits the merge-read: archive below, live tail above, one mint-ordered stream. Fold-before-trim is the no-loss ordering, and no-gap/no-overlap follows from it. The deep history is on the floor; the next dive is the floor itself, and the dial that turns it.

Next in the module: The door to persistence — the durability dial in depth, and the door to Echo Persistence.

References

Where the fold, the watermark, and the merge-read are grounded, and the durable engine they write into.

Sources

orbitinghail — Graft — the lazy, partial, page-replication engine echo_store builds natively on the BEAM, that the archive folds into.
Ong — CubDB — the append-only immutable B-tree the fold commits the archive pages onto.
Valkey — XTRIM — the count-only trim that makes fold-before-trim the sole defense against loss.
Kreps — The Log — the log whose offset/cursor the watermark W plays for the archive seam.

Related in this course

Retention & the archive — the module this dive belongs to.
Retention is a policy — the trim the fold runs before.
The door to persistence — the durable floor in depth.
The order theorem — the mint order this fold carries to disk.
Echo Persistence — the durable floor the fold writes into.
The Branded Component System · the persistence floor — the manuscript chapter this fold realizes.