jonnify·persistence /echo-persistence/platform/echomq-bus/retention-and-the-never-deleted-problem

Module 12 · Dive 11.2

Retention & the never-deleted problem.

A stream is a log, and XACK does not delete — it only clears an entry from a group’s pending list. So a busy stream grows forever unless something trims it, and trimming a log is exactly where you can lose data you meant to keep. EchoMQ makes retention a declared policy, not a side effect, and it has one open design question worth seeing in motion: when do you trim? Below, grow a stream, fold its tail to the archive, and watch a naive timer-driven trim outrun the fold and drop an un-archived entry.

Two frontiers on one log. The fold watermark is how far the archive consumer has copied entries to durable segments; the trim watermark is how far the stream has been cut. In fold-before-trim, trim cannot pass fold — only archived entries are removed. In trim-by-timer, trim advances on its own and can cut an un-folded entry: lost.
fold W trim W trimmed (cut) folded → segments (safe to trim) live tail (not yet archived)
cadence:
readout · fold copies the next live entry to durable segments; trim cuts from the left. In fold-before-trim the trim watermark cannot pass the fold watermark, so nothing un-archived is ever cut. Switch to trim by timer and trim past the fold to see the failure.
§1

Ack is not delete

emq.streams.md · emq3.4

The single fact behind this whole rung is that acknowledging an entry does not remove it. XACK clears the entry from one group’s pending list; the entry itself stays in the stream, visible to XRANGE and to any other group, forever. That is the right default — a log you can replay is the entire point of the tier — but it means a stream that is only ever appended to is a stream that only ever grows. The control is XTRIM, and EchoMQ treats it as retention-as-policy rather than an ad-hoc call: a stream declares its window once, and the two levers are MAXLEN ~ N (keep approximately the last N entries; the ~ lets the server trim on macro-node boundaries, which is far cheaper than exact) and MINID m (drop everything below a minimum id). Because a branded id’s sort order is mint order, MINID is also a time bound: Snowflake.min_for/1 turns an instant into the smallest id at-or-after it, so “keep the last hour” is a MINID trim, no scan required. Approximate MAXLEN is the default and exact trimming is opt-in, because for retention “roughly the last N” is what you actually want and exactness costs you the cheap boundary. None of this changes a reader inside the window: a deep read of entries that still exist is exact; the window only governs what no longer does.

§2

The open question: when do you trim?

streams.synthesis.md · fork F3.4-A

Trimming is safe only relative to what has been preserved elsewhere. The next rung (emq3.5) archives the trimmed tail into durable segments via the native EchoStore.Graft engine — the same engine Chapter III built — and the rule there is fold-before-trim: a dedicated fold consumer commits a slice to the archive, and only then may the stream cut up to the folded frontier. Couple the two and a trim can never remove an un-archived entry, because the trim watermark is bounded by the fold watermark by construction; that is the safe path the figure shows. The genuinely open question — recorded as fork F3.4-A — is the cadence of the general-purpose trim that is not the archive fold. One option drives trimming on the consumer beat and ties the trim watermark to the fold watermark, so safety is automatic but liveness and safety are now one knob. The other gives trimming a separate, named cadence and keeps safety as an explicit precondition, on the principle that you should not couple a liveness concern (how often we trim) to a safety concern (what we are allowed to trim). The Director’s reconciliation lands between them: ship the trim verb with its policy converged, make a general cadence opt-in and named rather than implicit, and let the emq3.5 fold consumer be the one place that does fold-before-trim. The naive failure the figure dramatizes — a timer that advances the cut past the fold — is precisely what naming the cadence is meant to prevent.

§3

References & sources