Module 12 · Dive 11.3 · closes Chapter IV’s bus arc
The Stream Tier ladder.
The Stream Tier is not a rewrite; it is six small rungs that add a log tier beside the job tier, each shipping under the same wire. The reason it is cheap to build is that the two hardest pieces already exist from Chapter IV: the sequence is already minted (a branded id’s byte order is its mint order) and latest-per-key is already law (the staleness fence plus newer-wins). What is left is mostly naming where order holds. The climb ends somewhere satisfying for this course: the archive folds straight into the Graft engine you spent Chapter III building, and a read can span memory and object storage as one log. See that merge below.
W it is a durable segment (Tigris, via EchoStore.Graft); at or above W it is in the live stream. A deep read that spans W stitches the segment portion to the tail portion in mint order — no gap, no overlap, because fold-before-trim plus the order theorem guarantee the seam is exactly W.W are segments; ids at/above W are the live tail. fold + trim advances W, moving one entry down to durable storage. deep read across W shows the merge: segments below W ++ tail at/above W, in mint order.Six rungs, one wire
emq.streams.md · streams.synthesis.mdThe verbs register as an additive minor on EchoMQ.Connector — the same connector that carries jobs — so the stream tier landed without a wire break, a job-only node and a stream-aware node on the same bus. The bus now speaks echomq:3.0.0: a major version is a coordinated cutover, taken once the tier was whole, because a shared fence-climb would brick co-tenants. The order theorem underwrites everything: a stream’s order is the branded-id sort, which is the mint order, with one named exception — a re-claimed pending entry returns out of real-time order, so the spec says exactly where order holds (the stream) and where it does not (re-claim).
- emq3.1The verbs —
XADD/XRANGE/XREADGROUP/XACK/XAUTOCLAIMregistered as a no-break minor. - emq3.2The writer law — hash-tagged streams, branded ids,
append == mintso byte order is time order. - emq3.3Readers — consumer groups + a polyglot handler; at-least-once, idempotent. Build-ready.
- emq3.4Retention as policy —
MAXLEN ~/MINIDby mint instant; open fork F3.4-A on cadence (Dive 11.2). - emq3.5The archive — a fold consumer commits trimmed slices to native
EchoStore.Graft, fold-before-trim; the merge-read below. - emq3.6Time travel — a mint-instant becomes a half-open
XRANGE [dt1,dt2)viaSnowflake.min_for/1; a Table hydrates from the tail as a changelog read, no compactor.
Two committed mechanics make the climb cheap rather than novel. The sequence is already minted: because every record carries a branded Snowflake whose bytes sort in creation order, the stream needs no separate sequence and “keep the last hour” is a key comparison, not a scan. And latest-per-key is already law: the staleness fence and newer-wins from Chapter IV mean a changelog-style read already resolves to the current value without a background compactor. The three delivery milestones — S1 the writer, S2 the readers, S3 the memory tier — are just the order in which these land.
One log, two homes
emq.streams.md · emq3.5The archive rung is where the bus and the engine you built finally meet. A dedicated fold StreamConsumer commits trimmed slices to the native EchoStore.Graft store via commit/3, and the engine streams those pages from CubDB out to Tigris — the same durable floor Chapter III’s engines lean on, which is why this supersedes the older external-replication idea (Litestream / the retired Shadow store) entirely. The merge-read is the payoff. Define the watermark W as the branded id of Graft’s folded frontier: everything below W has been folded to segments, everything at or above W is still in the live stream. A deep read of a range is then segments below W concatenated with the live tail at or above W — and there is provably no gap and no overlap at the seam, because fold-before-trim guarantees nothing is trimmed until it is folded (so nothing falls between the two homes) and the order theorem guarantees both halves sort by the same mint order (so the seam is exactly W). emq3.6 builds straight on this: a time window is a pair of mint instants, so a historical read is a half-open XRANGE over the merged log, and a Table rebuilt from the tail is just a changelog read resolved by the latest-per-key law — no compaction step anywhere. The course closes where it began: durability is one ordered log, and the only question is whether a given entry currently lives in memory or in object storage.
References & sources
- Echo records
- emq.streams.mdthe emq3.1–3.6 ladder, the writer law, the merge-read, time travel
- streams.synthesis.mdthe order theorem, the two committed mechanics, the milestones
- graft specs / graft.4.mdEchoStore.Graft — the archive the fold commits into
- External
- Redis StreamsXRANGE, the log model
- Snowflake IDtime-ordered ids, the mint-instant bound