jonnify·bcs /bcs/persistence/the-remote

B5.3 · dive 3 of 3

The portable remote.

The Streamer ships segments to Tigris in real time; Sync sends commit notices over the bus; the commit-LSN is the cursor; and the stream archive folds trimmed streams into the same floor.

A real-time streamer to Tigris, a cursor on the log, an archive of trimmed streams. Select a part.
Graft.Streamer → Tigrissegments + conditional commit objectsthe native replacement for LitestreamTigris — object storageconditional: no overwrite Graft.Sync — commit noticesover the bus; bytes via Tigriscommit-LSN = the cursora replica reads from the offsetrust echo_graft: Fjall + OpenDAL StreamArchive — trimmed streams fold into the floora reserved high page range, read beside the live tail on a watermark
readout · Select a part. The Streamer ships segments to Tigris; commit notices ride the bus while bytes go via Tigris; the commit-LSN is the cursor; the stream archive folds trimmed streams into the floor.
§1

The idea

echo_store · graft/streamer.ex, graft/sync.ex, graft/remote/tigris.ex, stream_archive.ex

The durable copy lives in object storage, and a streamer keeps it current. EchoStore.Graft.Streamer ships segments and conditional commit objects to Tigris in real time — the native replacement for Litestream, which streamed SQLite’s write-ahead pages to S3 the same way. Object storage becomes a transactional remote because the commit object is written conditionally: two writers that share a destination cannot overwrite each other, so the remote is safe without a coordinator.

The bus and the bytes travel separately. EchoStore.Graft.Sync sends a low-latency commit notice over EchoMQ.Connector — a small message that a new commit exists — while the pages themselves move through Tigris. A replica reads the notice, learns the new commit-LSN, and pulls the pages it needs; the commit-LSN is the cursor, the offset on the log that tells a reader how far it has caught up. The same idea ports: the Rust echo_graft pairs Fjall with OpenDAL, a different engine and a portable remote, for hosts that do not run the BEAM.

Deep history lands on the same floor. When EchoMQ trims a stream, EchoStore.StreamArchive folds the trimmed segments into this engine’s CubDB at a reserved high page range, far above any business page, and on to Tigris — deep history without resident memory. A reader merges the archived pages with the live tail on an engine-derived watermark, so a stream’s past is queryable beside its present, and what a system trims is kept, not lost.

§4

The doors

the sibling courses
/echomq

EchoMQ — the protocol, in depth

The bus B3 narrates, taught rung by rung: the keyspace, the Lua inventory, and the 3.0 Stream Tier — XADD, XREADGROUP, XACK — with conformance on Valkey.

 /redis-patterns

Redis Patterns Applied

The substrate patterns under the bus: sorted sets, atomic Lua moves, locks, and streams — the judgement layer.

 /echo-persistence

Echo Persistence — the durable floor

Beneath the volatile tiers: the Graft page engine built twice, the commit-LSN loop, and the stream archive on Tigris. Where Part I says state lives somewhere, this says where it lives durably.
§2

References

sources · related

Sources

Related