jonnify·persistence /echo-persistence/engines/rust

Module 8 · Chapter III — The engines

The Rust engine.

The platform’s second engine is echo_graft: a from-scratch fork seeded from upstream Graft (its SQLite layer stripped), written in Rust on Fjall and Apache OpenDAL, and run as a supervised sidecar the BEAM drives over EchoMQ. By the Operator’s COEXIST ruling it is not a rewrite of the native engine but a peer — it implements the same contract Module 3 defined (Volume, LSN, OCC commit, snapshot, the fence, the change feed) on a different substrate, reached over the bus rather than linked in-process. Tap a contract row to see how each engine satisfies it; the contract is identical, the internals diverge.

One contract, two engines. The spine is the EchoStore contract. Left is the native Elixir engine, right is Rust. Tap a row to contrast the two implementations of the same guarantee.
native · Elixir CubDB · BEAM tap a row → Rust Fjall · OpenDAL ← tap a row EchoStore contract
readout · Tap a contract row. Every guarantee is satisfied by both engines — the same Volume model, the same fence protocol, the same feed contract — but the substrate differs: a copy-on-write B-tree on the BEAM versus an LSM-tree in Rust.
§1

Why a second engine — and why a sidecar

graft.roadmap.md

Two engines exist because the contract from Module 3 is substrate-neutral: nothing in “Volume addressed by LSN, commit under OCC, fence with create-if-not-exists, feed the ordered changes” demands the BEAM. So the platform implements it twice. The native engine optimizes for tight Elixir integration — in-process ETS caches, GenServer serialization, no boundary crossing. The Rust engine, echo_graft, is seeded from the upstream Graft Volume runtime (MIT/Apache-2.0, a read-only idea source — no upstream compatibility is kept) with the SQLite extension (libgraft_ext) removed and three seams re-cut for this platform: a Tigris remote, branded-Snowflake identity at the edge, and an EchoMQ change-feed off the commit LSN. Crucially it is not linked into the BEAM during the spine: it runs as a supervised Rust sidecar (echo_graft_backend) the BEAM orchestrates over EchoMQ (RESP3) through a versioned wire (echo_graft_proto), with EchoStore.GraftBackend as the Elixir client — the bus is the contract, and the commit LSN is the synchronization cursor every advance publishes on the change-feed lane. Why fork rather than rebuild? Champ is bounded-loss and Oban is single-node unless Postgres replication is bolted on; Graft already solves the missing transactional-and-replicated quadrant, so reusing its core buys the consistency model without a multi-month build or the rejected SQLite/C-binding path. The work is sequenced eg.1 → eg.6: the core fork, the Tigris remote + fence (eg.2), branded-id identity + the EchoMQ feed (eg.3), the BEAM↔Rust backend + protocol (eg.4, Module 10), a low-latency group-commit tier (eg.5), and ship + the durability shootout beside Champ and Oban (eg.6). The three dives take the substrate apart — the LSM-tree, OpenDAL, and the in-memory feed — always against the same contract.

§2

The three dives

module 8
Dive 8.1

Fjall & the LSM-tree

The Rust engine’s local store. Writes land in a memtable, flush to immutable SSTables, and compact by merging — a different shape from CubDB’s B-tree, same append-only promise.

 Dive 8.2

OpenDAL & the portable remote

One operator, many backends. The engine writes segments through OpenDAL, so Tigris, S3, or local disk are a config change — and the fence is one if_not_exists call.

 Dive 8.3

The in-memory feed

The change feed served from a ring buffer broadcast to subscribers — with a fallback to segments for any consumer that falls off the window. Fast, with a safety net.
§3

Build & check

What you build

Take the five contract rows and write, for each, the native approach and the Rust approach in one line apiece. If the two columns differ but the row’s guarantee is identical, you have understood the split.

Check

Answer without code: name one thing that is identical across both engines and one thing that differs. “The fence protocol is identical; the local store structure differs (B-tree vs LSM)” means you have the module.

§4

References & sources