jonnify·persistence /echo-persistence/native-elixir/the-commit-log-outbox

Module 7 · Dive 7.3

The commit-log outbox.

The commit log earns its keep twice. Each entry is durable locally the instant it’s written — that’s recovery — and it is also a pending item in an outbox: the Committer drains the log to Tigris, uploading each segment with a conditional PUT so two committers can never double-write, then marks it shipped. One structure is both the durability log and the replication source. Append commits and drain them.

§1

Durable now, shipped soon

interactive
The commit log, draining. Every entry carries two badges — durable (always, once written) and its upload state (pending → shipped). The Committer drains the oldest pending entry to Tigris behind the fence. Append, then ship.
commit log = outbox durable locally · pending upload shipped through LSN 2 · pending: 2 Committer drain oldest idle Tigris shipped segments read pending PUT (fence)
readout · Append to add an entry — durable at once, pending upload. Ship to drain the oldest pending one: a conditional PUT to Tigris, then shipped. Turn on contention to watch the fence reject a duplicate write.
§2

One log, two jobs

graft.engine-split.design.md

Most systems keep a write-ahead log for crashes and a separate queue for replication, then fight to keep them consistent. The native engine refuses the duplication: the commit log is the outbox. A commit is durable the moment CubDB has it, so recovery (Module 6) needs nothing else; and the very same entries are the work list the Committer drains to object storage. Shipping is decoupled and idempotent — the Committer uploads a segment with a conditional PUT (create-if-not-exists, the fence from Module 3), so if another committer already wrote that segment the PUT fails the precondition and no second copy is created; the entry is marked shipped either way. A high-water mark tracks the highest contiguous shipped LSN, which is exactly what a replica needs to know where to start. Durability, replication, and the change feed all fall out of one append-only log — which is why this engine is the platform’s default. Chapter III continues with the Rust engine, which makes different bets under the same contracts.

§3

References & sources