Module 14 · Chapter IV — The platform
Why it beats classical scheduling.
The classical durable queue — Oban on PostgreSQL, cron on a table — is strict and well-understood, and it pays for that in two coins: it puts the database on the hot path of every dequeue, heartbeat, and ack, and it is single-node until you bolt on streaming replication. This module shows the one mechanism that explains the whole durability spectrum, and how EchoMQ’s commit-log-as-outbox reaches batch-class throughput while being the only tier that is both strict and replicated. Tap a queue below to place it.
K (Champ), or infinitely many (Memory, no durability). The commit-log-as-outbox rides the engine’s batch and is the only point that is strict and replicated — the dashed quadrant the others miss.Absolute job/s figures are indicative and single-machine; the teaching here is the mechanism — records amortized per fsync — not any one number.
What classical scheduling costs
graft.design.md · durability.exOban is the reference point because it is genuinely good: strict per-commit durability, mature tooling, a queue you can trust. Its bill comes in two parts. First, the database is on the hot path — every enqueue, every dequeue, every heartbeat, every ack is a PostgreSQL round-trip, so the store’s latency and availability gate the queue’s. Second, that durability is single-node: a committed job survives a crash of the worker, but not the loss of the database, unless you bolt on Postgres streaming replication and accept its operational weight. Underneath both sits the one mechanism this whole module turns on: durable throughput is set by how many records ride a single fsync. Oban’s single-insert path commits once per job — one record per fsync — and its insert_all path writes a transaction’s worth at once, closing most of the gap by spreading one fsync across the batch. Memory rides infinitely many records per fsync because it never fsyncs at all; Champ rides K because its checkpoint amortizes the fsync over K records. The spectrum is not four philosophies; it is one dial set to four positions. The platform’s job is to choose where each queue sits — and to make the strict-and-replicated end affordable, which is exactly the corner Oban cannot reach without leaving the single-node world.
The answer: the commit-log-as-outbox
graft.engine-split.design.md · ADR-AThe platform’s answer keeps the bus on Valkey — fast, reliable, volatile, off the durable path — and makes the durable record the engine’s own commit. This is the commit-log-as-outbox: built today as EchoStore.Durability.Graft, where an enqueue intent is simply a page commit in a reserved high LSN range (@obx_base = 1 <<< 48), and recovery replays that range above a watermark. Because the intent is a commit, it is atomic and durable in one act — there is no second system to disagree with, so the dual-write problem of the last module never arises. And because the engine is the Chapter III Graft engine, that single commit inherits everything that engine already proves: the OCC base-LSN check, the conditional-write fence, page-rollup replication to Tigris, and log-head recovery. So the enqueue is strict and replicated — the corner Oban misses — while riding the engine’s batch for throughput, and the bus stays fast because none of this is on the enqueue hot path. The same idea is where the platform is headed in the large (ADR-A: the commit-log-as-outbox as the journal, off SQL entirely). The balanced reading is the one the platform settled on: mitigate the single-instance database by making durability a replicated engine commit, and keep a reliable Valkey bus for speed. That is why it beats classical scheduling — not by being less strict, but by being strict in a place that also replicates and never touches the hot path.
The three dives
module 14The database on the hot path
The classical model traced: every dequeue, heartbeat, and ack a round-trip to PostgreSQL, strict but single-node, with throughput pinned to one record per fsync on the single-insert path.
Dive 13.2The commit log is the outbox
The robust solution: an enqueue intent is a page commit at @obx_base = 1 <<< 48, atomic and durable in one act, drained by the Committer, replayed above a watermark — the engine becomes the bus’s outbox.
The balanced decision
The verdict on two axes — durability and hot-path cost. Why the commit-log-as-outbox lands alone in the strict-and-replicated-and-off-the-hot-path quadrant the classical queue can’t reach.
Up next
chapter IVReferences & sources
- Echo records
- graft.design.mdthe measured durability spectrum (Memory / Champ / Oban / BullMQ); records-per-fsync as the single mechanism
- graft.engine-split.design.mdEchoStore.Durability.Graft — the outbox IS the commit log; @obx_base; the Committer drain
- echo_mq-v4-durability-adr.mdADR-A — commit-log-as-outbox: atomic and durable in one act, off SQL
- graft.roadmap.mdthe missing quadrant; Oban single-node unless replication is bolted on; the shootout
- External
- Obanthe classical Postgres-backed job queue
- Transaction logthe commit log the outbox reuses