Module 14 · Dive 13.1
The database on the hot path.
A classical durable queue earns its reliability by routing the whole job lifecycle through the database. Enqueue is an insert; claiming a job is an update; the heartbeat that keeps a long job’s lease alive is an update; the ack that finishes it is an update or delete. Every one is a round-trip to PostgreSQL, and every commit is an fsync. That is what makes it strict — and it is also the bill: the store’s latency gates the queue, the store’s availability gates the queue, and the store is single-node. Run a job through it below.
run job lifecycle to trace it.Absolute job/s figures are indicative and single-machine; the teaching is the mechanism — the database on the hot path, and records per fsync.
Everything is a round-trip
durability.exThe defining property of the classical durable queue is not that it stores jobs — every queue does — but where it stores the queue’s control state. In Oban, the jobs table holds the work and its lifecycle: a job is enqueued by an insert, claimed by an update under a lock, kept alive by periodic lease heartbeats, and completed by an update or delete. Each of those is a transaction against PostgreSQL, so the database sits on the hot path of every step, not just the durable write. That has two consequences worth stating plainly. The queue’s latency is the database’s latency — a slow commit slows claiming and acking, not only enqueue — and the queue’s availability is the database’s availability, because a worker that cannot reach Postgres cannot even mark progress on a job it already holds. And the durability, real as it is, is single-node: the committed jobs table survives a worker crash, but the loss of that one database is the loss of the queue, unless you run Postgres streaming replication and take on its failover complexity. None of this is a flaw in Oban; it is the direct cost of putting the control plane in the durable store. The platform’s move is to ask whether the durable write has to be on that path at all.
The throughput dial: records per fsync
graft.design.mdStrip the schema away and a durable queue’s throughput reduces to one number: how many durable records ride a single fsync. The classical single-insert path commits once per job, so it rides one record per fsync — the strictest and the slowest end. The insert_all path writes a transaction’s worth of jobs in one commit, riding a whole batch per fsync, and that single change closes most of the throughput gap — the same durability, far more jobs per second, paid for by holding a batch. This is the dial the hub’s spectrum turns: Memory rides infinitely many records per fsync because it never syncs, Champ rides K because its checkpoint amortizes the fsync over K records, and Oban rides one or a transaction depending on the path. The insight that matters for the next dive is that batching is not unique to any one system — anything that amortizes the fsync over more records moves right on this axis. What batching alone never adds is replication: an insert_all is still single-node. So the open corner, the one the classical queue cannot reach by tuning its batch size, is strict durability that is also replicated and also off the hot path. That corner is the subject of the next dive.
References & sources
- Echo records
- graft.design.mdthe measured spectrum; single-insert vs insert_all; records-per-fsync as the one dial
- graft.engine-split.design.mddurability.ex — the bus stays on Valkey, the hot path Oban puts every dequeue/heartbeat/ack through
- graft.roadmap.mdOban strict but single-node unless streaming replication is bolted on
- External
- Obanthe jobs table; the lifecycle
- PostgreSQL WAL reliabilityfsync on commit