jonnify·persistence /echo-persistence/platform/beats-classical-scheduling/the-database-on-the-hot-path

Module 14 · Dive 13.1

The database on the hot path.

A classical durable queue earns its reliability by routing the whole job lifecycle through the database. Enqueue is an insert; claiming a job is an update; the heartbeat that keeps a long job’s lease alive is an update; the ack that finishes it is an update or delete. Every one is a round-trip to PostgreSQL, and every commit is an fsync. That is what makes it strict — and it is also the bill: the store’s latency gates the queue, the store’s availability gates the queue, and the store is single-node. Run a job through it below.

Four steps, four round-trips. The worker cannot enqueue, claim, heartbeat, or ack without touching PostgreSQL — the database is on the hot path of the entire lifecycle, and each durable step is an fsync. Toggle the fsync mode to see the one dial that sets throughput.
worker (Oban) claims, runs, acks PostgreSQL fsync per commit single-node durability on the hot path 1 · enqueue (INSERT + fsync) 2 · dequeue (UPDATE claim) 3 · heartbeat (UPDATE lease) 4 · ack (UPDATE/DELETE + fsync) DB round-trips: 0 · fsyncs: 0
readout · every step of the lifecycle is a round-trip to the database, and the durable steps are fsyncs. The store gates the queue’s latency and availability, and it is single-node. run job lifecycle to trace it.

Absolute job/s figures are indicative and single-machine; the teaching is the mechanism — the database on the hot path, and records per fsync.

§1

Everything is a round-trip

durability.ex

The defining property of the classical durable queue is not that it stores jobs — every queue does — but where it stores the queue’s control state. In Oban, the jobs table holds the work and its lifecycle: a job is enqueued by an insert, claimed by an update under a lock, kept alive by periodic lease heartbeats, and completed by an update or delete. Each of those is a transaction against PostgreSQL, so the database sits on the hot path of every step, not just the durable write. That has two consequences worth stating plainly. The queue’s latency is the database’s latency — a slow commit slows claiming and acking, not only enqueue — and the queue’s availability is the database’s availability, because a worker that cannot reach Postgres cannot even mark progress on a job it already holds. And the durability, real as it is, is single-node: the committed jobs table survives a worker crash, but the loss of that one database is the loss of the queue, unless you run Postgres streaming replication and take on its failover complexity. None of this is a flaw in Oban; it is the direct cost of putting the control plane in the durable store. The platform’s move is to ask whether the durable write has to be on that path at all.

§2

The throughput dial: records per fsync

graft.design.md

Strip the schema away and a durable queue’s throughput reduces to one number: how many durable records ride a single fsync. The classical single-insert path commits once per job, so it rides one record per fsync — the strictest and the slowest end. The insert_all path writes a transaction’s worth of jobs in one commit, riding a whole batch per fsync, and that single change closes most of the throughput gap — the same durability, far more jobs per second, paid for by holding a batch. This is the dial the hub’s spectrum turns: Memory rides infinitely many records per fsync because it never syncs, Champ rides K because its checkpoint amortizes the fsync over K records, and Oban rides one or a transaction depending on the path. The insight that matters for the next dive is that batching is not unique to any one system — anything that amortizes the fsync over more records moves right on this axis. What batching alone never adds is replication: an insert_all is still single-node. So the open corner, the one the classical queue cannot reach by tuning its batch size, is strict durability that is also replicated and also off the hot path. That corner is the subject of the next dive.

§3

References & sources