jonnifyelixir
F7 · Deploy to Fly / F7.3 · the database / F7.3.3 · the boot-time DB coupling

F7.3 · deep dive 3 · the boot-time DB coupling

F7.3.3

The boot reads the log

Portal.Engine.init/1 folds the event log synchronously at startup — init(:ok), do: {:ok, load()}, and load/0 reaches Portal.EventStore.Postgres.read_stream/1, a real SELECT … FROM events under the connection pool’s 4‑second queue_timeout. So the boot makes one hard demand of the database before it can return: an unreachable one times out and init/1 raises — failed to start child Portal.Engine, exit 1, and Fly suspends the machine after ten restarts; a reachable‑but‑empty one boots straight into relation "events" does not exist. This dive observes that coupling and names its fix; the engine is left untouched.

We read echo/apps/portal/lib/portal/engine.ex and event_store/postgres.ex as built — the synchronous call chain (init/1 → load/0 → read_stream/1), the two crash modes (unreachable vs reachable‑but‑empty), the stacked precondition (reachable and migrated), and the carried hardening that touches Portal.Engine below the facade — named here, not built in F7.3.

The synchronous read

Portal.Engine is a single process that holds the Portal’s folded state in memory, and it rebuilds that state on every start by replaying the event log from the beginning. The replay is the first thing the process does, and it does it synchronously, before init/1 returns. The path is short and exact: init(:ok), do: {:ok, load()} calls load/0, which reads the whole stream through Portal.EventStore.adapter().read_stream(@stream) — the adapter/0 indirection that resolves to whichever store the release ships with. Under prod that store is Portal.EventStore.Postgres, so the read is a real query.

Portal.EventStore.Postgres.read_stream/1 answers with a single Repo.all(from e in Event, where: e.stream == ^stream, order_by: e.seq) — a SELECT … FROM events ordered by :seq, run through Portal.Repo over the connection pool. Because init/1 blocks on the result, the pool’s queue_timeout — the 4‑second window a checkout waits for a free connection — is the clock the whole boot runs against. If a connection cannot be had in that window, the read does not return; it raises, and a raise in init/1 means the process never starts.

Step the chain below to read what each call does, and to confirm that every link is synchronous — the boot is waiting on the SELECT, under the 4‑second queue_timeout, the entire time.

The synchronous boot read · step by step

1 · STARTUP Portal.Engine.init/1 {:ok, load()} 2 · REPLAY load/0 adapter().read_stream 3 · QUERY read_stream/1 SELECT … FROM events synchronous · init/1 blocks until the SELECT returns · 4s queue_timeout

The whole chain is synchronous: init/1 does not return until read_stream/1’s SELECT answers — so the database is a hard dependency of the boot, not a thing the engine reaches for later.

Two crash modes, one read

Because the boot blocks on that one read, two different database faults end the same way — the process fails to start. An unreachable database (the F7.2 reachability faults, left unresolved — a name that does not resolve, a port that does not answer) means the checkout never completes inside the queue_timeout; the read times out, init/1 raises, the supervisor reports failed to start child Portal.Engine, and the node exits non‑zero. Fly restarts the machine, the same read fails again, and after ten such restarts the platform suspends the machine. This is the floor under the gotcha‑2/3 crash stack: the same symptom family, reached through the same synchronous read.

A reachable‑but‑empty database — the trap this module names — fails one step later. The connection succeeds, so the queue_timeout never fires; but the SELECT … FROM events runs against a schema with no events table, and Postgrex surfaces relation "events" does not exist. The connection is fine; the table is not there. init/1 raises the same way, and the crash loop is the same.

So the precondition the boot demands is stacked, not single: the database must be reachable (the F7.2 half — a host that resolves and answers) and migrated (this module’s half — the schema applied release‑native, the F7.3.1 dive) before read_stream/1 can return a result. Either half missing, and the boot does not complete.

# echo/apps/portal/lib/portal/engine.ex — the synchronous boot read
def init(:ok), do: {:ok, load()}        # init/1 blocks on load() before it returns

defp load do
  {:ok, log} = Portal.EventStore.adapter().read_stream(@stream)
  # … fold the log back into state, synchronously …
end

# echo/apps/portal/lib/portal/event_store/postgres.ex — the real query
def read_stream(stream) when is_binary(stream) do
  {:ok, Repo.all(from e in Event, where: e.stream == ^stream, order_by: e.seq)}
end

# unreachable  → checkout exceeds the 4s queue_timeout → init/1 raises → exit 1 → suspend after 10
# reachable, empty → SELECT hits a missing table → relation "events" does not exist → init/1 raises

One synchronous read, two ways to fail: a timeout when the database is unreachable, a missing‑relation error when it is reachable but empty — so the boot demands both halves, reachable and migrated, at once.

The carried hardening — named, not built

The obvious fix is to stop coupling the boot to the database. A lazy or retried replay would let init/1 return immediately and fold the log a moment later — or retry the read a few times before giving up — so a momentary database blip during a deploy would no longer take the whole release down. That is a real and reasonable hardening, and it is the right shape. It is also not built in F7.3.

The reason is a boundary, not an oversight. A lazy or retried replay changes how Portal.Engine starts and folds its state — it edits the engine, which sits below the facade, beneath the deploy layer’s mandate. The deploy chapter’s rule is to ship the release and document what it depends on, not to reshape the domain underneath it; the engine is left untouched (F6.8.2-INV7). So this dive does what the deploy layer is allowed to do: it observes the coupling, names the fix, and stops there. No engine code changes in F7.3.

Until the hardening is built, the coupling stands, and so does the operational rule it implies: bring the database up and migrate it before the release boots. The interactive below contrasts the two behaviours — the synchronous coupling as built, and the lazy/retried replay that is carried — and badges the second clearly as not built.

As built vs the carried hardening

AS BUILT · SYNCHRONOUS REPLAY init/1 blocks on read_stream/1 DB unreachable or empty init raises exit 1 the current coupling · suspend after 10 restarts CARRIED — NOT BUILT

The hardening is named, not built: a lazy or retried replay would survive a momentary blip, but it touches Portal.Engine below the facade, so it stays outside F7.3’s mandate — the engine is unchanged, and the rule stands: migrate before the boot.

observed in F7.3

the boot’s synchronous coupling to the database — init/1 folds the log through read_stream/1, so an unreachable or unmigrated database crashes the release. Documented, with the engine left untouched.

carried, not built

a lazy or retried replay that lets init/1 return and re‑fold later — a change to Portal.Engine, below the facade and beyond the deploy layer’s mandate (F6.8.2-INV7). Named here; built elsewhere.

That closes F7.3: the schema is applied release‑native, with no mix on the artifact (F7.3.1), the empty‑database trap is sprung against the writable primary through the fly proxy path (F7.3.2), and the boot’s synchronous coupling is named with its hardening carried (here). The reachable half of the precondition was settled in F7.2 · runtime config & the IPv6 network; together they are the reachable and migrated the boot demands. Back to the module hub for the next module of F7.

References

Why a raising init/1 stops a process from starting, the supervisor restart behaviour behind the crash loop, the connection pool’s queue_timeout that bounds the synchronous read, and the driver that surfaces the missing‑relation and connection errors.

Sources

Related in this course