jonnify·bcs /bcs/cache/the-lane-that-remembers/two-memories-one-file

B4.4 · dive 1 — two memories, one file

Two memories in one file: the outbox and the last word.

One SQLite file per group, owned by one process, standing beside the bus. Inside it, two tables: intents is the outbox at the writer's edge, applied is the lane's last word per name at the applier's edge. The literature's answer to the dual-write trap is Richardson's transactional outbox — record the intent in local durable storage first, relay it to the broker after, and accept that the relay may deliver twice, so consumers "must be idempotent". This series is unusually well-armed for that acceptance, and the law's wording is exact: "the bus gets no WAL; the journal, which is not the bus, gets SQLite's."

Source: content/bcs4.4.md, quoting bcs_rung_4_4_check.out. This dive reads the opening of the record — the self-check, the SQLite header, H1, the windows derive, and H2.

The schema, on one sheet. One file per group, two tables inside it. Select a part of the surface to read what the manuscript and the H1 gate commit about it.
journal-<group>.db · named by the group's branded id intents · the outbox seq job id name version enq? applied · the last word per name name version a non-branded group refused at the door two groups two files on disk
readout · Select a part. The whole surface is the H1 gate's subject: one file, two tables, a refusal at the door — the identity contract holding at the journal the way it holds everywhere else.
§1

The transcript

bcs_rung_4_4_check.out · verbatim · this dive reads the opening: the header, H1, the windows derive, H2
content/echo_data/runtimes/elixir/bcs_rung_4_4_check.out · verbatim · the opening; the hub holds the record wholefrozen
02:15:17.826 [info] EchoData: contract self-check passed, codec=pure
header: Valkey on 6390 | SQLite 3.46.0 via exqlite, WAL, synchronous=NORMAL | Elixir 1.14.0 OTP 25 | schedulers 1
H1 files ok -- one journal file per group, named by the group's branded id -- two groups, two files on disk -- a non-branded group is refused at the door, and the schema carries the two memories: intents (the outbox) and applied (the lane's last word per name)
derive (windows): the writer's flow is record, enqueue, mark -- two crash seams; a death before the enqueue leaves a pending intent that replay enqueues; a death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it; full coverage by the applied memory makes replay a no-op
H2 windows ok -- both crash seams closed by machinery that already exists: the never-enqueued intent replayed onto the bus, the enqueued-but-unmarked one answered :duplicate at admission and was counted, both ended in the applied memory -- and once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0}
… (the memory, the loss, the compaction, and the price follow -- the full record is frozen on the module hub)

The opening [info] line and the SQLite header are part of the record: this rung is the first cache rung to run under MIX_ENV=prod mix run, on the runtime's first compiled dependency — exqlite, vendored as path deps under runtimes/elixir/vendor/, the NIF compiled from the bundled SQLite amalgamation, no system SQLite consulted.

§2

Two memories in one file

content/bcs4.4.md · What · Where · How

The intents table is the outbox: seq, the recorded job id, the name, the version, an enqueued flag, nothing else — "the cargo law in a schema". The applied table is the lane's last word per name: the newest version this group has ever applied, surviving the table, the node, and the bus. The file per group is journal-<group>.db under the declared directory, named by the group's branded id, and a non-branded group is refused at the door — the same typed refusal every surface in this series makes before it does any work.

The writer's edge is one verb, and the applier's edge is one handler — everything from 4.2's crash choreography still applies beneath it:

{:ok, _} =
  EchoStore.Journal.start_link(
    name: :limits_journal,
    group: group_id,            # the lane's PRT, one file per group
    table: "limits",
    dir: "/var/lib/echo/journals"
  )

{:ok, job_id} = Journal.intend_and_enqueue(:limits_journal, conn, ast_id, version)
{:ok, _} =
  EchoMQ.Consumer.start_link(
    queue: Coherence.queue("limits"),
    connector: [port: 6390],
    handler: Journal.handler(:limits_journal, :limits)
  )

Writers get the outbox in one verb, intend_and_enqueue/4, with the two seams inside it drilled rather than assumed. Consumers swap one handler for another: Journal.handler/2 wires the lane through the memory. The journal itself is owned by one process — the single writer as a storage discipline, which is what makes prepared-once statements trivially safe and SQLite's single-writer nature a fit instead of a fight.

§3

The seams, closed by what already exists

content/bcs4.4.md · What · Decisions

The writer's flow is record, enqueue, mark — two seams. The rung drills both at once — a never-enqueued intent and an enqueued-but-unmarked one — then calls replay, and the gate closes on machinery this series already shipped: the bus's admission dedup from Part III and newer-wins from 4.2. Step the drill below.

The two crash seams, walked. Record, enqueue, mark — and the two deaths between them, each closed by replay. Step through; the readout quotes the record where it lands.
seam ① · a death here seam ② · a death here the intent lands under the recorded id replay re-enqueues both ended here record seq, job id, name, version enqueue the job rides the lane mark enqueued = true, in the outbox intents the outbox the bus admission dedup · :duplicate applied the last word per name
readout · Press step to walk the H2 drill — record, the first seam, the second seam, and the replay that closes both with machinery that already exists.

Static walk — 1) record: the intent lands in the outbox under a recorded job id · 2) seam ①: a death before the enqueue leaves a pending intent · 3) replay enqueues it — the never-enqueued intent replayed onto the bus · 4) enqueue: the job rides the lane under its recorded id · 5) seam ②: a death after the enqueue but before the mark leaves the bus holding the job · 6) replay reuses the recorded id and the bus answers :duplicate at admission, counted · 7) both ended in the applied memory; once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0}.

Two decisions hold the seams open on purpose, then close them by replay. mark_enqueued is a separate call: folding it into the enqueue would erase the seam the drill exists to cover — "the outbox pattern's candor is that the seams are real", named, and closed by replay rather than wished away. And replay reuses recorded job ids: minting fresh ids would defeat the bus's admission dedup and deliver duplicates the consumer must absorb anyway; reusing the recorded id lets the bus answer :duplicate for whatever survived, which the record counts in the open.

{:ok, %{replayed: n, deduplicated: m}} = Journal.replay(:limits_journal, conn)

Replay is Richardson's at-least-once relay, with the duplicate handling this series shipped two chapters ago — the admission dedup of B3.2 · Jobs Are Entities and the newer-wins comparison of B4.2 · Coherence by Mint Time. The next dive stages what the outbox is for: the bus dying with thirty obligations on it.

§4

References

sources · related

Sources

Related