Codemojex · C8.3 · a dive of Production Deployment

Correctness is structural.

The runtime is one supervision tree, strategy: :one_for_one, and the children list reads as the dependency order. Delivery off the bus is at-least-once, and every handler is idempotent, so a crash-and-restart re-runs the work without double-paying. Nothing here is a retry policy bolted on at the edge — correctness lives in the shape of the tree, the lease on each job, and the guard inside each handler.

Grounding — echo/…/codemojex/application.ex (the tree + the consumer specs) · tables.ex · rooms.ex · scoring.ex · wallet.ex · design §Fault tolerance and correctness.

The supervision tree, in start order — select a band

the cache tier → {Codemojex.Tables, conn} is itself a supervisor with strategy: :rest_for_one: EchoStore.Directory starts first, then the :cm_games and :cm_emojisets tables. A Directory restart cascades downward, so the tables restart with it and re-register — never a table pointing at a dead directory. And cache writes are best-effort: a Valkey blip or a table restart can never fail the writer recording a game or a set.

§1

At-least-once, leased

application.ex · the consumer specs

Four EchoMQ.Consumer children drain four queues, and each consumer leases the job it is working. A crash stops the heartbeat, the lease expires, and the in-flight job becomes visible again on the queue rather than lost — while the supervisor restarts the consumer in place. The specs carry the whole contract in two numbers each, verbatim from the tree:

echo/…/codemojex/application.ex · the consumer leases source

# --- consumer specs --------------------------------------------------------

defp score_opts(conn) do
  [
    queue: Codemojex.ScoreWorker.queue(),
    handler: &Codemojex.ScoreWorker.handle/1,
    connector: conn,
    beat_ms: 100,
    lease_ms: 10_000
  ]
end

defp settle_opts(conn) do
  [
    queue: Codemojex.Settle.queue(),
    handler: &Codemojex.Settle.handle/1,
    connector: conn,
    beat_ms: 100,
    lease_ms: 10_000
  ]
end

defp notify_opts(conn) do
  [
    queue: Codemojex.NotificationWorker.queue(),
    handler: &Codemojex.NotificationWorker.handle/1,
    connector: conn,
    beat_ms: 50,
    lease_ms: 15_000
  ]
end

defp command_opts(conn) do
  [
    queue: Codemojex.EchoBot.commands_queue(),
    handler: &Codemojex.CommandWorker.handle/1,
    connector: conn,
    beat_ms: 100,
    lease_ms: 10_000
  ]
end

beat_ms is the poll-and-heartbeat cadence; lease_ms is how long a claimed job stays invisible before the bus re-exposes it — measured on the server clock, never a worker’s. The queues are the real ones: Codemojex.ScoreWorker.queue() is "cm", Codemojex.Settle.queue() is "cm-settle", Codemojex.NotificationWorker.queue() is "cm.notify", and Codemojex.EchoBot.commands_queue() names the inbound-command lane. The notifier differs on both numbers: it polls at 50 ms because a chat message is latency-sensitive, and holds a 15-second lease because its handler leaves the process boundary for Telegram’s API and classifies the result — ack, defer under the rate budget, or backoff — before acking.

§2

Why re-delivery is harmless

scoring.ex · rooms.ex · wallet.ex

At-least-once is only safe when a second run costs nothing. Every handler holds that property, each by a different mechanism: scoring is pure; settlement is guarded by the one-shot SET NX lock; the golden sealed pass is pure and idempotent, so a re-run pays identically; the wallet is a database transaction backed by a CHECK constraint. The lease closes the loop — the guards make a second run cheap, the lease makes it certain, and a crash becomes a delay, not a loss.

One job, delivered twice — select the guard

pure scoring → Codemojex.Scoring is a function — no process, no bus: 100 − 20·d per position, out of 600, linear only. The moduledoc carries the guarantee: “the same secret and guess always yield the same score, so a re-delivered guess re-scores identically.”

Extracted from scoring.ex (the purity guarantee is its moduledoc), rooms.ex (close_game/1 and close_sealed/2 — the one-shot key is cm:<game>:closed), wallet.ex (one Repo.transaction: lock the row with SELECT … FOR UPDATE, check the invariant, write, insert the paired ledger row), and the players_non_negative CHECK in the founding migration. Design §Fault tolerance and correctness records the same four guards.

§3

References

sources · related

Sources

Erlang/OTP — stdlib: supervisorthe dependency-ordered tree, one_for_one, and the rest_for_one cache tier
Erlang/OTP — stdlib: gen_serverthe supervised consumers and gateways, restarted in place
Kreps — The Logthe bus as the ordered log work is dropped onto and drained from

C8 — Production Deploymentthe hub — the release and the Valkey node beside this tree
EchoMQ — the coursethe leases and the server clock, in depth
echd — the weaveeach surface makes its own CAP trade over the same substrate