One command, then the machine

One fly deploy does two things in order. First it builds the image — the F7.1 Dockerfile, ending at mix release portal with an ENTRYPOINT ["bin/portal", "start"]. Then it ships that image to the machines under the [deploy] stanza’s strategy = "rolling": machines are replaced one at a time rather than all at once, so while one machine takes the new release the others keep answering, and the service is never fully down during the roll. If the new release fails its health checks, the [experimental] stanza’s auto_rollback = true reverts to the prior release, so a bad deploy does not leave the app stuck on a version that cannot pass its GET /health check.

The chapter’s one open decision lives in this same [deploy] stanza: there is no release_command. The Dockerfile records the intent in a comment — “the deploy runs bin/portal eval "Portal.Release.migrate()" (release-native migration, F6.8.2-D5) BEFORE this start” — but the intent is not wired. So the first bring-up migrated manually, through a fly proxy 15432:5432 onto the writable primary (the F7.3.2 path). Wiring release_command = "/app/bin/portal eval \"Portal.Release.migrate()\"" into [deploy] would make Fly run the migration in a temporary VM before the rolling release on every deploy — the recommended path. Both paths are real; which one ships is the Operator’s call.

Step through the deploy below: select a phase to read what fly deploy does at that point, and how the two paths — with and without a wired release_command — differ.

Once a release is rolled, the machine has a lifecycle of its own. A Fly machine is in one of three states. started is running and serving; the [[services]] stanza’s min_machines_running = 1 — paired with auto_stop_machines = false — keeps at least one machine started so the cluster and the Presence counts stay warm and a node is not reaped mid-session. stopped is scaled to zero on purpose; with auto_start_machines = true a stopped machine starts again on the next request. suspended is the failure state: a machine that crash-looped past its restart limit is suspended by Fly so it stops consuming restarts — it does not come back on its own.

Select a state below to read what it means, how a machine reaches it, and how it leaves.

The dominant failure mode is not new to this dive — it is the F7.3 boot coupling surfacing at the machine level. Portal.Engine.init/1 reads the event log synchronously at startup, so the release cannot finish booting until the database is both reachable and migrated. An unreachable or unmigrated database makes init/1 raise, the supervisor reports a failed child, the release exits with exit 1, and the machine restarts. After ten such restarts Fly gives up: the machine log carries machine has reached its max restart count of 10, and Fly suspendeds the machine (gotcha 4). The machine is now stopped consuming restarts, but it is also not serving — it does not recover on its own.

Recovery forks on a single question: is the underlying fault already fixed, or is the fix in the image or config? If the fault is already fixed — the database is now up and migrated — then fly machine restart <id> -a echo-portal un-suspends the machine: the same image boots, init/1 reads the log cleanly this time, and the machine returns to started. No rebuild is needed because nothing in the image or config was wrong. But if the fix lives in the image or config — the infra/inetrc COPY, or a socket_options: [:inet6] consume that the running release does not carry — then a restart boots the same broken image and loops again. That case needs a fresh fly deploy: a fly deploy rebuilds the image and rolls it, shipping both the inetrc and the socket-options fix in one new release.

Either way, you read the machine’s state before you choose. fly status shows the app’s machines and the current release; fly releases shows the release history and each one’s status; fly machine status <id> shows one machine’s state and its last exit. Pick the recovery path below and read the exact command it names.