R8.02.3 · Persistence, pooling & failover — dive 3
READONLY, reconnect & failover.
A record of truth has to stay reachable across a disruption. When the node a client is writing to fails over, the old connection points at the wrong server. This dive separates the signal that a failover happened from the discipline that recovers from it.
Grounding: EchoMQ.Connector (echo/apps/echo_wire/lib/echo_mq/connector.ex) for the reconnect discipline · the Valkey -READONLY reply for the signal (the pattern) · the engine is Valkey 9. Honest retarget: the connector handles failover by socket-drop-and-reconnect, not by parsing -READONLY.
-READONLY reply is the Valkey signal (the pattern); failing in-flight callers without replay,
the jittered backoff with re-fence, and the recovered connection are the real EchoMQ.Connector.Static sequence — signal → the failover is visible · fail in-flight →
at-most-once on the wire · backoff + re-fence → safe retry of the connection · reconnected
→ reconnects counter +1.
The -READONLY signal — the pattern
Valkey replicationIn a replicated Valkey deployment, writes go to the primary; replicas serve reads and refuse writes. If a
client keeps issuing writes to a node that has become a replica — which is exactly what happens for a
moment after a failover, before the client has detected the change — the server answers with a
-READONLY error: “You can’t write against a read only replica.”
That reply is a signal: the topology changed under the client. A resilient client treats it as the cue to stop writing to this node and reconnect to the new primary. How it finds the new primary depends on the deployment — a Sentinel client asks the sentinels, a Cluster client re-resolves the slot’s owner. The signal is the pattern; the rediscovery mechanism is deployment-specific.
# after a failover, a write to the demoted node is refused: > SET emq:{orders}:pending 1 (error) READONLY You can't write against a read only replica. # the signal: reconnect to the promoted primary.
Notes on ValkeyA replica refuses writes and answers
-READONLY; a client that receives it has been talking to a node that is no longer the primary, and a
Sentinel- or Cluster-aware client re-resolves the promoted primary to reconnect to —
valkey.io/topics/replication.
What the connector actually does
connector.ex · the honest retargetEchoMQ.Connector does not parse -READONLY, and it does not do
Sentinel discovery. Being honest about that is the whole point of this dive: the module name
“READONLY-reconnect” overstates what is in the code. The connector handles failover the way a
single-owner socket client should — by socket-drop and supervised reconnect. When the
wire to the old node drops (a failover tears the connection down, or the node restarts), the connector notices,
fails its in-flight callers, and reconnects on a backoff.
The shipped deploy is a standalone Fly node, so a failover here is a restart and the reconnect goes back to
the same server — but the discipline is exactly the one a Sentinel- or Cluster-aware client layers its
rediscovery on top of. So the grounding here is not a -READONLY handler; it is the
reconnect discipline the connector ships — three real properties that make reconnection
safe.
- the pattern
- the
-READONLYreply signals a failover; reconnect to the new primary (Sentinel/Cluster re-resolves it). - the grounding
- the connector reconnects on a dropped socket — capped jittered backoff, re-fence, in-flight fail-not-replay.
The three production properties
connector.ex · moduledoc + schedule/1The connector’s moduledoc names them. The reconnect itself: “supervised reconnect with
capped jittered backoff, re-fencing on every reconnect.” That is real in schedule/1 —
each attempt waits the current backoff plus a random jitter, and the backoff doubles up to a ceiling. The
jitter keeps a fleet of reconnecting clients from synchronizing into a thundering herd; the cap keeps the wait
bounded. Re-fencing means the wire-version fence is re-checked before the first command on the new connection,
so a reconnect can never resume against a server speaking a different protocol.
# capped jittered backoff: wait backoff + jitter, then double up to a cap.
defp schedule(%{sock: nil} = s) do
jitter = :rand.uniform(div(s.backoff, 2) + 1)
Process.send_after(self(), :reconnect, s.backoff + jitter)
%{s | backoff: min(s.backoff * 2, s.backoff_max)}
end
The in-flight policy is the load-bearing safety property: “in-flight callers failed with
:disconnected on socket loss — never replayed, because the connector cannot know what is
idempotent.” When the socket drops, every caller waiting on a reply gets a clean
{:error, :disconnected} — and the command is not re-sent on the new
connection. The connector refuses to replay because it cannot know whether the command was idempotent: a write
that already landed before the drop would be applied twice if replayed. So the wire is at-most-once,
and idempotency is pushed up to the caller, which is the only place with the information a safe retry needs. A
complete that already ran must not run again; the retry decision belongs to the caller.
The early-warning property: “an idle heartbeat that PINGs a quiet wire so dead peers are noticed before the next caller pays for the discovery.” A connection that has been idle might already be dead — the peer gone, the path broken — and nobody knows until the next caller tries to use it and waits for a timeout. The connector arms an idle heartbeat that PINGs a quiet wire, so a dead peer surfaces as a failed PING and triggers the reconnect before a real caller is made to pay the discovery cost.
The connector also exposes the failover state to observers: a reconnects counter (slot 4 of its
counters) increments on each successful reconnect, and stats/1 reports
status: :reconnecting while the socket is down and :connected once it is back.
Treat -READONLY (or a dropped socket) as the cue that the topology changed; reconnect with
backoff; never replay an in-flight write whose idempotency is unknown.
EchoMQ.Connector ships capped jittered backoff with re-fence on reconnect, fails in-flight
callers :disconnected (at-most-once on the wire), and PINGs an idle wire so dead peers surface
early; the standalone Fly deploy reconnects to the restarted node.
Failover recovery is a discipline, not a single error code. The connector does not parse
-READONLY — it drops, fails in-flight callers loudly, re-fences, and reconnects on a jittered
backoff, and it pushes idempotency to the caller because that is the only place with the information a safe retry
needs.
Where this completes the posture
restart · load · failoverThis is the third guarantee of operating the bus as a record of truth. The persistence policy (dive 1) keeps
the data across a restart; the pool (dive 2) serves it under load; this reconnect discipline keeps each
connection reachable across a failover. Because a pool member is an EchoMQ.Connector, the
pool inherits this discipline for free: one member’s socket dropping fails only that member’s
in-flight callers and reconnects on its own, while the rest of the pool keeps serving.
The reconnect, observed
The connector exposes its failover behaviour as telemetry — the reconnects counter, the
status field, the pipeline-stop events. The conformance, telemetry, and benchmark that turn these
signals into evidence are the EchoMQ Proof pillar. The standalone Fly node this
reconnects to is where Codemojex runs on Fly; the durability dial one tier below the
volatile bus is /echo-persistence. This dive stays on the reconnect discipline.
status: :reconnecting ← while the socket is down (else :connected) reconnects: 3 ← counter slot 4, +1 per successful reconnect wire_errors: 0 ← the re-fence keeps a stale protocol out
Three guarantees, one record of truth: durability keeps it across a restart, the pool serves it under load, and the reconnect discipline reaches it across a failover — and the pool inherits this third one because a member is a connector.
Recap — the module whole
return → R8.02 hubThe Valkey -READONLY reply is the signal that a failover happened (the pattern);
EchoMQ.Connector grounds the reconnect discipline (the real surface): capped jittered backoff with
re-fence, in-flight callers failed :disconnected and never replayed, and an idle heartbeat that
surfaces dead peers early. The connector does not parse -READONLY — it drops and reconnects.
With this dive the module is whole: the bus survives a restart (dive 1), serves under load (dive 2), and rides a
failover (here) — the three guarantees that keep a record of truth one. Return to the
module hub or the
chapter.
References
sources · relatedSources
- Valkey — Replication — primary/replica roles and the
-READONLYreply a client gets writing to a replica after a failover. - Valkey — Sentinel — automatic failover and how a Sentinel-aware client re-resolves the new primary the reconnect targets.
- antirez — Redis Sentinel design — the Redis creator’s notes on failover signalling and client rediscovery.
- Redis — Documentation — the high-availability and client-reconnection topics this discipline draws on.
Related in this course
- R8.02 · Persistence, pooling & failoverthe module hub
- R8.02.2 · Pool sizingthe pool whose members inherit this discipline
- R7.01 · Redis as a primary databasethe record of truth this keeps reachable
- /echomq · the Proof pillarconformance, telemetry, the reconnects counter
- /bcs · Production on Flythe standalone Fly node this reconnects to
- /echo-persistencethe durability floor beneath the volatile tier