Persistence, pooling & failover.

R7.01 made the case that the bus is the record of truth: the JOB HASH carries state, attempts, and payload, and Postgres holds only what must be transactional. This module takes that record into production and answers the operational question R7.01 left open — how does it stay one.

A record of truth is not a data model; it is a data model plus three operational guarantees. It must survive a restart so the data outlives the process, serve under load so one server feeds many callers without a head-of-line stall, and ride a failover so a dropped wire recovers cleanly. Each guarantee lives on a different real surface, and this module reads each one. The pattern is R7.01; the subject here is the operations.

The first guarantee is durability: the data outlives the process. The real valkey.conf makes the append-only log the single source of durability. appendonly yes records every keyspace-changing write to a log on disk; appendfsync everysec flushes that log on a background thread once a second, bounding crash loss to about one second of writes; save "" turns snapshotting off so it never competes for a second fork; aof-use-rdb-preamble yes keeps the rewrite fast. propagation-error-behavior panic fails a write rather than acknowledge data that may not survive, and maxmemory-policy noeviction rejects writes loudly under memory pressure instead of dropping a job — the R7.01 tie.

Dive 1 takes this block apart line by line. It is distinct from R8.01’s persistence-safe-settings, which is the kernel layer (the vm.overcommit_memory sysctl for the save fork); here it is the Valkey persistence policy itself. The two are halves of one posture — cross-reference, not duplicate.

The second guarantee is throughput under concurrency. EchoMQ.Pool is a fixed pool of pipelined connectors with lock-free round-robin dispatch: next/1 advances an :atomics counter and a caller hits the next member; the default size: 4. The sharp production insight is the one most pool sizing gets wrong. Valkey runs commands single-threaded — the real valkey.conf sets io-threads 1 and its comment reads “command execution is single-threaded.” So a pool does not buy N× server parallelism — it hides round-trip latency by keeping the one command thread fed via pipelining.

Size for pipeline depth and RTT-hiding, not for cores; over-sizing only adds sockets the single thread serializes anyway. Dive 2 derives that insight. It is the same EchoMQ.Pool R6.05 teaches as the concurrency primitive — here it is sized for production rather than introduced.

The third guarantee is reachability across a disruption. When a primary fails over to a replica, a client that keeps writing to the old node gets a Valkey -READONLY reply — the signal that a failover happened. A resilient client must reconnect to the new primary. EchoMQ.Connector grounds the reconnect discipline: supervised reconnect with capped jittered backoff, re-fencing on every reconnect; in-flight callers failed with :disconnected on socket loss — never replayed, because the connector cannot know what is idempotent; and an idle heartbeat that PINGs a quiet wire so dead peers are noticed before the next caller pays for the discovery.

The shipped deploy is a standalone Fly node, so a failover here is a socket drop and a reconnect to the same restarted server; the discipline is the same one a Sentinel- or Cluster-aware client uses to re-resolve a new primary. Dive 3 is honest about the seam: the connector handles failover by socket-drop and reconnect, not by parsing -READONLY — the -READONLY reply is the signal (the pattern), the connector’s reconnect is the real grounding.

The three surfaces are one posture seen from three layers. A restart hits the persistence policy: the append-only log replays and the record is whole. A load spike hits the pool: the four pipelined connectors keep the single command thread fed without one slow caller stalling the rest. A failover hits the connector: the socket drops, in-flight callers get a clean :disconnected, and the supervised reconnect re-fences and resumes. Because a pool member is an EchoMQ.Connector, the pool inherits the reconnect discipline for free.

Each dive takes one guarantee and the real surface that delivers it, in the arc survive → serve → ride: the engine’s persistence policy, the connector pool sized for a single-threaded server, and the connector’s reconnect discipline. Read them in order — durability starts.