R8.02.2 · Persistence, pooling & failover — dive 2

Pool sizing — pipeline depth, not cores.

A pool is the standard answer to serving many callers over a server. The standard intuition for sizing it — one connection per CPU core — is wrong for Valkey, and getting it wrong wastes sockets without buying throughput.

Grounding: EchoMQ.Pool (echo/apps/echo_mq/lib/echo_mq/pool.ex) · the real infra/valkey/conf/valkey.conf (io-threads 1, the single-thread comment) · the engine is Valkey 9. The same pool R6.05 introduces as the concurrency primitive — here it is sized for production.

One command thread, fed by N pipelines. Step the pool size and read what each connector adds — and where adding more stops buying throughput.

readout · Step the pool size. One connector leaves the thread idle during each round trip; a few pipelines keep it fed and hide the round-trip latency; over-sizing adds sockets the single thread serializes anyway.

Static verdicts — 1 → the thread idles between round trips · a few (size: 4) → the thread stays fed, RTT hidden · over-sized → more sockets, no more throughput.

§1

EchoMQ.Pool — N pipelined connectors

pool.ex · moduledoc + next/1

EchoMQ.Pool is a fixed pool of pipelined connectors with lock-free round-robin dispatch. Two design choices carry the production behaviour. First, each member is already a pipeline — a single EchoMQ.Connector keeps a FIFO of in-flight commands on one socket, so it does not block waiting for a reply before sending the next. Second, dispatch is lock-free — there is no checkout. next/1 advances an :atomics counter and a caller hits the next member directly. The default is size: 4.

echo/apps/echo_mq/lib/echo_mq/pool.ex · the shape and the dispatchfrozen

# A fixed pool of pipelined connectors with lock-free round-robin
# dispatch. Each member is already a pipeline, so a small pool
# multiplies throughput without checkout ceremony: callers hit the
# next member by an atomic counter and the member's own FIFO does
# the rest.

size = Keyword.get(opts, :size, 4)

defp next(name) do
  {size, at} = :persistent_term.get({__MODULE__, name})
  member(name, rem(:atomics.add_get(at, 1, 1), size) + 1)
end

Notes on ValkeyA single connection can hold many commands in flight at once — this is pipelining, and it is what lets one socket keep the server busy without waiting for each reply — redis.io/docs/…/pipelining.

A pool member is a pipeline, and dispatch is a lock-free atomic increment. A small pool multiplies throughput with no checkout ceremony — the question is only how many members the server can actually use.

§2

The single-threaded server

valkey.conf · io-threads 1

Here is the fact that decides how to size it. Valkey executes commands on a single thread. The real valkey.conf sets io-threads 1 and its comment reads “command execution is single-threaded.” No matter how many connections you open, the server runs one command at a time.

So a pool does not buy N× server parallelism. Opening four connections does not let the server run four commands at once — it runs them one after another on the same thread. What the pool buys is something else: it hides round-trip latency. With one connection, a caller that waits for each reply before sending the next leaves the command thread idle during every network round trip. With a pipeline, commands stack on the wire and the one command thread stays fed — it always has the next command ready when it finishes the last. The pool is several pipelines, so several callers can keep the thread fed concurrently without coordinating.

# valkey.conf — command execution is single-threaded.
# the second shared vCPU is for background work, not I/O threads.
io-threads 1

The distinction is the whole dive:

does NOT buy: N× server parallelism — the one command thread runs one command at a time regardless of pool size.
DOES buy: round-trip latency hidden — enough pipelines keep that single thread continuously fed across the network round trip.

Notes on ValkeyExtra I/O threads on a shared-CPU machine busy-wait and burn the CPU budget; the engine keeps one command thread and reserves the second vCPU for background helpers, so pool size is about feeding that one thread, not adding more — valkey.io/topics/admin.

§3

The sizing rule

depth, not cores

The rule that follows: size for pipeline depth and RTT-hiding, not for cores. Enough connectors to keep the single command thread continuously fed across the network round trip is enough. Beyond that, each extra connection adds a socket the single thread serializes anyway — more memory and more file descriptors, no more throughput. size: 4 is a small pool because a small pool is what RTT-hiding needs; over-sizing it to match core count is sizing for parallelism the server does not have.

the pattern

Size a connection pool for pipeline depth — enough to keep a single command thread fed across the round trip — not for cores the server does not use. Over-sizing adds sockets, not throughput.

its EchoMQ application

EchoMQ.Pool round-robins size: 4 pipelined connectors lock-free via :atomics; over a Valkey with io-threads 1, four pipelines hide RTT for the codemojex bot workers without contending for parallelism the server never had.

Pool sizing is a question about the network, not the CPU. A pool hides the round trip on a single-threaded server; size it for the depth that keeps the thread fed, and stop.

§4

The same pool, sized — the R6.05 tie

R6.05 · worker-concurrency

R6.05 (worker-concurrency) introduces this exact EchoMQ.Pool as the concurrency primitive: how workers share connectors to drain a queue. This dive does not re-teach that — it sizes the same Pool for production, with the single-threaded-server fact as the sizing rule. R6.05 is the what; this is the how-big.

the door · the pool in the production posture

Where the pool meets the rest

The pool serves the load; the persistence policy (dive 1) keeps the data across a restart; the connector (dive 3) keeps each member reachable across a failover. Because a member of the pool is an EchoMQ.Connector, the pool inherits the connector’s reconnect discipline — a member whose socket drops fails its in-flight callers :disconnected and reconnects on its own, while the others keep serving. The EchoMQ protocol depth — lanes, leases, the Lua — is the dedicated /echomq course; this dive stays on the sizing. For where the whole pool runs against a Fly Valkey, see Codemojex on Fly.

The pool is one of three surfaces in one posture. Sized for RTT-hiding, it serves the load; as a set of connectors, it inherits the failover discipline of dive 3 for free.

§5

Recap — feed the one thread

next → R8.02.3 · readonly-reconnect-failover

EchoMQ.Pool is N pipelined connectors round-robined lock-free via :atomics, default size: 4. Because Valkey runs commands on a single thread, the pool hides round-trip latency rather than buying parallelism the server does not have — so it is sized for pipeline depth, not cores. That is the load half of operating the bus as a record of truth. Next, the failover half: the connector’s reconnect discipline, which every pool member inherits.

§6

References

sources · related

Sources

Redis — Pipelining — why a pipeline hides round-trip latency, the premise behind sizing for depth rather than cores.
Valkey — Administration — the single command thread and why extra I/O threads burn a shared-CPU budget.
antirez — On Redis latency — the Redis creator’s notes on the single command thread and where latency comes from.
Redis — Documentation — the connection and performance topics pool sizing draws on.

Related in this course

R8.02 · Persistence, pooling & failoverthe module hub
R6.05 · Worker concurrencythe same EchoMQ.Pool as the primitive
R8.02.1 · RDB and AOFthe durability this load runs over
R8.02.3 · READONLY-reconnect failoverthe reconnect each member inherits
/echomq · the Proof pillarthe production evidence the bus carries
/bcs · Production on Flywhere this pool runs