jonnify·redis /redis-patterns/flow-control/batches/chunking-across-a-pool

R6.04.2 · Batches — dive

Chunking across a pool.

One pipelined connection is already fast; a pool of them is faster, because the flush round-robins across several sockets. The trick that makes it safe is that the script cache is server-global — load the body once on any member and every member can run it by SHA. This dive is about flushing a batch through EchoMQ.Pool, and the one property that keeps a round-robin EVALSHA from faulting.

Grounding: echo/apps/echo_mq/lib/echo_mq/pool.ex (a fixed pool of pipelined connectors, lock-free round-robin via :atomics.add_get, default size 4) · jobs.ex enqueue_many/4 with via: EchoMQ.Pool routes both wire steps through the pool.

One dispatcher, four pipelined members. Select a member to read it. A caller hits the next member by an atomic counter; each member is already a pipeline, so the pool multiplies throughput without a checkout step.
next/1atomic counter member 1 · pipeline member 2 · pipeline member 3 · pipeline member 4 · pipeline one server :6390
readout · Select a member. The dispatcher next/1 picks the next member with a lock-free atomic add-and-get, so consecutive calls spread across the four members and wrap. Each member is its own pipelined socket to the one server — the pool widens the wire without checkout ceremony.
§1

A pool of pipelines

pool.ex · next/1

EchoMQ.Pool is a fixed pool of pipelined connectors with lock-free round-robin dispatch. Each member is already a pipeline, so a small pool multiplies throughput without checkout ceremony: a caller hits the next member by an atomic counter, and that member's own FIFO does the rest. One supervisor, N connectors, one dispatcher.

The default size is four. next/1 picks the next member with an atomic add-and-get — no lock, no contention:

EchoMQ.Pool · echo/apps/echo_mq — next/1: lock-free round-robin by an atomic counterverbatim
defp next(name) do
  {size, at} = :persistent_term.get({__MODULE__, name})
  member(name, rem(:atomics.add_get(at, 1, 1), size) + 1)
end

:atomics.add_get bumps a shared counter and returns the new value in one hardware-atomic step; the remainder maps it onto a member index. Two callers racing to dispatch get two different members, with no coordination between them. command/3, pipeline/3, and eval/5 all route through this next/1, so every call spreads across the pool.

A pool is N pipelined connectors and one dispatcher. next/1 hands out the next member by a lock-free atomic counter, so dispatch costs no lock and no checkout — the member's own FIFO carries the pipeline.

§2

Fronting the flush with the pool

jobs.ex · opts[:via]

enqueue_many/4 takes the pool through opts[:via]. The default via is the single connector — so the single-argument form is unchanged — and passing via: EchoMQ.Pool routes both wire steps through the pool instead: the SCRIPT LOAD, and the EchoWire.Pipe flush.

EchoMQ.Jobs.enqueue_many(conn, "cm", pairs, via: EchoMQ.Pool)
# → SCRIPT LOAD on one member; the EVALSHA batch flushed round-robin via Pool.pipeline/3

The pipe carries the via reference and flushes through Pool.pipeline/3 when a pool is supplied. The reference is carried, never inspected — the batch does not look inside the pool, it only dispatches through it.

One option moves both wire steps onto the pool. The default keeps the single connector; passing the pool as via round-robins the load, and the pool reference is carried through and never inspected.

§3

The fault that does not happen

jobs.ex · server-global SCRIPT LOAD

Here is the hazard the design has to answer. enqueue_many/4 loads the script with SCRIPT LOAD once, on whichever member the call lands on. Then it sends the batch as EVALSHA — run-by-SHA — round-robin across members. If the cache were per connection, every member except the one that loaded the body would answer the EVALSHA with NOSCRIPT, because they had never seen the script.

That fault does not happen, because Valkey's script cache is server-global. There is one server, and one cache in it; the SHA the first member loaded is resolvable for every connection to that server, pooled or not. So a single SCRIPT LOAD on any member makes the subsequent round-robin EVALSHA resolve on all of them.

EchoMQ.Jobs · echo/apps/echo_mq — enqueue_many/4: the two wire steps through `via`verbatim
via = Keyword.get(opts, :via, Connector)
{:ok, _} = via.command(conn, ["SCRIPT", "LOAD", @enqueue.source])   ← cache once — server-global

# ... then the EVALSHA batch, assembled into a Pipe and flushed
# round-robin through Pool.pipeline/3 — every member resolves the SHA

The SCRIPT LOAD caches the script on whichever member it lands on; because the cache is server-global, that one load makes the SHA resolvable on every member, which is why the round-robin EVALSHA never faults across members. The pool spreads the flush; the shared cache keeps it correct.

the pattern

Multiply a pipelined flush across several connections without coordination — and keep a cached script resolvable on all of them.

its EchoMQ application

EchoMQ.Pool dispatches each call to the next member by a lock-free atomic counter; enqueue_many/4 with via: EchoMQ.Pool flushes the EVALSHA batch round-robin, and the server-global script cache keeps every member's EVALSHA resolvable from one SCRIPT LOAD.

Per-connection cache against server-global cache. Select a case to read what a round-robin EVALSHA does under each. Only the second case is how Valkey actually behaves.
per-connection cache load on member 1 member 1: EVALSHA ok members 2–4: NOSCRIPT server-global cache load on any member all members: EVALSHA ok one cache, one server Valkey is the right column · one SCRIPT LOAD covers every connection to the server
readout · Select a case. If the cache were per connection, only the loading member could run the EVALSHA and the rest would NOSCRIPT-fault. Valkey's cache is server-global, so one SCRIPT LOAD on any member makes the SHA resolvable on every member — the round-robin EVALSHA always resolves.

Notes on ValkeySCRIPT LOAD caches a script body in the server's script cache and returns its SHA1; the cache is per server, not per connection, so a body loaded on one connection is runnable by EVALSHA on every connection to that server. A SHA the server has not cached answers NOSCRIPT — valkey.io/commands/script-load.

§4

Chunking, and a consumer

echo/apps/codemojex

Round-robin spreads a batch across members, but each member's flush still queues on that member's FIFO. A single enormous batch flushed through one member would grow that member's FIFO unbounded; chunking is the answer. Split a huge batch into chunks and flush each chunk, so no single member holds an overlong pipeline at once and the work spreads across the pool. The chunk size is a throughput-versus-memory knob: larger chunks amortise more round-trip per flush, smaller chunks bound the in-flight queue depth.

In codemojex — a Telegram emoji-guessing game on the same stack — many bot workers admit guesses concurrently. A burst that arrives all at once is the case where the pool earns its width: the flat-queue batch over emq:{cm}:pending flushes round-robin across the pool's members, each guess a JOB-keyed idempotent enqueue, and the one SCRIPT LOAD keeps every member's EVALSHA resolvable.

/echomq/queue

Pool width at the dequeue point

The Queue pillar tunes the pool on both sides: the producer flush that admits a burst and the consumer claim that drains it, the member count that sets concurrent in-flight.

 /bcs/bus/jobs-and-lanes

The bus the pool talks to

Part B3.2 builds the Valkey-native bus — the one server every pool member connects to, and whose server-global script cache the round-robin EVALSHA relies on.

 R6.04.1

The single flush this multiplies

One pipelined flush turns N round-trips into one; the pool flushes that across N members, so a burst spreads instead of queueing on one socket.
§5

References

sources · related

Sources

  • Valkey — SCRIPT LOAD — cache a script body and return its SHA; the cache is server-global, which is the property the pool relies on.
  • Valkey — EVALSHA — run a cached script by SHA, with NOSCRIPT when the body is not cached; the fault the server-global cache avoids across pool members.
  • Valkey — Pipelining — why each member being a pipeline already multiplies throughput before the pool widens it.

Related in this course