jonnify·redis /redis-patterns/flow-control/batches

R6.04 · Flow Control & Scale — module hub

Batches — many writes, one round-trip.

Batching turns many small writes into one network round-trip: assemble the commands, send them together, read the replies together — paying the wire cost once instead of once per item. The real surface is EchoMQ.Jobs.enqueue_many/4, and it is a pipeline, not a transaction: each item is atomic and idempotent, the batch returns a verdict per item, and there is no rollback.

Technique, not a fresh catalog pattern: it applies atomic-updates (R2.01) — the per-item idempotent enqueue script — run many times in one flush. Grounding: echo/apps/echo_mq · EchoMQ.Jobs.enqueue_many/4 through EchoWire.Pipe, the @enqueue script, EchoMQ.Lanes.bclaim/3 over emq:{q}:pending. Consumer: codemojex — a burst of guess JOBs in one flush.

A batch enqueue, end to end. Select a step to read what it does. The script is cached once, an EVALSHA per pair is assembled into a pipe, the pipe is flushed in one round-trip, and a verdict comes back per item in input order.
one wire round-trip carries every EVALSHA· the replies return in one frame SCRIPT LOADcache once Pipe.commandEVALSHA / pair Pipe.execone round-trip mapper item
readout · Select a step. The enqueue script is cached once and its SHA returned; an EVALSHA per pair is appended to an EchoWire.Pipe; the pipe is flushed in one round-trip; and the replies are mapped to a verdict per item, in input order.
§1

What batching is, and what it is not

pipelining vs transactions

Two different ideas hide under “do many things at once,” and conflating them is the common mistake.

  • Pipelining is a transport optimisation. The client stops waiting for each reply before sending the next command; it writes N commands to the socket back to back and then reads N replies. The server still runs N commands — pipelining changes when the bytes travel, not how much work the server does. The win is wall-clock round-trip time.
  • A transaction (MULTI/EXEC) is an atomicity guarantee. The commands run as one indivisible unit; either every queued command applies or, on the narrow conditions MULTI rolls back, none do. The win is all-or-nothing semantics.

enqueue_many/4 is the first, not the second. It pipelines N idempotent enqueue scripts and returns a list of N verdicts. There is no rollback: if the third item is a duplicate, the first two are still enqueued and the verdict list says so plainly at position three. That is a deliberate design choice, examined in the third dive.

Batching here is pipelining, not a transaction. The wire is paid once for the whole batch; the server still runs every script; and the reply is a verdict per item, not a single all-or-nothing result.

§2

One round-trip, N scripts

jobs.ex · enqueue_many/4

EchoMQ.Jobs.enqueue_many(conn, queue, pairs, opts \\ []) takes a list of {id, payload} pairs and admits them in one wire flush. It runs in two wire steps: it SCRIPT LOADs the enqueue body once — Valkey's script cache is server-global, so one load makes the SHA resolvable for every command that follows — then it appends one EVALSHA per pair to an EchoWire.Pipe and flushes the whole pipe in one exec.

EchoMQ.Jobs.enqueue_many(conn, "cm", pairs)
# → SCRIPT LOAD once, then one EVALSHA per pair flushed together through EchoWire.Pipe
# → [:enqueued, :duplicate, :enqueued, {:error, :kind}, ...]  (one verdict per pair, in order)

Each pair runs the same script the single enqueue/4 runs — the per-item atomic move from the atomic-updates pattern. The script kind-gates the id, refuses a duplicate, and writes the row and the pending entry on the server in one step:

EchoMQ.Jobs · echo/apps/echo_mq — the @enqueue script each batch member runsverbatim
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')   ← kind-gate
end
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0                                                       ← duplicate: no-op
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])                              ← row + pending insert
return 1

So each member is: kind-gate (JOB-namespaced) → duplicate refusal (EXISTS returns 0) → row write plus pending insert → 1. The batch pipelines many of these. The idempotency is per item — :duplicate is one item's verdict, never an abort of the rest — which is exactly why the move is a pipeline and not a single MULTI/EXEC.

A batch is one round-trip carrying N scripts. The wire cost is paid once; the server runs the same N enqueue scripts it would for N separate calls; and the per-item idempotency is what lets the batch be a pipeline rather than a transaction.

§3

The verdict per item

jobs.ex · the reply map

The reply is a list, in input order, one entry per pair:

  • :enqueued — the script returned 1: the row and the pending entry were written.
  • :duplicate — the script returned 0: a row already existed at that id, so nothing changed.
  • {:error, :kind} — the id was not JOB-namespaced; the script refused it with EMQKIND and that one item failed.

An empty pairs list answers {:error, :empty_pipeline} — there is nothing to flush, and the empty pipe refuses rather than send an empty frame. A non-empty batch always returns the verdict list; reading it is how a caller learns which items landed.

built ready to read
R6.04.1built

Round-trip elimination

N enqueues cost N× the round-trip; enqueue_many is one flush. The win is wall-clock round-trip time, not server work — the server still runs every script.
R6.04.2built

Chunking across a pool

via: EchoMQ.Pool round-robins the flush across N pipelined members; a server-global SCRIPT LOAD keeps every member's EVALSHA resolvable.
R6.04.3built

Partial-failure handling

A verdict per item against MULTI/EXEC rollback: each item atomic, the batch not transactional — the honest model, by design.
§4

Applied in EchoMQ

echo/apps/echo_mq

The whole technique is EchoMQ.Jobs.enqueue_many/4, and it is the atomic-updates pattern run in bulk: the same script, the same row shape, the same idempotency as the single enqueue/4, pipelined so the wire is paid once. The batch is not transactional, and that is the design — per-item idempotent dedup beats whole-batch rollback for an at-least-once queue, because a re-sent batch settles to the same state rather than failing wholesale.

The mirror of a batch enqueue is a batch claim. EchoMQ.Lanes.bclaim/3 rotates to a serviceable lane and serves up to a batch of heads in one script, every member leased on one shared TIME deadline read once for the whole batch. The non-grouped spine is the count-variant ZPOPMIN loop, which clamps the count to k = min(requested, depth) so it never over-pops:

EchoMQ.Jobs · echo/apps/echo_mq — the @bclaim spine: one TIME lease, k = min(requested, depth)verbatim
local depth = redis.call('ZCARD', KEYS[1])
if depth == 0 then return {} end
local k = tonumber(ARGV[3])
if depth < k then k = depth end                            ← never over-pops
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)            ← one lease for the batch
local served = {}
for _ = 1, k do
  local popped = redis.call('ZPOPMIN', KEYS[1])
  if #popped == 0 then break end
  local id = popped[1]
  local jk = ARGV[1] .. id
  local att = redis.call('HINCRBY', jk, 'attempts', 1)
  redis.call('HSET', jk, 'state', 'active')
  redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)
  served[#served + 1] = {id, redis.call('HGET', jk, 'payload'), att}
end
return served

An under-fill returns the short list rather than refusing. The batch is a claim unit, not a resolution unit — the members are claimed together, but each is settled on its own through the byte-frozen completion and retry paths, so one poisoned member is isolated to its own retry while the rest of the batch settles.

the pattern

Many writes, one round-trip, no rollback: assemble the commands, send them together, read the replies together — and read a result for each.

its EchoMQ application

EchoMQ.Jobs.enqueue_many/4: the per-item idempotent @enqueue script SCRIPT LOADed once (server-global cache), an EVALSHA per pair assembled through EchoWire.Pipe and flushed in one round-trip, a verdict per item (:enqueued / :duplicate / {:error, :kind}) in input order.

“Enqueue is one idempotent script: kind policy, duplicate refusal, row write, and pending insertion happen on the server in one atomic step.” A batch pipelines many of these — the atomicity is the item, the round-trip is the batch. — The Branded Component System, Part B3 · Jobs and lanes
The throughput problem → the technique. Select a problem to read the technique that answers it and the real EchoMQ surface that carries it.
the problem the technique & surface per-job round-trip pipeline · enqueue_many/4 one socket saturates pool · via: EchoMQ.Pool one item is bad per-item verdict · the reply map per-claim fetch batch claim · bclaim/3
readout · Select a problem. Each throughput problem maps to one move: a per-job round-trip to pipelining the batch, a saturated socket to a connector pool, a bad item to a per-item verdict, and a per-claim fetch to a batch claim — all real EchoMQ surfaces.

A consumer makes the burst concrete. In codemojex — a Telegram emoji-guessing game on the same stack — a single guess is admitted by Codemojex.Guesses.submit/3, which mints a branded JOB, builds the guess payload, and enqueues it. When the bot ingests a burst of player commands at once, the flat-queue shape of the same move admits the whole burst in one flush — enqueue_many(conn, "cm", pairs) over emq:{cm}:pending — so one wire round-trip admits the burst and each guess-JOB is idempotent at its own JOB-keyed row.

Wiring the batch claim into the dequeue path at scale — the consumer that drives bclaim/3, the pool width, the lease recovery — is the queue's scaling layer, taught in the EchoMQ course rather than here.

/echomq/queue

The batch at the dequeue point

The Queue pillar wires the batch claim into the consumer: bclaim/3 amortising the per-claim fetch, the pool width that feeds it, and the lease recovery for a batch on one deadline.

 /bcs/bus/jobs-and-lanes

Enqueue is one atomic script

Part B3.2 builds the job row and the pending set the enqueue script writes — the per-item atomic move a batch pipelines many times in one flush.

 R2

The atomic move each item runs

The per-item enqueue is the atomic-updates pattern: a read-modify-write done in one server-side step. The coordination chapter builds it; a batch runs it in bulk.

Notes on ValkeyPipelining is not a transaction: the commands in a pipeline are sent without waiting for each reply, but they are not isolated and do not roll back as a unit. All-or-nothing is MULTI/EXEC; a pipeline returns a reply for every command, which is why a batch enqueue can report a verdict per item — valkey.io/topics/pipelining.

§5

When to use / when to avoid

the decision

Reach for a pipelined batch when:

  • A producer admits many jobs at once and the per-job round-trip is the bottleneck — a burst of events, a fan-out, a backfill.
  • Each item is independently meaningful: a duplicate or a bad id on one item should not undo the others. Per-item verdicts are exactly what you want to read back.
  • The work is idempotent at the item level, so a re-sent batch settles to the same state. The enqueue script's EXISTS guard gives this for free.

Avoid a pipelined batch — or reach for a real transaction instead — when:

  • The items must be all-or-nothing: either every write applies or none does. That is a transaction's job, not a pipeline's; a pipeline has no rollback.
  • The batch is so large that one connection's FIFO would grow unbounded; chunk it across a pool, or cap the chunk size.
  • There is only ever one item; the batch machinery adds assembly with no round-trip to amortise.

A pipelined batch buys the round-trip back and keeps per-item visibility; it does not buy all-or-nothing. Reach for it when items are independent and the producer may re-send — reach for a transaction when the items are only meaningful together.

§6

References

sources · related

Sources

  • Valkey — Pipelining — sending multiple commands without waiting for each reply; the transport optimisation behind a batch enqueue, and why it is not a transaction.
  • Valkey — EVALSHA — run a cached script by its SHA; the per-item command a batch assembles, resolvable on any connection once the body is loaded.
  • Valkey — SCRIPT LOAD — cache a script body and return its SHA; the server-global cache that lets a pool's round-robin EVALSHA never fault.
  • Valkey — Transactions — MULTI/EXEC and the all-or-nothing guarantee a pipeline deliberately does not provide.
  • Valkey — ZADD — the score-0 insert that puts a job's branded id into the pending set in the enqueue script.

Related in this course