The Queue · module 03
Batches
A batch is not a new kind of work. It is the same per-item rule, applied to many at once.
To admit many jobs is to run the @enqueue transition many times in
one wire flush; to compose many flows is to land each through the single add/3,
one flow at a time; to checkpoint many leases is to re-score every live member under
one server-clock read. Each bulk verb is the bulk twin of a single verb taught elsewhere, and each
answers in input order: a verdict per job, the flows that landed, the leases it did not extend.
Batching changes the round-trip count, never the rule each item obeys. Pick a verb below, then take the three
dives.
Three bulk verbs, one rule
Three verbs on the Queue surface admit, compose, and extend many at once. Each is the bulk twin of a single verb, each saves round trips, and each keeps the single verb's contract. Pick a verb to read its handle, the single verb it batches, its wire shape, the shape it returns, and its native twin — the readout is a pure function of the verb chosen.
The bulk verbs · pick one
batches: enqueue/5 (admit one job)
wire: SCRIPT LOAD @enqueue once, then one EVALSHA per item, gathered in an EchoWire.Pipe and flushed once
returns: {:ok, [:enqueued | :duplicate | {:error, :kind} | {:error, {:server, other}}]}, one verdict per item in input order; an empty list answers {:error, :empty_pipeline}
native: no batch verb; each item's @enqueue has the single twin EMQ.ENQUEUE
Each bulk verb is the bulk twin of a single verb: it saves round trips and keeps the contract. A batch is a wire optimization, not a semantic one.
The dives
Three dives read the bulk verbs in depth, each grounded in real echo/apps/echo_mq
code: admitting many in one flush, composing many flows one at a time, and extending many leases under one
clock read.
Enqueue many
enqueue_many/4 loads @enqueue once with SCRIPT LOAD, then flushes one EVALSHA per item through an EchoWire.Pipe — every reply mapped, in input order, under the same gate, row and idempotency as enqueue/5.
Bulk flows
Flows.add_bulk/3 lands many flows, each through the single add/3, one at a time — a refused same-queue flow writes nothing, a cross-queue flow that stops partway leaves its parent held, and the batch returns the flows that landed.
Batch lease extension
extend_locks/4 re-scores every live lease under one server-clock read and returns the failed list — contrasted with the single extend_lock/5, and with the native twin EMQ.EXTENDS.
Two executors
Of the three, only the batch lease extension has a native twin, EMQ.EXTENDS; the
pipelined enqueue has no batch verb (its per-item script, @enqueue, has the single twin
EMQ.ENQUEUE), and bulk flows run only as scripts. The Queue's batch claim,
claim_batch/4, pops up to size pending jobs in one
@bclaim2 script under one clock read, and it too runs only as a script. The native
module itself is taught in The native module.
Pattern & implementation
Where Redis Patterns Applied teaches pipelining — many commands in one round trip, replies in order — this module is where the move lands across three verbs.
Pipelining sends many commands in one round trip and reads the replies in order, so N operations cost one network turn instead of N. Reliable queues is the near side of the door.
EchoMQ.Jobs.enqueue_many/4 loads @enqueue once and
pipelines the EVALSHA calls through an EchoWire.Pipe;
EchoMQ.Flows.add_bulk/3 issues add/3 per flow;
EchoMQ.Jobs.extend_locks/4 re-scores every live member in one script — each keeping
the single verb's contract.
Take the dives in order, or jump to batch lease extension if you already know admission. The next module in The Queue covers the control plane beyond enqueue and claim.
References
Sources
- Redis — SCRIPT LOAD — load the script body once, before the pipeline of EVALSHA calls.
- Redis — EVALSHA — load-once, run-by-SHA dispatch the bulk verbs pipeline.
- Valkey — ZADD — the sorted-set re-score the batch lease extension performs per live member.
- Valkey — Documentation — the substrate of record EchoMQ is backed by.
Related in this course
- Enqueue many — admit many in one flush.
- Bulk flows — compose many flows, one at a time.
- Batch lease extension — extend many leases under one clock read.
- The Queue — the pillar this module belongs to.
- The native module — the native module and its EMQ.* commands.
- redis-patterns · Reliable queues — the near side of the door.