jonnify·redis /redis-patterns/flow-control/batches/partial-failure-handling

R6.04.3 · Batches — dive

Partial-failure handling.

A pipelined batch is not all-or-nothing. enqueue_many/4 returns a verdict for every item in input order — some enqueued, some duplicate, some refused — and an empty batch refuses outright. That is a deliberate choice over MULTI/EXEC rollback, and for an at-least-once queue it is the better one.

Grounding: echo/apps/echo_mq/lib/echo_mq/jobs.ex · EchoMQ.Jobs.enqueue_many/4 maps each reply to :enqueued / :duplicate / {:error, :kind}; an empty pairs list answers {:error, :empty_pipeline}. This dive corrects the catalog “all-or-nothing bulk enqueue” framing.

A batch of five, two of them refused. Select an item to read its verdict. The third is a duplicate and the fifth has a bad id; the other three enqueue, and the batch does not unwind.
one verdict per item, in input order · no rollback item 1:enqueued item 2:enqueued item 3:duplicate item 4:enqueued item 5:kind items 1, 2, 4 landed · item 3 was a no-op · item 5 was refused — none undone
readout · Select an item. Each of the five gets its own verdict at its own position: three :enqueued, one :duplicate (a no-op), one {:error, :kind} (refused). The refusals do not undo the items that landed — the batch is a pipeline, not a transaction.
§1

The picture to unlearn

bulk enqueue is not a transaction

“Bulk enqueue” sounds like a transaction: send a batch, and either the whole batch lands or none of it does. That is the MULTI/EXEC model, and it is not what enqueue_many/4 does. There is no rollback. If the third item in a batch is a duplicate and the fifth has a malformed id, the first, second, and fourth are still enqueued — the batch does not unwind. What the caller gets back is not a single success-or-failure; it is a list of verdicts, one per item, in input order.

The catalog name invites a transaction; the real surface is a pipeline. A bad item is one item's verdict, never an abort of the rest, and there is nothing to roll back.

§2

The verdict list

jobs.ex · the reply map

Each {id, payload} pair runs the same idempotent enqueue script, and the script's return maps to a verdict:

  • :enqueued — the script returned 1: the row and the pending entry were written.
  • :duplicate — the script returned 0: a row already existed at that id, so the script made no change.
  • {:error, :kind} — the id was not JOB-namespaced; the script refused it with EMQKIND.

enqueue_many/4 maps the flushed replies straight onto these:

EchoMQ.Jobs · echo/apps/echo_mq — enqueue_many/4: the per-item verdict mapverbatim
with {:ok, results} <- Pipe.exec(pipe) do
  {:ok,
   Enum.map(results, fn
     1 -> :enqueued
     0 -> :duplicate
     {:error_reply, "EMQKIND" <> _} -> {:error, :kind}
   end)}
end

The list has the same length and order as the input, so position n in the verdict list is the outcome of pair n. A caller reads it to learn exactly which items landed and which were refused — and the refusals do not contaminate the rest. One duplicate among a thousand new jobs enqueues nine hundred ninety-nine and reports the one.

The empty batch is the boundary case: enqueue_many(conn, queue, []) answers {:error, :empty_pipeline}. There is nothing to flush, and the empty pipe refuses rather than send a frame with no commands.

The reply is a list of verdicts, same length and order as the input. Each item's outcome sits at its own position; an empty batch answers {:error, :empty_pipeline} rather than flush nothing.

§3

Atomic per item, not per batch

jobs.ex · the @enqueue script

The subtlety worth being precise about: every item is atomic, but not the batch. Each member runs the enqueue script, and that script is one indivisible server-side step — the kind-gate, the EXISTS check, the HSET, and the ZADD either all happen for that item or none do. An item never lands half-written. What the batch gives up is atomicity across items: there is no point at which all items are applied together or none are.

EchoMQ.Jobs · echo/apps/echo_mq — the @enqueue script: one indivisible step per itemverbatim
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

So the unit of atomicity is the item; the unit of the round-trip saving is the batch. They are different scopes on purpose — each item is all-or-nothing on its own, the batch is not all-or-nothing across items.

Two scopes, not one. Select a scope to read what is guaranteed at it. The item is atomic; the batch is not.
the item — atomic the @enqueue script is one indivisible step the batch — not atomic items settle on their own; no all-or-nothing point atomicity is the item · the round-trip saving is the batch
readout · Select a scope. At the item scope the enqueue script is atomic — all of its steps apply or none do. At the batch scope there is no atomicity across items — each settles independently, which is exactly what the verdict list reports.

Atomicity is the item, not the batch. Each enqueue script applies all-or-nothing on its own item; the batch never has a moment where all items are applied together or none are.

§4

Why per-item beats rollback here

echo/apps/echo_mq · at-least-once

For an at-least-once queue, the per-item model is the stronger one, not a compromise.

The enqueue script is idempotent: a second enqueue of the same id is a :duplicate, a no-op. So if a batch is sent again — a retry after a timeout, a redelivery, a producer that is not sure the first send landed — the items that already enqueued report :duplicate and the items that did not enqueue land. The batch converges to the intended state. A transaction would instead fail the whole retried batch because some members now conflict, and the producer would have to reason about which items to strip before resending.

Per-item idempotent dedup beats whole-batch rollback because the failure mode of a queue producer is a re-sent batch, and the per-item model handles a re-send by settling rather than by aborting. The price is that the caller reads a verdict list instead of a single boolean — which is information it wanted anyway.

the pattern

A batch that reports per-item rather than rolling back: each item independently meaningful, a re-send converging instead of failing.

its EchoMQ application

enqueue_many/4 maps the flushed replies to :enqueued / :duplicate / {:error, :kind} in input order; an empty batch is {:error, :empty_pipeline}. Each item is atomic via the enqueue script; the batch is not transactional, by design.

The two models, side by side:

pipelined batch (enqueue_many/4)a transaction (MULTI/EXEC)
atomicityper item (the enqueue script)across the whole batch
on a bad itemthat item's verdict; the rest landthe conditions that roll back undo the batch
replya list of verdicts, in input orderone result for the batch
a re-sent batchconverges — duplicates are no-opsconflicts; the batch must be reshaped
the winone round-trip, per-item visibilityall-or-nothing semantics

The two are different tools. A pipeline is right when items are independent and the producer may re-send; a transaction is right when the items are only meaningful together and must not partially apply. enqueue_many/4 chose the pipeline because a queue's producers re-send, and convergence is worth more than rollback.

A consumer makes it concrete. In codemojex — a Telegram emoji-guessing game on the same stack — a single guess is admitted by Codemojex.Guesses.submit/3, which mints a JOB and enqueues it. When a burst is admitted in one flush, a stray duplicate or a malformed id in the burst fails only itself: the rest of the guesses enqueue, and the verdict list says which. If the bot re-sends a burst it is unsure landed, the already-enqueued guesses report :duplicate and the missing ones land — the burst converges.

/echomq/queue

Bursts through the consumer

The Queue pillar drives bursts at scale: the producer flush, the verdict list a worker reads, and the claim that drains what landed — the at-least-once delivery the per-item model serves.

 /bcs/bus/jobs-and-lanes

The idempotent enqueue

Part B3.2 builds the enqueue script whose EXISTS guard makes a re-sent item a no-op — the per-item idempotency convergence depends on.

 R2

The atomic-updates pattern

The per-item enqueue is a read-modify-write done in one server-side step. The coordination chapter builds atomic-updates; this batch runs it once per item.

Notes on ValkeyA transaction (MULTI/EXEC) queues commands and runs them as one isolated unit; a pipeline sends commands without waiting and they are neither isolated nor rolled back as a unit. Choosing a pipeline means choosing per-command replies and no rollback — the trade a batch enqueue makes on purpose — valkey.io/topics/transactions.

§5

References

sources · related

Sources

  • Valkey — Transactions — MULTI/EXEC and the all-or-nothing model a pipeline deliberately does not provide.
  • Valkey — Pipelining — the transport model that returns a reply per command, the basis of the per-item verdict list.
  • Valkey — EVALSHA — the per-item command; an error reply on one item is one item's verdict, not a batch abort.

Related in this course