R3.02.1 · At-least-once — dive 1

At-least-once semantics.

One or more times, never zero. A reliable queue holds the job until its acknowledgement — the only choice that loses no job, and the choice that lets a job run twice.

Three guarantees bound every queue, set by one decision: does the job leave before the work or after the acknowledgement. Select a delivery mode below, over a fixed run where the worker crashes after the effect, before the acknowledgement, and read messages lost against messages run.

One run, the worker crashes after the effect. Select a delivery mode; the readout reports lost against run.

readout · Select a delivery mode to read messages lost against messages run.

Static verdicts — at-most-once → 1 lost, 0 run · at-least-once → 0 lost, 2 run · exactly-once → not a deliverable guarantee.

§1

The three guarantees

reliable-queue.md.txt · Delivery Guarantees

Three delivery guarantees bound every queue design, and the spectrum is set by one decision: when does the job leave the queue, before the work or after the acknowledgement.

At-most-once. The job is removed before the work runs — RPOP and process. A crash mid-job loses the job: zero deliveries. The guarantee is that a job is delivered no more than once; the price is that it can be delivered zero times. Acceptable only when a dropped job is harmless.

At-least-once. The job is held until the acknowledgement — leased to a worker and only settled after the work confirms. A crash mid-job redelivers the job: one delivery or more. The guarantee is that a job is delivered at least once; the price is that it can be delivered twice. This is the reliable queue.

Exactly-once. One delivery and exactly one. This is what everyone wants, and it is not achievable as a delivery guarantee, because the acknowledgement itself can be lost — the third dive in this module is that argument. What is buildable is an exactly-once effect, by making the consumer idempotent on top of at-least-once delivery.

at-most-once: job leaves before the work; a crash loses it. Zero or one delivery, never a duplicate.
at-least-once: job held until the acknowledgement; a crash redelivers it. One or more deliveries, never zero.
exactly-once: one delivery and exactly one. Not a deliverable guarantee — build it as an idempotent effect.

A reliable queue must keep the job until it is acknowledged, so it lands on at-least-once — the only point on the spectrum that never drops a job.

§2

Why the lease-until-ack move yields at-least-once

echo_mq/jobs.ex · claim

R3.01 moved a job into an in-flight position under a lock instead of popping it out. EchoMQ does the same with a lease, not a list pop: EchoMQ.Jobs.claim/3 pops the oldest pending member, marks it active, and scores it on the active set at now + lease_ms — now read from the server clock (TIME), so the deadline is the server's, not the caller's. A healthy worker settles the job before the lease expires. When a worker crashes, it settles nothing, the lease deadline passes, and EchoMQ.Jobs.reap/2 finds the job still in the active set past its deadline and returns it to pending.

The recovery path is what makes the queue reliable, and it is precisely why a job can run twice. A leased job is never lost — that is the win. But a leased job whose deadline passed is redelivered — that is the cost. The two are the same mechanism viewed from two angles. The recovery cannot be kept while the redelivery is removed, because removing the recovery is exactly what loses the job.

# EchoMQ.Jobs.@claim — lease the oldest pending job on the server clock (real, trimmed)
local popped = redis.call('ZPOPMIN', KEYS[1])        -- the oldest pending member
if #popped == 0 then return {} end
local id = popped[1]
local att = redis.call('HINCRBY', jk, 'attempts', 1) -- the fencing token, minted at the lease
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')                          -- the SERVER clock, never the caller's
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)  -- score on the active set at the lease deadline

Hold the job until the acknowledgement, and a crash leaves it recoverable rather than lost; recover it, and the same job runs again. The recovery is the reliability and the redelivery both.

§3

The duplicate window

reliable-queue.md.txt · Recovery from Failures

The duplicate is not a rare corner; it has a precise window. A worker leases a job. It applies the effect — charges the card, writes the row. Then it crashes after the effect but before the acknowledgement (complete/5). The effect is done; Valkey has no record of it. The lease deadline passes. The reaper finds a job past its deadline and returns it to pending. A second worker runs it again. The card is charged twice.

This window is irreducible. No reordering closes it: if the acknowledgement comes first, the job leaves the active set before the work, and a crash now loses it (back to at-most-once). The acknowledgement has to come after the work, which means there is always a moment where the work is done and the acknowledgement is not. A crash in that moment redelivers. The window is small, but it is never zero — and over enough jobs, a never-zero probability fires.

Hover or step the crash point across the run below. Only a crash inside the gap between the effect and the acknowledgement produces a duplicate; a crash before the effect runs nothing, a crash after the acknowledgement is clean.

The crash point across one run. Hover a step or move the slider; the readout names the outcome.

crash at step 2

Static verdicts — crash at lease → nothing ran, clean re-run · crash at effect or complete → the window, charged twice · crash at done → clean.

The duplicate window is the gap between the effect and the acknowledgement, and no reordering closes it: the acknowledgement must follow the work, so a crash between them always redelivers.

§4

In EchoMQ — the worker loop that is at-least-once

echo_mq/consumer.ex · drain

EchoMQ.Consumer is the at-least-once loop in real code. It drains the queue through Lanes.claim on a default :lease_ms of 30 000, runs the handler, and settles: on :ok it calls EchoMQ.Jobs.complete(conn, queue, id, att), on an error or a raise it calls EchoMQ.Jobs.retry(…). The att is the attempts token the claim minted; it fences the settlement, so only the current lease-holder may complete or retry the job. A worker that crashes between the handler returning and complete/5 running settles nothing — the lease expires, reap/2 returns the job to pending, and the loop redelivers it. Held-until-acknowledged plus recover-on-expired-lease is exactly at-least-once: never lost, possibly twice.

# EchoMQ.Consumer.drain/1 — claim, handle, settle on the fencing token (real, trimmed)
case Lanes.claim(s.conn, s.queue, s.lease_ms) do
  :empty -> :ok
  {:ok, {id, payload, att, group}} ->
    verdict = try do s.handler.(%{id: id, payload: payload, attempts: att, group: group}) rescue _ -> {:error, "raised"} end
    case verdict do
      :ok           -> Jobs.complete(s.conn, s.queue, id, att)   # settle with the attempts token
      {:error, why} -> Jobs.retry(s.conn, s.queue, id, att, s.retry_delay_ms, s.max_attempts, why)
    end
end

the leased state machine, against live Valkey

Never lost, possibly twice

The lease-until-ack guarantee is the manuscript's, stated for the bus — the committed B3.2 figure walks the leased state machine:

bcs.3.md · B3.2 Jobs and lanesfrozen

Jobs.enqueue   writes a waiting job
Jobs.claim     leases it to a worker and makes it active
Jobs.complete  finishes it · Jobs.retry returns it to waiting with a backoff
the lease      ← a worker that dies holding a job loses the lease,
                 and the job is reclaimable rather than lost

the pattern

Hold the job until the acknowledgement and recover a crashed one: at-least-once delivery — never lost, possibly twice. The duplicate is structural, not a defect.

its EchoMQ application

Jobs.claim/3 leases the job on the server clock (TIME), and Jobs.reap/2 returns an expired lease to pending — the redelivery that makes the guarantee at-least-once.

Notes on ValkeyThe lease deadline is computed inside the claim script from the server's own TIME, so every worker leases against one clock — a caller with a fast or slow local clock cannot widen or shrink the window. valkey.io/commands/time

EchoMQ holds a job leased until the acknowledgement and returns an expired lease to pending; that is at-least-once in real code — never lost, possibly twice.

§5

References

sources · related

Sources

Redis — RPOPLPUSHthe original atomic move into the in-flight list, held until acknowledged
Redis — LMOVEthe modern successor with explicit RIGHT/LEFT directions
Valkey — ZPOPMINthe oldest-first pop the EchoMQ claim script runs over the mint-ordered pending set
Redis — Redis queuethe queue and its delivery-guarantee overview

Related in this course

R3.02 · At-least-oncethe module hub
R3.02.2 · Idempotent consumersthe next dive: absorb the duplicate at the consumer
R3.02.3 · Why exactly-once is a liewhy no protocol closes the window
R3 · The reliable queuethe move-under-lock that yields at-least-once
/echomq/queuethe leased state machine and the worker loop in depth