R3.02.3 · At-least-once — dive 3
Why exactly-once is a lie.
The acknowledgement can be lost. A queue that picks no-loss must redeliver on a missing acknowledgement, and no protocol closes that gap — so exactly-once delivery is unreachable.
Exactly-once delivery is sold as a feature; it is not one. The acknowledgement that would confirm a single delivery can itself be lost, and “the work never ran” and “the acknowledgement was lost” leave the queue in the same state. Select ack lost below and read why the redelivery is unavoidable. What gets built instead is an exactly-once effect: at-least-once delivery on an idempotent consumer.
Static verdicts — ack arrives → the queue settles, one effect, clean · ack lost → the queue must redeliver to avoid loss, so the work runs twice.
The acknowledgement gap
reliable-queue.md.txt · Delivery GuaranteesThe argument is the two-generals problem in queue clothing. A worker leases a job, does the work, and sends an acknowledgement. The queue settles the job when the acknowledgement arrives. Now ask: what happens if the acknowledgement is lost in transit, or the worker crashes the instant after sending it.
The queue is in a position with no clean exit. It leased the job and received no acknowledgement. Two histories produce that exact state, and from the outside the two are indistinguishable:
The work never ran (a real loss), so the job must be redelivered. Or the work ran and the acknowledgement was lost, so redelivering runs the work twice.
To guarantee no loss, the queue must redeliver on a missing acknowledgement. But redelivering on a missing acknowledgement runs the second history twice. There is no third behaviour: redeliver and risk a duplicate, or do not redeliver and risk a loss. A confirmation of the confirmation does not help — that message can be lost too, and the regress never bottoms out. The gap is structural, not an implementation gap.
worker: lease job -> do work -> send ACK ──✗ (ACK lost here) queue: leased job, no ACK on record -> must redeliver to avoid loss -> work runs twice # "work never ran" and "ACK was lost" leave the queue in the same state
So a queue picks a side. A reliable queue picks no-loss, which forces redeliver-on-missing-ack, which is at-least-once. The duplicate is the price of never losing a job, and it is paid at the acknowledgement gap. In EchoMQ the gap is concrete: a worker holds a job leased on the server clock, and a lease whose deadline passes is reaped back to pending whether the work ran or not — the reaper cannot tell the two histories apart.
Two histories produce the same “no acknowledgement on record” state, indistinguishable from the outside; choosing no-loss forces redeliver-on-missing-ack, which is at-least-once.
Exactly-once effect is the real target
reliable-queue.md.txt · the idempotency lineExactly-once delivery is unreachable; exactly-once effect is reachable, and it is what the phrase should mean. Hold at-least-once delivery — never lose a job — and make the consumer idempotent so a redelivery changes nothing the second time. The effect happens once even though the delivery may happen twice. Systems that advertise “exactly-once” deliver this: at-least-once transport plus an idempotent or deduplicating consumer. The honest framing moves the guarantee from the delivery to the effect, where it can actually hold.
This reframing is not a downgrade. It is more honest and it is more robust, because it puts the duplicate-handling where the duplicate is observable — at the effect, keyed by the job's identity — rather than pretending a transport layer can resolve a lost acknowledgement.
Move the guarantee from the delivery to the effect: at-least-once delivery plus an idempotent consumer is the exactly-once that actually holds.
Three moments, none of them exactly-once delivery
echo_mq/jobs.ex · enqueue · completeThe sharpest error in this area is to point at a deduplication check and call it exactly-once. EchoMQ has three real mechanisms across a job's life, and not one of them is exactly-once delivery, because no mechanism can be. Hover or select a moment below to read what it does and what it does not.
Admission, at enqueue. EchoMQ.Jobs.enqueue/4 refuses a duplicate branded
JOB id — if redis.call('EXISTS', KEYS[1]) == 1 then return 0, the host reads
that 0 as {:ok, :duplicate}. It collapses two enqueues that carry the same id to one
row. It says nothing about a single job being delivered twice by the recovery path.
The fence, at settlement. complete/5 and retry/7 read the job's
attempts value and refuse a token mismatch (EMQSTALE), so a worker whose lease was
reaped and re-claimed by another worker cannot settle the job out from under its new owner. That fence keeps
two workers from corrupting one job's state; it does not stop the job from being delivered twice.
Idempotency, at the effect. The consumer's own marker or naturally idempotent effect absorbs a recovery-path redelivery. This is the only one of the three that makes the effect exactly-once — and it lives in the consumer, not the transport.
Static verdicts — admission collapses duplicate enqueues · the fence keeps one lease-holder · only the effect's idempotency makes the effect exactly-once.
# the three moments — none of them is exactly-once delivery enqueue: Jobs.enqueue -> EXISTS KEYS[1] == 1 ? return 0 (:duplicate) # admission: collapse same-id enqueues settle: Jobs.complete -> attempts token mismatch ? EMQSTALE # fence: only the lease-holder settles effect: the consumer's own idempotency (a marker, or a natural no-op) # absorb a recovery-path redelivery
Admission collapses duplicate submissions; the fence keeps one lease-holder; only the consumer's idempotency applies one delivery's effect. None is exactly-once delivery, because no mechanism can be.
In EchoMQ — admission and the fence are not exactly-once
echo/apps/codemojex · ScoreWorkerThe grounding is precise, and it is the discipline of this whole module. EchoMQ's branded-id refusal is real
producer-side admission, and the attempts token is a real settlement fence; both are useful and
neither is consumer exactly-once. The exactly-once effect a reliable queue ships is the consumer's,
made by an effect-time marker or a naturally idempotent effect — as the previous dive built with the
codemojex scorer, where a guess for an unknown game answers :ok and a re-scored guess writes the
same GES.
Exactly-once delivery is impossible because the acknowledgement can be lost; build an exactly-once effect on at-least-once delivery and an idempotent consumer.
The branded JOB id collapses duplicate enqueues and the attempts token
fences a settlement — neither is consumer exactly-once; the exactly-once effect is
Codemojex.ScoreWorker's, made by an idempotent GES write.
Notes on ValkeyThe duplicate-admission guard is one
EXISTS KEYS[1] inside the atomic EVAL, so the existence check and the row write happen
on the server in one indivisible step — but no engine command can confirm a lost acknowledgement, which is
why admission is not exactly-once. valkey.io/commands/exists
EchoMQ's branded id is real admission and the attempts token is a real fence; neither
is consumer exactly-once, and no mechanism on the producer makes the delivery exactly-once.
Admission, the fence, the lifecycle
The full recovery coordination — the lease on the server clock, the attempts fence, the
reaper, and the acknowledge/finish lifecycle — is the EchoMQ Queue pillar. This dive
teaches why exactly-once delivery cannot exist; that pillar teaches the lifecycle that makes the effect
exactly-once anyway.
References
sources · relatedSources
- Redis — SETthe
NXmarker behind the consumer-side exactly-once effect - Valkey — EXISTSthe existence check the enqueue script runs to refuse a duplicate branded id
- Sanfilippo, S. — RPOPLPUSH, the reliable queuethe design note for the reliable-queue move, by the creator of Redis
- Redis — Redis queuethe queue and its delivery-guarantee framing
Related in this course
- R3.02 · At-least-oncethe module hub
- R3.02.1 · At-least-once semanticsthe three guarantees and the structural duplicate
- R3.02.2 · Idempotent consumersthe exactly-once effect, built
- R3 · The reliable queuethe family in one place
- /echomq/queueadmission, the fence, and the lifecycle in depth