jonnify·redis /redis-patterns/queues/at-least-once

R3.02 · Reliable Queues

At-least-once.

Guarantee at-least-once message delivery using LMOVE to atomically transfer messages to a processing list, enabling recovery if consumers crash before completing work.

R3.01 built the move that loses no job: a worker leases a job into an in-flight position, so a crash leaves it parked and recoverable. That move makes redelivery possible — and that is the point. The consequence is precise: a reliable queue delivers a job one or more times, never zero. A finish-then-crash before the acknowledgement makes a duplicate inevitable, so the responsibility for absorbing it moves to the consumer, which must be idempotent. The grounding is the real EchoMQ worker path in echo/apps/echo_mq: the branded JOB id is the idempotency key, and the worked consumer is codemojex.

Delivered N, effect once Recovery from failures Delivery guarantees In EchoMQ Dives References
§1

Delivered N times, effect once

echo/apps/codemojex · ScoreWorker

A reliable queue can hand the same job to a worker more than once. What separates a correct consumer from a broken one is whether the effect stays at one no matter how many times the job arrives. The figure below replays one guess job up to three times against two consumers: an idempotent one that scores the same emojis to the same GES and counts the attempt under a marker, and a non-idempotent one that re-counts on every delivery. Hover or select a consumer; the readout reports deliveries and effects side by side.

Replay one job N times. Select a consumer and a delivery count; the readout runs the replay and counts the effects.
2
deliveries of one guess JOB deliver 1 count deliver 2 skip deliver 3 2 deliveries → 1 effect the marker absorbs the redeliveries · the attempt counts once
readout · Select a consumer to replay one guess job and count the effects.

Static verdicts — idempotent → N deliveries, 1 effect (the marker absorbs the redeliveries) · non-idempotent → N deliveries, N effects (every delivery re-counts).

Delivered N times, effect once. A reliable queue is at-least-once; an idempotent consumer is what turns that into an exactly-once effect.

§2

Recovery from Failures

reliable-queue.md.txt · Recovery from Failures

A separate monitor process — a reaper — periodically scans the in-flight set for stalled jobs. For each job that has sat there longer than its lease, it treats the worker as dead and returns the job to the main queue. R3.01 grounded this in EchoMQ's stalled sweep. The recovery path is what makes the queue reliable, and it is also what makes redelivery inevitable.

The window is precise. A worker leases a job, applies the effect — charges a card, writes a row, sends a mail — and then crashes before it can acknowledge the job. The effect already happened; the acknowledgement did not. The reaper finds a job whose lease expired, returns it to the main queue, and a second worker runs it again. The effect runs twice.

This guarantees “at-least-once” delivery — messages may be processed multiple times but are never lost.

That sentence from the source is the whole bargain. The dependency chain is exact: R3.01's in-flight move makes redelivery possible (that is the design); a finish-then-crash before the acknowledgement makes it inevitable; therefore the consumer must be idempotent. There is no fourth option that removes the duplicate without reintroducing the loss.

The recovery path that never loses a job is the same path that can redeliver one. Keep the recovery, accept the duplicate, and move the work to the consumer.

§3

Delivery Guarantees

reliable-queue.md.txt · Delivery Guarantees

This pattern provides at-least-once delivery, which is two properties stated together. Messages are never lost — a job always exists in some place (the pending set, the active set), so no crash drops it. Messages may be processed multiple times — a worker that crashes after the effect but before the acknowledgement leaves the job leased, and recovery runs it again.

The three guarantees that bound the design space are worth naming together.

GuaranteeWhen the job leavesFailure outcome
At-most-oncebefore the work (RPOP)a crash loses the job — zero deliveries
At-least-onceafter the acknowledgement (lease, then complete)a crash redelivers — one or more deliveries
Exactly-once—not a deliverable guarantee — build it as an idempotent effect

A reliable queue must keep the job until it is acknowledged, so it is at-least-once — not a defect of the implementation, but the only point on the spectrum that loses no job.

For exactly-once semantics, make message handlers idempotent — processing the same message twice should produce the same result as processing it once.

This is where the work moves to the consumer. Idempotency converts at-least-once delivery into an exactly-once effect. Some effects are naturally idempotent — a SET key value overwrites to the same value however many times it runs. Others are not — an INCR, a charge, a row insert — and those need a guard: a marker claimed with SET … NX keyed by a stable id, so the second delivery finds the marker and skips.

A reliable queue must keep the job until it is acknowledged, so it is at-least-once — the only point on the spectrum that loses no job.

§4

In EchoMQ — the branded JOB id is the idempotency key

echo_mq/jobs.ex · enqueue

EchoMQ's idempotency starts at the identity, not at a side key. A job is a branded JOB entity, and its identity is the key: EchoMQ.Jobs.enqueue/4 runs one idempotent Lua script that admits by kind, refuses a duplicate id, and writes the row and the pending entry atomically. The duplicate refusal is a single line, and the host maps its 0 to {:ok, :duplicate}. Two enqueues that carry the same branded JOB id collapse to one row, because the id is the key, gated at EchoMQ.Keyspace.job_key/2.

# EchoMQ.Jobs.enqueue/4 — one idempotent script; the branded JOB id is the key (real)
@enqueue Script.new(:enqueue, """
  if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
    return redis.error_reply('EMQKIND job id must be JOB-namespaced')
  end
  if redis.call('EXISTS', KEYS[1]) == 1 then
    return 0                                  -- duplicate id: refuse → host reads {:ok, :duplicate}
  end
  redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
  redis.call('ZADD', KEYS[2], 0, ARGV[1])
  return 1
  """)

That refusal is producer-side admission: it stops two enqueues of the same id from both creating work. It is not consumer exactly-once, because the recovery path can still redeliver a job that entered the queue once — a worker that leases a job, applies the effect, and crashes before the acknowledgement loses its lease, and the reaper returns the job to pending for another worker. The consumer absorbs that redelivery by being idempotent: the worker loop hands the handler the job and, on :ok, calls EchoMQ.Jobs.complete/5 with the attempts token; the token is the per-claim fence, so only the current lease-holder may settle the job. Producer admission and consumer idempotency are two disciplines protecting two moments — the enqueue and the effect.

the pattern

A reliable queue is at-least-once; the consumer must be idempotent. The exactly-once effect is the consumer's to build — a naturally idempotent effect, or a marker claimed at effect-time.

in EchoMQ & codemojex

The branded JOB id refuses a duplicate enqueue (Jobs.enqueue → :duplicate), while Codemojex.ScoreWorker absorbs a duplicate delivery — it scores a guess to the same GES and answers :ok for an unknown game, a drop, never a retry loop.

EchoMQ's branded id refuses a duplicate submission; an idempotent consumer absorbs a duplicate delivery. Two disciplines, two moments.

Notes on ValkeyThe enqueue script's duplicate guard is one EXISTS KEYS[1] inside the atomic EVAL, so the kind check, the existence check, the row write and the pending insert happen on the server in one indivisible step — no window where two enqueues of one id both write. valkey.io/commands/exists

/echomq/queue

The leased state machine, in depth

The worker fetch loop, the lease on the server clock, the attempts fence, and the lanes the consumer drains are the EchoMQ Queue pillar — the emq:{q}: protocol taught end to end. This module teaches the at-least-once bargain; that pillar teaches the engine.

 /echo-persistence

Where a dead-lettered job rests

A job that exhausts its tries moves to the morgue; a trimmed stream of completed work folds into the durable floor — EchoStore.StreamArchive onto CubDB and Tigris. The durability dial is the persistence course.
§5

The three dives

three dives · in order

Each dive takes one face of the bargain, in order: the three guarantees and why the lease-until-ack move yields at-least-once; making the consumer idempotent so a redelivery changes nothing; and the acknowledgement-gap argument that proves exactly-once delivery cannot exist.

built ready to read
R3.02.1built

At-least-once semantics

The three delivery guarantees, why the lease-until-ack move yields at-least-once, and the finish-then-crash-before-ack window. Never lost; possibly twice.
R3.02.2built

Idempotent consumers

A naturally idempotent effect versus a marker for an accumulating count. Worked example: a codemojex guess scored twice writes one GES.
R3.02.3built

Why exactly-once is a lie

The acknowledgement-gap argument: exactly-once delivery is impossible, exactly-once effect is what you build. The branded-id refusal is admission.

After R3.02, the Reliable Queues chapter continues with stalled recovery, the atomic state machine, blocking pickup, and a workshop. Return to the Reliable Queues chapter to continue.

§6

References

sources · related

Sources

  • Redis — SETthe NX option behind an effect-time idempotency marker: claim the key only if absent
  • Redis — RPOPLPUSHthe at-least-once move into the in-flight list that makes redelivery possible
  • Valkey — EXISTSthe existence check the enqueue script runs to refuse a duplicate branded id
  • Redis — Redis queuethe queue and its delivery-guarantee overview

Related in this course