jonnify·redis /redis-patterns/queues/at-least-once/idempotent-consumers

R3.02.2 · At-least-once — dive 2

Idempotent consumers.

Score twice, scored once. At-least-once delivery hands the consumer a job that may arrive again; an idempotent effect runs once however many times the job arrives.

The fix for a duplicate delivery is not exactly-once transport — that is impossible — but an idempotent effect: running the same job twice produces the same result as once. A codemojex guess is the cleanest case: scoring the same emojis writes the same GES, and the one accumulating part, the per-game attempt count, takes a marker. Select a delivery count below, then read the GES write and the attempt count hold.

Score one guess, then replay. Select a delivery count; the readout runs the replay and reports the GES write and the attempt count.
GES write (game + player — naturally idempotent) 1 GES attempt count (NX marker guards INCR) 1 delivered 1× · one GES written · one attempt counted score twice = scored once:the GES write absorbs the redelivery, the marker guards the count

Static verdicts — any delivery count → 1 GES (the write is naturally idempotent) and 1 attempt (the marker guards the count).

§1

Naturally idempotent effects

reliable-queue.md.txt · the idempotency line

An effect is idempotent when applying it twice changes nothing the second time. SET key value is the clearest case: it overwrites to the same value however many times it runs, so a redelivered job that sets the same value is harmless. Set membership is another — adding a member already in the set is a no-op. Writing a row by a fixed primary key with an upsert is idempotent; so is sending mail keyed by a dedup id the provider honours.

A naturally idempotent consumer needs no extra machinery. The redelivery runs the effect again, the effect produces the same state, and at-least-once delivery has already become an exactly-once effect at no cost. The discipline is to recognise which effects are naturally idempotent and lean on them — choosing a SET over an INCR, an upsert over a blind insert — before reaching for a marker.

# naturally idempotent — a redelivery changes nothing
SET game:G:secret <value>       # run once or twice: the secret is the same either way
SADD room:R:players PLR0...      # add a member already in the set: a no-op

The cheapest idempotency is the effect that is already idempotent: a SET or a set add absorbs a redelivery for free. Reach for those before reaching for a marker.

§2

Non-idempotent effects need a marker

redis.io/commands/set · NX

The hard cases are effects where the second run does change the state. INCR counter is the canonical one — two deliveries increment it twice. A card charge, a row insert with a fresh id, an outbound message without a dedup id: each accumulates on every redelivery. A non-idempotent consumer double-counts on the duplicate the previous dive proved is structural.

The guard is a marker keyed by a stable id. Before the effect runs, claim the marker with SET … NX — set only if absent. If the claim succeeds, this is the first delivery: run the effect. If the claim fails, the marker is already set, so the effect already ran on an earlier delivery: skip it. The marker carries the exactly-once decision that the effect itself cannot.

Hover or select an effect below to compare a naturally idempotent SET against an accumulating INCR over two deliveries; the readout names where the marker is needed.

Two deliveries · SET versus INCR. Hover an effect; the readout runs both deliveries and reports the end state.
SET key v (naturally idempotent) deliver 1: v · deliver 2: v = v · idempotent for free INCR counter (accumulates) deliver 1: +1 · deliver 2: +1 counter = 2 a SET is safe to redeliver as-is; an INCR needs the NX marker to be redeliver-safe

Static verdicts — SET → same value both deliveries, no marker · INCR with no marker → counter = 2 · INCR under an NX marker → counter = 1.

# a non-idempotent effect, guarded by an NX marker keyed by the stable job id
if SET seen:JOB0... 1 NX PX 86400000:    # claim the marker; NX = only if absent
    count_the_attempt()                  # first delivery: count once
else:
    skip()                               # a redelivery: the marker is set, do nothing

A naturally idempotent effect is redeliver-safe as-is; an accumulating one is not, and an NX marker keyed by the job id is what makes it so — best-effort, not a transaction.

§3

The worked example — a codemojex guess

echo/apps/codemojex · ScoreWorker

A codemojex guess is the cleanest illustration. A player submits six emojis; Codemojex.Guesses.submit/3 mints a branded JOB id, charges the wallet, and enqueues the job on the player's lane (Lanes.enqueue). The scoring consumer is the authority: Codemojex.ScoreWorker.handle/1 drains the lane, reads the game's secret through the cache, scores the six emojis with the pure linear engine, and writes a GES guess. The scoring itself is naturally idempotent — the same secret and the same six emojis produce the same score every time the job runs.

The one part that accumulates is the per-game attempt counter, an INCR on cm:<game>:attempts. Counting an attempt on every redelivery would over-count. That is the INCR-shaped effect, and a marker keyed by the stable JOB id is what makes it redeliver-safe: claim seen:<JOB> with SET … NX; on the first delivery the claim succeeds and the attempt is counted once; on a redelivery the claim fails and the count is skipped. The score rides on its natural idempotency; the attempt count rides on the marker. Score twice, scored once.

ScoreWorker carries the discipline at its boundary too: a guess for an unknown game answers :ok — a drop, never a retry loop. A redelivery that no longer maps to a live game settles cleanly rather than looping, because the consumer treats the same identity the same way each time it handles it.

# Codemojex.ScoreWorker.handle/1 — score the guess, write a GES (real, trimmed)
def handle(%{id: job_id, payload: payload, group: player}) do
  {:guess, game, ^player, emojis} = :erlang.binary_to_term(payload)
  case Cache.fetch_game(game) do
    %{secret: secret} = g ->
      s   = Scoring.score(secret, emojis)            # naturally idempotent: same secret + emojis -> same score
      gid = EchoData.BrandedId.generate!("GES")
      Store.put_guess(gid, %{game: game, player: player, emojis: emojis, points: s.total})
      Cmd.incr("cm:" <> game <> ":attempts") |> Wire.run(conn)   # accumulating — guard with a marker
    _ -> :ok                                          # unknown game: a drop, never a retry loop
  end
end
the pattern

Absorb at-least-once redelivery by making the consumer's effect idempotent — natural idempotency where the effect allows it, an NX marker where it does not.

in a codemojex guess

The scoring is naturally idempotent (same secret and emojis, same GES), the per-game attempt count is guarded by an NX marker keyed by the JOB id, and an unknown game answers :ok — so scoring a guess twice equals scoring it once.

Score twice, scored once: the GES write absorbs the redelivery, the marker guards the attempt count, and an unknown game settles to :ok rather than looping.

§4

A note on producer-side admission

echo_mq/jobs.ex · enqueue

EchoMQ refuses a duplicate at enqueue, and it is a different mechanism from the consumer marker above. EchoMQ.Jobs.enqueue/4 runs one idempotent script whose first guard after the kind check is if redis.call('EXISTS', KEYS[1]) == 1 then return 0 end — two enqueues of the same branded JOB id collapse to one row, and the host reads the 0 as {:ok, :duplicate}. That is producer-side admission; it stops two enqueues of the same id from both creating work. It does not make a single in-flight job exactly-once at the consumer, because the recovery path can still redeliver a job that entered the queue once. The consumer marker in this dive is claimed at effect-time, keyed by the job's identity, and is what absorbs that redelivery. Two markers, two moments — the third dive separates them in full.

Notes on ValkeySET … NX PX <ms> claims the marker and gives it an expiry in one command, so the guard cannot leave an immortal key if the consumer dies between two commands. valkey.io/commands/set

/echomq/queue

The lifecycle the consumer settles into

The acknowledge/finish lifecycle that closes an in-flight job out — Jobs.complete, the attempts fence, the lane the worker drains — is the EchoMQ Queue pillar. This dive teaches the idempotent consumer; that pillar teaches the lifecycle the consumer acknowledges into.
§5

References

sources · related

Sources

Related in this course