jonnify·redis /redis-patterns/queues/stalled-recovery/atomic-vs-non-atomic

R3.03.3 · Stalled recovery — dive · centrepiece

Atomic vs non-atomic — one EVALSHA vs the round-trip loop.

Stalled recovery detects a stalled job and moves it. As one indivisible step inside the server, it is exact. As a sequence of round trips from the application, a second sweep can slip between detect and move and reclaim the same job twice.

Grounding: echo/apps/echo_mq/lib/echo_mq/stalled.ex · @sweep_stalled — ZRANGEBYSCORE + ZREM + ZADD + HINCRBY inside one EVALSHA, the server single thread. The non-atomic loop is a generic app-side contrast.

Two sweeps over one stalled job. Select a mode to see how the two sweeps interleave. In atomic mode the move is one step; in non-atomic mode sweep B slips in between.
sweep A sweep B A: detect+move+count (one EVALSHA) B: empty, A already done A: detect B: detect A: move B: move (again)
readout · Select a mode. Atomic: the move is one EVALSHA, sweep B finds the job already gone — recover count 1. Non-atomic: B slips between A's detect and move — recover count 2.
§1

The atomic form — one EVALSHA

stalled.ex · @sweep_stalled

EchoMQ.Stalled runs the whole recovery as one cached Lua script, dispatched EVALSHA-first. Read, decide, and write happen on the server's single thread, so nothing interleaves between them.

local exp = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, lim)  -- detect
for _, id in ipairs(exp) do
  redis.call('ZREM', KEYS[1], id)                       -- claim it out of active
  local jk = p .. 'job:' .. id
  local st = redis.call('HINCRBY', jk, 'stalled', 1)    -- count in-script
  if st >= maxst then
    redis.call('HSET', jk, 'state', 'dead')
    redis.call('ZADD', KEYS[3], 0, id)                  -- dead
  else
    redis.call('ZADD', KEYS[2], 0, id)                  -- pending
    redis.call('HSET', jk, 'state', 'pending')
  end
end

The ZREM that removes a job from active and the ZADD that puts it in pending are in the same script. A job is never in both sets and never in neither — between the two writes no other command runs. A second sweep that starts while this one is mid-loop does not begin until this EVALSHA returns, because the server runs one script at a time. Each expired lease is reclaimed exactly once per sweep, and the stalled and attempts counters are incremented by the script, never read-modify-written by a client.

The script is the lock. Detect, decide, and move are one step, so there is no window between the check and the move for a second sweep to act in.

§2

The non-atomic form — the round-trip loop

the cautionary contrast

The same recovery, written app-side, is a sequence of separate round trips: list the in-flight set, probe each member's liveness, then move the stalled ones. This is the cautionary contrast — not a strawman, the genuinely common shape of an app-side recovery, and exactly what the atomic Lua exists to avoid.

in_flight = ZRANGEBYSCORE active -inf now      # round trip 1: detect
for id in in_flight:
  if probe_says_stalled(id):                   # round trip 2: re-check
    ZREM active id                             # round trip 3: remove
    ZADD pending 0 id                          # round trip 4: re-queue
    n = HGET job:id stalled                     # read
    HSET job:id stalled (n + 1)                 # increment, write — a lost-update race

Every line is a separate command, and the application's state lives between them. That gap is where correctness leaks. The atomic script collapses all six lines into one indivisible step; the loop leaves five gaps a second sweep can run inside.

Notes on ValkeyA Lua script runs as one indivisible step on the server's single command thread, so two scripts never interleave — the loop above has no such guarantee between its separate commands — redis.io/commands/eval.

§3

The double-recovery window

the reliable-queue core

Two sweeps run on a cadence; on a busy queue they overlap. In the non-atomic loop, sweep A reads the in-flight set and finds job J stalled. Before A reaches its ZREM, sweep B also reads the set and also finds J stalled. Now both move it: A removes it from active and pushes it to pending; B removes it again (a no-op) and pushes it again. J is now in pending twice — claimed twice, run twice.

The same gap corrupts the count: A reads stalled = 2, B reads stalled = 2, both write 3, and one increment is lost — a job that has stalled four times records two. The atomic form has no window: a single sweep's detect-and-move is one script; a concurrent sweep cannot start mid-loop, and the count is bumped in-script on the value the script itself read.

Where the count is incremented. Select a form to read what the stalled field reaches after two overlapping sweeps over one job.
in-script client-side HINCRBY x2 → 2 correct, no lost update read 0, read 0 → 1 one increment lost
readout · Select a form. The in-script HINCRBY reaches the correct count; the client read-modify-write loses an increment when two sweeps read the same value.
the pattern

The detect-and-move must be one indivisible step, or two overlapping sweeps redeliver the same job and lose an increment on the attempt counter.

its EchoMQ application

@sweep_stalled is that one step: ZRANGEBYSCORE + ZREM + ZADD + HINCRBY inside one EVALSHA, on the server's single thread; the count is bumped in-script, never read-modify-written by a client.

Atomic recovery is not a performance choice; it is the only form that survives two sweeps running at once.

The atomic sweep that recovers a job is the same one that dead-letters it: at max_stalled the job lands in emq:{q}:dead in the same script that detected it. From there its history is durable. When the bus trims the stream, EchoStore.StreamArchive folds the trimmed segments into the Graft floor and on to Tigris — so a job that exhausted its recoveries is kept, not lost.

/echomq/queue

The script dispatch, in depth

The Queue pillar teaches the EVALSHA cache, the inline Script.new/2 Lua, and the state machine the sweep is one transition of.

 /echo-persistence

Where a dead-lettered job lands

The durability dial: EchoStore.StreamArchive folds trimmed stream segments into the Graft floor on CubDB and on to Tigris — the floor a given-up job's history reaches.
§4

References

sources · related

Sources

  • Redis — EVALSHA — the cached script run as one indivisible step; the dispatch the connector uses first.
  • Redis — EVAL / scripting — the server runs one script at a time, so two sweeps serialize.
  • Valkey — ZRANGEBYSCORE — the detect read at the top of both forms; only the atomic form fences the move that follows it.
  • Valkey — HINCRBY — the in-script increment that avoids the lost-update race a client-side read-modify-write opens.

Related in this course