jonnify·redis /redis-patterns/queues/atomic-state-machine/read-decide-write-in-one-evalsha

R3.04.2 · Atomic state machine — dive 2

Read-decide-write in one EVALSHA.

The completion is a read-modify-write: read the fencing token, decide where the job goes, write the move. Split into round trips, a sweep slips between read and write; run as one EVALSHA, it is indivisible.

Grounding: echo_mq/jobs.ex · the inline @complete / @retry / @claim scripts (EchoMQ.Script.new/2) · EchoMQ.Connector.eval/5 (one EVALSHA each) — the row's attempts the fencing token. Engine: Valkey.

Two ways to finish a job, with a reaper running. Select a step to light it; the readout names the outcome.
completing worker reaper (EchoMQ.Stalled) HGET attempts reaper re-queues ZREM + DEL row select a step
readout · Select a step. As separate commands, the reaper interleaves between the worker's read and write; as one EVALSHA, the whole read-decide-write runs before any other command.

Static verdicts — split → the reaper re-queues the job between the read and the write, so it is handled twice · one EVALSHA → the script runs to completion first, so the job is already retired before the reaper runs — handled once.

§1

The transition is a read-modify-write

atomic-updates.md.txt · the race

Completing a job is not one write. It is a read, a decision, and several writes:

read
the row's attempts token. A worker whose lease was reaped and re-claimed by another holds a stale token, and must not complete a job the new holder now owns.
decide
does the token still match? On a retry, is the attempt past max — schedule again, or dead-letter?
write
remove the id from active, record the result or error on the row, move the row's state, bump the metric.

That is a read-modify-write across more than one key. Done as separate commands, the gaps between them are open. A lease-expiry sweep reads the same job still in active past its deadline and returns it to pending for redelivery — between the completing worker's read and its write. The job is retired by one process and re-queued by the other, or its lease is released by one while the other still treats it as held. At-least-once becomes a mess by accident.

A queue completion reads the token, branches on it, and writes across several keys — a read-modify-write the reaper can interleave unless it is one step.

§2

One EVALSHA closes every gap

jobs.ex · @complete

A Lua script runs to completion before Valkey serves the next command. So folding the whole read-decide-write into one script and dispatching it with a single EVALSHA removes every gap. The read, the decision, and the writes are one indivisible step; no sweep and no second completion can interleave. In EchoMQ each transition is one inline script, declared with EchoMQ.Script.new(name, lua) and dispatched by EchoMQ.Connector.eval/5. The @complete body is the read-decide-write in one call:

# @complete (jobs.ex), trimmed — read the token, decide, write, all in one EVALSHA
local att = redis.call('HGET', KEYS[2], 'attempts')   # read the fencing token
if not att then return 0 end                          # the row is gone
if att ~= ARGV[2] then                                # decide: a stale holder is refused
  return redis.error_reply('EMQSTALE complete token mismatch')
end
local was_active = redis.call('ZREM', KEYS[1], ARGV[1])  # write: release the lease
redis.call('DEL', KEYS[2])                               # retire the row

Every key the script touches is declared in KEYS (the v2 law) — KEYS[1] the active set, KEYS[2] the row — both braced emq:{q}: keys built host-side by EchoMQ.Keyspace, so they share one slot. ARGV carries values only.

The read, the branch, and every write are one EVALSHA — the script is the lock, and the reaper cannot slip between the worker's read and its write.

§3

The fencing token

jobs.ex · @claim, @retry

The decision that makes the transition safe under redelivery is the token check. claim/3 mints the token by HINCRBY-ing the row's attempts when it leases the job; the worker carries that value, and every later transition passes it as ARGV[2]. A worker whose lease was reaped and re-claimed holds the old value, so its @complete or @retry reads a mismatched token and is refused with EMQSTALE token mismatch — it changes nothing. The completing worker and the reaper cannot both move the same job, because the script reads the token inside the same atomic step that does the writes.

Pick a token below and run the completion. A matching token retires the job; a stale token — a worker whose lease was reaped and re-claimed, so its attempts moved on — is refused, and the job is untouched.

Run @complete with a token. The row's current attempts is 2; select the token the worker holds.

Static verdicts — token 2 → matches attempts, the job is retired · token 1 → mismatch, EMQSTALE complete token mismatch, the job is untouched.

@retry carries the same fence and adds the branch: below max it sets state = scheduled and ZADDs the row into schedule on the server clock; at max it sets state = dead and ZADDs the row into dead, bumping metrics:failed. One read, one decision, the writes — one EVALSHA.

Notes on ValkeyA script runs to completion in the single command thread, so a token read and the writes that depend on it cannot be split by another client — valkey.io/commands/eval.

the application · the token-fenced retry

Below max, schedule; at max, dead-letter

The @retry body branches on the same token, then on the attempt count — the real morgue transition:

echo/apps/echo_mq/lib/echo_mq/jobs.ex · @retry, the branchfrozen
if att ~= ARGV[2] then
  return redis.error_reply('EMQSTALE retry token mismatch')
end
if tonumber(att) >= tonumber(ARGV[4]) then        ← at max attempts
  redis.call('HSET', KEYS[4], 'state', 'dead')
  redis.call('ZADD', KEYS[3], 0, ARGV[1])         ← into the dead set
  return 'dead'
end
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[3]), ARGV[1])   ← else reschedule
the pattern

A multi-key read-modify-write must be one server-side script, or concurrent clients interleave. Fold the read, the decision, and every write into one Lua call.

its EchoMQ application

Each transition is one inline EchoMQ.Script.new/2, dispatched by EchoMQ.Connector.eval/5 as one EVALSHA; the row's attempts fencing token, minted by claim/3, makes the script idempotent so a stale holder cannot move the job.

One EVALSHA per transition, with the attempts token read inside it — the worker and the reaper cannot both finish one job.

§4

Recap — one step, one token

next → R3.04.3

Each lifecycle transition folds its read, decision, and writes into one inline EchoMQ.Script.new/2, run as a single EVALSHA; the row's attempts fencing token keeps a stale holder from finishing the job twice. The next dive shows how that cached script reaches the server — by SHA, with a NOSCRIPT fallback.

§5

References

sources · related

Sources

Related in this course