jonnify·redis /redis-patterns/coordination/atomic-updates/lua-for-logic

R2.01.2 · Atomic updates — dive 2

The script is the lock.

A Lua script runs to the end with nothing else interleaved. A multi-step conditional with side effects becomes one indivisible move — the way EchoMQ moves every job between states.

Grounding: echo_mq/lib/echo_mq/jobs.ex · @claim (jobs.ex:126), @complete (jobs.ex:139), @retry (jobs.ex:173) · echo_wire/connector.ex:63 eval/5 (EVALSHA-first). Committed figure: bcs3.3.md.

The atomic move — one script vs a non-atomic sequence. Select a mode to light its path and read whether the job is double-picked.
worker Atoken a1 worker Btoken b2 Valkeyj7 in pending active0 jobs one EVAL — indivisible GET/SET — B double-picks
readout · Select a mode. One Lua script picks the job once; a GET/SET sequence lets a second worker read the same pending state and pick it twice.

Static verdicts — one Lua script → A runs the whole move, B finds j7 gone, active holds 1, double-picked: no · GET/SET sequence → B reads pending between A’s steps, both write, active holds 2, double-picked: yes.

§1

One script, no window

atomic-updates.md.txt · Pattern 2

A WATCH/MULTI/EXEC transaction reads the value outside the server and branches on the client; a Lua script moves the whole read-branch-write into the server. Redis evaluates a script on a single thread to completion before the next command on that connection runs, so a script that reads a key, branches on its value, and writes several keys is one atomic step. No other command lands in the middle.

That is the window a non-atomic sequence cannot close. A bare GET, then a branch, then a SET leaves a gap: between the read and the write, another client can read the same key and act on the same stale value. Both pass the check; both write; one update is lost — or, in a queue, one job is picked twice. The script removes the gap because there is no point inside it for another command to run.

EVAL script numkeys k… a…
Run a Lua script atomically. KEYS are the keys it touches, ARGV the arguments.
redis.call(…)
Issue a command from inside the script. Every call runs within the one atomic execution.
atomic step
The whole script runs to the end with no other command interleaved on the server.
the window
The gap a non-atomic GET…SET leaves, where a second client can act on a value about to change.

The optimistic-locking alternative — WATCH with a retry loop — is the sibling dive, R2.01.1 · WATCH / MULTI / EXEC. This dive uses one Lua script to close the window.

§2

Compare-and-set, then a conditional with side effects

atomic-updates.md.txt · Compare-and-Set

The smallest atomic conditional is compare-and-set: write only if the current value still matches what was read. The GET, the comparison, and the SET are one operation, so the set is conditioned on a value still current at the instant it is written — the property a separate read and write cannot promise.

-- CAS: set only if the current value still matches the expected one
if redis.call('GET', KEYS[1]) == ARGV[1] then
  redis.call('SET', KEYS[1], ARGV[2])
  return 1
end
return 0

The pattern scales past a single key: a script can read a field, branch on it, write the field, and append a log entry — all atomic, so any read returns either the whole before-state or the whole after-state, never a half-applied one. Pick a case below to run the compare-and-set: the readout shows the read value, whether the expected value matched, and whether the SET ran.

Run a compare-and-set — match writes, mismatch holds. Select a case to read the read value, the comparison, and whether the SET ran.
readout · Select a case. A match writes and returns 1; a changed value or an absent key skips the SET and returns 0.

Static verdicts — value matches → GET returns "open", SET runs, returns 1 · value changed → GET returns "locked", mismatch, skip, returns 0 · key absent → GET returns nil, mismatch, skip, returns 0.

When the logic is conditional and touches several keys, the script is the lock: the check and the writes run with no command between them.

§3

How EchoMQ’s transitions are single Lua scripts

echo_mq/jobs.ex:126 · bcs3.3.md

This is the chapter’s central bridge. EchoMQ moves a job between states with a Lua script, because moving a job is a multi-step conditional with side effects: pop the source set, increment a counter, stamp a lease, return the job. EchoMQ.Jobs.claim/3 runs the @claim script (jobs.ex:126) through EchoMQ.Connector.eval/5 — ZPOPMIN the oldest pending id, HINCRBY the attempts counter (the fencing token), set the row active, and ZADD the lease on the server clock, all in one atomic step:

-- @claim (echo_mq/lib/echo_mq/jobs.ex:126) — pop, fence, lease, return; one atomic step.
local popped = redis.call('ZPOPMIN', KEYS[1])
if #popped == 0 then return {} end
local id = popped[1]
local jk = ARGV[1] .. id
local att = redis.call('HINCRBY', jk, 'attempts', 1)
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)
return {id, redis.call('HGET', jk, 'payload'), att}

Three facts live here. The clock is the server’s — redis.call('TIME') inside the script, so leases never see client skew. The token is minted by HINCRBY — monotone per job by construction, the property the fencing argument requires. And the keys are declared in KEYS (KEYS[1] pending, KEYS[2] active); the job key is built from the prefix argument, the one sanctioned exception, safe because every key the prefix can produce shares the queue’s hashtag and lands on one slot.

The other transitions are the same shape and verify that token before they touch anything. @complete (jobs.ex:139) refuses a stale token, then retires the row:

-- @complete (echo_mq/lib/echo_mq/jobs.ex:139) — only the current token holder may retire the row.
local att = redis.call('HGET', KEYS[2], 'attempts')
if not att then return 0 end
if att ~= ARGV[2] then
  return redis.error_reply('EMQSTALE complete token mismatch')
end

@retry (jobs.ex:173) carries the same EMQSTALE retry token mismatch fence and dead-letters a job past max_attempts. The committed manuscript record reads the law in six words: one job, two lives, one counter — the same integer that counts the attempts fences each life against the last.

content/bcs3.3.md · the fence, performing · against Valkeyfrozen
complete, wrong token         EMQSTALE; the lease holder's work survives the zombie's complete
complete, right token          complete with the right token retires the row -- nothing remains
attempts                       one job, two lives, one counter   ← HINCRBY mints the fence

Notes on ValkeyA Lua script runs atomically and reads one frozen server clock via redis.call('TIME'), so the pop, the increment, and the lease land together or not at all — valkey.io/topics/programmability.

§4

EVALSHA, NOSCRIPT, SCRIPT LOAD — the script cache

echo_wire/connector.ex:63 · bcsA.md

Sending a kilobyte of Lua on every call would waste bandwidth. Redis caches scripts by the SHA1 of their body, so a client sends the full source once and a 40-byte hash thereafter. EVALSHA runs a cached script; if it is not in the cache, the server replies NOSCRIPT, the client retries with SCRIPT LOAD (which caches the body) and re-issues the EVALSHA, and every later call hits the cache.

EchoMQ.Connector.eval/5 (connector.ex:63) is that dispatch verbatim: EVALSHA first, and on a NOSCRIPT error, SCRIPT LOAD then re-run by SHA. The SHA1 is precomputed once at EchoMQ.Script.new/2. Run the call sequence below: the first call to a fresh server misses, pays the source once, and caches it; every later call hits and sends only the SHA.

Step the call sequence — first miss loads, rest hit. Run the next call or reset; the readout tallies hits, misses, and bytes sent.

Static verdict — the body is sent once on the first NOSCRIPT (the source); every later call sends only the 40-byte SHA. The committed connector record holds script_loads=1 across a busy session.

content/bcsA.md · the script path, against Valkey on :6390frozen
script path                   script_loads=1   ← exactly one NOSCRIPT load, then EVALSHA from cache
pipelined EVALSHA             161192 ops/s   ← the declared-keys law at speed

The atomicity is EVALSHA running the whole script on the server; the SHA cache is the cost layered on top — the body once, the hash forever after.

§5

The pattern, and its application

echo_mq/jobs.ex
the pattern

A Lua script runs atomically: a read, a branch, and several writes with no command interleaved. One of three Redis read-modify-write tools, alongside WATCH/MULTI/EXEC and the shadow-key swap.

its EchoMQ application

@claim, @complete, @retry (echo_mq/lib/echo_mq/jobs.ex) are the real job transitions — each one inline Lua script, each atomic, the attempts counter the fencing token, so no two workers pick the same job.

When the logic is conditional and touches several keys, the script is the lock; EchoMQ’s transitions prove it.

the door · the EchoMQ course

The full Lua bundle is the next course

EchoMQ’s full Lua bundle — every transition script, the EVALSHA/NOSCRIPT dispatch internals, and the governance that keeps the transitions immutable — is the dedicated EchoMQ course. This dive cites three real transitions as proof the pattern is real; the depth is the next course. The functional-Elixir & OTP craft behind the echo umbrella is the /elixir course.

§6

Recap — the script is the lock

next → R2.01.3

A Lua script runs to the end with nothing interleaved, so a conditional with side effects becomes one atomic step: compare-and-set, the field-plus-log update, the job transition. EchoMQ moves every job between states with one such script — @claim over pending and active, @complete and @retry token-fenced. The SHA cache (EVALSHA → NOSCRIPT → SCRIPT LOAD) sends the body once and the hash thereafter. The next dive swaps a whole value at once instead, and admits a whole batch in one wire flush.

§7

References

sources · related

Sources

Related in this course