jonnify·redis /redis-patterns/queues/states-as-locations

R3 · dive 2

States as locations.

A job's state is the emq:{q}: key its branded JOB id lives in — and a transition moves the id to the next key, in one atomic Lua move.

A naive queue stores a status column and updates it as the job moves, so the location and the status can disagree after a crash. EchoMQ holds the identity in a Valkey location and keeps a matching state field on the row, written in the same atomic move. EchoMQ.Keyspace builds every location as emq:{q}:<type> and gates the branded JOB id at the key builder; EchoMQ.Jobs holds the verbs. Fire a transition below and watch the id hop from one emq:{q}: key to the next.

The job id hops from one emq:{q}: key to the next. Hover or select a transition; the readout names the key the id now lives in and its type.
emq:{cm}:pendingZSETJOB emq:{cm}:activeZSET emq:{cm}:scheduleZSET emq:{cm}:deadZSET done · row retiredDEL
§1

The state is the location

echo_mq/keyspace.ex

A naive queue stores a status column on the job record and updates it as the job moves. Two facts can then disagree: where the job actually sits and what its status claims. EchoMQ holds the job's identity in a Valkey location and keeps a matching state field on the row, written in the same atomic move — so the two can never drift apart.

The pending set emq:{q}:pending is a score-0 sorted set, so byte order is mint order; the active set is scored by each member's lease deadline; the schedule set by a future run-at; the dead set holds terminal ids; and the row hash carries the state, attempts, and payload fields:

# a job's state is the emq:{q}: key its branded JOB id lives in
emq:{cm}:pending          # ZSET · score 0, byte order is mint order
emq:{cm}:active           # ZSET · scored by lease deadline (server TIME)
emq:{cm}:schedule         # ZSET · scored by a future run-at
emq:{cm}:dead             # ZSET · terminal, dead-lettered ids
emq:{cm}:job:JOB0Nt…      # HASH · state, attempts, payload

EchoMQ.Keyspace.queue_key/2 builds each location as emq:{q}:<type>, with the queue name braced as the cluster hash-tag so every key of one queue lands on one slot. EchoMQ.Keyspace.job_key/2 builds the row key and gates the branded JOB id at the key builder — it raises if the id is not a valid branded id, so an ill-formed identity never reaches the keyspace. The hero figure runs exactly this: fire a transition and the id hops to its new key.

A job's state is not a field that can lie about where the job is — it is the emq:{q}: key the gated JOB id lives in, with a matching row field written in the same move.

§2

The whole transition is one atomic Lua move

echo_mq/jobs.ex · @complete

Moving a job from active to done is not one write. The transition reads the row's attempts to fence the caller, checks the token, removes the id from the active set, retires the row, and bumps a metric — touching several keys. Done as separate commands, a crash partway leaves the job in two places, or in none.

EchoMQ runs each transition as one inline EchoMQ.Script.new/2 body, dispatched by EchoMQ.Connector.eval/4 as a single EVALSHA (EVALSHA-first, falling back to EVAL on NOSCRIPT). EchoMQ.Jobs.complete/5 is the active→done move — all in one script that runs to completion with no other client interleaving, so it applies in full or not at all. This is the R2.01 atomic-update pattern at lifecycle scale.

-- @complete (inline) — active → done, token-fenced, in one EVALSHA
local att = redis.call('HGET', KEYS[2], 'attempts')   -- KEYS[2] = the row
if not att then return 0 end
if att ~= ARGV[2] then
  return redis.error_reply('EMQSTALE complete token mismatch')
end
redis.call('ZREM', KEYS[1], ARGV[1])              -- KEYS[1] = emq:{q}:active
redis.call('DEL', KEYS[2])                        -- retire the row
redis.call('HINCRBY', ARGV[3] .. 'metrics:completed', 'count', 1)
return 1
One EVALSHA vs N round-trips — where a crash tears the lifecycle. Select how the transition is issued.
one EVALSHA · 1 round trip EVALSHA complete/5 → fence · ZREM · DEL · metric all-or-nothing · no torn window N separate commands · 4 round trips HGET attempts · ZREM active · DEL row · HINCRBY metric runs clean when nothing fails between the commands

A lifecycle move touches several keys, and a queue survives a crash only if that move is all-or-nothing — one inline Lua script run as one EVALSHA, not a sequence a crash can split.

§3

Every key is declared, gated, and on one slot

echo_mq/keyspace.ex · job_key/2

The atomic move has a discipline behind it. Every key the script touches is passed in KEYS[] — the active set and the row are declared keys, not derived inside the script from a data value. Every key of one queue carries the same {q} brace, so the whole transition lands on one cluster slot and Valkey will run the multi-key script. And the branded JOB id is gated at EchoMQ.Keyspace.job_key/2 before any key is built. The script is the protocol: the caller sends the SHA and the keys, the body lives in Valkey, and the transition is indivisible.

Notes on Valkeya Lua script may only touch keys in KEYS[] that hash to one slot — the braced emq:{q}: keyspace puts every key of one queue on the same slot, which is what makes the multi-key transition legal under cluster mode — redis.io cluster spec · hash tags.

§4

In EchoMQ — the lifecycle the worker runs

echo/apps/echo_mq

The pattern and the application line up directly. A job's state is its emq:{q}: location plus a matching state field written in the same move; every transition is one inline EchoMQ.Script.new/2 body run by EchoMQ.Connector.eval/4 as a single EVALSHA; and the branded JOB id is gated at the key builder. EchoMQ.Jobs holds the verbs — claim/3 (pending→active), complete/5 (active→done), retry/7 (active→schedule or →dead), reap/2 (active→pending), promote/3 (schedule→pending).

the pattern

A job's state is its Valkey location, the whole transition is one atomic Lua move that cannot tear, and every key it touches is declared and on one slot.

its EchoMQ application

EchoMQ.Keyspace builds the gated emq:{q}: locations; each EchoMQ.Jobs verb is one inline Script.new/2 body run by EchoMQ.Connector.eval/4 as a single EVALSHA, token-fenced and all-or-nothing.

States are emq:{q}: locations, a transition is one atomic Lua move run by SHA, and every key is declared and gated — the reliable lifecycle EchoMQ's worker runs.

→ EchoMQ · /echomq/queue

The state machine and the inline Lua

The full lifecycle — every transition's script, the fencing-token protocol, the schedule and dead sets, fair lanes — is the dedicated EchoMQ Queue pillar. This dive teaches the pattern; that course teaches the engine that runs it.

/echomq/queue · the EchoMQ Queue pillar

§5

References

sources · related

Sources

  • Redis — Scripting with Lua — a script runs to completion with no interleaving, so a transition applies in full or not at all.
  • Redis — EVALSHA — run a cached script by its SHA; a NOSCRIPT reply falls back to EVAL, the connector's EVALSHA-first dispatch.
  • Valkey — ZREM — removes the id from one state set, the move out of a location.
  • Redis — Cluster key hash tags — the {q} brace that lands every key of one queue on one slot, so a multi-key script is legal.

Related in this course