The record hash · dive 01 · the field set
The hash and its fields
One write builds the row. Every field after it has a named writer.
The job record is one Valkey HASH, addressed by
EchoMQ.Keyspace.job_key/2 at emq:{q}:job:<id> and born
from exactly one HSET — the same one on either executor:
@enqueue in Lua and EMQ.ENQUEUE in the native module. The field
list is the record's shape, fixed and named, so any speaker reads each field by its name with no schema
lookup. Pick a field on the right to read its role, its birth value, and its writers on each
executor.
The field set · pick a field
One hash, a fixed field set
A job is an entity, and its state is one Valkey hash. Admission builds it with exactly one write, and the two
executors write the same one: @enqueue in Lua and EMQ.ENQUEUE
in the native module both run HSET <row> state pending attempts 0 payload
<payload>. There is no id field, because the id is the key; no
created_at, because the branded id is itself time-ordered; no second copy of anything.
The birth row is the smallest row that carries a job.
What makes the field set part of the protocol, not an implementation detail, is that every transition reads and
writes the row by these names, on either executor. When a worker claims the job, the claim reads
payload by that name; when a completion is fenced, it reads
attempts by that name. The rest of the field set, with its writers on each executor:
| Field | Role | Scripted executor | Native module |
|---|---|---|---|
state | the lifecycle position | @enqueue pending; @schedule scheduled; @claim2 active; @retry scheduled or dead; @delay scheduled; @promote and @reap pending; the stall sweep pending or dead; @reprocess2 pending; a flow parent is held awaiting_children | the same values from EMQ.ENQUEUE, EMQ.SCHEDULE, EMQ.CLAIM, EMQ.FAIL, EMQ.DELAY, EMQ.PROMOTE, EMQ.STALLED.CHECK, EMQ.REQUEUE, EMQ.REPROCESS |
attempts | the fencing token | '0' at admission; @claim2 runs HINCRBY attempts 1 | '0' at admission; only EMQ.CLAIM increments it |
payload | the opaque cargo | ARGV[2] at admission; update_data/4 replaces it | ARGV[2] at admission |
last_error | the latest failure | @retry on every failure; the stall sweep's death writes 'stalled'; @reprocess2 clears it | EMQ.FAIL; EMQ.STALLED.CHECK's death writes 'stalled'; EMQ.REPROCESS clears it |
stalled | the stall count | the stall sweep (EchoMQ.Stalled), on every expired lease | EMQ.STALLED.CHECK, on counted expiries only |
progress | the progress value | update_progress/4 | no native verb |
group | the lane | lanes (EchoMQ.Lanes) | none: EMQ.COMPLETE, EMQ.FAIL, EMQ.DELAY, EMQ.PROMOTE, EMQ.REQUEUE, EMQ.REPROCESS and EMQ.STALLED.CHECK refuse a grouped row |
parent, parent_queue, parent_policy | the flow links | flows (EchoMQ.Flows) | none: EMQ.COMPLETE and EMQ.FAIL refuse a flow row |
lease_epoch | the lease's server epoch | never written | EMQ.CLAIM, EMQ.EXTEND, EMQ.EXTENDS stamp the server run_id |
failures | the retry budget | never written (@retry caps on attempts) | EMQ.FAIL counts each verdict and caps on it |
| (the lease) | not a field | the job's score in emq:{q}:active: now + lease_ms, with now from TIME | the same score, now from the server's cached clock |
Two rows, then, mark the executors apart: lease_epoch and
failures exist only on a row the native module has touched. The parity harness that
compares the two executors treats these two fields as declared differences and judges them against their
declaration.
Three named fields at birth, one hash, one write on either executor. Every later field has a named writer, and the native module's two additions are declared.
The write — two beats
Beat one — the named handle. EchoMQ.Jobs.enqueue/4 checks the
queue name, builds the keys it touches — the row and the pending set — and runs
@enqueue through its dispatch (EchoMQ.Connector unless
opts names another). The row key comes from job_key/2, a gated
builder that raises on an ill-formed branded id, so a bad id never reaches the wire.
# echo_mq — EchoMQ.Jobs: the named handle for admission. # job_key/2 addresses the row at emq:{q}:job:<id> and raises on an ill-formed # branded id; queue_key/2 builds the pending set. Both are built host-side and # handed in declared. The values (id and payload) ride ARGV, never the keys. def enqueue(conn, queue, job_id, payload, opts \\ []) when is_binary(job_id) and is_binary(payload) do Keyspace.valid_queue!(queue) # The dispatch is carried, never inspected: EchoMQ.Connector by default. via = Keyword.get(opts, :via, Connector) keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")] case via.eval(conn, @enqueue, keys, [job_id, payload]) do {:ok, 1} -> {:ok, :enqueued} {:ok, 0} -> {:ok, :duplicate} {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind} other -> other end end
Beat two — the script body. @enqueue gates the kind (the first
three bytes must be JOB, else EMQKIND), returns
0 for a row that exists, then writes the record with one HSET
— three names, three birth values — and the pending entry with one ZADD.
-- the @enqueue script: the record write, every key declared in KEYS[] -- The kind gate: the first three bytes must be JOB, else refused (EMQKIND). if string.sub(ARGV[1], 1, 3) ~= 'JOB' then return redis.error_reply('EMQKIND job id must be JOB-namespaced') end -- Idempotent: a row that already exists is a no-op (a duplicate enqueue). if redis.call('EXISTS', KEYS[1]) == 1 then return 0 end -- THE FIELD SET: one HSET, three named fields, in field/value pairs. -- 'state' 'pending' born at the head of the lifecycle -- 'attempts' '0' born with no token minted -- 'payload' ARGV[2] the caller's bytes, stored verbatim redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2]) -- The pending set: the id is the member, so byte order is mint order. redis.call('ZADD', KEYS[2], 0, ARGV[1]) return 1
The native twin. EMQ.ENQUEUE's handler runs the same transition in a
different order: shape, key layout, the wire-identity fence, the JOB gate, the
duplicate check, the pending set's type — and only then the writes, the same HSET
and ZADD. A refusal writes nothing.
// trafikey — EMQ.ENQUEUE, the native twin of @enqueue: one command, the same row. // EMQ.ENQUEUE 2 emq:{q}:job:<id> emq:{q}:pending <id> <payload> pub(crate) fn enqueue(ctx: &Context, args: Vec<ValkeyString>) -> ValkeyResult { // Shape, then the exact key layout: KEYS[1] must be emq:{<q>}:job: + ARGV[1]. let s = host::shape("EMQ.ENQUEUE", &args, 2)?; s.expect_argc(2)?; if !enqueue_layout(&s) { return Err(s.layout_error()); } // The fence: {emq}:version must hold the module's wire identity. Read only. host::fence(ctx)?; let (row, pending, id, payload) = (s.key(1), s.key(2), s.arg(1), s.arg(2)); // The same kind gate as the Lua: the first three bytes are JOB. if !keyspace::is_job_id(id) { return Err(errors::emq(errors::KIND_JOB)); } // Idempotent: an existing row answers 0 and writes nothing. match host::read(ctx, cmd::EXISTS, &[row])? { Reply::Int(0) => {} Reply::Int(_) => return Ok(ValkeyValue::Integer(0)), _ => return Err(errors::emq(errors::HOST_REPLY)), } // The pending set's type is checked before the first write: a refusal writes nothing. host::expect_type(ctx, pending, b"zset")?; let w = host::begin_writes(ctx)?; // THE FIELD SET: the same HSET as @enqueue, the same three names and birth values. w.call( cmd::HSET, &[ row, b"state", b"pending", b"attempts", b"0", b"payload", payload, ], )?; w.call(cmd::ZADD, &[pending, b"0", id])?; Ok(ValkeyValue::Integer(1)) }
The HSET is the field set, literally: the same three names and the same birth values on
either executor. Every later field in the table arrives with a later transition, and each has a named
writer.
Reading the row back
The same names are what a reader asks for. A reader naming a field that exists gets its value; a reader naming a field that is not on the record — or misspelling one — gets nil, because the protocol is the exact strings. The sample row is one job after enqueue, a claim, one failed try, the promotion and a second claim, as the scripts left it or as the native module left it. Read a field off either row and compare the answers.
HGET the sample row · which executor, which field name?
One entity, one hash, a small fixed field set — the key is the address, the fields are the contract. Data Modeling & Memory (R7) teaches the move in Redis Patterns Applied; Patterns become protocol is the near side.
job_key/2 addresses the row; @enqueue's
HSET and EMQ.ENQUEUE's HSET fix its
three birth fields; every later field has a named writer, and the native module's two additions are
declared.
The record is the exact field strings. Name payload and the row answers;
name data and it answers nil; name failures and only a row the
native module touched answers.
Recap
A job is a Valkey hash at emq:{q}:job:<id>, born with three named fields written
by one atomic HSET — state,
attempts, payload — on either executor. Later transitions
add last_error and stalled on both; on the scripted side the
progress verb, lanes and flows add progress, group and the flow
links; the native module adds lease_epoch and failures. The
lease is the active-set score, not a field. The names are the protocol; the shape is the contract.
Next in this module: The state and attempts — the lifecycle position every transition rewrites, the fencing token, and the retry budget each executor caps on.
References
Where the record's data type and the writes that build and read it are documented.
Sources
- Valkey — HSET — the one write that builds the three-field row.
- Valkey — HGET — reading a field back by its exact name.
- Valkey — ZSCORE — where the lease lives: a score in the active set.
- Valkey — INFO — the
run_idthe native module stamps aslease_epoch. - Valkey — Documentation — the substrate of record.
- Redis — Hashes — the hash data type the record is modeled on.
Related in this course
- The record hash — the module hub.
- The state and attempts — the next dive.
- The native module — the module whose
EMQ.*commands write the native column. - The Protocol — the chapter landing.
- The protocol below the line — where the field set first appears.
- Data Modeling & Memory (R7) — the data-modeling chapter.