jonnifyechomq
EchoMQ / The Protocol / The record hash / The hash and its fields

The record hash · dive 01 · the field set

The hash and its fields

One write builds the row. Every field after it has a named writer.

The job record is one Valkey HASH, addressed by EchoMQ.Keyspace.job_key/2 at emq:{q}:job:<id> and born from exactly one HSET — the same one on either executor: @enqueue in Lua and EMQ.ENQUEUE in the native module. The field list is the record's shape, fixed and named, so any speaker reads each field by its name with no schema lookup. Pick a field on the right to read its role, its birth value, and its writers on each executor.

The field set · pick a field

HASH key emq:{orders}:job:JOB0f3k… born · one HSET on either executor state "pending" lifecycle position attempts "0" fencing token payload "<caller bytes>" opaque cargo later · both executors last_error — latest failure stalled — stall count later · the scripted executor only progress — progress value group — the lane parent, parent_queue, parent_policy flow links later · the native module only lease_epoch — lease’s server epoch failures — retry budget the lease is not a field: a score in emq:{q}:active
state — the lifecycle position. Born: "pending". Scripts: @enqueue "pending"; @schedule "scheduled"; @claim2 "active"; @retry "scheduled" or "dead"; @delay "scheduled"; @promote and @reap "pending"; the stall sweep "pending" or "dead"; @reprocess2 "pending"; a flow parent is held "awaiting_children". Native module: the same values from EMQ.ENQUEUE, EMQ.SCHEDULE, EMQ.CLAIM, EMQ.FAIL, EMQ.DELAY, EMQ.PROMOTE, EMQ.STALLED.CHECK, EMQ.REQUEUE and EMQ.REPROCESS.

One hash, a fixed field set

A job is an entity, and its state is one Valkey hash. Admission builds it with exactly one write, and the two executors write the same one: @enqueue in Lua and EMQ.ENQUEUE in the native module both run HSET <row> state pending attempts 0 payload <payload>. There is no id field, because the id is the key; no created_at, because the branded id is itself time-ordered; no second copy of anything. The birth row is the smallest row that carries a job.

What makes the field set part of the protocol, not an implementation detail, is that every transition reads and writes the row by these names, on either executor. When a worker claims the job, the claim reads payload by that name; when a completion is fenced, it reads attempts by that name. The rest of the field set, with its writers on each executor:

The field set and its writers
FieldRoleScripted executorNative module
statethe lifecycle position@enqueue pending; @schedule scheduled; @claim2 active; @retry scheduled or dead; @delay scheduled; @promote and @reap pending; the stall sweep pending or dead; @reprocess2 pending; a flow parent is held awaiting_childrenthe same values from EMQ.ENQUEUE, EMQ.SCHEDULE, EMQ.CLAIM, EMQ.FAIL, EMQ.DELAY, EMQ.PROMOTE, EMQ.STALLED.CHECK, EMQ.REQUEUE, EMQ.REPROCESS
attemptsthe fencing token'0' at admission; @claim2 runs HINCRBY attempts 1'0' at admission; only EMQ.CLAIM increments it
payloadthe opaque cargoARGV[2] at admission; update_data/4 replaces itARGV[2] at admission
last_errorthe latest failure@retry on every failure; the stall sweep's death writes 'stalled'; @reprocess2 clears itEMQ.FAIL; EMQ.STALLED.CHECK's death writes 'stalled'; EMQ.REPROCESS clears it
stalledthe stall countthe stall sweep (EchoMQ.Stalled), on every expired leaseEMQ.STALLED.CHECK, on counted expiries only
progressthe progress valueupdate_progress/4no native verb
groupthe lanelanes (EchoMQ.Lanes)none: EMQ.COMPLETE, EMQ.FAIL, EMQ.DELAY, EMQ.PROMOTE, EMQ.REQUEUE, EMQ.REPROCESS and EMQ.STALLED.CHECK refuse a grouped row
parent, parent_queue, parent_policythe flow linksflows (EchoMQ.Flows)none: EMQ.COMPLETE and EMQ.FAIL refuse a flow row
lease_epochthe lease's server epochnever writtenEMQ.CLAIM, EMQ.EXTEND, EMQ.EXTENDS stamp the server run_id
failuresthe retry budgetnever written (@retry caps on attempts)EMQ.FAIL counts each verdict and caps on it
(the lease)not a fieldthe job's score in emq:{q}:active: now + lease_ms, with now from TIMEthe same score, now from the server's cached clock

Two rows, then, mark the executors apart: lease_epoch and failures exist only on a row the native module has touched. The parity harness that compares the two executors treats these two fields as declared differences and judges them against their declaration.

Three named fields at birth, one hash, one write on either executor. Every later field has a named writer, and the native module's two additions are declared.

The write — two beats

Beat one — the named handle. EchoMQ.Jobs.enqueue/4 checks the queue name, builds the keys it touches — the row and the pending set — and runs @enqueue through its dispatch (EchoMQ.Connector unless opts names another). The row key comes from job_key/2, a gated builder that raises on an ill-formed branded id, so a bad id never reaches the wire.

# echo_mq — EchoMQ.Jobs: the named handle for admission.
# job_key/2 addresses the row at emq:{q}:job:<id> and raises on an ill-formed
# branded id; queue_key/2 builds the pending set. Both are built host-side and
# handed in declared. The values (id and payload) ride ARGV, never the keys.
def enqueue(conn, queue, job_id, payload, opts \\ [])
    when is_binary(job_id) and is_binary(payload) do
  Keyspace.valid_queue!(queue)
  # The dispatch is carried, never inspected: EchoMQ.Connector by default.
  via = Keyword.get(opts, :via, Connector)
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case via.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}
    {:ok, 0} -> {:ok, :duplicate}
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}
    other -> other
  end
end

Beat two — the script body. @enqueue gates the kind (the first three bytes must be JOB, else EMQKIND), returns 0 for a row that exists, then writes the record with one HSET — three names, three birth values — and the pending entry with one ZADD.

-- the @enqueue script: the record write, every key declared in KEYS[]
-- The kind gate: the first three bytes must be JOB, else refused (EMQKIND).
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
-- Idempotent: a row that already exists is a no-op (a duplicate enqueue).
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
-- THE FIELD SET: one HSET, three named fields, in field/value pairs.
--   'state'    'pending'  born at the head of the lifecycle
--   'attempts' '0'        born with no token minted
--   'payload'  ARGV[2]    the caller's bytes, stored verbatim
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
-- The pending set: the id is the member, so byte order is mint order.
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

The native twin. EMQ.ENQUEUE's handler runs the same transition in a different order: shape, key layout, the wire-identity fence, the JOB gate, the duplicate check, the pending set's type — and only then the writes, the same HSET and ZADD. A refusal writes nothing.

// trafikey — EMQ.ENQUEUE, the native twin of @enqueue: one command, the same row.
// EMQ.ENQUEUE 2 emq:{q}:job:<id> emq:{q}:pending <id> <payload>
pub(crate) fn enqueue(ctx: &Context, args: Vec<ValkeyString>) -> ValkeyResult {
    // Shape, then the exact key layout: KEYS[1] must be emq:{<q>}:job: + ARGV[1].
    let s = host::shape("EMQ.ENQUEUE", &args, 2)?;
    s.expect_argc(2)?;
    if !enqueue_layout(&s) {
        return Err(s.layout_error());
    }
    // The fence: {emq}:version must hold the module's wire identity. Read only.
    host::fence(ctx)?;
    let (row, pending, id, payload) = (s.key(1), s.key(2), s.arg(1), s.arg(2));
    // The same kind gate as the Lua: the first three bytes are JOB.
    if !keyspace::is_job_id(id) {
        return Err(errors::emq(errors::KIND_JOB));
    }
    // Idempotent: an existing row answers 0 and writes nothing.
    match host::read(ctx, cmd::EXISTS, &[row])? {
        Reply::Int(0) => {}
        Reply::Int(_) => return Ok(ValkeyValue::Integer(0)),
        _ => return Err(errors::emq(errors::HOST_REPLY)),
    }
    // The pending set's type is checked before the first write: a refusal writes nothing.
    host::expect_type(ctx, pending, b"zset")?;
    let w = host::begin_writes(ctx)?;
    // THE FIELD SET: the same HSET as @enqueue, the same three names and birth values.
    w.call(
        cmd::HSET,
        &[
            row,
            b"state",
            b"pending",
            b"attempts",
            b"0",
            b"payload",
            payload,
        ],
    )?;
    w.call(cmd::ZADD, &[pending, b"0", id])?;
    Ok(ValkeyValue::Integer(1))
}

The HSET is the field set, literally: the same three names and the same birth values on either executor. Every later field in the table arrives with a later transition, and each has a named writer.

Reading the row back

The same names are what a reader asks for. A reader naming a field that exists gets its value; a reader naming a field that is not on the record — or misspelling one — gets nil, because the protocol is the exact strings. The sample row is one job after enqueue, a claim, one failed try, the promotion and a second claim, as the scripts left it or as the native module left it. Read a field off either row and compare the answers.

HGET the sample row · which executor, which field name?

The scripts’ row: HGET row "state" → "active". The second claim wrote it.
the pattern (Redis Patterns Applied)

One entity, one hash, a small fixed field set — the key is the address, the fields are the contract. Data Modeling & Memory (R7) teaches the move in Redis Patterns Applied; Patterns become protocol is the near side.

the implementation (echo_mq + trafikey)

job_key/2 addresses the row; @enqueue's HSET and EMQ.ENQUEUE's HSET fix its three birth fields; every later field has a named writer, and the native module's two additions are declared.

The record is the exact field strings. Name payload and the row answers; name data and it answers nil; name failures and only a row the native module touched answers.

Recap

A job is a Valkey hash at emq:{q}:job:<id>, born with three named fields written by one atomic HSET — state, attempts, payload — on either executor. Later transitions add last_error and stalled on both; on the scripted side the progress verb, lanes and flows add progress, group and the flow links; the native module adds lease_epoch and failures. The lease is the active-set score, not a field. The names are the protocol; the shape is the contract.

Next in this module: The state and attempts — the lifecycle position every transition rewrites, the fencing token, and the retry budget each executor caps on.

References

Where the record's data type and the writes that build and read it are documented.

Sources

Related in this course