jonnifyechomq
EchoMQ / The Protocol / The record hash / The payload

The record hash · dive 03 · the cargo

The payload

The third field is the caller's. The protocol stores it, moves it, delivers it — and never parses it.

The third field, payload, is the caller's data, and no transition parses it. Admission takes it as ARGV[2] and stores it verbatim; the claim hands it back unchanged — on either executor. Between those two points the protocol does nothing with it: it is opaque cargo, addressed, moved, and delivered but never parsed, validated, or branched on. That is what lets one protocol carry any job for any consumer. Round-trip a sample payload on the right and read it back, byte for byte.

Enqueue then claim · pick a payload

enqueue · HSET @enqueue · EMQ.ENQUEUE payload ← ARGV[2] claim · HGET @claim2 · EMQ.CLAIM payload → worker verbatim in: HSET payload {"sku":"A-19","qty":3} out: HGET payload → {"sku":"A-19","qty":3}
Enqueue a JSON object: stored "{"sku":"A-19","qty":3}". The claim returns "{"sku":"A-19","qty":3}". Identical: stored and returned byte for byte on either executor; no transition parsed it.

Opaque cargo

The payload is the caller's, and the protocol holds no opinion about its bytes. It is not parsed on the way in, not validated, not indexed, not branched on; it is stored as one string and returned as the same string. The system's job is to address it, move it through the lifecycle, and deliver it to a worker — not to understand it.

That is deliberate, and it is what makes the protocol general. Because no transition reads into the value, the same queue serves any job for any consumer, in whatever encoding the caller and the worker agree on — JSON, a bare string, a packed binary. The cargo law: the bus carries; the caller decides what.

Both executors keep the law. EMQ.ENQUEUE writes the argument as given in the same HSET, and EMQ.CLAIM returns the stored bytes as a bulk reply: the native commands treat the payload as bytes from end to end. update_data/4 can replace the field whole, but the contents stay the caller's.

The payload is opaque cargo: stored, moved, delivered, never parsed, on either executor. The protocol holds no opinion about the bytes, so it carries any job.

Store and return — two beats

Beat one — the named handle. EchoMQ.Jobs.enqueue/4 requires the payload to be a binary (when … and is_binary(payload)) and passes it as ARGV[2]. The guard makes the field always hold a string; the contents stay the caller's.

# echo_mq — EchoMQ.Jobs: the named handle for admission.
# The is_binary(payload) guard means the field always holds a string, but its
# CONTENTS are the caller's. The payload rides ARGV[2]; no script parses it.
def enqueue(conn, queue, job_id, payload, opts \\ [])
    when is_binary(job_id) and is_binary(payload) do
  Keyspace.valid_queue!(queue)
  via = Keyword.get(opts, :via, Connector)
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case via.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}
    {:ok, 0} -> {:ok, :duplicate}
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}
    other -> other
  end
end

Beat two — the two scripts. @enqueue stores the payload with its one HSET, untouched; @claim2 returns it with HGET, untouched, after deriving the row key from the declared base (KEYS[3] .. 'job:' .. id). Side by side, the round trip is store-then-return; neither script parses the value.

-- @enqueue, the store: its one HSET. The payload is ARGV[2], written
-- verbatim as the third field.
-- …
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
-- …

-- @claim2, the return: the whole body. The row key derives from the
-- declared base KEYS[3]; the payload is read back, never parsed.
local popped = redis.call('ZPOPMIN', KEYS[1])
if #popped == 0 then return {} end
local id = popped[1]
local jk = KEYS[3] .. 'job:' .. id
if redis.call('EXISTS', jk) == 0 then return {} end
local att = redis.call('HINCRBY', jk, 'attempts', 1)
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[1]), id)
-- The worker is handed HGET payload: byte for byte what admission stored.
return {id, redis.call('HGET', jk, 'payload'), att}

Nothing between the HSET and the HGET parses the value. The protocol's contract over the record is the field names and the meanings of state and attempts (and, natively, failures and lease_epoch); payload is a passthrough.

A row with no payload. Every admission writes the field — enqueue, scheduled admission, lanes and flows all put payload in the same HSET — so a row without one lies outside the admission path. If a claim meets such a row, both executors reply [id, nil, att]: EMQ.CLAIM returns a nil for the missing field, and in the script HGET yields Lua false, which converts to a nil reply without ending the table. The parity harness leaves this case out of its comparison.

Opaque vs protocol — which fields the protocol interprets

Some fields carry meaning the protocol defines; one it only carries. state and attempts are protocol fields on both executors — the transitions write state with the lifecycle positions the protocol defines, and every holder's call compares attempts with its token. Natively, failures and lease_epoch are protocol fields too: EMQ.FAIL caps on failures, and EMQ.STALLED.CHECK compares lease_epoch with the server's run_id. payload is the opaque field: a transition moves it and never parses it. Pick a field and read which the system interprets.

Does the protocol interpret this field?

payload — the opaque cargo, on both executors. Carried, never parsed. @enqueue and EMQ.ENQUEUE store ARGV[2] with HSET; @claim2 and EMQ.CLAIM return it; no transition parses the value.
the pattern (Redis Patterns Applied)

Keep the payload opaque to the store: let the value be the caller's, and the key and the control fields be the system's. Data Modeling & Memory (R7) teaches the move in Redis Patterns Applied; Patterns become protocol is the near side.

the implementation (echo_mq + trafikey)

payload is ARGV[2], stored verbatim by @enqueue and EMQ.ENQUEUE and returned verbatim by @claim2 and EMQ.CLAIM; the transitions interpret only the control fields. The cargo law, in both executors.

The protocol interprets state and attempts, and natively failures and lease_epoch; payload it only carries. The value is the caller's, so one protocol fits any job.

Recap

payload is the caller's opaque cargo — stored verbatim at admission, returned verbatim by the claim on either executor, never parsed. The protocol interprets the key and the control fields; the value is the caller's. That is what lets one protocol carry any job. With all three birth fields read, the record is complete — the chapter's remaining modules hold this shape fixed with the Lua layer, the native module and the immutability discipline.

Back to The record hash hub, or on to the rest of The Protocol — the Lua layer, the native module and the immutability that hold the field set fixed.

References

Where the writes that store and return the payload are documented.

Sources

Related in this course