The protocol below the line

One line. Below it the keys and the transitions are shared and fixed; above it, the executor and the language are yours.

The three pillars interoperate, and any runtime can drive them, for one reason: the protocol lives below the language line. Read EchoMQ as a stack — L0 Valkey, L1 the keys and field names, L2 the atomic transitions, L3 the executor, L4 the API. The shared line falls between L2 and L3. Pick a layer on the right to read its scope and whether it is shared. This dive draws the line on a real key and a real transition; the full key taxonomy and both executors are The Protocol.

The stack The line, on a real key The line, on a real transition Change a field, who breaks? References

The layer stack · pick a layer

L2 Atomic transitions — scope: Lua scripts, and for the core job transitions native EMQ.* commands inside the server. Status: SHARED — below the line, fixed for every runtime. one indivisible step per transition, bound by the same keys, fields and replies on either executor.

The stack, and where the line falls

What makes one wire serve three pillars, in any language? A line. EchoMQ reads as a stack. L0 is Valkey itself — the engine the system runs on, but does not contain. L1 is the data layer: which key holds which structure, and what the field names are. L2 is the set of atomic transitions that move a record between states, each one an indivisible step inside the server. A transition runs as a Lua script the connector sends; each core job transition can also run as a native EMQ.* command that the trafikey module registers in the server, while lanes, flows and the reaper run only as scripts. Either way a transition is bound by the same keys, the same fields and the same replies. Together, L1 and L2 are the protocol — a Valkey data convention plus the transitions that operate on it.

L3 is the executor: the code that hands a transition to the server. The scripted connector loads each script once and runs it by SHA with EVALSHA, reloading on a NOSCRIPT miss; a native client sends the command name and loads nothing. In the canonical Elixir, L3 is EchoMQ.Connector, the scripted executor. L4 is the language API a caller invokes — EchoMQ.Jobs.enqueue/4 in Elixir. L3 and L4 are where a runtime is free to differ. The whole point of the model is that the freedom is confined to those two layers: the shared line falls between L2 and L3.

L4 · API: The language method a caller invokes. Method names, parameter order, and return types vary per runtime.
L3 · executor: The code that hands each transition to the server: scripts loaded once and run by SHA (EVALSHA, reload on NOSCRIPT), or native commands sent by name. Connection handling and dispatch vary per runtime.
L2 · transitions: The atomic steps that change record state: Lua scripts, and for the core job transitions native EMQ.* commands inside the server. Same keys, same fields, same replies on either executor; shared by everything that speaks the wire.
L1 · keys: The keys (emq:{q}:<type>) and the field names (state, attempts, payload). One agreed convention, shared; the native module keeps the same field names and adds two of its own, lease_epoch and failures.
L0 · Valkey: The engine itself, external. One server for the scripted connector, which does not route by slot, though every key of a transition shares its queue's slot; a standalone server for the native module, which refuses a cluster-enabled one.

This dive draws the line on one key and one transition. The full key taxonomy, the record, every transition and both executors are the subject of The Protocol, the foundation chapter on the course map.

The line, on a real key

The data layer is concrete: a real key the system builds for every queue. EchoMQ.Keyspace.queue_key/2 composes the per-queue key emq:{q}:<type> — the literal emq:, the queue name wrapped in braces, then the type suffix.

# echo_mq — EchoMQ.Keyspace
# Per-queue keys are emq:{q}:<type>. The braces are not decoration —
# they are the cluster hashtag, and they pin every key of one queue to
# one slot.
def queue_key(queue, type) when is_binary(queue) and is_binary(type),
  do: IO.iodata_to_binary(["emq:{", queue, "}:", type])

The {q} is the hashtag. A Valkey cluster routes a key by a CRC16 of the substring inside the first {...}, so wrapping the queue name pins every key of that queue — emq:{orders}:pending, emq:{orders}:active, emq:{orders}:job:<id> — to one slot. One atomic transition can touch the pending set, the active set, and the job row in a single step because they are guaranteed co-resident. On a standalone server, the kind the native module serves, every key is local anyway; the hashtag is what keeps the scripts legal on a cluster. That key format is L1: the same on either executor, wherever the protocol runs. Change the literal emq:, or move the braces, and you are speaking a different wire.

The line, on a real transition — two beats, one command

The state transitions are L2. They are not application code that happens to touch Valkey — they run inside the engine, atomically, as a Lua script the connector sends or, for a core job transition, as a native command compiled into the server. The wire's key rule is what keeps a script portable: every key it touches is declared in KEYS[] or derived from a declared KEYS[n] root, so each one carries the queue's hashtag. @enqueue declares both keys it touches; several older scripts, @complete among them, build some keys from a queue base passed in ARGV, which the rule does not count as a declared root. Read the enqueue transition in two beats, then as one native command.

Beat one — the named handle. EchoMQ.Jobs.enqueue/4 checks the queue name, builds the two keys the script touches and runs the @enqueue script through its executor, EchoMQ.Connector by default. The keys are built host-side, by the keyspace; the script receives them declared.

# echo_mq — EchoMQ.Jobs
# The host builds the two keys the script touches — the job row and the
# pending set — and hands them in declared. The executor is `via`:
# EchoMQ.Connector unless opts name another, EVALSHA first, one SCRIPT LOAD
# on NOSCRIPT. KEYS = [row, pending]; the id and the payload ride ARGV.
def enqueue(conn, queue, job_id, payload, opts \\ [])
    when is_binary(job_id) and is_binary(payload) do
  Keyspace.valid_queue!(queue)
  via = Keyword.get(opts, :via, Connector)
  # job_key/2 raises unless job_id is a valid branded id: the full check is host-side
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case via.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}
    {:ok, 0} -> {:ok, :duplicate}
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}
    other -> other
  end
end

Beat two — the script body. @enqueue is the transition: admit by kind, refuse a duplicate, write the row, insert the pending entry — one atomic step. KEYS[1] is the job row, KEYS[2] the pending set; ARGV[1] is the branded id, ARGV[2] the payload.

-- the @enqueue script (L2) — one atomic transition, both keys declared
-- The kind gate: the id's first three bytes must be JOB, refused at the wire.
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
-- Idempotent: a row that already exists is a no-op (a duplicate enqueue).
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
-- The three-field row: state, attempts, payload — the shape every speaker reads.
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
-- The pending set: the id is the member, so byte order is mint order.
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

One command — the same transition, native. On a node that loads the native module, the enqueue transition is one command. The command name takes the place of the SHA; the keys and the arguments are the same.

# the scripted executor: EchoMQ.Connector sends the script by its SHA
EVALSHA <sha> 2 emq:{q}:job:<id> emq:{q}:pending <id> <payload>

# the native module: the command name replaces the SHA; nothing to load
EMQ.ENQUEUE 2 emq:{q}:job:<id> emq:{q}:pending <id> <payload>

What the excerpts show is the boundary: the KEYS/ARGV contract, the JOB kind gate, the three named fields. None of it is language-specific, and none of it belongs to one executor: the native command checks the same JOB prefix, answers a duplicate with the same 0 and writes the same three fields; it also checks the key layout and the wire identity in {emq}:version before it writes. The executor is L3, above the line: the scripted connector loads a script once and runs it by SHA, a native client sends the command name and loads nothing, and each runtime writes that its own way.

Because the keys, the fields and the transitions are the whole agreement, any runtime that speaks them is a peer — and with the native module, the core job transitions take no script at all, only the command. So EchoMQ is supported in, and ported to, other runtimes; the depth of this course is the canonical one, Elixir.

Draw the line between L2 and L3, and a Valkey convention becomes a protocol three pillars share and any runtime can speak. Below it: one key format, one set of atomic transitions, run as scripts and, for the core job transitions, as native commands. Above it: your executor and your language.

Why L1 and L2 are fixed — change a field, who breaks?

The fields state, attempts, payload are L1, and so is the key prefix. They are the contract, not an implementation detail, and both executors are bound to them: the scripted claim and the native claim both read payload. Rename payload to data in one writer and its rows carry a field no claim reads, on either executor. The data is not lost; it is invisible across the wire. The same is true of the key: change emq: and the scripted jobs land in a keyspace no claim looks in, while the native commands refuse the keys outright. The L4 verb is the opposite — it is above the line, so renaming it breaks nothing. Make one change and read what each executor does.

Change one thing · do both executors still find the job?

below the line

above the line

Change (above the line): rename the L4 verb enqueue → push. Scripted executor: HOLDS — the same keys, the same script, the same row. Native module: HOLDS — the same command line, the same row. Why: the verb is above the line; nothing the server sees has changed.

the pattern (Redis Patterns Applied)

A Valkey convention becomes a protocol when the agreement is pushed below the line — the keys plus the atomic transitions — and the runtime is left above it. Redis Patterns Applied teaches that move; this is the system that applies it.

the implementation (echo_mq)

EchoMQ.Keyspace.queue_key/2 fixes the key emq:{q}:<type> with the {q} hashtag; the @enqueue script and its native twin EMQ.ENQUEUE fix the transition, the same two keys in KEYS[] on either executor. L1 and L2, shared and fixed.

Below the line is the contract; above it is the code. A field rename below the line breaks both executors at once; an L4 rename above it changes nothing on the wire.

Recap — the line, the key, the script

The protocol below the line is the keys, the fields and the atomic transitions. EchoMQ.Keyspace.queue_key/2 builds emq:{q}:<type>, where the {q} hashtag pins one queue to one slot so one atomic transition can touch its sets and rows together. The @enqueue transition declares both keys it touches, gates the JOB prefix, and writes the three-field row and the pending entry in one step — as a script the connector runs by SHA, or as the native EMQ.ENQUEUE. That shared substrate — L1 and L2 — is fixed; the executor (L3) and the API (L4) are the runtime's own. The next dive reads where EchoMQ lands: the door from Redis Patterns Applied and the BCS family.

Next in the Overview: The door & the BCS family — the bidirectional Redis-Patterns door, and what owning the queue, the bus, and the cache means for the systems built on them.

References

Where the layer model is grounded, and where the keyspace and scripting layers are documented.

Sources

Valkey — Documentation — the BSD-licensed, foundation-governed store EchoMQ is backed by; the substrate of record.
Valkey — EVALSHA — the load-once, run-by-SHA dispatch the scripted executor uses to run an L2 script.
Redis — Keyspace & hash tags — cluster key routing by the hashtag inside {...}, the L1 mechanism the {q} brace uses.
Valkey — Cluster specification — the {hashtag}→hash-slot routing the per-queue-hashtag keyspace is built for: every key of a queue on one slot, where a multi-key script is legal.

Related in this course

Overview — the chapter this dive belongs to; the three pillars over one wire.
The Protocol — the owned keyspace, the record hash, the Lua layer and the native module in depth (the chapter the line opens onto).
redis-patterns · Patterns become protocol — the near side of the door: the move from convention to protocol.
The Branded Component System — the architecture EchoMQ is the bus of; the JOB namespace the @enqueue gate checks is part of its branded-id contract.
Course home — the full six-chapter map.