jonnifyechomq
EchoMQ / The Protocol / The owned keyspace / The braced grammar

The owned keyspace · dive 01 · the grammar

The braced grammar

One shape for every key: emq, the queue in braces, the type. The host builds it; the server checks it.

Every key EchoMQ writes has the same shape — emq:{q}:<type>. This is the L1 grammar, and it has two halves. On the host, EchoMQ.Keyspace builds it: queue_key/2 for the per-queue key, job_key/2 for the job row, and valid_queue!/1 to refuse a name that would break it. On the server, the native module checks it: it rebuilds a call's keys from one queue name and refuses any mismatch before it writes. Every key of one queue carries the same {q} hashtag.

The shape queue_key/2 job_key/2 The queue name The server-side check Build and check a key Pattern → implementation References

A per-queue key · pick a segment

emq: {orders} :pending literal hashtag (the queue) type queue_key("orders", "pending")
emq: — the literal. the fixed prefix of every key: the host writes it, and the native layout check requires it byte for byte.

The shape, and why it is total

A protocol's data layer is its agreement about where things live. EchoMQ's agreement is one string shape: emq:{q}:<type> — the literal emq:, the queue name wrapped in braces, a colon, then the type. The type is the role of the key within the queue: pending (the ready set), active (the claimed set), schedule (the delayed set), job:<id> (a record row).

What makes it a protocol and not a habit is that the grammar is total, on both halves of the wire. On the host, EchoMQ.Keyspace has exactly two productions for queue state — queue_key/2 and job_key/2 — and the job row is itself a per-queue key with a validated id appended. A transition that needs a row derives it from a declared base by the same grammar: @claim2 appends job: and the popped id to KEYS[3]. On the server, the native module accepts the keys only in this layout. So one queue named orders owns a whole family of keys that all share the {orders} hashtag.

emq:{orders}:pending
the ready set — ids waiting to be claimed.
emq:{orders}:active
the claimed set — ids a worker holds.
emq:{orders}:schedule
the delayed set — ids parked until a due time.
emq:{orders}:job:<id>
the record row — one job's hash.

Every key in that family carries {orders}, so every key routes to one slot, and a transition can move a job between them in one atomic step: one EMQ.* command on the native module.

queue_key/2 — the per-queue production

queue_key/2 composes the per-queue key as iodata, then flattens it to a binary in one pass. The guard fixes both arguments as binaries, so a non-string queue or type fails at the boundary rather than producing a malformed key.

# echo_mq — EchoMQ.Keyspace
# Per-queue keys are emq:{q}:<type>. The braces are not decoration —
# they are the cluster hashtag, and they pin every key of one queue to
# one slot. Built as iodata, then flattened once.
def queue_key(queue, type) when is_binary(queue) and is_binary(type),
  do: IO.iodata_to_binary(["emq:{", queue, "}:", type])

The iodata list ["emq:{", queue, "}:", type] is the grammar made literal: the brace opens immediately after emq: and closes immediately before the type colon, so the queue name is always the hashtag and nothing else is. There is no string interpolation that a caller could subvert — the structure is in the list, not in a format string.

job_key/2 — the job row, gated

The job row is a per-queue key whose type is job: with a validated branded id appended. It is the one production that refuses its input: an id that is not a valid branded id is not turned into a key — the function raises.

# echo_mq — EchoMQ.Keyspace
# The job row is a per-queue key whose type is "job:" plus the branded id.
# The id is GATED first: the keyspace refuses to address an unbranded row,
# so a malformed id never becomes a key.
def job_key(queue, branded) when is_binary(branded) do
  if EchoData.BrandedId.valid?(branded) do
    queue_key(queue, "job:") <> branded
  else
    raise ArgumentError, "job_key requires a valid branded id"
  end
end

So job_key("orders", "JOB0Nb1VTbfnu4") yields emq:{orders}:job:JOB0Nb1VTbfnu4. The branded id is the long part of the key by design — 14 bytes, a 3-byte namespace plus an 11-byte Base62 payload — and valid?/1 is the full check: this host-side call is the only place all 14 bytes are validated. Inside a transition the id is gated again on its 3-byte JOB prefix only, by the script and by the native command alike; that gate is Immutability & branded ids. Here the point is the grammar: the row's address is always a per-queue key, so it co-locates with the queue's sets.

valid_queue?/1 — the name the braces hold

A brace inside a queue name would move the hashtag. The cluster hashes the substring inside the first {...}, so a name such as a}b builds emq:{a}b}:pending, whose hashtag is a: its keys would route as if they belonged to the queue a. A NUL would corrupt the NUL-joined flow-outbox entries. So the host refuses both, and the admission verbs of EchoMQ.Jobs and EchoMQ.Lanes call the raising form before any wire.

# echo_mq — EchoMQ.Keyspace
# A queue name is safe for the grammar unless it carries a brace or a NUL:
# a brace would silently change the hashtag the slot routing extracts.
def valid_queue?(queue) when is_binary(queue) do
  not String.contains?(queue, ["{", "}", <<0>>])
end

def valid_queue?(_), do: false

# …
# The admission verbs call this first. A bad name is a caller bug, so it
# raises at the boundary, before any wire: a loud error, never a silent
# change of slot.
def valid_queue!(queue) do
  if valid_queue?(queue) do
    :ok
  else
    raise ArgumentError,
          "a queue name must not contain a brace or NUL (§6 charset): #{inspect(queue)}"
  end
end

The host check accepts an empty name: queue_key("", "pending") builds emq:{}:pending, and the scripts run against it on a standalone server. The native layout check is stricter.

The server-side half — the native layout check

The native module receives a transition's keys as declared arguments, in the order the scripts take them: EMQ.ENQUEUE 2 emq:{q}:job:<id> emq:{q}:pending <id> <payload>. Before it reads the wire identity, and before any domain check, it checks the layout. Its keyspace mirror carries the rule: split a key into its queue name and its type, then accept the queue only when the type is the one the verb lists and the name is non-empty.

// trafikey — the native keyspace mirror
// Split emq:{<q>}:<type>. The key must start with the literal "emq:{",
// the name runs to the FIRST "}", the type follows "}:", and the name
// must pass the same test as valid_queue?/1: no "{", "}" or NUL.
pub fn split_queue_key(key: &[u8]) -> Option<(&[u8], &[u8])> {
    let rest = key.strip_prefix(b"emq:{")?;
    let close = rest.iter().position(|&b| b == b'}')?;
    let (queue, tail) = rest.split_at(close);
    let ty = tail.strip_prefix(b"}:")?;
    valid_queue(queue).then_some((queue, ty))
}

// …
impl<'a> Queue<'a> {
    // The queue of `key` when `key` is exactly emq:{<q>}:<ty>: the type must
    // be the one the verb lists, and the name must be NON-EMPTY. An empty
    // emq:{}: carries no hashtag, so it would spread one queue over slots.
    pub fn from_key(key: &'a [u8], ty: &[u8]) -> Option<Self> {
        match split_queue_key(key) {
            Some((name, t)) if t == ty && !name.is_empty() => Some(Self { name }),
            _ => None,
        }
    }
    // … the other builders (base, job, meta) rebuild keys from `name`.
}

EMQ.ENQUEUE names its pair, and its handler checks it right after the argument shape:

// trafikey — EMQ.ENQUEUE's layout
// KEYS[2] must be a queue's pending key; KEYS[1] must equal that same
// queue's row for ARGV[1], rebuilt and compared byte for byte. One name,
// one hashtag: the layout check is an equality.
fn enqueue_layout<S: Bytes>(s: &Shape<'_, S>) -> bool {
    Queue::from_key(s.key(2), b"pending").is_some_and(|q| s.key(1) == q.job(s.arg(1)).as_slice())
}

// …
// The handler's first steps: shape, then layout, then the fence. A layout
// refusal comes before {emq}:version is read, and nothing is written.
pub(crate) fn enqueue(ctx: &Context, args: Vec<ValkeyString>) -> ValkeyResult {
    let s = host::shape("EMQ.ENQUEUE", &args, 2)?;
    s.expect_argc(2)?;
    if !enqueue_layout(&s) {
        return Err(s.layout_error());
    }
    host::fence(ctx)?;
    // … then the JOB gate, the duplicate check, the type check, the write
    // guard, and only then the writes.
}

The layout refuses a key that does not start with emq:{ or lacks the }: after the name; a name with {, } or NUL; an empty name, which the host builder composes and the scripts accept on a standalone server; a type other than the one the verb lists; and a row key built for another queue, or for another id than ARGV[1]. The refusal is the layout error, and nothing is written:

EMQ.ENQUEUE 2 emq:{}:job:JOB0Nb1VTbfnu4 emq:{}:pending JOB0Nb1VTbfnu4 <payload>
ERR EMQ.ENQUEUE keys do not match the wire-4 layout

The canon states the rule the same way: each verb lists its exact keys, compared byte for byte, with one non-empty queue name per call, and this check is also the cluster hashtag check, because every declared key shares one {q}.

Build a key — and check it on both sides

The grammar is mechanical: give a queue name and a type and the key is determined. Type a queue and pick a type below. The page composes the key exactly as queue_key/2 does — and, for the job: type, appends a sample branded id the way job_key/2 does — then runs both halves of the check: the host's valid_queue?/1, and a port of the native split_queue_key and Queue::from_key (for the job row, ENQUEUE's pair: the row for the id and the queue's pending key). It reads back the hashtag, so you can see the brace is the routing key. Try the presets: an empty name passes the host and fails the server; a}b fails both; a colon is legal.

queue_key / job_key · compose a key, check both halves

queue_key("orders", "pending")
→ emq:{orders}:pending
hashtag: orders — every key of this queue shares it, so all route to one slot
host · valid_queue?("orders") → true — the admission verbs proceed
native · layout → accepted — the name "orders", the type "pending"; the fence is next

A queue name and a type determine the key — and the key always carries the queue as its hashtag. The host refuses a name that would move the brace; the server refuses any shape but this one.

Pattern → implementation

the pattern (Redis Patterns Applied)

A key naming convention becomes a contract when every writer agrees on the exact string shape — the prefix, the separators, the field positions — and on the hashtag that co-locates a group. Redis Patterns Applied R2.05 teaches it.

the implementation (echo_mq + the native module)

EchoMQ.Keyspace.queue_key/2 and job_key/2 make emq:{q}:<type> the only way the host builds a key, and the native module refuses any other shape on the server, so the convention holds on both sides of the wire — the grammar is the protocol, not a style guide.

The grammar is the data-layer half of the protocol. Build every key through one module to one braced shape, check it again where it lands, and "where things live" stops being a convention and becomes the wire.

Recap — one shape, two halves

The braced grammar is emq:{q}:<type>. On the host, queue_key/2 builds the per-queue key as iodata, job_key/2 builds the job row from a per-queue key plus a validated branded id, and valid_queue!/1 refuses a brace or NUL in the name before any wire. On the server, the native module recovers one queue name from the declared keys, rebuilds the rest, and refuses any mismatch — an empty name included — before the fence, with nothing written. Every key of one queue shares the {q} hashtag. The next dive reads what that hashtag computes: the cluster slot.

Next in this module: The hashtag & the slot — the brace as the cluster hashtag, and slot/1 = CRC16-XMODEM % 16384.

References

Where the grammar and the hashtag convention are documented, where the record it addresses is written, and how a module's commands come to check it on the server.

Sources

Related in this course