EchoMQ / The Protocol / Immutability & branded ids / The version fence

Immutability · dive 03 · how

The version fence

An executor is checked against the node's wire before it writes: once per connection, or once per command.

The immutable line protects the data; the branded-id gate protects identity. One thing is left: the node. An executor that does not speak the wire a node holds must not write to it. The fence is one reserved key — {emq}:version — holding the node's wire identity, and the two executors hold it two ways. The scripted connector — EchoMQ.Connector on its default wire 4 — claims or verifies it at connect; the native module's write verbs verify it on every write and never write it. Pick a stored value on the right and read both verdicts.

Two executors, one value The fences, on real code The module fence The reserve map Pattern & implementation References

Both fences · pick the stored value

{emq}:version is absent
scripted connector, at connect: claim + run — :ok. After SET {emq}:version echomq:4.0.0 NX and a confirming GET; of two connectors racing an empty node exactly one write lands. Afterwards: the key holds "echomq:4.0.0".
native write verb, on every write: refused — EMQWIRE expected echomq:6.0.0 but {emq}:version is absent. A refusal writes nothing; the module’s value is written administratively, by EMQ.INSTALL. Afterwards: unchanged.

Two executors, one stored value

The keys and the transitions are fixed for a wire; identity is gated. But an executor that does not speak the wire a node holds could still write rows that wire's readers cannot understand. The fence closes that gap. It is one reserved key, {emq}:version, holding the node's wire identity, and the two executors fence on it in two ways.

The scripted connector claims or verifies, at connect. EchoMQ.Connector, on its default wire 4, runs the fence once per connection, after the boot sequence (HELLO or AUTH, SELECT, CLIENT SETNAME) and before the socket serves a caller. A matching value runs. An absent key is claimed — SET … NX, then a confirming GET — so of two connectors racing an empty node exactly one write lands, and both read the winner's value. Any other value is {:error, {:version_fence, got}}: at start the connector stops, and on a reconnect the refusal is fatal, so the connection is never retried against a node it does not match.

The native write verbs verify on every write, and never write. Every EMQ.* write verb runs the fence after its shape and layout checks and before any domain check. The fence opens {emq}:version for read as the caller, so ACL key rules apply to it, reads it, and requires the module's identity. An absent key, another value or a wrong type is refused, and a refusal writes nothing. The module serves standalone servers only; on a cluster-enabled server its fence refuses before it reads.

Each executor refuses the other's value. A scripted connector on a node that holds the module's identity stops; a native write verb on a node that holds the scripts' identity refuses. A scripted claim on an empty node writes the scripts' identity, after which every native write verb refuses. One node holds one fence value, so the two executors keep separate nodes: the module runs on its own node, and the Echo service node, whose connectors keep the default wire 4, loads no module. The same connector opened with wire: 6 is the module's client instead: at connect it checks EMQ.VERSION, then reads the key, requires the module's identity and never writes it. The simulator above runs one stored value through both lanes.

The fences, on real code

The reserved key is built by the keyspace. {emq}: is the core's own space — cross-queue keys live there, disjoint from any queue's emq:{q}: keys — and the version key is one of them.

# echo_mq — EchoMQ.Keyspace
# {emq}: is the core's reserved space — cross-queue keys live here, disjoint
# from any queue's emq:{q}: keys. The version key is one of them.
@reserve "{emq}:"
@version_key @reserve <> "version"
# …
@spec version_key() :: binary()
def version_key, do: @version_key

The scripted fence, at connect. @wire_version is the identity this connector speaks on wire 4 — a code constant, not a label. The fence is dispatched on the connection's wire; its wire-4 clause reads the version key once and branches: a match runs, an absent key is claimed with SET … NX and confirmed, and any other value refuses the connection.

# echo_wire — EchoMQ.Connector
# @wire_version is the identity this connector speaks; it is a code constant,
# not a label. The fence reads {emq}:version once: a match runs, an absent key
# is claimed with SET NX then re-read to confirm the claim, and any other
# value refuses the connection. This is the wire-4 clause; a wire: 6
# connection runs fence(sock, buf, 6), which only reads.
@wire_version "echomq:4.0.0"
# …
defp fence(sock, buf, 4) do
  vkey = Keyspace.version_key()

  with {:ok, current, buf2} <- sync(sock, ["GET", vkey], buf) do
    case current do
      @wire_version ->
        :ok

      nil ->
        with {:ok, _, buf3} <- sync(sock, ["SET", vkey, @wire_version, "NX"], buf2),
             {:ok, @wire_version, _} <- sync(sock, ["GET", vkey], buf3) do
          :ok
        else
          {:ok, got, _} -> {:error, {:version_fence, got}}
          err -> err
        end

      got ->
        {:error, {:version_fence, got}}
    end
  end
end

A refusal stops the process. At start, init/1 turns the fence's error into {:stop, reason}; on a reconnect, the handler stops instead of scheduling another attempt. A wrong-typed key is no exception: its WRONGTYPE error reply comes back as the value, and the fence refuses it like any other. The fatal set also holds the three refusals of the wire-6 handshake: the module absent, an EMQ.VERSION error or malformed reply, and a failed client rule.

# echo_wire — EchoMQ.Connector, the fatal set and the :reconnect handler
# The refusals a reconnect does not retry: the server answered, and its
# answer will not change by retrying.
@fatal_connect [:version_fence, :auth_refused, :boot_refused, :emq_absent, :emq_version, :r14]
# …
# A fence refusal is fatal: the connector stops instead of scheduling another
# attempt against a node whose wire it does not speak.
{:error, fatal} when elem(fatal, 0) in @fatal_connect ->
  {:stop, fatal, s}

The native fence, in two beats. Beat one — the named handle. host::fence is the call every EMQ.* write verb makes after its shape and layout checks — EMQ.ENQUEUE, EMQ.CLAIM, EMQ.COMPLETE and the rest. The two controls stand apart: EMQ.VERSION makes no such call, and EMQ.INSTALL makes its own decision on the same key. The key and the one value it accepts are constants, the native mirror of the keyspace:

// trafikey — keyspace, the native mirror of EchoMQ.Keyspace
// the fence key, as Keyspace.version_key/0 builds it
pub const VERSION_KEY: &[u8] = b"{emq}:version";
// the reserve, as Keyspace.reserve/1 prefixes it
pub const RESERVE: &[u8] = b"{emq}:";
// the only value the fence accepts
pub const EPOCH: &[u8] = b"echomq:6.0.0";

Beat two — the fence body. It refuses a cluster-enabled server first, then opens the key for read as the caller and makes one GET. Only the module's value passes, and nothing here writes.

// trafikey — host::fence, run by every write verb before its domain checks
pub fn fence(ctx: &Context) -> Result<Caller, ValkeyError> {
    standalone()?; // D6.4: a cluster refuses before the fence read
    // {emq}:version is not among the call's declared keys, so the module
    // opens it for read as the caller: ACL key rules apply to the fence too.
    let caller = Caller::of(ctx)?;
    caller.open(keyspace::VERSION_KEY, Access::Read)?;
    // One GET. The module's value passes and hands the caller on to the
    // handler; anything else refuses. Nothing here writes.
    match read(ctx, cmd::GET, &[keyspace::VERSION_KEY])? {
        Reply::Bulk(b) if b == keyspace::EPOCH => Ok(caller),
        Reply::Bulk(b) => Err(errors::wire_found(&b)),
        Reply::Null => Err(errors::emq(errors::WIRE_ABSENT)),
        _ => Err(errors::emq(errors::HOST_REPLY)),
    }
}

The two refusals of the fence read, as the module spells them, in the EMQWIRE error class. The cluster refusal above is a third text in the same class.

// trafikey — errors, the two fence-read texts (the class is EMQWIRE)
pub const WIRE_ABSENT: &str = "EMQWIRE expected echomq:6.0.0 but {emq}:version is absent";
// …
pub fn wire_found(brand: &[u8]) -> ValkeyError {
    ValkeyError::String(format!(
        "EMQWIRE expected echomq:6.0.0 found {}",
        printable(brand)
    ))
}

Who writes the module's value. Ordinary clients only read it. On a node that loads the native module, the value is installed administratively, before ordinary clients connect, by EMQ.INSTALL 1 {emq}:version. The caller cannot name a value: the command writes the module's constant or nothing. Its decision, over its one GET:

// trafikey — EMQ.INSTALL, the decision before its single write
// Absent: install (one SET, reply 1). The module's own value: nothing to do
// (reply 0). Any other string: the same EMQWIRE refusal as the fence. The
// value written is the constant EPOCH; the caller cannot name another.
fn decide(stored: Result<Reply, ValkeyError>) -> Result<Step, ValkeyError> {
    match stored? {
        Reply::Null => Ok(Step::Install),
        Reply::Bulk(brand) if brand == keyspace::EPOCH => Ok(Step::Equal),
        Reply::Bulk(brand) => Err(errors::wire_found(&brand)),
        _ => Err(errors::emq(errors::HOST_REPLY)),
    }
}

absent: one SET of the module's value; reply 1
the module's value: nothing written; reply 0
another string: refused with the fence's own EMQWIRE … found text
a non-string: the server's WRONGTYPE, passed through
flags: write deny-oom admin deny-script, in the ACL categories write admin dangerous slow

The SET … NX is the scripted race guard: NX writes only if the key is absent, so of two connectors claiming an empty node exactly one succeeds, and the confirming GET makes the loser read the winner's value — which either matches (it runs) or does not (it is refused). The native fence has no race to guard, because it never writes; its value arrives once, administratively, before ordinary clients connect.

One reserved key, two ways to hold it: the scripts claim or verify it once per connection, the module's write verbs verify it once per command and never write it, and each refuses the other's value.

The module fence — which build serves the node

{emq}:version says which wire a node holds; it says nothing about which module build serves it. EMQ.VERSION does. It is read-only, reads no key and runs no fence, and its reply is a six-entry map:

wire: echomq:6.0.0, the value the module's fence requires
module: trafikey
module_version: the crate version, 0.1.0
module_major: an integer, 0
build: reported, never compared; unset unless the builder sets one
capabilities: the capability tokens of the registered commands, in table order — EMQ.CLAIM lists two (claim, claim.pause), EMQ.VERSION and EMQ.INSTALL none: seventeen tokens, enqueue claim claim.pause extend extends complete.flat schedule promote reschedule delay fail.flat reprocess requeue stalled.check append failures.incr publish, the fourteen of the job core and then one for each bus verb

The client rule. Require wire equal, module_major equal, and every capability the client needs listed. A token is listed only while its command is registered, so a client that passes the check never meets unknown command. EchoMQ.Connector applies it when it opens with wire: 6: it requires wire echomq:6.0.0, module_major 0 and the fourteen job-core tokens, ignores the bus tokens, and only then reads {emq}:version.

TRA.CAPABILITIES reports the module's build facts as eight pairs: module, version, stage, api_version, bitmap_commands, shared_api, bundled_lua and lua_scripts.

The key fences the wire; EMQ.VERSION describes the module. The rule for a client is to check the wire, the major and every capability it needs before it writes.

The reserve map — where does this key live?

Why does the version key belong in {emq}: and not in a queue's space? Because it is not a queue's data — it describes the whole wire. The reserve is first-byte-disjoint from any queue's emq:{q}: keys, so the cross-queue version record is never confused with a queue's pending set or job row. Pick a key and read which space it lands in and why.

Queue space or reserve · pick a key

key {emq}:version → the core reserve {emq}:. hashtag: emq. it describes the whole wire, not one queue — a cross-queue record, so it lives in the core reserve {emq}:.

the pattern (Redis Patterns Applied)

A single reserved key, claimed atomically, can guard a whole connection — SET … NX is the atomic claim, and an atomic check-then-act is what makes the guard safe under a race. Coordination teaches that atomic claim.

the implementation (echo_wire + trafikey)

EchoMQ.Connector.fence/3 reads Keyspace.version_key/0 → {emq}:version at connect; on wire 4, absent → SET … NX + confirm, match → run, mismatch → {:error, {:version_fence, got}} (fatal on reconnect); on wire 6 it only reads and requires the module's value. trafikey's host::fence reads the same key as the caller on every write verb and refuses with EMQWIRE; it never writes, and EMQ.INSTALL writes the module's value administratively.

An executor is checked against the node's wire before it writes, so an executor that does not match the wire never writes to it.

Recap — the third guard

The fence is one reserved key, {emq}:version (EchoMQ.Keyspace.version_key/0), and two ways to hold it. On its default wire 4 the connector claims or verifies it at connect (EchoMQ.Connector.fence/3): a match runs, an absent key is claimed with SET … NX and confirmed, a mismatch stops the connector. The native write verbs verify it on every write (host::fence), as the caller, and never write it; EMQ.INSTALL writes the module's value administratively, and EMQ.VERSION lets a client check the module itself, as a wire 6 connector does before it reads the key. Each executor refuses the other's value, so they keep separate nodes.

An executor is checked against the node's wire before it writes — once per connection for the scripts, once per command for the native module.

Next: back to the module hub, then The native module — the core job transitions compiled into the server, where the second fence lives.

References

Where the atomic claim, the read both fences make, the key permissions and the module mechanism are documented.

Sources

Valkey — SET — the NX atomic claim the scripted fence uses to win an empty-node race uncontested.
Valkey — GET — the one read both fences make.
Valkey — ACL — the key permissions the native fence read is checked against.
Valkey — Modules Introduction — how the native module registers its commands at load.
Valkey — Documentation — the BSD-licensed store EchoMQ is backed by; the substrate of record.

Related in this course

The owned keyspace — the keyspace and the {emq}: reserve in depth.
The branded-id gate — the previous dive: identity gated in the transition.
The native module — the core job transitions compiled into the server, where the second fence lives.
The Protocol — the chapter landing.
redis-patterns R2 · Coordination — the atomic claim, the near side of the door.