Trafikist · Valkey and the native module

Valkey plus one module

One Rust module inside one Valkey server: the job core of EchoMQ as native commands, beside the Lua scripts it is ported from.

Trafikist is a Valkey server with one native module loaded into it. The module, trafikey, is Rust compiled into a shared library that Valkey loads at startup with loadmodule. It refuses the load unless the host is a Valkey 9 server, every host API it calls is present and it was given no arguments. Then it registers two command families from one table: EMQ.*, the job core of the EchoMQ protocol and three bus verbs, and TRA.*, metadata and bitmaps on ordinary strings. The job core runs natively beside the Lua scripts, which stay as wire 4, the compatibility path and the reference the native handlers are ported from. The Protocol’s native module introduced the command form and the fence; this part reads the module in depth: how it loads, the grammar its commands run, the job core it carries, and where the specs take it.

One module in one server One table, two families Beside the scripts The modules The bridge in References

One module in one server

A Valkey module is native code the server loads into its own process; the commands it registers run inside the server like the built-in ones. trafikey is one Rust crate built into one shared library. The server loads it at startup with a loadmodule directive, on a node of its own: the Echo service node loads no module.

Before anything registers, a load gate checks three things: the host is a Valkey 9 server, every host API the module calls is present, and no module arguments were given. Any other answer refuses the load. The reason is in the release build: it aborts on a panic, so a missing API has to refuse the load rather than kill the node. Once the gate passes, init sets every row’s ACL categories and refuses the load on any mismatch. The declaration the server loads names the gate, the init and nothing else:

// trafikey — valkey_module!, the declaration the server loads
valkey_module! {
    name: "trafikey",
    // the module version, an integer; the crate version EMQ.VERSION reports
    // and the TRA metadata version are two separate strings
    version: 100,
    allocator: (ValkeyAlloc, ValkeyAlloc),
    // no module data type: the default build keeps every value in Valkey's own types
    data_types: [],
    // the load gate runs before anything registers, then init
    preload: preload,
    init: init,
    // empty: every command registers from its row in the registration table
    commands: [],
}

A gate before anything registers, and an init that refuses on any mismatch: a node runs the module its table describes, or it does not load the module at all.

One table, two families

The module serves exactly the commands its registration table lists. Each row carries a command’s name, family, arity, flags, key-specs, ACL categories and capability tokens, and everything derives from it: the registration itself, the ACL categories init sets, the capability list EMQ.VERSION reports and the bitmap count TRA.CAPABILITIES reports. The default build has twenty-four rows in two families.

EMQ.* carries the job core of the EchoMQ protocol and three bus verbs: the read-only EMQ.VERSION, thirteen job-core write verbs that carry the flat job lifecycle with fourteen capability tokens, the three bus verbs EMQ.APPEND, EMQ.FAILURES.INCR and EMQ.PUBLISH, one token each, for a stream append, a dead-letter failure count and the bus's durable publish, and the administrative EMQ.INSTALL. Every job-core and bus write verb runs one grammar: shape, layout, the fence on {emq}:version, domain checks, one write guard, then writes that replicate as plain effects. Its own refusals come from one closed registry of error texts; the rest are the server's own.

TRA.* is metadata and bitmaps on ordinary strings: the keyless TRA.CAPABILITIES and TRA.INFO, and TRA.SETBIT, TRA.GETBIT, TRA.STRLEN and TRA.BITCOUNT over one string key. TRA runs a grammar of its own: no fence and no brand read, the host checks arity, a bitmap command on any key type other than a string answers WRONGTYPE, and TRA.SETBIT replicates the command verbatim where EMQ replicates effects. Pick a family to read its rows.

The module map · pick a family

EMQ.* · the job core of the EchoMQ protocol and three bus verbs · 18 commands
rows, in table order: EMQ.VERSION, EMQ.ENQUEUE, EMQ.CLAIM, EMQ.EXTEND, EMQ.EXTENDS, EMQ.COMPLETE, EMQ.SCHEDULE, EMQ.PROMOTE, EMQ.RESCHEDULE, EMQ.DELAY, EMQ.FAIL, EMQ.REPROCESS, EMQ.REQUEUE, EMQ.STALLED.CHECK, EMQ.APPEND, EMQ.FAILURES.INCR, EMQ.PUBLISH, EMQ.INSTALL
kinds: 1 read-only · 13 job-core write verbs · 3 bus write verbs · 1 administrative
capability tokens (17, derived from the rows): enqueue claim claim.pause extend extends complete.flat schedule promote reschedule delay fail.flat reprocess requeue stalled.check append failures.incr publish
grammar, every job-core and bus write verb: shape → layout → the fence on {emq}:version → domain checks → one write guard → writes replicated as plain effects
refusals: its own texts from one closed registry; WRONGTYPE and the HINCRBY and INCR texts are the server’s

Two rows of the table, one from each family, verbatim from the registry:

// trafikey — registry::TABLE, one row from each family
// an EMQ row: a job-core write verb, EVAL's key-spec, one capability token
emq_enqueue => crate::emq::jobs::enqueue, Row { name: c"EMQ.ENQUEUE", family: Family::Emq, arity: 6,
    flags: "write deny-oom fast", keys: EVAL_KEYS, acl: c"write fast", caps: &["enqueue"],
    summary: Some("Admit a flat JOB-namespaced job: row and pending entry, idempotent (@enqueue)"),
    complexity: Some("O(log N)"), since: Some("0.1.0") };
// …
// a TRA row: a bitmap command, a one-key spec on its string, the bitmap token
tra_setbit => crate::tra::bitmap::setbit, Row { name: c"TRA.SETBIT", family: Family::Tra, arity: 4,
    flags: "write deny-oom",
    keys: &[Keys { flags: &["READ_WRITE", "ACCESS", "UPDATE"], index: 1, find: ARGV1 }],
    acl: c"write bitmap slow", caps: &["bitmap"],
    summary: Some("Set one MSB-first bit of a string; returns the previous bit"),
    complexity: Some("O(1) or O(N) in counted bytes"), since: Some("0.1.0-tra7") };

Both families read their capability lists through one function over the same table:

// trafikey — registry::capabilities, the derivation both families read
// a family's rows in table order, each registered row's tokens in turn: EMQ.VERSION
// joins EMQ's tokens into one string; TRA.CAPABILITIES counts TRA's bitmap tokens
pub fn capabilities(
    family: Family,
    registered: impl Fn(&CStr) -> bool,
) -> impl Iterator<Item = &'static str> {
    TABLE
        .iter()
        .filter(move |r| r.family == family && registered(r.name))
        .flat_map(|r| r.caps.iter().copied())
}

Seventeen tokens on sixteen commands: fourteen on the thirteen job-core verbs, where EMQ.CLAIM carries two, claim and claim.pause, then one for each bus verb, append, failures.incr and publish; EMQ.VERSION and EMQ.INSTALL carry none. A token is advertised only while its command is registered, so a capability always names a served command.

A command exists by having its row in the table; its registration, its ACL categories and its place in a capability list all come from that row.

Beside the scripts

The wire-4 Lua scripts in echo/apps/echo_mq run every production transition: every production caller connects on wire 4, the connector’s default, and EchoMQ.Connector evaluates the scripts. A connection opened with wire: 6 speaks to the module instead. It calls EMQ.VERSION and requires the fourteen job-core tokens, reads the brand echomq:6.0.0 and never seeds it, and sends each of the eleven job verbs its scripts declare as the native command in place of EVALSHA; two read-only scripts keep EVALSHA, built-in commands pass as they are, and every undeclared script is refused before anything is sent. No Elixir code sends the three bus verbs, on either wire; the Go client sends EMQ.APPEND and EMQ.PUBLISH. The native commands answer on a module-loaded node whose {emq}:version holds echomq:6.0.0.

The native handlers are ports of those scripts. The row summaries name the scripts they port: @enqueue, @claim2 with the pause read, @schedule, @promote, @delay, @retry, @reprocess2, @sweep_stalled; EMQ.RESCHEDULE and EMQ.REQUEUE have no scripted twin and are held to their specification. The three bus verbs port no script: their summaries name the Elixir functions whose built-in commands they port, EchoMQ.Stream.append_id and append_batch, EchoMQ.Bus.DLQ.record_failure and the durable tier of EchoMQ.Bus.publish. Each matches that sequence on an ordinary call and differs by design at its declared edges: a batch writes all of its entries or none, an id not after the stream’s top is refused before any write, the failure count and its TTL land in one turn, and a publish checks the envelope. The native clock is the command’s cached server time, the value TIME reports. The wire-4 error texts are byte-exact copies of the Elixir source, and a unit test re-reads that tree. A differential lab runs each case on both executors and compares the reply and every key under the queue’s hashtag. The native side writes two row fields the scripts do not, lease_epoch and failures. Lanes, flows and the reaper run only as scripts: the native verbs serve flat jobs, a native complete, fail, delay, promote, reprocess, requeue or stall sweep refuses a lane row with an EMQSTATE text, and nothing in the module blocks, ticks or reaps.

The two executors keep two brands. A default wire: 4 connection seeds echomq:4.0.0 when the brand is absent and stops on any other value, echomq:6.0.0 included; every job-core and bus write verb requires echomq:6.0.0 and refuses an absent or different value. So the scripted path keeps a node of its own, and an Elixir connection serves the module’s node only on wire: 6, which no production caller passes; the conformance task takes it as --wire 6.

Lua’s role follows. The architecture gives Lua two roles, “composition of supported script-safe commands, and an explicitly versioned compatibility/reference path”, and says it “should not remain a second production implementation of every native transition”. So the native module takes over the transitions, and the scripts stay as wire 4, the compatibility path and the reference every native handler is held to.

Two executors, one protocol: the module carries the job core natively, and the scripts keep wire 4 running and every native handler honest.

The modules

Five modules, each a hub and three dives, then a workshop. The first three read what runs today: how the module loads and registers, the grammar every job-core and bus write verb runs, and the job core it carries. Blocking, timers and the bus and TRA and native values pair what runs today with what the specs define, and the workshop runs one transition on both executors.

Valkey and the module

A module is native code Valkey loads at startup; trafikey refuses the load unless the host, the APIs and its own table check out.

The command grammar

Every job-core and bus write verb runs the same order: shape, fence, reads, one guard, writes; its own refusal texts come from one closed registry.

The job core, natively

Fourteen capability tokens on thirteen commands carry the flat job lifecycle natively.

Blocking, timers and the bus

Today a client drives every sweep and nothing in the module blocks or ticks; three bus verbs append, publish and count failures natively, and the specs move waiting and time into the server.

TRA and native values

TRA is the module’s second family: metadata, bitmaps on ordinary strings, a C ABI and bundled Lua.

Workshop

Run one transition, enqueue then claim, on both executors and compare what each leaves in the keyspace.

The bridge in

This part assumes the lesson of the Protocol’s native module and goes below it. Where that module sets one native command beside its script, this part opens the module that serves the command.

the Protocol · the native module

The core job transitions compiled into the server: the EVALSHA line with the sha replaced by the command name, a read-only fence on {emq}:version, and the claim on both executors. The native module teaches it.

Trafikist · this part

The module itself: the load gate and the registration table, the grammar and its closed registry, the fourteen capability tokens with lease_epoch and failures, TRA beside EMQ, and the direction the specs set for blocking, timers, the bus and native values.

Take the modules in order, starting with Valkey and the module.

References

Sources

Valkey — Modules introduction — what a Valkey module is: native code loaded into the server that registers commands.
Valkey — Modules API reference — command registration, key-specs and ACL categories, the surface the registration table drives.
Valkey — MODULE LOAD — the runtime form of what loadmodule does at startup.
Valkey — EVALSHA — how the wire-4 scripts run, and the tail every native command keeps.
Valkey — SETBIT — the built-in bit command beside the TRA bitmap family.
Valkey — ACL — the categories init sets from every row.

Related in this course

The native module — the bridge in: the command form, the fence, one transition on both executors.
One transition, two executors — the claim as a script and as a native command.
The Queue — the job lifecycle both executors carry.
redis-patterns · Patterns become protocol — the move this part takes one step further.