jonnifyechomq

Valkey and the module · dive 01

Loading the module

One gate before anything registers, then one init that runs its steps in a fixed order and stops at the first that fails.

The module's declaration names two functions. preload runs first, before any command exists, and asks one question: can the module run on this host? It requires three things at once: every host API the module calls, no module arguments, and a Valkey 9 server. Then the table's commands register, and init runs its steps in order: the ACL categories of every row, the server's run_id, the server mode, the shared C API. Set the gate's three inputs on the right, then step through init below.

The preload gate · set the host

apis args.is_empty() v.major == 9 AND Status::Ok
preload → Status::Ok
apis: true · args.is_empty(): true · v.major == 9: true
next: every row of the table registers its command, then init runs

The declaration: two hooks, no commands, no value types

Everything the server learns about the module starts at one macro. valkey_module! names the module, its allocator, the two hooks the load runs, and two lists, both empty.

// trafikey — the module declaration
valkey_module! {
    name: "trafikey",
    // the declaration's own number; EMQ.VERSION and TRA.CAPABILITIES report other strings
    version: 100,
    allocator: (ValkeyAlloc, ValkeyAlloc),
    // no module value type in the default build
    data_types: [],
    // the gate, before anything registers
    preload: preload,
    // the steps after every row has registered
    init: init,
    // no command from the macro: each one comes from a row of registry::TABLE
    commands: [],
}

data_types: [] means the default build registers no module value type; the persistence probe type exists only under a crate feature that no shipped artifact enables. commands: [] means no command comes from the macro: each one comes from a row of the registration table, which the next dive reads. And version: 100 is the declaration's own number, one of three separate version strings: EMQ.VERSION reports the crate's version and its major as module_version and module_major, and TRA.CAPABILITIES reports a version of its own. A client checks module_major.

Two hooks and two empty lists: the macro declares when the module checks, and the table declares what it serves.

The gate: three conditions, one match arm

preload gathers two facts before it answers. registry::host_apis_present() checks every host function pointer the module calls; ctx.get_server_version() asks the server for its version. The one arm that answers Status::Ok needs all three conditions at once. Every other case answers Status::Err, a version the host cannot report included, and the load is refused before a single command exists.

// trafikey — preload, the gate the declaration names
/// Load gate, before anything registers: no module arguments, a Valkey 9.x host, and every
/// host API this module calls present. raw wrappers unwrap the host pointers and the release
/// profile aborts on panic, so a missing API must refuse the load instead of killing the node.
fn preload(ctx: &Context, args: &[ValkeyString]) -> Status {
    // one boolean over the whole API list (read below)
    let apis = registry::host_apis_present();
    // the server's own version; an Err here falls to the refusing arm
    match ctx.get_server_version() {
        // the API list, the loadmodule line's arguments, the major: all three or nothing
        Ok(v) if apis && args.is_empty() && v.major == 9 => Status::Ok,
        _ => Status::Err,
    }
}

The doc comment gives the reason the gate refuses rather than trusting the host. The release profile aborts on a panic, and that abort is no recovery: it keeps a panic from unwinding across the C boundary by ending the process, and inside a loaded module the process is the node.

# trafikist — the workspace release profile
[profile.release]
# …
panic = "abort"
# …
# Release abort semantics prevent unwinding across FFI; they do not provide panic recovery.

The major-version condition lives in preload because the binding the crate compiles against cannot carry it. That binding is pinned, and its compatibility feature is the Valkey 8 level; the module's own requirement, a Valkey 9 host, is enforced at load:

# trafikey — the host binding the crate compiles against
[dependencies]
# Pinned host binding: crates.io valkey-module 0.1.14 (checksummed; its feature set predates git 9ed1bbf,
# which adds a 9-0 feature). Bindgen runs over the crate's bundled header; the 9.x gate is in preload().
valkey-module = { version = "=0.1.14", default-features = false, features = ["min-valkey-compatibility-version-8-0"] }

The gate is the one place a host is judged: compiled against a binding at the Valkey 8 level, loaded only on a Valkey 9 server, with arguments refused outright.

The API half of the gate

host_apis_present is one boolean over every host function the module calls, in groups: key and string access, replies, verbatim replication, keyspace notifications, the shared-API export, command registration, module data types, then the raw host adapter the EMQ.* handlers use (Call and its reply readers, the cached clock, the replica-traffic check), the caller's ACL check, the server-info reads behind the epoch, and the context flags behind the server mode.

// trafikey — registry::host_apis_present, the API half of the gate
macro_rules! all_present {
    ($($api:ident),* $(,)?) => { true $(&& unsafe { (*std::ptr::addr_of!(raw::$api)).is_some() })* };
}

/// The preload API gate: every host API this module calls is present. raw wrappers unwrap the
/// host pointers and the release profile aborts on panic, so a missing API must refuse the load.
pub fn host_apis_present() -> bool {
    all_present!(
        RedisModule_OpenKey,
        // …
        RedisModule_CreateCommand,
        RedisModule_GetCommand,
        RedisModule_SetCommandInfo,
        RedisModule_SetCommandACLCategories,
        // …
        // emq: the raw host adapter (emq/host.rs) and its replies.
        RedisModule_Call,
        // …
        RedisModule_CachedMicroseconds,
        RedisModule_AvoidReplicaTraffic,
        // …
        // the caller's ACL check, for keys outside a command's key-spec
        RedisModule_ACLCheckKeyPermissions,
        // …
        // the three server-info reads behind the epoch init loads
        RedisModule_GetServerInfo,
        RedisModule_ServerInfoGetFieldC,
        RedisModule_FreeServerInfo,
        // the context flags behind the server mode init records
        RedisModule_GetContextFlags,
    )
}

The macro expands to true && … over one presence test per name, so a single absent pointer makes the whole gate false. The list also says what the module never calls: it names no blocking-client API, no ready-key signal and no timer. Nothing in the module blocks a client or runs on a clock of its own; every sweep runs when a client calls it.

The API list is the module's whole contract with its host, checked once, before the first registration.

init, in order

Once the gate has passed and the table's commands have registered, init runs its steps in a fixed order. A step that fails refuses the load at once, and the steps after it never run.

// trafikey — init, after every row has registered
fn init(ctx: &Context, _: &[ValkeyString]) -> Status {
    // step 0, only in a build with the probe-type feature: the probe's module data type
    #[cfg(feature = "probe-type")]
    if probe_type::PROBE_TYPE.create_data_type(ctx.ctx).is_err() {
        return Status::Err;
    }
    // step 1: every row's ACL categories; a table that differs from the registered set refuses
    if registry::set_acl_categories(ctx) == Status::Err {
        return Status::Err;
    }
    // step 2: the server epoch, run_id, read once and held in memory; no key is written
    if let Err(e) = emq::epoch::load(ctx) {
        ctx.log_warning(&format!(
            "trafikey: emq epoch not loaded: {}",
            emq::errors::text(&e)
        ));
        return Status::Err;
    }
    // step 3: the server mode, read once; this step cannot refuse
    emq::host::load_mode(ctx);
    // step 4: the versioned shared C API, exported under its name for other modules
    registry::export_shared_api(
        ctx,
        c"trafikey_get_api_v1",
        tra::trafikey_get_api_v1 as *const std::ffi::c_void,
    );
    Status::Ok
}

Step 1, the ACL categories. Every row's categories are set on its registered command, and the step first proves the table and the registered set equal: distinct names, one registered command per row, each found in this module. The next dive reads it with the table.

Step 2, the epoch. The server's run_id is read from INFO server once, checked, and held for the life of the process:

// trafikey — emq::epoch::load, init's step 2
/// Reads `INFO server` `run_id` once (host.rs [`super::host::server_run_id`]) and caches it for
/// the process. A second call keeps the first value (one process, one `run_id`). A missing host
/// API or a malformed value refuses typed, and module init then refuses the load.
pub(crate) fn load(ctx: &Context) -> Result<(), ValkeyError> {
    let id = host::server_run_id(ctx)?;
    // exactly 40 lowercase hex characters, as the server prints a run_id
    if !valid_run_id(&id) {
        return Err(errors::emq(errors::HOST_CALL));
    }
    let _ = RUN_ID.set(id);
    Ok(())
}

Every lease the job core writes records that value, and only the stall sweep reads it, to tell a restart from a stall; The job core natively reads that use. Step 3 caches whether the server is cluster-enabled, a startup-only setting, so the value cannot change while the module is loaded; the third dive reads what follows from it. Step 4 exports one name and one pointer, the versioned C API another module fetches to reach the bundled TRA Lua sources; TRA and native values reads it.

// trafikey — registry::export_shared_api, init's step 4: one name, one pointer
pub fn export_shared_api(ctx: &Context, name: &CStr, api: *const c_void) {
    unsafe { raw::export_shared_api(ctx.ctx, api, name.as_ptr()) };
}

init, step by step · pick a build and an outcome

probefeature only ACLstep 1 epochstep 2 modestep 3 shared APIstep 4 Ok the module is loaded
default build · every step passes — Status::Ok
step 0, probe type: compiled out (no probe-type feature)
step 1, ACL categories: set on 24 rows; the table equals the registered set
step 2, epoch: run_id read, 40 lowercase hex characters, held in memory
step 3, server mode: recorded
step 4, shared API: trafikey_get_api_v1 exported
log: nothing

A fixed order with early returns: by the time init answers Status::Ok, the categories, the epoch and the mode are all in place.

the pattern (Redis Patterns Applied)

An immutable core checks at boot and refuses to serve on a value it cannot trust, so a wrong deployment fails loudly at start instead of quietly at the first request. The immutable core teaches it.

the implementation (trafikey)

preload refuses a non-9 host, a missing host API or any argument before a command exists; init refuses a table mismatch, a rejected category or an unreadable run_id before the load completes.

Recap — loading the module

The declaration names a gate and an init, registers no value type and lists no command. The gate requires every host API, no arguments and a Valkey 9 server, all at once, before any command exists. Then every row registers, and init sets the ACL categories, reads the epoch, records the server mode and exports the shared API, in that order, refusing the load at the first step that fails. The next dive reads the table every command comes from.

Next in this module: The registration table — one row per command, and everything that derives from it.

References

Where the load is grounded, and the server behaviour it relies on.

Sources

Related in this course