jonnifyechomq

TRA and native values · dive 03

The native value type

The default build registers no module data type. The one data type in the source, traprobe1, compiles only behind a feature that is off by default, and it exists to prove that a module-owned value survives RDB, AOF and a restart. A native bitmap value type is designed, not built.

A module data type is how a Valkey module owns a value: the server keeps an opaque pointer under the key and calls the module’s callbacks to save the value to an RDB file, load it back, rewrite it into the AOF and free it. Every value trafikey’s default build writes is one of the server’s own types: a TRA bitmap is a string, an EMQ job row is a hash. The probe type is the exception, compiled only on request, and its callbacks are the recorded proof that a module-owned value persists. Pick a value to read who owns its bytes and who writes it to disk; then walk the probe through a save, a rewrite and a restart.

No data type in the default build

The module declaration lists its data types, and the list is empty. The default build creates none anywhere else either: the one call that creates a data type sits behind the probe-type feature. So every value the default build writes is one of the server’s own types. A TRA bitmap is a string: the server keeps its bytes, saves it to RDB, writes it on an AOF rewrite and frees it, as it does for any string, and GET reads it with or without the module. An EMQ job row is a hash the module writes through the server’s own commands. The manifest ends on the same fact: no bitmap data type is implemented in the crate.

// trafikey — the module declaration: the list of data types is empty
valkey_module! {
    name: "trafikey",
    version: 100,
    allocator: (ValkeyAlloc, ValkeyAlloc),
    data_types: [],
    preload: preload,
    init: init,
    commands: [],
}

Who owns a value · pick one

the server’s own types module data types stringa TRA bitmap hashan EMQ job row traprobe1probe build only RDBthe server, as a string AOFTRA.SETBIT, verbatim without the modulea string: GET reads it a TRA bitmap: the server owns the value, the module owns the command
a TRA bitmap · TYPE string
owner: the server: a string, written in place by TRA.SETBIT
RDB: written by the server, as a string
AOF: TRA.SETBIT as the client sent it; a rewrite writes the string as it writes any string
a restart without the module: the RDB copy loads as an ordinary string, and GET reads the bytes

In the default build the server owns every value; the module owns commands, a C table and three Lua sources.

Off by default

The crate’s feature table declares probe-type with no dependencies and declares no default set, so a plain build leaves the probe out. The repository’s build line is plain: it builds the package and passes no feature.

# trafikey — Cargo.toml, the feature table and the closing comment
[features]
# the persistence probe for a module data type, never part of a shipped artifact;
# no default feature is declared, so a plain build leaves it off
probe-type = []
# …
# The cdylib is loaded by Valkey; no standalone RESP listener or bitmap data type is implemented here.
# deploy — build-local.sh, the build line: the package, no feature flag
cargo build --locked --release -p trafikey

The feature gates three places. The probe_type module compiles only under it. The first step of init, also under it, creates the data type, and a failure refuses the load. Two rows of the registration table, also under it, register TRAPROBE.SET and TRAPROBE.GET in a family of their own, Probe, with no capability token, so neither EMQ.VERSION nor TRA.CAPABILITIES reports them. A default build has neither the type nor the two commands.

// trafikey — lib.rs: the probe module and init's first step, both under the feature
mod emq;
#[cfg(feature = "probe-type")]
mod probe_type;
mod registry;
mod tra;
// …
fn init(ctx: &Context, _: &[ValkeyString]) -> Status {
    #[cfg(feature = "probe-type")]
    if probe_type::PROBE_TYPE.create_data_type(ctx.ctx).is_err() {
        return Status::Err;
    }
    // … every later step of init, the shared API export last
    Status::Ok
}
// trafikey — registry::TABLE, the two probe rows: family Probe, no capability token
    #[cfg(feature = "probe-type")]
    probe_set => crate::probe_type::set, Row { name: c"TRAPROBE.SET", family: Family::Probe, arity: 3,
        flags: "write deny-oom", keys: &[Keys { flags: &["READ_WRITE", "UPDATE"], index: 1, find: ARGV1 }],
        acl: c"write slow", caps: &[], summary: None, complexity: None, since: None };
    #[cfg(feature = "probe-type")]
    probe_get => crate::probe_type::get, Row { name: c"TRAPROBE.GET", family: Family::Probe, arity: 2,
        flags: "readonly fast", keys: &[Keys { flags: &["READ_ONLY", "ACCESS"], index: 1, find: ARGV1 }],
        acl: c"read fast", caps: &[], summary: None, complexity: None, since: None };

One feature switches the probe on in three places at once, and nothing in the repository’s build switches it on.

Four callbacks, two commands

The type is named traprobe1, nine characters as the module API requires, with an encoding version of 0. Its value is a box of bytes in module memory. It fills four callbacks: rdb_save writes the bytes as one string buffer; rdb_load reads them back and returns NULL for any encoding version but 0; aof_rewrite emits one TRAPROBE.SET key bytes command per value; free drops the box. It reports no digest and no memory usage.

// trafikey — probe_type: the traprobe1 type and its four callbacks
pub struct Probe(Vec<u8>);

pub static PROBE_TYPE: ValkeyType = ValkeyType::new(
    "traprobe1",
    0,
    raw::RedisModuleTypeMethods {
        version: raw::REDISMODULE_TYPE_METHOD_VERSION as u64,
        rdb_load: Some(rdb_load),
        rdb_save: Some(rdb_save),
        aof_rewrite: Some(aof_rewrite),
        free: Some(free),
        digest: None,
        mem_usage: None,
        // … aux_*, free_effort, unlink, copy, defrag and the rest: None or 0
    },
);

unsafe extern "C" fn rdb_load(rdb: *mut raw::RedisModuleIO, encver: c_int) -> *mut c_void {
    if encver != 0 {
        return std::ptr::null_mut();
    }
    match raw::load_string_buffer(rdb) {
        Ok(buffer) => Box::into_raw(Box::new(Probe(buffer.as_ref().to_vec()))).cast(),
        Err(_) => std::ptr::null_mut(),
    }
}
unsafe extern "C" fn rdb_save(rdb: *mut raw::RedisModuleIO, value: *mut c_void) {
    raw::save_slice(rdb, unsafe { &(*value.cast::<Probe>()).0 });
}
unsafe extern "C" fn aof_rewrite(
    aof: *mut raw::RedisModuleIO,
    key: *mut raw::RedisModuleString,
    value: *mut c_void,
) {
    let bytes = unsafe { &(*value.cast::<Probe>()).0 };
    #[expect(
        clippy::unwrap_used,
        reason = "the preload gate refuses the load without EmitAOF; aof_rewrite has no error channel"
    )]
    unsafe {
        raw::RedisModule_EmitAOF.unwrap()(
            aof,
            c"TRAPROBE.SET".as_ptr(),
            c"sb".as_ptr(),
            key,
            bytes.as_ptr(),
            bytes.len(),
        );
    }
}
unsafe extern "C" fn free(value: *mut c_void) {
    drop(unsafe { Box::from_raw(value.cast::<Probe>()) });
}

TRAPROBE.SET key bytes stores a new box under the key through the binding’s typed key API, replicates verbatim and replies OK. TRAPROBE.GET key returns the bytes, or a null reply for a missing key. On a key of any other type the binding knows, its own check answers Existing key has wrong Valkey type, not the standard WRONGTYPE reply, and the record keeps that as a finding for any handler built on the typed key API. That check first converts the raw key type, so on a Valkey 9.2 path hash key it is the abort trap that Bitmaps on strings reads.

// trafikey — probe_type: TRAPROBE.SET and TRAPROBE.GET
pub(crate) fn set(ctx: &Context, args: Vec<ValkeyString>) -> ValkeyResult {
    // The host enforces arity 3 before the handler runs; the else arm is its arity reply.
    let [_, name, payload] = args.as_slice() else {
        return Err(ValkeyError::WrongArity);
    };
    ctx.open_key_writable(name)
        .set_value(&PROBE_TYPE, Probe(payload.as_slice().to_vec()))?;
    ctx.replicate_verbatim();
    Ok(ValkeyValue::SimpleStringStatic("OK"))
}
pub(crate) fn get(ctx: &Context, args: Vec<ValkeyString>) -> ValkeyResult {
    // The host enforces arity 2 before the handler runs; the else arm is its arity reply.
    let [_, name] = args.as_slice() else {
        return Err(ValkeyError::WrongArity);
    };
    Ok(match ctx.open_key(name).get_value::<Probe>(&PROBE_TYPE)? {
        Some(probe) => ValkeyValue::StringBuffer(probe.0.clone()),
        None => ValkeyValue::Null,
    })
}

The probe’s evidence is a recorded scratch run on Valkey 9.1.2, on scratch ports apart from the stand’s two nodes. TRAPROBE.SET pk "hello\x01world", SAVE and a restart gave back TYPE pk as traprobe1 and the bytes exactly. With appendonly yes and aof-use-rdb-preamble no, BGREWRITEAOF wrote a base file holding two TRAPROBE.SET commands from the rewrite callback, and a restart restored both keys. The control started a server without the module on the same dump.rdb, and the server refused it: The RDB file contains module data I can't load: no matching module type 'traprobe1'.

The probe, step by step · pick a step

commandSAVE, then restart Probe(Vec<u8>)rebuilt, 11 bytes on diskdump.rdb rdb_save rdb_load aof_rewrite free RDB: rdb_save, then rdb_load at the restart
SAVE, then restart · recorded on scratch Valkey 9.1.2
what runs: rdb_save writes the bytes as one string buffer; at the restart rdb_load checks that the encoding version is 0 and rebuilds the box
callbacks: rdb_save, rdb_load
reply: TYPE pk → traprobe1; TRAPROBE.GET pk → the same 11 bytes

The probe proves the mechanism, a module value through RDB, AOF and a restart; it carries no bitmap and ships in no artifact.

Pattern & implementation

Where Redis Patterns Applied keeps one bitmap string per day and names the cost of a sparse id space, this dive reads a module whose bitmaps are that same flat string.

the pattern (Redis Patterns Applied)

One bitmap per day, a bit per user id, counted and combined; a bitmap allocates to its highest offset, so a sparse id space wastes memory. Daily-active patterns teaches it.

the implementation (trafikey)

A TRA bitmap is the server’s string: TRA.SETBIT at a high offset grows it with zero bytes up to that offset. The module’s one data type is a persistence probe, compiled only on request.

This dive closes the module. Return to the hub for the whole family in one picture, or go on to the workshop.

References

Sources

Related in this course