Trafikist · module 03

The job core, natively

Thirteen commands, fourteen capability tokens: the flat job lifecycle, run inside the server beside the scripts it was ported from.

The job core is the part of EchoMQ every job passes through: admission, the claim and its lease, completion, scheduling, failure and recovery. trafikey runs that core natively. Thirteen EMQ.* write verbs move a flat job between the four sets of its queue, and EMQ.VERSION advertises fourteen capability tokens for them: one per verb, and a second one for the pause read folded into the claim. Each verb takes the keys of the script it was ported from or, for the two verbs with no scripted twin, of its specification. On an ordinary path a ported verb replies as its script does and makes the same writes, plus the lease stamp and the failure count below. On the edges it differs by design: it refuses a row that is not active where the script would act on it, writes nothing on a repeated terminal failure, creates no row for a ghost id, refuses lane and flow rows, and checks every argument and key type before its first write. The row carries three counters with three separate jobs, attempts, failures and stalled, and every lease carries the server's process identity. EchoMQ's connector sends eleven of these commands when it opens with wire: 6; on its default wire: 4 connection, EchoMQ.Jobs evaluates the scripts.

The flat lifecycle Fourteen tokens Three counters and a stamp Who calls them The dives References

Thirteen verbs, one flat lifecycle

A flat job lives in one of four sorted sets under its queue's braced prefix — pending, active, schedule, dead — and the state field of its row names the same place: pending, active, scheduled, dead. Completion deletes the row. Every native verb is one move between those places, or a re-score inside one, and every check it makes runs before its first write.

Flat is the boundary of the native job core. Seven verbs refuse a lane row (a row with a group field), and EMQ.COMPLETE and EMQ.FAIL also refuse a flow child (a row with a parent field), each with an EMQSTATE … not native text and nothing written. Lanes, flows and the reaper run as scripts only. Pick a command to see the move it makes, the tokens it advertises, the writes it plans, the counters it touches and the script it mirrors.

The flat lifecycle · pick a command

EMQ.CLAIM · pending → active
tokens: claim, claim.pause
writes: ZPOPMIN pending; HINCRBY row attempts 1; HSET row state active lease_epoch <run_id>; ZADD active <now + lease_ms> id
counters: attempts +1, the holder’s new token; lease_epoch stamped
scripted twin: EchoMQ.Jobs @claim2, with paused?/2 folded in
reply: [id, payload, attempts]; an empty array when the queue is paused, pending is empty or the head has no row

One command, one move: the native verb plans every write before it makes the first, and the verbs that settle, move or recover a job refuse a lane row.

Fourteen tokens on thirteen verbs

Each verb's row in the registration table carries its capability tokens, and EMQ.VERSION answers them as one space-separated text in the table's row order. The crate's own test pins the full list: the fourteen job-core tokens, then the three tokens of the bus verbs, whose rows follow the job core in the table:

// trafikey — the token list as the crate's test pins it: every EMQ row registered,
// the tokens joined with one space, in the order of the registration table
assert_eq!(caps(|_| true), "enqueue claim claim.pause extend extends complete.flat \
                            schedule promote reschedule delay fail.flat reprocess requeue stalled.check \
                            append failures.incr publish");

Twelve verbs advertise one token each; the claim advertises two, claim for the lease and claim.pause for the pause read folded into it. EMQ.VERSION carries none: it is the command that lists them. A token is advertised only while its command is registered, so a client that finds stalled.check in the list can call EMQ.STALLED.CHECK. Two tokens carry their boundary in a suffix: complete.flat and fail.flat name completion and failure of flat rows, and no row carries a token for grouped or flow completion. The last three tokens, append, failures.incr and publish, belong to the bus verbs that the native bus reads; a wire: 6 connection requires the fourteen job-core tokens and ignores the rest. The first dive derives the list from the table and follows a client through the check it makes before its first call.

The list is a reading of the table, not a second table: a token exists because a registered row carries it.

Three counters and a stamp

The scripts' row holds state, attempts and payload, plus stalled once the stall sweep has counted it. The native row keeps all of them and adds two fields the scripts never write: failures and lease_epoch. The four fields that count or stamp have four separate jobs:

The counters on a native job row
field	written by	read by	its one job
`attempts`	EMQ.ENQUEUE and EMQ.SCHEDULE write `0`; EMQ.CLAIM adds 1	every verb that takes a token: EXTEND, EXTENDS, COMPLETE, DELAY, FAIL, REQUEUE	the fencing token a holder presents, compared as bytes
`failures`	EMQ.FAIL adds 1	EMQ.FAIL	the retry budget, capped by `max_attempts`
`stalled`	EMQ.STALLED.CHECK adds 1 on a counted expiry	EMQ.STALLED.CHECK	the stall budget, capped by `max_stalled`
`lease_epoch`	EMQ.CLAIM, EMQ.EXTEND, EMQ.EXTENDS write the server's `run_id`	EMQ.STALLED.CHECK only	tells an expiry that crossed a server restart from a worker's stall

The scripted retry caps on attempts, the claim count. The native EMQ.FAIL caps on failures instead, so a lease recovered by the sweep or a job released by EMQ.REQUEUE spends none of the retry budget, and attempts stays a token and nothing else. EMQ.REQUEUE writes no counter at all, and EMQ.REPROCESS keeps every counter when it returns a dead job to pending. The second dive follows the stamp from the server's process identity to the one command that reads it; the third follows the three budgets and the one native recovery verb.

One counter per question: who holds the job, how often it failed, how often its lease ran out under the same server process.

Who calls them today

EchoMQ's connector sends them. A connection opened with wire: 6 calls EMQ.VERSION and requires the fourteen job-core tokens among those it lists, then reads the {emq}:version brand, which must already hold echomq:6.0.0; it never writes the brand. From then on, a job-core script that names its verb travels as that verb, with the script's numkeys, KEYS and ARGV unchanged. Eleven scripts in EchoMQ.Jobs and EchoMQ.Stalled name one; two scripts in EchoMQ.Metrics, the counts and the state lookup, are declared reads and keep EVALSHA; any other script is refused with {:wire6_unsupported, name}, so no Lua write reaches a wire-6 node. The default connection is still wire: 4, which evaluates every script on a node branded echomq:4.0.0, and every production caller opens that default; the conformance task opens wire: 6 when asked to. EMQ.REQUEUE and EMQ.RESCHEDULE have no script to stand in for, so no Elixir code sends them.

# what a wire 6 connection sends for a claim: the EVALSHA tail, the sha replaced by the verb
EMQ.CLAIM 3 emq:{orders}:pending emq:{orders}:active emq:{orders}: 30000
# the reply: [id, payload, attempts]; an empty array when the queue is paused or pending is empty

The two executors keep separate nodes, because each refuses the other's brand value, and nothing in the module runs on its own: EMQ.PROMOTE and EMQ.STALLED.CHECK move time-driven work only when a client sends them. The Protocol's native module introduced the command form and the fence; this module reads the job core that runs behind them. Who sends those two commands, and when, is the subject of Blocking, timers and the bus.

One library, two executors: the connection's wire decides whether a job-core call runs as its script or as its native verb.

The dives

Three dives read the job core in depth, each grounded in the real trafikey handlers and registry beside the scripts in echo/apps/echo_mq.

built page ready to read

01built

Fourteen capabilities

The token list derived from the registered rows in table order; the test that pins it; the check a client makes on it before its first call.

02built

The lease epoch

Valkey's run_id, read once at load and stamped by every lease write; the stall sweep reads it, alone, to tell a restart from a stall.

03built

Failures and recovery

attempts fences, failures caps FAIL, stalled caps the sweep; EMQ.STALLED.CHECK is the one native recovery verb, and the reaper has no native twin.

Pattern & implementation

the pattern (Redis Patterns Applied)

A job's state is the set it sits in, and a transition is one atomic move between sets, read and written in one server-side step. The atomic state machine and Stalled recovery teach it.

the implementation (trafikey)

Thirteen EMQ.* verbs move a flat job between pending, active, schedule and dead, one native command per move, with three budgets on the row and the server's process identity on every lease.

Take the dives in order. The first starts from the table; the last ends at the one recovery verb the module serves.

Up next

→built

Fourteen capabilities

Start the module: the token list and the client's check.

←built

One transition, two executors

The Protocol's claim on both executors, the lesson this module assumes.

References

Sources

Valkey — Modules introduction — native code the server loads, registering its own commands.
Valkey — Modules API reference — command registration and the calls a handler makes into the server.
Valkey — ZADD — the score that places a job in pending, active, schedule or dead.
Valkey — HINCRBY — the increment behind attempts, failures and stalled.
Valkey — INFO — the run_id field of INFO server, the stamp on every native lease.

Related in this course

The command grammar — the previous module: the check order every one of these verbs runs.
The native module — the Protocol's introduction to the module: the command form and the fence.
The state and attempts — the row fields both executors write.
The four sets — pending, active, schedule and dead in the Queue.
Completion and recovery — the scripted exits from active.
redis-patterns · The atomic state machine — states as locations, the near side of the door.