A jonnify course
EchoMQ In Depth
One Valkey, one wire you own — a queue, a bus, and a cache, from a single protocol.
EchoMQ is one Valkey-native job system you own end to end. Below the language line sits the
protocol: the keys, the record fields and the atomic transitions, run by two executors — the
inline Lua scripts that the Elixir connector sends by EVALSHA on its default wire 4, and the native
trafikey module, where each core job transition is one EMQ.*
command inside the server, which the same connector sends on wire 6. Above it
stand the three surfaces you reach for every day: the Queue that distributes work, the
Bus that broadcasts signals and keeps a replayable event log, and the Cache that
serves reads.
This course teaches their internals in depth, and Elixir-canonical — the real keys, the real Lua
and native commands, and the real module functions, lifted onto the page and annotated — and what they mean for
every system in the BCS family that rides them. Because the protocol lives below the language
line, any runtime that sends the commands is a peer (with the native module, the core job transitions take no
script), so EchoMQ is supported in and ported to others. It is the course on the far side of every
“→ EchoMQ” door in Redis Patterns Applied.
One system, three pillars
EchoMQ is not a library you bolt onto an app — it is a protocol you own. The keys, the record
fields and the atomic transitions that run against Valkey are the system, whether a Lua script or a
native EMQ.* command runs them; everything a caller touches sits above them. Draw the
line where the agreement lives: below it, the keys and the atomic transitions, fixed and shared; above it, the
language you happen to write in. Push the agreement below that line and three surfaces fall out of one wire —
the Queue (distribute work: enqueue, claim, retry, complete), the Bus
(broadcast signals: pub/sub events and a retained, replayable log), and the Cache (serve
reads: an L1 ETS / L2 Valkey near-cache kept coherent on the Bus).
Why Valkey. EchoMQ is backed by Valkey — the BSD-licensed, foundation-governed fork with Redis-semantics scripting, a module API, AOF durability, and hash-field TTL. It is the neutral substrate of record, the store every durability and coherence claim is phrased against, and it outlives any one vendor's licensing mood.
The course map · pick a chapter
One system, three pillars; the protocol below the line; the door and the BCS family.
applies ← redis-patterns
One protocol below the line; the Queue, the Bus, and the Cache above it — open the chapter you need.
What's inside · the map
The whole system, chapter by chapter. Read the foundation first — the Overview, then the
Protocol that the three pillars share. Then take the pillar you need: the Queue, the
Bus, the Cache. The Proof chapter shows the system holds —
conformance and telemetry, with the benchmark named as its frontier. Trafikist then opens the native executor in depth: Valkey and the trafikey module. Each chapter closes with a hands-on workshop.
The system & its protocol
What EchoMQ is, and the wire every pillar stands on — the owned keyspace, the record hash, the native module, and the immutability that lets one protocol serve three surfaces. Taught in depth on the real code.
Overview
Open chapter →One system, three pillars; the protocol below the line; the Redis-Patterns door and what the pillars mean for the BCS family.
← redis-patterns
The three pillars
The Queue, the Bus, the Cache — one system, three surfaces over one wire.
The protocol below the line
Keys, fields and transitions shared and fixed; the language above. Why that line makes it polyglot and coherent.
The door & the BCS family
The bidirectional Redis-Patterns door, and what EchoMQ means for the systems built on it.
The Protocol
Open chapter →The substrate the three pillars share: the owned keyspace, the record hash, immutability and branded ids, and the native module.
← redis-patterns R0.3 · R2
The owned keyspace
emq:{q}: with a hashtag per queue, the {emq}: reserve, and the slot the hashtag pins.
The record hash
The job row: the HASH, the fields, and what each holds — part of the protocol.
Immutability & branded ids
The immutable line below the language; the branded-id gate and the version fences.
The native module
trafikey: each core job transition one EMQ.* command inside the server, behind a read-only fence.
Workshop
Decode a real key, trace one declaration to its EMQ.* command, and read the retired Lua script beside it.
What EchoMQ does
The three surfaces over the one wire — distribute work, broadcast signals, serve reads. Each applies a Redis pattern you can follow back to Redis Patterns Applied, and each is grounded in the real Elixir code.
The Queue
Open chapter →Distribute work: the lifecycle, jobs · lanes · the consumer, fair groups, batches, lifecycle controls, and flows.
← redis-patterns R3 · R4 · R6
The lifecycle
The state machine: enqueue → claim → complete or retry; one atomic transition per step.
Jobs, lanes & the consumer
The producer surface, fair grouped lanes, and the supervised consumer loop.
Batches
Bulk in one flush: enqueue-many, bulk flows, and batch lease-extension.
Lifecycle controls
Scheduling and recurrence, cooperative cancel and checkpoints, and the operator plane.
Flows
Orchestration: parent and child jobs, fan-in on completion, cross-queue delivery.
Workshop
Run a job through every state; compose a flow and watch the fan-in complete.
The Bus
Open chapter →Broadcast signals: pub/sub events, the retained replayable event log, consumer groups, time-travel and the archive.
← redis-patterns R5 · R6
The events channel
The event seam: subscribe to a queue's lifecycle, publish a signal, decoupled one-to-many.
The stream log
The retained, replayable stream; append is mint order; read at your own offset.
Consumer groups
At-least-once delivery, crash re-delivery, and the polyglot reader seam.
Time-travel
Read the log by a mint instant: a DateTime becomes an id range bound.
Retention & the archive
Bound the log by length or age; fold the trimmed segments to the durable floor.
Workshop
Publish, append, replay by instant, then watch a retention sweep fold to disk.
The Cache
Open chapter →Serve reads: a two-layer cache-aside near-cache, single-flight filled, jittered-TTL swept, coherent on the Bus.
← redis-patterns R1 · R5
Cache-aside, two layers
fetch over L1 ETS at memory speed and L2 Valkey on a miss; the branded key.
Single-flight & jittered TTL
One loader for a thundering herd; cohorts that never expire in step.
Coherence — newer wins on the Bus
Version-guarded fills; invalidation rides the Bus, newer mint wins.
Workshop
Hit at ETS speed; invalidate from another node; survive a herd with one fill.
The system holds
How the wire is kept honest across every runtime — the black-box conformance suite and the telemetry catalog, with the benchmark named as the frontier.
The Proof
Open chapter →Conformance and telemetry & the read plane — the evidence the protocol is correct and observable; the benchmark is the named frontier.
← redis-patterns R8
The conformance suite
The scenarios that keep the owned wire honest across runtimes — each asserts an externally visible verdict.
Telemetry & the read plane
The lifecycle metered via :telemetry at zero cost when absent; a pure-read plane over the as-built sets.
Benchmark — the frontier
Throughput and latency under load — named as the frontier; no page.
Workshop
Run the suite, attach a meter and watch the events flow, then read the plane.
Valkey and the module
The second executor in depth — how Valkey loads the trafikey module, the grammar every
job-core write verb runs, the flat job lifecycle it carries natively, who moves time today, and its second
family, TRA. Taught on the real Rust.
Trafikist
Open chapter →Valkey and the trafikey module: loading and registration, the command grammar, the job core natively, blocking, timers and the bus, and TRA and native values.
← redis-patterns R0.3
Valkey and the module
Native code Valkey loads at startup; the load gate, one registration table, standalone only.
The command grammar
Shape, fence, reads, one guard, writes; its own refusals from one closed registry.
The job core, natively
Fourteen capability tokens on thirteen commands carry the flat job lifecycle.
Blocking, timers and the bus
Today a client drives every sweep; nothing in the module blocks, ticks or publishes to the bus.
TRA and native values
Metadata and bitmaps on ordinary strings, a C ABI with the bundled Lua, and a feature-gated persistence probe.
Workshop
Run one enqueue and one claim on both executors and compare what each leaves in the keyspace.
Canonical in Elixir, polyglot by design
This course teaches EchoMQ through its canonical Elixir implementation — the real modules, the real Lua and the native commands, lifted onto the page and annotated. That is where the depth is. But the depth is portable: because the protocol is the keys, the fields and the transitions, not the library, any runtime that sends them is a peer, and with the native module it carries no script for the core job transitions. EchoMQ is supported in and ported to other runtimes; what you learn here — the keyspace, the record, the transitions, the lifecycle — is the same wire they speak. The pillars are taught once, in Elixir, and they hold wherever the protocol runs.
Learn the wire once, in Elixir; it is the same wire every peer runtime speaks.
Why the BCS architecture builds on EchoMQ
EchoMQ is not a library bolted onto an app — it is the queue, the bus, and the cache of the Branded Component System. The BCS course teaches the architecture: the 14-byte branded snowflake, the grammar-total keyspace law, fair lanes, and a coherent cache-aside near-cache. EchoMQ is that architecture, built — the bus (jobs, lanes, the consumer) and the cache (the L1/L2 near-cache, kept coherent on the bus) are the same modules this course opens up.
So a system built on the BCS architecture inherits its queue, its cache, and its coherence from EchoMQ for free. The worked example is codemojex, the BCS worked project: an emoji-guessing game whose scoring jobs, fair-lane guess claims, and round cache all ride EchoMQ. Read this course to learn the bus; read the BCS course to see the architecture it completes.
Learn EchoMQ here; see where it lands — the whole branded architecture — in the BCS course.
References
Sources
- Redis — Documentation — the data structures and commands the protocol is built from.
- Valkey — ZPOPMIN — the oldest-pending pop a claim is built on: the job leaves the pending set and takes a lease in the active set.
- Valkey — Modules Introduction — how a module registers its own commands inside the server, as the native module registers each core job transition as one
EMQ.*command. - llmstxt.org — The llms.txt convention — the machine-readable map format the course follows.
Related in this course
- Overview — one system, three pillars, and the door.
- The Protocol — the owned keyspace, the native module, immutability.
- Trafikist — Valkey and the trafikey module in depth.
- Redis Patterns Applied — the near side of the door; the pattern source.
- The Branded Component System — the architecture EchoMQ completes.