R8.06 · Production & Operations — module hub
Operating EchoMQ.
Operating the bus in production is three operator moves on top of the application code: colocate the keys, observe the bus, and scale the fleet. Each lands on a real surface, and each previews the dedicated EchoMQ course.
The bridge to the dedicated EchoMQ course — preview the operations, door forward for the protocol. Grounding: EchoMQ.Keyspace.slot/1 (echo/apps/echo_mq/lib/echo_mq/keyspace.ex) · EchoMQ.Meter (meter.ex) · the branded-id conformance (echo/apps/echo_data/lib/echo_data/branded_id.ex). The engine is Valkey 9.
Static verdicts — colocate → Keyspace.slot/1 pins a queue to one of
16384 slots · observe → EchoMQ.Meter feeds Prometheus and OpenTelemetry · scale
→ the conformant id is one canon across five runtimes.
The operator's view
colocate · observe · scaleEarlier R8 modules tuned the host and made the bus a record of truth. This module takes the operator's seat and asks what running the bus at scale actually requires — three moves on top of the application code. First, colocate the keys so a multi-key Lua script stays legal when the keyspace is split across a cluster. Second, observe the bus so throughput, latency, and failures are visible through the tools already running. Third, scale the fleet so more workers — eventually in more languages — can drain the same queues.
Each move lands on one real surface, and each is a preview of the dedicated EchoMQ course rather than a re-teaching of the protocol. The module is honest about what ships: the bus is Elixir, and the polyglot fleet is the conformance target the wire and the id contract make reachable. The subject here is the operations.
Operating a job system in production: colocate related keys so multi-key commands stay legal at scale, observe the system through a standard telemetry surface, and scale the worker fleet over one wire.
EchoMQ.Keyspace.slot/1 (the {q} hashtag → one of 16384 cluster slots)
· EchoMQ.Meter ([:emq, …] telemetry → Prometheus / OpenTelemetry)
· the branded-id conformance (one id canon → a polyglot fleet over one wire).
A job system in production is the application plus three operator moves. Colocation keeps the multi-key scripts legal at scale, the telemetry tree makes the bus observable through standard exporters, and the conformant id is the shared contract that lets a fleet of workers drain the same queues over the same wire.
Colocate — the cluster slot
EchoMQ.Keyspace · slot/1The first move is placement. In Valkey Cluster the keyspace is split across 16384 hash slots, and a single
command — including a Lua EVAL — may only touch keys in one slot, or the server answers
CROSSSLOT. EchoMQ makes that constraint disappear by construction: every key of a queue is born
braced — emq:{q}:pending, emq:{q}:active, emq:{q}:job:<JOB>
— and the slot is computed from the substring inside the first {…}, the hashtag.
EchoMQ.Keyspace.slot/1 is rem(crc16(hashtag(key), 0), 16384): CRC16-XMODEM over the
hashtag, modulo 16384, the cluster specification's algorithm computed client-side so the
connector can route without a server round trip. Because every key of one queue shares the brace, all hash to
one slot, so a multi-key script over a queue is always CROSSSLOT-safe.
The source pins one known vector: slot("123456789") == 12739. Dive 1 takes slot/1
and hashtag/1 apart and shows why the braced keyspace keeps a queue's Lua legal at cluster scale.
# CRC16-XMODEM over the {q} hashtag % 16384 — the cluster-spec # algorithm, computed client-side so the connector can route. # Known vector: slot("123456789") == 12739. def slot(key) when is_binary(key), do: rem(crc16(hashtag(key), 0), 16384) def hashtag(key) do with [_, rest] <- :binary.split(key, "{"), [tag, _] when tag != "" <- :binary.split(rest, "}") do tag ← the substring inside the first {…} else _ -> key end end
Notes on ValkeyValkey Cluster splits the keyspace into 16384 hash
slots; a key that contains a {…} hashtag is placed by the bytes inside the braces alone, so a
key family sharing one hashtag lands on one slot — which is what lets a multi-key command stay in a single
slot — valkey.io/topics/cluster-spec.
Observe — the telemetry tree
EchoMQ.Meter · meter.exThe second move is observability. EchoMQ.Meter is the bus's :telemetry surface
— attach, attach_many, emit, span — with every
event re-rooted [:emq | suffix] and every emission guarded so it costs nothing when
:telemetry is not loaded. The lifecycle event tree is fixed:
[:emq, :job, :add|:start|:complete|:fail|:retry],
[:emq, :worker, :start|:stop], and [:emq, :rate_limit, :hit] — plus the
connector's own [:emq, :connector, …], one tree. span/3 wraps a call in
start, stop, and exception events, the standard OpenTelemetry span shape.
Attach a Prometheus exporter or an OpenTelemetry bridge to the [:emq, …] tree and
throughput, latency, and failure are metered the standard Elixir way — no bus-specific agent. For a quick
read at the terminal, EchoMQ.Dashboard is a cat-able ANSI operator view, read-only over the
EchoMQ.Metrics pure-read plane. Dive 2 attaches an exporter to the tree.
# Re-rooted [:emq | suffix]; guarded zero-cost when :telemetry is absent. [:emq, :job, :add | :start | :complete | :fail | :retry] ← a job's life [:emq, :worker, :start | :stop] ← a worker's life [:emq, :rate_limit, :hit] ← a limit fired [:emq, :connector, …] ← the wire's own events
Notes on ValkeyThe bus emits plain :telemetry events, so a
Prometheus exporter or an OpenTelemetry bridge attaches the standard Elixir way; the server side complements it
with INFO and latency metrics scraped from Valkey itself —
valkey.io/topics/admin.
Scale the fleet — the conformant id
branded id · conformantThe third move is scale. A 14-character branded id has four properties — typed,
ordered, placed, and conformant — and the fourth is what scales a fleet.
Conformant means one source of truth and one vector file make encode and decode identical across five runtimes:
Elixir, Node, Go, PostgreSQL, and WASM. Which language runs a worker is a deployment detail. The runtime proves
it at boot — self_check! mints, encodes, decodes, and hashes a known id down both the native
and pure paths and asserts the committed vectors. A node that cannot reproduce them never reaches its first
message.
The honest frame: the shipped bus is Elixir, and a Go or Node worker fleet does not ship today. The polyglot fleet is the conformance target — the wire protocol and the id contract are exactly what make it reachable. Dive 3 reads the boot-asserted vectors and is the door to the dedicated EchoMQ course, where the polyglot protocol and the full Lua inventory live.
The bridge to the EchoMQ course
The three moves preview the bus a production operator runs; the protocol that makes a second runtime a peer on the wire — the polyglot codec, the full Lua inventory, the conformance suite — is the dedicated EchoMQ Proof pillar, and the scaling layer of lanes and the worker fleet is the Queue pillar. For the architecture this colocated, conformant keyspace expresses, see the bus; for where it runs in production, Codemojex on Fly. The id vectors the fleet shares, as the runtime asserts them at boot:
placement("USR0KHTOWnGLuC") → 234878118 ← placed: native & pure agree
parse("USR0NgWEfAEJfs") → {:ok, "USR", 320636799581945856}
decode("USRzzzzzzzzzzz") → :error ← overflow refused, not wrapped
The USR namespace is the manuscript's illustrative brand for the id contract; a real application
uses its own — codemojex mints PLR, ROM, GAM, GES,
and JOB.
The three dives
colocate → observe → scaleEach dive takes one move and the real surface that delivers it, in the arc colocate → observe → scale: the keyspace's cluster slot, the bus's telemetry tree, and the identity's conformance. Read them in order — placement first.
Cluster colocation
EchoMQ.Keyspace.slot/1 and hashtag/1: CRC16-XMODEM over the
{q} hashtag, the vector slot("123456789") == 12739, and why the braced keyspace
keeps a queue's Lua CROSSSLOT-safe.
Prometheus and OpenTelemetry
EchoMQ.Meter's [:emq, …] event tree and span/3
shape: attaching a Prometheus exporter or an OpenTelemetry bridge to a zero-cost telemetry surface.
The polyglot fleet
The branded id's conformant property and the boot-asserted vectors: one id canon across five runtimes, the door to the dedicated EchoMQ course.
Three dives, one operator posture: colocate the keys, observe the throughput, scale the fleet — the cluster slot, the telemetry tree, and the conformant id working as one.
R8.06 is the bridge to the dedicated EchoMQ course. Return to the Production & Operations chapter to continue, open the EchoMQ Proof pillar for the conformance and telemetry in depth, or the bus for the architecture this keyspace expresses.
References
sources · relatedSources
- Valkey — Cluster specification — the 16384 hash slots and how a
{hashtag}forces a key family onto one slot, the basis forslot/1. - Valkey — CLUSTER KEYSLOT — the command that returns a key's slot, the server-side counterpart to the client-side CRC16 computation.
- OpenTelemetry — Traces — the span model
EchoMQ.Meter.span/3follows with its start/stop/exception events. - Prometheus — Documentation — the metrics model an exporter maps the
[:emq, …]telemetry tree onto.
Related in this course
- R8.02 · Persistence, pooling & failoverthe same pool and connector in production
- R7.04 · Bitmap patternsthe same hash32 placement as a bit offset
- /echomq · the Proof pillarconformance, telemetry, production evidence
- /echomq · the Queue pillarlanes, leases, the worker fleet in depth
- /bcs · The busthe braced keyspace as the architecture
- /bcs · Production on Flywhere the bus runs