jonnify·redis /redis-patterns/production-operations/operating-echomq

R8.06 · Production & Operations — module hub

Operating EchoMQ.

Operating the bus in production is three operator moves on top of the application code: colocate the keys, observe the bus, and scale the fleet. Each lands on a real surface, and each previews the dedicated EchoMQ course.

The bridge to the dedicated EchoMQ course — preview the operations, door forward for the protocol. Grounding: EchoMQ.Keyspace.slot/1 (echo/apps/echo_mq/lib/echo_mq/keyspace.ex) · EchoMQ.Meter (meter.ex) · the branded-id conformance (echo/apps/echo_data/lib/echo_data/branded_id.ex). The engine is Valkey 9.

One bus in production, three operator moves. Select a move to light the real surface that delivers it; the readout names the surface.
the busin production colocate Keyspace.slot{q} hashtag · 16384 slots observe EchoMQ.Meter[:emq, …] telemetry scale fleet branded-id conformanceone canon · five runtimes
readout · Select a move. The bus is colocated by the keyspace's hashtag, observed through the telemetry tree, and scaled across runtimes by the conformant id.

Static verdicts — colocate → Keyspace.slot/1 pins a queue to one of 16384 slots · observe → EchoMQ.Meter feeds Prometheus and OpenTelemetry · scale → the conformant id is one canon across five runtimes.

§1

The operator's view

colocate · observe · scale

Earlier R8 modules tuned the host and made the bus a record of truth. This module takes the operator's seat and asks what running the bus at scale actually requires — three moves on top of the application code. First, colocate the keys so a multi-key Lua script stays legal when the keyspace is split across a cluster. Second, observe the bus so throughput, latency, and failures are visible through the tools already running. Third, scale the fleet so more workers — eventually in more languages — can drain the same queues.

Each move lands on one real surface, and each is a preview of the dedicated EchoMQ course rather than a re-teaching of the protocol. The module is honest about what ships: the bus is Elixir, and the polyglot fleet is the conformance target the wire and the id contract make reachable. The subject here is the operations.

the pattern

Operating a job system in production: colocate related keys so multi-key commands stay legal at scale, observe the system through a standard telemetry surface, and scale the worker fleet over one wire.

its EchoMQ application

EchoMQ.Keyspace.slot/1 (the {q} hashtag → one of 16384 cluster slots) · EchoMQ.Meter ([:emq, …] telemetry → Prometheus / OpenTelemetry) · the branded-id conformance (one id canon → a polyglot fleet over one wire).

A job system in production is the application plus three operator moves. Colocation keeps the multi-key scripts legal at scale, the telemetry tree makes the bus observable through standard exporters, and the conformant id is the shared contract that lets a fleet of workers drain the same queues over the same wire.

§2

Colocate — the cluster slot

EchoMQ.Keyspace · slot/1

The first move is placement. In Valkey Cluster the keyspace is split across 16384 hash slots, and a single command — including a Lua EVAL — may only touch keys in one slot, or the server answers CROSSSLOT. EchoMQ makes that constraint disappear by construction: every key of a queue is born braced — emq:{q}:pending, emq:{q}:active, emq:{q}:job:<JOB> — and the slot is computed from the substring inside the first {…}, the hashtag. EchoMQ.Keyspace.slot/1 is rem(crc16(hashtag(key), 0), 16384): CRC16-XMODEM over the hashtag, modulo 16384, the cluster specification's algorithm computed client-side so the connector can route without a server round trip. Because every key of one queue shares the brace, all hash to one slot, so a multi-key script over a queue is always CROSSSLOT-safe.

The source pins one known vector: slot("123456789") == 12739. Dive 1 takes slot/1 and hashtag/1 apart and shows why the braced keyspace keeps a queue's Lua legal at cluster scale.

echo/apps/echo_mq/lib/echo_mq/keyspace.ex · the cluster slot, computed client-sidefrozen
# CRC16-XMODEM over the {q} hashtag % 16384 — the cluster-spec
# algorithm, computed client-side so the connector can route.
# Known vector: slot("123456789") == 12739.
def slot(key) when is_binary(key), do: rem(crc16(hashtag(key), 0), 16384)

def hashtag(key) do
  with [_, rest] <- :binary.split(key, "{"),
       [tag, _] when tag != "" <- :binary.split(rest, "}") do
    tag                              ← the substring inside the first {…}
  else
    _ -> key
  end
end

Notes on ValkeyValkey Cluster splits the keyspace into 16384 hash slots; a key that contains a {…} hashtag is placed by the bytes inside the braces alone, so a key family sharing one hashtag lands on one slot — which is what lets a multi-key command stay in a single slot — valkey.io/topics/cluster-spec.

§3

Observe — the telemetry tree

EchoMQ.Meter · meter.ex

The second move is observability. EchoMQ.Meter is the bus's :telemetry surface — attach, attach_many, emit, span — with every event re-rooted [:emq | suffix] and every emission guarded so it costs nothing when :telemetry is not loaded. The lifecycle event tree is fixed: [:emq, :job, :add|:start|:complete|:fail|:retry], [:emq, :worker, :start|:stop], and [:emq, :rate_limit, :hit] — plus the connector's own [:emq, :connector, …], one tree. span/3 wraps a call in start, stop, and exception events, the standard OpenTelemetry span shape.

Attach a Prometheus exporter or an OpenTelemetry bridge to the [:emq, …] tree and throughput, latency, and failure are metered the standard Elixir way — no bus-specific agent. For a quick read at the terminal, EchoMQ.Dashboard is a cat-able ANSI operator view, read-only over the EchoMQ.Metrics pure-read plane. Dive 2 attaches an exporter to the tree.

echo/apps/echo_mq/lib/echo_mq/meter.ex · the lifecycle event treefrozen
# Re-rooted [:emq | suffix]; guarded zero-cost when :telemetry is absent.
[:emq, :job, :add | :start | :complete | :fail | :retry]   ← a job's life
[:emq, :worker, :start | :stop]                       ← a worker's life
[:emq, :rate_limit, :hit]                            ← a limit fired
[:emq, :connector, …]                              ← the wire's own events

Notes on ValkeyThe bus emits plain :telemetry events, so a Prometheus exporter or an OpenTelemetry bridge attaches the standard Elixir way; the server side complements it with INFO and latency metrics scraped from Valkey itself — valkey.io/topics/admin.

§4

Scale the fleet — the conformant id

branded id · conformant

The third move is scale. A 14-character branded id has four properties — typed, ordered, placed, and conformant — and the fourth is what scales a fleet. Conformant means one source of truth and one vector file make encode and decode identical across five runtimes: Elixir, Node, Go, PostgreSQL, and WASM. Which language runs a worker is a deployment detail. The runtime proves it at boot — self_check! mints, encodes, decodes, and hashes a known id down both the native and pure paths and asserts the committed vectors. A node that cannot reproduce them never reaches its first message.

The honest frame: the shipped bus is Elixir, and a Go or Node worker fleet does not ship today. The polyglot fleet is the conformance target — the wire protocol and the id contract are exactly what make it reachable. Dive 3 reads the boot-asserted vectors and is the door to the dedicated EchoMQ course, where the polyglot protocol and the full Lua inventory live.

the door · the operations meet the protocol

The bridge to the EchoMQ course

The three moves preview the bus a production operator runs; the protocol that makes a second runtime a peer on the wire — the polyglot codec, the full Lua inventory, the conformance suite — is the dedicated EchoMQ Proof pillar, and the scaling layer of lanes and the worker fleet is the Queue pillar. For the architecture this colocated, conformant keyspace expresses, see the bus; for where it runs in production, Codemojex on Fly. The id vectors the fleet shares, as the runtime asserts them at boot:

echo/apps/echo_data/lib/echo_data/branded_id.ex · self_check! — the conformance vectorsfrozen
placement("USR0KHTOWnGLuC")  →  234878118        ← placed: native & pure agree
parse("USR0NgWEfAEJfs")      →  {:ok, "USR", 320636799581945856}
decode("USRzzzzzzzzzzz")     →  :error           ← overflow refused, not wrapped

The USR namespace is the manuscript's illustrative brand for the id contract; a real application uses its own — codemojex mints PLR, ROM, GAM, GES, and JOB.

§5

The three dives

colocate → observe → scale

Each dive takes one move and the real surface that delivers it, in the arc colocate → observe → scale: the keyspace's cluster slot, the bus's telemetry tree, and the identity's conformance. Read them in order — placement first.

Three dives, one operator posture: colocate the keys, observe the throughput, scale the fleet — the cluster slot, the telemetry tree, and the conformant id working as one.

R8.06 is the bridge to the dedicated EchoMQ course. Return to the Production & Operations chapter to continue, open the EchoMQ Proof pillar for the conformance and telemetry in depth, or the bus for the architecture this keyspace expresses.

§6

References

sources · related

Sources

Related in this course