R8.06.2 · Operating EchoMQ — dive 2

Prometheus & OpenTelemetry.

A bus you cannot see is a bus you cannot operate. The operator needs three numbers — throughput, latency, failure — through the tools already running, not a bus-specific agent. The bus emits a standard telemetry tree.

Grounding: EchoMQ.Meter (echo/apps/echo_mq/lib/echo_mq/meter.ex) — a :telemetry surface re-rooted [:emq | suffix], guarded zero-cost when :telemetry is absent · EchoMQ.Dashboard (dashboard.ex) · the engine is Valkey 9.

One event tree, one exporter. Select a part of the [:emq, …] tree and read what a Prometheus or OpenTelemetry exporter makes of it.

readout · Select an event group. Every group is rooted [:emq, …], so one attached exporter meters the whole bus — throughput, latency, and failure — the standard Elixir way.

Static verdicts — job events meter throughput and latency · worker events meter concurrency · rate_limit meters back-pressure · connector events meter the wire; one exporter reads them all.

§1

The event tree

meter.ex · [:emq | suffix]

EchoMQ.Meter re-roots every event under [:emq | suffix] — one tree for the whole bus. The lifecycle events are fixed, and the connector already fires its own under the same root, so an operator attaches one handler set and sees the whole bus: [:emq, :job, :add|:start|:complete|:fail|:retry] for a job's life, [:emq, :worker, :start|:stop] for a worker's life, [:emq, :rate_limit, :hit] when a limit fires, and [:emq, :connector, …] for the wire.

emit/3 takes an atom name or a suffix list and executes it rooted [:emq | …]. Every emission is guarded: it runs only when :erlang.function_exported(:telemetry, :execute, 3) is true, so a deployment without :telemetry pays nothing — the event never fires and the handler is harmless.

echo/apps/echo_mq/lib/echo_mq/meter.ex · attach + emit, guarded zero-costfrozen

# Re-rooted [:emq | suffix]; guarded zero-cost when :telemetry is absent.
@root :emq
defp loaded?, do: :erlang.function_exported(:telemetry, :execute, 3)

def attach(handler_id, event_suffix, handler_fn, config \\ nil) do
  if loaded?() do
    apply(:telemetry, :attach, [handler_id, [@root | event_suffix], handler_fn, config])
  else
    :ok                              ← no :telemetry → a harmless no-op
  end
end

def emit(event_suffix, measurements, metadata) when is_list(event_suffix) do
  if loaded?(), do: apply(:telemetry, :execute, [[@root | event_suffix], measurements, metadata])
  :ok
end

Notes on ValkeyThe application telemetry pairs with the server side: INFO reports memory, clients, and per-command stats, and the latency tooling reports spikes — so the operator reads the bus from the application events and the engine from Valkey's own metrics — valkey.io/topics/admin.

The bus emits one [:emq, …] event tree, guarded so it costs nothing without :telemetry. One attached handler set sees a job's life, a worker's life, rate-limit pressure, and the wire — the whole bus from one root.

§2

The span shape — OpenTelemetry

meter.ex · span/3

span/3 wraps a function call in start, stop, and exception events around it — the standard :telemetry.span shape, which is exactly the shape an OpenTelemetry bridge maps to a span. The wrapped function always runs; only the events are guarded zero-cost. On a normal return the stop event carries the duration; on an exception the exception event carries the duration plus the reason and stacktrace, and the error re-raises. Step the three outcomes below.

start, then stop or exception. Select an outcome and read which events span/3 emits — the OpenTelemetry span lifecycle.

readout · Select an outcome. span/3 always emits start; then stop with the duration on a normal return, or exception with the duration, reason, and stacktrace before re-raising.

Static verdicts — :start opens the span · :stop closes it with the duration · :exception closes it with the duration, reason, and stacktrace, then re-raises.

Notes on ValkeyAn OpenTelemetry trace stitches a job's span to the spans of the commands it ran, so a slow operation is traced from the worker through the wire to the engine — the standard distributed-tracing read of a job's execution — opentelemetry.io/docs/…/traces.

span/3 is the standard span shape: start, then stop or exception. An OpenTelemetry bridge turns each call into a span with its timing and its error attached, with no bus-specific tracing code.

§3

Attaching an exporter

telemetry_metrics_prometheus · the dashboard

Because the surface is plain :telemetry, the exporters are the standard Elixir ones. Attach telemetry_metrics_prometheus to the [:emq, …] tree and the bus exposes a /metrics endpoint Prometheus scrapes: counts off [:emq, :job, :complete] and [:emq, :job, :fail], latency off the :duration measurement, rate-limit pressure off [:emq, :rate_limit, :hit]. Attach an OpenTelemetry bridge and the span/3 events become traces. The bus needs no custom agent — it speaks the same telemetry every other Elixir library does.

The two reads, side by side:

Prometheus: scrape counters and histograms off the event measurements — throughput, latency, failure as time series.
OpenTelemetry: turn each span/3 call into a trace span — a job's execution stitched across worker, wire, and engine.

# attach an exporter to the [:emq, …] tree — the standard Elixir way.
EchoMQ.Meter.attach_many("prom", [
  [:job, :complete], [:job, :fail], [:rate_limit, :hit]
], &handle/4)
# the events are rooted [:emq | suffix] for you; the handler feeds Prometheus.

For a quick read at the terminal there is EchoMQ.Dashboard: a cat-able ANSI operator view, read-only over the EchoMQ.Metrics pure-read plane. It reimplements no read and opens no new wire — operator tooling, not a bus surface, and the right tool when the answer is “what is the queue doing right now” rather than a scraped time series.

§4

The bridge — events to observability

pattern → application

the pattern

Instrument a system through a standard telemetry surface: emit named events with measurements and metadata, and let standard exporters map them to metrics and traces.

its EchoMQ application

EchoMQ.Meter re-roots every event [:emq, …], guarded zero-cost when :telemetry is absent; a Prometheus exporter or an OpenTelemetry bridge attaches to the tree, and span/3 is the OpenTelemetry span shape.

Observability is a surface, not an agent. The bus emits a fixed [:emq, …] event tree the standard Elixir exporters already understand, so throughput, latency, and failure are visible through Prometheus and OpenTelemetry with no bus-specific tooling.

§5

The production angle

one dashboard, every queue

In production the operator attaches the exporters at boot and reads the bus through the same Grafana and tracing tools the rest of the system uses. There is no separate pane of glass for the job system — its events are on the same tree as everything else.

the door · codemojex on one dashboard

Where the telemetry runs

For codemojex, the bot workers draining the cm queue fire the same [:emq, :job, …] events, so a notify job's throughput and failure are on the same dashboard as every other queue — no special case for the game's workers. The deeper proof surface — the conformance suite and the production evidence the bus carries — is the dedicated EchoMQ Proof pillar. For where the metered bus runs in production, see Codemojex on Fly.

The bus is observable through the tools already running. One exporter on the [:emq, …] tree puts every queue — the game's included — on the same dashboard, with no bus-specific agent.

§6

Recap — the bus speaks telemetry

next → R8.06.3 · the-polyglot-fleet

EchoMQ.Meter re-roots a fixed event tree [:emq, …], guarded zero-cost when :telemetry is absent, and span/3 is the OpenTelemetry span shape. A Prometheus exporter or an OpenTelemetry bridge attaches to the tree, and EchoMQ.Dashboard answers the right-now question at the terminal. That is the observability move of operating the bus. Next, the scale move: the conformant id that lets a fleet of workers — eventually in five runtimes — share the wire.

§7

References

sources · related

Sources

OpenTelemetry — Traces — the span model span/3's start/stop/exception events follow.
Prometheus — Documentation — the metrics model an exporter maps the [:emq, …] telemetry tree onto.
Valkey — Administration — the server-side INFO and latency metrics that complement the application telemetry.
Redis — Documentation — the observability and latency-monitoring topics the metered bus draws on.

Related in this course

R8.06 · Operating EchoMQthe module hub
R8.06.1 · Cluster colocationplacing the bus the exporter observes
R8.06.3 · The polyglot fleetthe fleet whose work this surface meters
/echomq · the Proof pillarconformance, telemetry, production evidence
/bcs · Production on Flywhere the metered bus runs
/bcs · The busthe architecture the telemetry observes