Telemetry & the read plane · dive 01

The telemetry surface

Re-root the standard :telemetry surface under one namespace, so one attach reaches the whole bus.

A running bus reports on itself by emitting events as work happens, the standard Elixir :telemetry way. EchoMQ.Meter takes a suffix — [:job, :start] — and re-roots it under the bus namespace [:emq, …], so every event the system fires lives on one tree. The connector already fires [:emq, :connector, …]; the lifecycle fires [:emq, :job, …]; one attach prefix reaches both. On top of emit and span, named lifecycle emitters fire a job's beats by name. Pick a beat on the right to see the event it fires.

Re-rooted, one tree The lifecycle emitters span, measure & re-raise Pattern & implementation References

A lifecycle beat · the event it fires

Pick a lifecycle beat below to see the event it fires.

Re-rooted under `[:emq, …]` — one tree

Telemetry events are lists of atoms. EchoMQ.Meter takes a suffix — [:job, :start], [:job, :complete] — and roots it under the bus namespace, so the host sees [:emq, :job, :start]. The connector already fires [:emq, :connector, …], so the meter's events and the connector's events are the same tree: one attach prefix ([:emq, …]) reaches the whole bus. That is the point of the re-root — the host subscribes to one namespace and gets every signal the system emits.

attach/4 is a convenience over :telemetry.attach/4: it prepends :emq to the suffix and hands off. attach_many/4 does the same for a list of suffixes in one call — the usual way to meter a whole lifecycle at once.

# echo_mq — EchoMQ.Meter
# @root is :emq — every event the bus fires is rooted [:emq | suffix], so the
# connector's [:emq, :connector, …] and the lifecycle's [:emq, :job, …] are ONE
# tree. attach/4 is a convenience over :telemetry.attach/4: prepend the root,
# hand off. (With no :telemetry loaded this answers :ok as a no-op — next dive.)
@root :emq

def attach(handler_id, event_suffix, handler_fn, config \\ nil) do
  if loaded?() do
    apply(:telemetry, :attach, [handler_id, [@root | event_suffix], handler_fn, config])
  else
    :ok
  end
end

# attach to several events at once — the usual way to meter a whole lifecycle.
def attach_many(handler_id, event_suffixes, handler_fn, config \\ nil) do
  if loaded?() do
    events = Enum.map(event_suffixes, fn suffix -> [@root | suffix] end)
    apply(:telemetry, :attach_many, [handler_id, events, handler_fn, config])
  else
    :ok
  end
end

One namespace, one tree: the host attaches to [:emq, …] once and hears the connector and the whole job lifecycle — the re-root is what makes the bus a single source of signals.

The lifecycle emitters — the named beats of a job

emit/3 fires one event; span/3 wraps a function call in start/stop/exception events. On top of those, EchoMQ.Meter exposes named lifecycle emitters so a job's beats are fired by name with the right measurements and metadata already shaped:

job_added/4 → [:emq, :job, :add] with a queue_time measurement. job_started/4 → [:emq, :job, :start] with system_time and the worker pid. job_completed/5 → [:emq, :job, :complete] with duration. job_failed/6 → [:emq, :job, :fail] with duration and the error.

Each is a thin call into emit/3: the measurements are the numbers a handler aggregates (a histogram of duration, a counter of :fail), and the metadata is the context (queue, job_id, job_name, worker). Because they all route through emit/3, they all inherit the zero-cost guard the next dive is about — a host that has not opted into :telemetry pays nothing for these calls in the lifecycle.

# echo_mq — EchoMQ.Meter
# The named lifecycle beats. Each is a thin emit/3 with the measurements a handler
# aggregates (duration, queue_time) and the metadata that contexts them (queue,
# job_id, job_name, worker). All rooted [:emq, :job, …] — one tree with the
# connector's events.
def job_started(queue, job_id, job_name, worker_pid) do
  emit([:job, :start], %{system_time: System.system_time()}, %{
    queue: queue, job_id: job_id, job_name: job_name, worker: worker_pid
  })
end

def job_completed(queue, job_id, job_name, worker_pid, duration) do
  emit([:job, :complete], %{duration: duration}, %{
    queue: queue, job_id: job_id, job_name: job_name, worker: worker_pid
  })
end

def job_failed(queue, job_id, job_name, worker_pid, duration, error) do
  emit([:job, :fail], %{duration: duration}, %{
    queue: queue, job_id: job_id, job_name: job_name, worker: worker_pid, error: error
  })
end

The split between measurements and metadata is the :telemetry convention, and it is load-bearing: a handler aggregates the measurements (sum the duration, count the :fail) and groups by the metadata (per queue, per job_name). The emitters shape both so a host's handler is a one-liner.

`span/3` — measure a block, surface a raise

Where the emitters mark a single instant, span/3 measures a block. It is the standard :telemetry.span shape, re-rooted: it fires [:emq | suffix, :start] before the function, [:emq | suffix, :stop] with a measured duration after it succeeds, and [:emq | suffix, :exception] (with the kind, reason, and stacktrace) if it raises — then re-raises, so the span never swallows the error. Step the block below to watch the events fire around it.

span/3 · the events around a block

The one rule that makes span/3 safe to wrap anything: the wrapped function always runs — only the events are guarded zero-cost. With :telemetry absent the span reduces to fun.(); the work is never skipped because nobody is metering, and a raise is never swallowed because the span re-raises.

Pattern & implementation

the pattern (Redis Patterns Applied)

Instrument the running system so an operator can see throughput, latency, and failures — without changing the work. Production Operations teaches running the tier.

the implementation (echo_mq)

EchoMQ.Meter re-roots the standard :telemetry surface under [:emq, …], names the lifecycle beats, and span/3 measures a block while re-raising any error — one tree the host attaches once to meter the whole bus.

The pattern says: instrument the system without disturbing it. The implementation gives you one namespace to attach to — the connector and the whole job lifecycle on a single tree.

Recap — one tree, named beats, a measured block

EchoMQ.Meter is the push side of the self-report: attach / attach_many subscribe a handler to the re-rooted [:emq, …] tree; the named lifecycle emitters fire a job's beats with the measurements a handler aggregates and the metadata that groups them; and span/3 measures a block while always running it and re-raising any error. One namespace, one attach, the whole bus. The next dive is the property that makes all of this free when a host does not want it.

Next in the module: Zero cost when absent — why every one of these emitters costs nothing unless a host opts into :telemetry.

References

Where the telemetry surface and its re-rooted event tree are grounded, and the metering convention they rest on.

Sources

Elixir — the :telemetry library — the metering surface the Meter re-roots under [:emq, …].
Erlang/OTP — :telemetry.execute/3 — the execute the lifecycle emitters call.
Erlang/OTP — :telemetry.span/3 — the start/stop/exception shape span/3 re-roots.

Related in this course

Telemetry & the read plane — the module this dive belongs to.
Zero cost when absent — why these emitters cost nothing unless a host opts in.
The read plane — the pull side that answers state by reading.
The Queue — the job lifecycle these events are the beats of.
The Branded Component System · together — the manuscript chapter where the four libraries are one umbrella.