Telemetry & the read plane · dive 03

The read plane

Answer a question from the structures the bus already keeps — and move nothing in the asking.

Where the meter pushes events as work happens, the read plane pulls: a caller asks, the bus answers from the structures it already keeps. The defining property is one sentence — every verb observes; none mutates. A metric read never moves a set member, re-schedules a job, or spends a token. EchoMQ.Metrics answers counts per state, a job's state by which structure holds it, throughput, and the rate-gate — and a metric that perturbed the thing it measured would be worse than no metric at all. Pick a state on the right to read its count off the structure beneath it.

Counts per state A job's state Throughput, the gate, lane depth Pattern & implementation References

A state · the structure it is counted off

Pick a state below to see the structure its count is read off.

Counts per state — read the structures, move nothing

get_counts/3 answers a map of state → count over the bus's as-built structures. The four live states — pending, active, schedule, dead — are sorted sets, so their count is a ZCARD of each set. The two terminal outcomes — completed, failed — leave no set behind (completion deletes the job), so their count is read from a small terminal-outcome counter the completion and dead-letter transitions keep. One read, two kinds of structure, no member touched.

The verb is also closed: it validates every requested state name against the registered set first. An unrecognized name is {:error, {:unknown_state, name}} — never an open read that quietly concatenates a caller-chosen string into a key. The closed registry is the same discipline the whole protocol rests on: the bus answers a known set of questions, precisely.

# echo_mq — EchoMQ.Metrics
# Count per requested state. Validate the names against the closed registry FIRST
# (an unknown name is a typed error, never an open read). The set states read
# ZCARD of their sorted set; the metric states read the terminal-outcome counter
# (completion deletes the job, so it leaves no set). The queue base is the declared
# slot root, so even a metric-only request pins the {q} slot. A pure read.
def get_counts(conn, queue, states) when is_binary(queue) and is_list(states) do
  with :ok <- validate_states(states) do
    {sets, metrics} = Enum.split_with(states, &(&1 in @set_states))
    base = Keyspace.queue_key(queue, "")
    keys = [base | Enum.map(sets, &Keyspace.queue_key(queue, &1))]

    case Connector.eval(conn, @counts, keys, metrics) do
      {:ok, counts} ->
        ordered = sets ++ metrics
        {:ok, ordered |> Enum.zip(counts) |> Map.new()}

      other -> other
    end
  end
end

A count is a ZCARD, not a scan; a terminal outcome is a counter, not a resurrected set — the read plane reports the shape the structures already hold, and moves none of it.

A job's state — which structure holds the id

get_job_state/3 answers a different question: for one branded id, which state is it in? The read checks the four set keys in order — pending, active, scheduled, dead — and answers the first that holds the id by its score. If no set holds it, the read consults the row itself: a flow parent held out of every set carries awaiting_children on its row; a row that exists but sits in no set (an id in flight between transitions) answers unknown; and no row at all answers absent. The id is gated by BrandedId.valid?/1 at the key builder, so an ill-formed id is refused before any wire command is sent. Pick an id to see which structure answers.

An id · which structure answers

The closed-set discipline appears again on the way out: the wire string maps through an explicit table to its atom, never String.to_existing_atom. An unexpected wire string is a typed error, not an open conversion that could raise — the read plane stays honest even about its own answers.

# echo_mq — EchoMQ.Metrics
# The job's state by which structure holds the id. The four set keys are declared
# in KEYS; the id is gated at the key builder. The wire string maps through a
# closed table to its atom (never to_existing_atom) — an unexpected string is a
# typed error, never an open conversion. A pure read.
def get_job_state(conn, queue, job_id) when is_binary(job_id) do
  keys =
    Enum.map(@set_states, &Keyspace.queue_key(queue, &1)) ++ [Keyspace.job_key(queue, job_id)]

  case Connector.eval(conn, @state_lookup, keys, [job_id]) do
    {:ok, state} when is_binary(state) ->
      case @lookup_states do
        %{^state => atom} -> {:ok, atom}
        _ -> {:error, {:unknown_state, state}}
      end

    other -> other
  end
end

The rest of the plane — throughput, the rate gate, lane depth

The same observe-never-mutate rule shapes the rest of the verbs:

get_metrics/3 — the terminal-transition throughput for :completed or :failed: the count the completion/dead-letter transition tallied, read honestly (the series length is reported as it stands, never a phantom). get_rate_limit_ttl/3 — the remaining rate-limit TTL in milliseconds (0 = not limited): the limiter read against the configured max. A read of the gate, not a spend of it. is_maxed/2 — the concurrency gate as a read-and-refuse: where the active set is at the configured ceiling, the refusal leads with the EMQRATE wire class, mapped to {:error, :rate}; otherwise :ok. No state transition, no member moved — a read of the ceiling and a typed refusal. lane_depth/3 — the pending depth behind one group identity, delegating to the as-built Lanes.depth, which gates the group with BrandedId.

Across all of them, every read script declares its keys — the queue base is passed as a declared KEYS root (so the {q} slot is pinned even for a read that requests no set key), never a base concatenated in-script from an argument. That is the protocol's slot-soundness law applied to the read plane: a multi-key read stays legal on a cluster, and a state name a caller chose can never become a key.

Throughput is a tallied counter; the rate-gate is a read-and-refuse; a lane depth is a count behind a gated id — the whole plane answers from what the bus already holds, and every read declares the keys it touches, so an observation never becomes a mutation and never becomes an open key.

Pattern & implementation

the pattern (Redis Patterns Applied)

Read the system's state for an operator — counts, throughput, the rate-gate — without changing the work. Production Operations teaches running the tier; the atomicity these reads rest on is Coordination.

the implementation (echo_mq)

EchoMQ.Metrics is pure-read: get_counts/3, get_job_state/3, get_metrics/3, get_rate_limit_ttl / is_maxed/2, lane_depth/3 — every verb observes, every read declares its keys, and an unregistered state name is a typed error, never an open read.

The pattern says: read the state without disturbing it. The implementation enforces it — every verb is a read, every read declares its keys, and the closed registry refuses anything it does not recognize.

Recap — observe, never mutate

The read plane is the pull side of the self-report: get_counts/3 reads per-state counts off the four sorted sets and the terminal-outcome counters; get_job_state/3 locates one branded id by which structure holds it; get_metrics/3, get_rate_limit_ttl, is_maxed/2, and lane_depth/3 answer throughput, the gate, and lane depth. Every verb observes, none mutates, and every read declares its keys. With the meter's push side, that is the whole pillar's middle claim: the proven system reports on itself, honestly, without being asked to change.

Back to Telemetry & the read plane to fold the module, or on to the Proof landing for the conformance suite and the benchmark frontier.

References

Where the read plane's verbs and the structures they observe are grounded, and the discipline of reading state without perturbing it.

Sources

Valkey — ZCARD — the cardinality get_counts/3 reads each state set by.
Valkey — ZSCORE — the membership test get_job_state/3 resolves an id by.
Valkey — HGET — the terminal-outcome counter and the row fields the plane reads.
Beck — Test-Driven Development — observing the externally visible state, never perturbing it.

Related in this course

Telemetry & the read plane — the module this dive belongs to.
The telemetry surface — the push side that emits as work happens.
The Queue — the four state sets and the lifecycle the read plane reads.
The Protocol — the declared-keys law every read script obeys.
Coordination — the atomicity these reads rest on.
The Branded Component System · together — the manuscript chapter where the four libraries are one umbrella.