jonnifyechomq

The events channel · dive 02

Publish after the verdict

The announcement belongs to the host, not the script that decides.

An event is published after a transition decides, from the host — not inside the script that decides. publish/5 gates the id at the key builder, builds a small cjson object by hand, and issues one PUBLISH emq:{q}:events direct through EchoMQ.Connector.command/3 — no new script, no Lua. That one placement choice is what keeps the byte-frozen transition scripts byte-unchanged. See the choice on the right, then read the code.

Why host-side · two placements

the transition script @complete — bytes UNCHANGED, SHA frozen freeze holds the publish publish/5 — host-side, AFTER the verdict PUBLISH emq:{q}:events (direct, no Lua) the verdict is the script's; the announcement is the host's

Where the publish lives — and why it is host-side

A queue transition (complete, retry, the stalled reaper) is an atomic Lua script. The publish does not go inside it. The calling code runs the transition, reads its verdict, and then — host-side, in Elixir — calls publish/5 to announce what happened. The reason is a hard invariant of this bus: the transition scripts are byte-frozen. Adding a PUBLISH to a script would change its bytes (and its SHA), which is exactly what the freeze forbids. Publishing from the host keeps every script byte-for-byte identical to what shipped, while still broadcasting the outcome.

(One event is published inline because its transition already does: progress. The @update_progress seam established the emq:{q}:events channel and the cjson payload shape; every other lifecycle event reuses that channel from the host.)

# echo_mq — EchoMQ.Events.publish/5 (host-side, after the verdict)
# Announce a lifecycle event AFTER its transition has decided. The transition
# scripts are byte-frozen, so the publish lives here in the host, not in the
# script — the scripts stay byte-unchanged. One PUBLISH, issued DIRECT (no Lua).
def publish(conn, queue, event, job_id, extra \ []) do
  # gate the id at the key builder (INV5) — raises on an ill-formed id BEFORE the
  # wire, so a malformed JOB id can never reach a PUBLISH
  _ = Keyspace.job_key(queue, job_id)
  payload = encode_event(event, job_id, extra)

  case Connector.command(conn, ["PUBLISH", channel(queue), payload]) do
    {:ok, _n} -> :ok                  # best-effort — n is the receiver count (may be 0)
    other -> other
  end
end

# the channel — emq:{q}:events, the SAME suffix the @update_progress seam publishes on
def channel(queue), do: Keyspace.queue_key(queue, "events")

Two facts carry weight. First, the id is gated at the key builder before anything is sent: Keyspace.job_key(queue, job_id) raises if the job id is ill-formed, so a malformed branded id cannot reach a PUBLISH. The return value is discarded — the call exists purely for its gate. Second, the publish is best-effort: PUBLISH answers the number of receivers, which the function throws away and maps to :ok. Zero receivers is success, not error.

Split the work: the script decides, the host announces. That one line of separation is what lets the bus broadcast every outcome without ever editing a frozen transition.

The payload — flat cjson, built by hand

The payload is a tiny JSON object of flat string fields: the event name, the job id, and any extra keyword pairs (e.g. [progress: "50"]). The bus carries no JSON dependency, so the object is built by string concatenation, with the two JSON-significant characters escaped.

# echo_mq — EchoMQ.Events.encode_event/3 (the flat cjson payload, built by hand)
# {"event":"<name>","job":"<id>", …extra} — flat string fields only. No JSON
# library on the bus; the object is concatenated, with \ and " escaped.
defp encode_event(event, job_id, extra) do
  fields =
    [{"event", to_string(event)}, {"job", job_id}] ++
      Enum.map(extra, fn {k, v} -> {to_string(k), to_string(v)} end)

  body =
    fields
    |> Enum.map(fn {k, v} -> ~s("#{k}":"#{escape(v)}") end)
    |> Enum.join(",")

  "{" <> body <> "}"
end

defp escape(v) do
  v
  |> String.replace("\\", "\\\\")   # backslash first
  |> String.replace("\"", "\\\"")   # then the quote
end

No JSON library on the bus — only a concatenated object with the two significant characters escaped. Small payloads, zero dependency.

Reading the name back — a scan, not a parser

On the receive side, the consumer needs the event name. It does not parse the JSON. Because the payload is hand-built and cjson key order is not guaranteed, the name is read by a substring scan for the event field, and the matched string is resolved with String.to_existing_atom/1. That last choice is the safety one: an unknown name answers :unknown rather than minting an atom from the wire. Atoms are not garbage collected; turning arbitrary wire bytes into atoms is an unbounded-memory hazard, so the function only ever returns an atom that already exists.

# echo_mq — EchoMQ.Events.event_name/1 (read the name by scan; never mint an atom)
# The bus has no JSON parser and cjson key order is not guaranteed — so read the
# "event" field by substring scan. Resolve with to_existing_atom: an UNKNOWN
# name answers :unknown, never minting an atom from the wire (atoms aren't GC'd).
def event_name(payload) when is_binary(payload) do
  case Regex.run(~r/"event"\s*:\s*"([^"]+)"/, payload) do
    [_, name] ->
      try do
        String.to_existing_atom(name)     # only an atom that already exists
      rescue
        ArgumentError -> :unknown          # unknown name → :unknown, not a new atom
      end

    _ ->
      :unknown
  end
end

Read the name by scan, resolve it to an existing atom or :unknown — never mint an atom from the wire. A hand-built payload deserves a hand-built, safe read.

The publish round-trip

Pick a verdict and watch the whole round-trip: publish/5 gates the id, builds the cjson payload with encode_event/3, and issues PUBLISH emq:{q}:events; then event_name/1 scans the same payload back to the atom. The unknown option injects a name no atom exists for, so the read answers :unknown. The strings shown are exactly what the code produces — pure, no network.

publish/5 → encode_event/3 → event_name/1

encode_event → PUBLISH · event_name (scan) ·

The id is gated before the payload is built — an ill-formed job id raises at job_key/2 and never reaches the wire. The read is a scan, so it tolerates any key order and refuses any name it does not already know.

Pattern & implementation

the pattern (Redis Patterns Applied)

Emit a domain event when state changes, so other parts of the system can react without coupling to the writer. R5 · Streams & Events teaches the events side of the pattern.

the implementation (echo_mq)

publish/5 emits after the transition's verdict, host-side, gating the id at the key builder and issuing one PUBLISH emq:{q}:events direct — so the byte-frozen transition scripts stay byte-unchanged.

The pattern says emit on the change. The implementation emits from the host after the change, so the change's script never has to move.

Recap — the host announces

A lifecycle event is published host-side, after a transition's verdict, by publish/5 — one PUBLISH emq:{q}:events through EchoMQ.Connector.command/3, no Lua. The id is gated at the key builder before the wire; the cjson payload is built by hand (no JSON dependency); the receiver reads the name by substring scan and resolves it with String.to_existing_atom/1, answering :unknown rather than minting an atom. The next dive: the guarantee the channel makes — at-most-once — and its control-plane sibling, EchoMQ.Cancel.

Next in the module: Fire-and-forget — what at-most-once costs, the resubscribe and the stream as the two mitigations, and the cooperative cancel token.

References

Where the host-side publish, the hand-built payload, and the safe name read are grounded, and the command beneath them.

Sources

Related in this course