jonnifyechomq

The stream log · dive 01

The host-side mint

The writer mints the id itself, derives the entry id from it, and appends direct — no Lua, nothing to spoof.

To append to the stream is to run append/4: mint an EVT record id host-side, derive the explicit XADD id from it by field correspondence, issue XADD direct, and return {:ok, branded} — the receipt. Two facts carry the rest: the writer owns the mint, so there is nothing for a caller to spoof, and the entry id is a function of the mint, not of when the wire happened to accept the write. Step the four moves on the right, then read the code.

The append The id derivation Pattern & implementation References

The append · step the moves

move 1 mint: next_branded("EVT") → branded (host-side) move 2 derive: xadd_id/1 → "<ms>-<tail22>" move 3 append: XADD <key> <xadd_id> id <branded> … move 4 receipt: return {:ok, branded} no Lua — XADD issued direct through the connector

The append, and the kind door

The happy path. append/4 mints an EVT id with EchoData.Snowflake.next_branded/1, then delegates to append_id/5. The writer owns the mint, so the id is born in its own process — there is no "give me an id and I'll trust it" path on this route. fields is a claims-only flat pair list.

# echo_mq — EchoMQ.Stream
# append/4: mint an EVT id host-side, then delegate to the kind-checked append.
# The writer OWNS the mint (next_branded/1) — nothing to spoof. fields is a
# claims-only flat pair list ([{k, v}, …] or a pre-flattened [k, v, …]).
def append(conn, queue, name, fields)
    when is_binary(queue) and is_binary(name) and is_list(fields) do
  branded = EchoData.Snowflake.next_branded(Id.kind())   # Id.kind() == "EVT"
  append_id(conn, queue, name, branded, fields)
end

The kind door. append_id/5 runs the kind check first: a malformed or wrong-namespace id raises before any wire — the same policy-before-existence move the job-key builder makes. Only an EVT id reaches XADD, which is issued direct with the branded id stored as the entry's id field. The id≤top rejection becomes {:error, :nonmonotonic}; any other reply passes through verbatim.

# append_id/5: the host-side KIND DOOR (INV2). The kind check runs FIRST —
# a malformed or wrong-namespace id RAISES before any wire, symmetric with the
# job-key builder: policy before existence before write.
def append_id(conn, queue, name, branded, fields)
    when is_binary(queue) and is_binary(name) and is_list(fields) do
  case Id.xadd_id(branded) do
    {:ok, xadd_id} ->
      key = stream_key(queue, name)
      # XADD <key> <xadd_id> id <branded> <fields…> — issued DIRECT, no Lua.
      # The branded id is stored as the entry's "id" field (claims-only).
      parts = ["XADD", key, xadd_id, "id", branded | flatten(fields)]

      case Connector.command(conn, parts) do
        {:ok, id} when is_binary(id) -> {:ok, branded}                  # the receipt
        {:ok, {:error_reply, @id_too_small}} -> {:error, :nonmonotonic}  # id ≤ top
        {:ok, {:error_reply, msg}} -> {:error, {:error_reply, msg}}     # verbatim
        {:error, _} = err -> err
      end

    {:error, :kind} ->
      raise ArgumentError, "EchoMQ.Stream admits one brand per stream (EVT)"

    {:error, :malformed} ->
      raise ArgumentError, "append_id requires a valid branded id"
  end
end

An append is not "write a row." It is: mint the id yourself, derive the entry id from it, append it in one direct command, and hand back the id as the receipt — so the writer, not the caller, owns the truth of what was appended.

The id derivation

The entry id is not a fresh * autogenerated id — it is derived from the branded record id, so the position in the stream is a function of the mint. Read the branded id you minted and see the explicit XADD id it maps to: the real Unix-millisecond high, the 22-bit node|seq tail low. The model is the real xadd_id/1 arithmetic over a fixed id — pure, no network. The decode runs in your browser.

EVT id → XADD id · the A1 mapping

branded EVT id · · → XADD entry id · ·

The <ms> field is the real Unix-millisecond of the mint, not the raw epoch-relative field — that is what lets a later wall-clock read map a DateTime to a bound that lands on the right entries. The <tail22> carries the node, so two writers in the same millisecond with the same sequence stay distinct. A well-formed id of the wrong namespace returns {:error, :kind} — the door the writer raises on.

Pattern & implementation

the pattern (Redis Patterns Applied)

Append-only logging writes each event to a stream and lets readers replay it; the entry id orders the log. Streams & Events teaches the append-only log and event sourcing.

the implementation (echo_mq)

append/4 mints an EVT id host-side, derives the explicit XADD id by field correspondence, and stores the branded id as the entry's id field — the receipt is the canonical id, not an opaque offset.

The pattern says: append, and let the id order the log. The implementation makes the writer mint that id — so the log position is a function of the mint, and the receipt is the canonical id the caller keeps.

Recap — mint, derive, append, receipt

append/4 mints an EVT id host-side, derives the explicit entry id with xadd_id/1, XADDs it direct, and returns {:ok, branded} — the receipt. append_id/5 is the kind door, raising before the wire on a malformed or wrong-namespace id. The next dive proves why the order the writer mints in is exactly the order the stream reads back.

Next in the module: The order theorem — why stream order, id sort, and mint order are one fact, proven by construction, and why a violation surfaces as :nonmonotonic.

References

Where the host-side mint, the explicit-id append, and the kind door are grounded, and the command the writer uses.

Sources

Related in this course