jonnifyechomq

The stream log · dive 03

The claims-only id

The branded id is a stored field, not merely a position — so any reader recovers the canonical receipt without re-encoding it.

Every XADD entry already has an entry-id position. So why does the writer also store the branded id as a field, id <branded>? Because the position is a wire detail and the branded id is the canonical receipt. The contract is claims-only: a reader — including a polyglot reader on another runtime — recovers the 14-byte branded id by reading the id field, without re-encoding it from the position. read/6 is the minimal un-grouped XRANGE read-back that returns {branded, fields_map} in mint order. Step the read on the right, then see the round trip close.

The read-back The round trip Two doors out Pattern & implementation References

The read-back · step the moves

move 1 XRANGE <key> <from> <to> [COUNT n] move 2 parse [[xadd_id, [field, value, …]], …] move 3 recover {branded = map["id"], fields_map} in MINT ORDER — the branded id is the receipt, not the position the minimal read-back — NOT a consumer group

The minimal read-back

read/6 is the minimal un-grouped read-back — the order-theorem proof surface. It is deliberately not a consumer group (XREADGROUP and at-least-once delivery are a later module); it is the plain range read that recovers entries in mint order. from and to default to the full range "-"/"+".

# echo_mq — EchoMQ.Stream
# read/6: the MINIMAL un-grouped read-back. Wraps XRANGE <key> <from> <to>
# [COUNT n] and parses the nested-array reply into {branded, fields_map} tuples
# IN MINT ORDER. This is the order-theorem proof surface, NOT a consumer group.
def read(conn, queue, name, from \\ "-", to \\ "+", count \\ nil)
    when is_binary(queue) and is_binary(name) do
  key = stream_key(queue, name)
  parts = ["XRANGE", key, from, to] ++ if(count, do: ["COUNT", Integer.to_string(count)], else: [])

  case Connector.command(conn, parts) do
    {:ok, entries} when is_list(entries) -> {:ok, Enum.map(entries, &parse_entry/1)}
    {:error, _} = err -> err
  end
end

The claims-only recovery. The reply shape is [[xadd_id, [field, value, …]], …]. parse_entry/1 turns each entry into the canonical {branded, fields_map} tuple — the stored id field becomes the branded receipt, the remaining pairs become the payload map. The xadd_id position is discarded; the branded id is the canonical id the reader sorts by.

# echo_mq — EchoMQ.Stream
# Parse one XRANGE entry [xadd_id, [field, value, …]] into {branded, map}:
# the branded record id is the stored "id" field (the claims-only receipt the
# reader sorts by), the remaining pairs are the payload as a map. The xadd_id
# is only the wire position; the BRANDED id is the canonical id.
defp parse_entry([_xadd_id, kv]) when is_list(kv) do
  map = pairs(kv)
  {Map.get(map, "id"), Map.delete(map, "id")}
end

defp pairs([k, v | rest]), do: Map.put(pairs(rest), k, v)
defp pairs([]), do: %{}

The position is how Valkey indexes the entry; the stored id field is the truth the writer minted. They agree by the order theorem — but the reader reads the receipt, not the index.

The round trip

A round trip closes: append/4 returns {:ok, branded}; read/6 returns the same branded id, recovered from the entry's id field, with the rest as the payload map — in mint order. Pick how many records were appended and read them back: the receipts the writer handed out are exactly the ids the reader recovers. The model is the real parse_entry/1 shape over a fixed set of records — pure, no network.

append → read · the receipts come back in mint order

3
the stream — appended in mint order read/6 → {branded, fields_map} … ·

The stream key both ends use is stream_key/2 — emq:{q}:stream:<name> via the shipped total Keyspace.queue_key/2, no grammar edit. The stream shares the queue's {q} hashtag slot, so a queue and its stream co-locate on one of 16384 Valkey Cluster slots.

Two doors out of the plain read

The plain read/6 is the floor. Two named capabilities are built on top of it in later modules — the same surface, read two more ways.

Time-travel — read by a mint instant

read_since/5 and read_window/6 read the log by a mint instant: a %DateTime{} becomes an XRANGE bound (minid_floor/1 the lower floor, maxid_ceil/1 the inclusive upper inverse), and the read delegates to read/6 — zero new Lua. The same order theorem is what makes a time window a server-side range filter rather than a scan-and-compare. The depth is the Time-travel module of this pillar.

Retention & the archive — the door to persistence

A log that only grows is a leak, so trim/4 bounds it by length (XTRIM MAXLEN) or by age (XTRIM MINID). What a stream trims is not lost: EchoStore.StreamArchive folds the trimmed segments into the durable Graft floor — readable beside the live tail through a watermark merge-read. That durable floor is taught in full in Echo Persistence.

Pattern & implementation

the pattern (Redis Patterns Applied)

Reading a stream recovers entries and their payloads in order; a consumer replays from a position. Streams & Events teaches the read side of the log.

the implementation (echo_mq)

The canonical id is a stored field, not the wire position, so any reader recovers the 14-byte branded receipt without re-encoding — the claims-only contract. read/6 returns {branded, fields_map} in mint order; time-travel and the archive build on it.

The pattern says: read the log back in order. The implementation makes the branded id a field the reader recovers directly — so the receipt survives the round trip, and the plain read becomes the floor under time-travel and the archive.

Recap — the id is a field, the read is the floor

The branded id is stored as the entry's id field — the claims-only contract — so read/6 recovers the canonical receipt and the payload map in mint order, closing the round trip from append/4. The plain read is the floor under two named doors: time-travel by mint instant and retention folding into the durable archive — the door to Echo Persistence. That completes the writer; the rest of the pillar reads it.

Back to The stream log — consumer groups, time-travel, and retention are the pillar's later modules, each reading this writer.

References

Where the claims-only id field, the minimal read-back, and the forward doors are grounded, and the command the read uses.

Sources

Related in this course