jonnify·redis /redis-patterns/streams-events/streams-event-sourcing/the-append-only-log

R5.01.1 · dive 1

The append-only log.

The log is the source of truth: every state change is appended, never updated in place. A Redis Stream is the structure — XADD adds an entry to the end and returns its id.

A Redis Stream grows one way: XADD appends an entry to the end and returns its time-ordered id. There is no in-place update of a written entry — the log only grows forward, and a later append always carries a larger id. That single property, append never overwrite, is what makes it the right structure for event sourcing. Append placed, filled, closed; the readout shows the growing log and its ids.

XADD appends; the log grows forward only. Append the next event; the readout reports the log and the rising ids.
oldest newest · XADD appends here —— —— —— —— the log is empty — append the first event
§1

A stream is an append-only log

the-append-only-log.md · the structure

A Redis Stream is an ordered sequence of entries, each a small set of field-value pairs under an id. The only way to add to it is XADD, which appends to the end and returns the new entry's id. There is no in-place update of a written entry: a stream grows forward only. That single property — append, never overwrite — is what makes it the right structure for event sourcing.

XADD orders * type placed amount 4200   # append; * mints a time-ordered id -> "…000-0"
XADD orders * type filled amount 4200   # append the next event           -> "…001-0"

The id is <milliseconds>-<sequence>: the millisecond the entry was added, then a sequence number that breaks ties within the same millisecond. A later append always carries a larger id than an earlier one, so the id orders the log. The * asks the server to mint the id; an explicit id may be supplied instead, and must exceed the current top of the stream or XADD rejects it.

A stream is an append-only sequence: XADD adds at the end under a rising id, nothing is overwritten, and the id orders the log.

§2

Every transition is an event

the-append-only-log.md · the history

Event sourcing records each change as its own entry. An order placed, an order filled, an order cancelled — three events, appended in the order they happened, never collapsed into one mutable "order" row. The log holds the whole history; the present is whatever the history adds up to. Because nothing is overwritten, the log is an audit trail for free: it says not only what is true now but how it came to be true.

Each state change is its own appended entry, so the log is the whole history — an audit trail for free, not a single mutable value.

§3

The append rejects a smaller id

echo_mq/stream.ex · the order kept

A stream's entry id must strictly exceed its current top. XADD with an explicit id at or below the top is refused — the server answers an error rather than break the order. This is the property that keeps the log a log: the id sequence is strictly increasing, so reading by id reads in append order. A writer that mints ids in order never trips it; one that does trips it loudly, and the error is not swallowed.

The entry id is <ms>-<seq>, and it must exceed the top. Select an append against a stream whose top is …001-0; read whether it is accepted.
candidate entry id vs the stream top …001-0 ms …002 seq 0 id select a candidate id ms = the milliseconds part · seq = the tie-break within one ms a larger id is accepted; an equal or smaller id is rejected (id ≤ top)

The id must strictly exceed the top, so the sequence is strictly increasing and reading by id reads in append order; an equal or smaller id is refused, not silently dropped.

§4

In EchoMQ — EchoMQ.Stream.append/4

echo_mq/stream.ex · append/4

EchoMQ.Stream is the writer of EchoMQ's Stream Tier. EchoMQ.Stream.append(conn, queue, name, fields) does the append in four steps, in real code in echo/apps/echo_mq: it mints an EVT-branded record id host-side, derives the explicit XADD id from it, issues XADD emq:{q}:stream:<name> <xadd_id> id <branded> <fields…>, and returns {:ok, branded} — the branded id is the receipt.

# EchoMQ.Stream.append/4 — the append, shaped to the wire (real)
EchoMQ.Stream.append(conn, "orders", "events", [{"type", "placed"}, {"amount", "4200"}])
#  1. mint an EVT-branded record id host-side  (EchoData.Snowflake.next_branded("EVT"))
#  2. derive the explicit XADD id from it       (Stream.Id.xadd_id/1 — "<ms>-<tail22>")
#  3. XADD emq:{orders}:stream:events <xadd_id> id <branded> type placed amount 4200
#  4. -> {:ok, branded}    (the branded id is the receipt)

The writer owns the mint, so there is nothing to spoof — the id comes from the host, not a client. The 14-byte branded id is stored as the stream entry's id field, so a reader in any language gets the canonical id without re-encoding it. The append returns {:ok, branded}; a Valkey rejection of an id at or below the top is mapped to {:error, :nonmonotonic} — never swallowed, never retried with *, because that error is the wire telling the truth that an upstream mint-order violation happened.

The order theorem holds because a single writer per stream mints strictly increasing snowflakes: the next id always exceeds the stream top, so no XADD rejection is possible. Stream order equals id sort equals mint order — the log is ordered by append the way the property store is ordered by mint.

Notes on ValkeyXADD key * field value appends an entry and returns its <ms>-<seq> id; an explicit id at or below the stream top is refused with an error rather than reordered — valkey.io/commands/xadd.

the pattern

Append every state change as an immutable entry to a log; the id orders it, and a smaller id is refused so the sequence only grows.

its EchoMQ application

EchoMQ.Stream.append/4 mints an EVT id, derives the XADD id, issues XADD emq:{q}:stream:<name>, and returns the branded id; an id≤top rejection surfaces as {:error, :nonmonotonic}.

EchoMQ's append is the pattern in real code: mint an ordered id, append the entry, return the receipt, and surface a mint-order violation instead of breaking the log.

/echomq/bus

A door, not a depth

The batch append, the explicit kind door on a caller-supplied id, and the way the writer's monotone mint proves the order theorem are the subject of the dedicated EchoMQ course. From here, this dive opens onto the Bus pillar — the writer and the retained, replayable log in depth. This dive teaches the append; that course teaches the writer that runs it.

§5

References

sources · related

Sources

Related in this course