jonnify·persistence /echo-persistence/platform/echomq-bus

Module 12 · Chapter IV — The platform

The EchoMQ bus.

Ten modules built the durable floor; this one teaches the thing that sits on it. EchoMQ’s newest tier is a stream tier — append-only event streams with consumer groups — built on the same ValKey/Redis Streams verbs a field team would reach for (the converted Arcjet account is this module’s reference). The lesson is the consumer-group lifecycle: an entry is read into a group’s pending list, acknowledged per group, and re-claimed on crash — and exactly where ordering survives that and where it doesn’t. Drive a stream, a group, and a crash below.

One stream, one group, the PEL. The producer appends in mint order (the branded id is the sort key). A consumer reads an entry into the group’s pending list; XACK clears it; a crashed consumer’s entry is re-claimed by a peer — returning out of delivery order. That last move is the named exception to the order theorem.
producer XADD stream · append = mint order (branded id = sort key) acked entries stay in the stream (acked ≠ deleted) until retention trims or the archive folds them XREADGROUP > "process" consumer group PEL · pending entries list (per group) the order theorem: stream order = id order (XRANGE / XREADGROUP >) the exception: a re-claimed PEL entry returns out of order
readout · XADD appends to the stream in mint order. XREADGROUP > hands the next entry to a consumer and moves it to the PEL (pending, not removed). XACK clears it from the PEL for this group. Crash c1 and XAUTOCLAIM re-hands its pending entry to c2 — re-delivery, out of order.
§1

The verbs, and the edge that appears with groups

Arcjet field account, annotated

For a single reader, a stream is three verbs: XADD to append, XREAD to consume, XDEL to remove when done. The edge appears the moment two independent consumers need the same entries. You can no longer delete on completion — a fast group would pull entries out from under a slow one. The coordination primitive is the consumer group: an entry read through a group (XREADGROUP GROUP g c >) is placed on that group’s Pending Entries List, a consumer drains its own PEL first on rejoin, and XACK guarantees an entry will not be re-delivered within that group. The trap the field account names precisely: an acknowledged entry is not deleted — it stays in the stream until something trims it, and left alone it fills the box. EchoMQ keeps these verbs exactly — XADD, XRANGE, XREADGROUP, XACK, XAUTOCLAIM — as additive registrations on the existing EchoMQ.Connector, a protocol minor on the same RESP3 wire and house port as the jobs, the bus, and the cache. No second system to run: that was the whole objection to Kafka.

§2

What EchoMQ adds: invariants, not verbs

emq.streams.md

EchoMQ claims the small end explicitly — event streams, bounded retention, a handful of groups per stream — and keeps its databases (Tables, the journal, Postgres) beside the log, which is what makes keyed compaction someone else’s problem. Its contribution is to turn each edge the field account discovered into something named. First, an order theorem: stream order equals branded-id sort order equals mint order, because every record’s id is minted so its byte order is its time order. But a consumer group adds a second axis — a re-claimed PEL entry returns to a consumer out of real-time order, the honest cost of at-least-once — and the spec names exactly where the theorem holds (the stream) and where it cannot (a re-claim), so a spec asserting “order preserved” can never be a false green. Second, retention is policy, not a bespoke process: a declared per-stream window (MAXLEN ~ approx, MINID by mint instant) rather than a Janitor trimming by estimate. Third, the archive: a fold consumer commits each trimmed slice into the native EchoStore.Graft engine before it trims (fold-before-trim), and the engine streams those pages to Tigris — so the field account’s “acked entries are never deleted” fear is answered by folding them to durable object storage, not by hoarding RAM. The three dives walk the verbs, the retention problem, and the full six-rung tier.

§3

The three dives

module 12
§4

Build & check

What you build

Take the field account’s three pains — the multi-group deletion problem, the never-deleted accumulation, the trim-by-estimate Janitor — and name the EchoMQ answer to each: the PEL plus per-group XACK; the archive fold (emq3.5); declared retention policy (emq3.4). If each pain maps to an invariant, a policy, or a fold, you have the module.

Check

State exactly where the order theorem holds and where it breaks, and why “acked” does not mean “deleted.” “Order holds on the stream, not on a re-claimed PEL entry; acked entries fold to Tigris rather than vanish” means you have it.

§5

References & sources