echd — Echo on Encore · E3.2 · dive

Consumer Groups

A topic is a stream; a subscription is a group. The group partitions the log — one entry to one consumer — and remembers who has what, so work is shared without being duplicated.

Vision · real substrate

echd speaks in the present tense over a build ladder. Every module, verb, key, and port named on this page is real; the composition into one standing platform is the vision the ladder builds.

§1

The dive

docs/echo_mq/emq.streams.md:29-42

PubSub broadcasts: every subscriber sees every message. A consumer group does the opposite — it partitions. XREADGROUP hands each new entry to exactly one consumer in the group, so ten workers on one topic share the load and no entry is processed twice. That single change is what turns a notification channel into a work queue: the group is the unit of parallelism, and adding a consumer adds throughput without any coordination between the workers themselves.

The group can do this because it holds two pieces of state, on the server, beside the stream. The first is a shared cursor — the last-delivered id — which XREADGROUP advances as it hands out never-seen entries. The second is a pending-entries list, one per consumer, recording what that consumer was given but has not acknowledged. The cursor answers "what is new?"; the pending list answers "what is still owed?" Neither lives in a client, so a worker can crash and the group still knows exactly where it was.

§2

The cursor and the pending list

docs/echo_mq/emq.streams.md:29-42

One group over one stream · the log, partitioned · hover to read

Partition, not broadcast. XREADGROUP gives each new entry to exactly one consumer in the group, so C1 and C2 drain disjoint halves of the same stream. This is the difference between a topic and a queue: a plain XREAD lets every reader see everything; the group splits the work.

The shared cursor. The group keeps one last-delivered-id. Reading with the special > id asks for entries never handed to anyone and advances the cursor; reading an explicit id re-reads a consumer's own history. The cursor is the group's memory of what is new.

The per-consumer pending list. Every entry a consumer receives is added to its pending-entries list and stays there until XACK. The cursor tracks progress; the pending list tracks debt. Because both live with the stream, a crashed consumer loses nothing — its debt is still recorded.

Hover or focus a part to read it; click to pin. Without scripting, all three read in sequence.

§3

Why the state lives on the server

docs/echo_mq/emq.streams.md:29-42

The tempting shortcut is to track delivery in the consumer: read the stream, remember an offset locally, move on. It works until the consumer restarts, and then it either re-reads everything or skips whatever arrived while it was down. The consumer group refuses that trade by keeping the cursor and the pending lists with the stream, so the truth about "what is new" and "what is owed" survives any single worker. A consumer is disposable; the group is not.

That is also what makes the group the natural unit of scaling. Because the server, not the client, owns the partition, you add throughput by adding a consumer name to the group and reading — no rebalancing protocol, no shared client state, no leader. Two consumers or twenty, the group hands each one entries no other consumer will see, and the pending lists keep every hand-off honest. The mechanism that supersedes PubSub is the same mechanism that lets the tier scale: one group, many consumers, one partitioned log.

§4

The rule

one line to carry

The group rule — a subscription is a consumer group, and the group partitions the log: one entry to one consumer. The cursor remembers what is new and the per-consumer pending list remembers what is owed — both on the server, not the client — so workers share a topic without double-delivery and a crash loses nothing.

§5