jonnify·redis /redis-patterns/streams-events/pubsub

R5.03 · Streams & Events — module hub

Pub/Sub vs Streams — fire-and-forget, or remembered.

Two channels carry an event in Valkey, and they make opposite promises. Pub/Sub is fire-and-forget: PUBLISH reaches whoever is subscribed at that instant and reaches nobody else. A Stream is a durable, replayable log: XADD writes the entry, hands back a receipt, and a reader who was not there can read it later. The pattern is choosing — a channel a subscriber must be present for, against a log a reader can resume.

Grounding: echo/apps/echo_mq · EchoMQ.Events (PUBLISH emq:{q}:events, at-most-once) · EchoMQ.Stream.append/4 (the {:ok, branded} receipt) · EchoMQ.Cancel (the worked fire-and-forget control signal). Consumer: codemojex.

One event, two promises. Select a behaviour to read how a publish and an append differ. The publish reaches present subscribers and forgets; the append writes a durable entry a late reader can still read.
PUBLISH · Events XADD · Stream stored? late crash receipt nothing keptentry retained missedread later goneresume by cursor subscriber count{:ok, branded}
readout · Select a behaviour. A publish reaches the subscribers present at that instant and stores nothing; an append writes a durable entry and returns a receipt — so a reader who was absent reads it later.
§1

Two promises

valkey · pubsub vs streams

A Redis channel is a delivery, not a store. PUBLISH channel payload sends the payload to every connection currently subscribed to channel and returns the number of subscribers it reached. There is no buffer, no backlog, no cursor: a subscriber that connects one millisecond later sees nothing of what came before, and a publish that reaches a subscriber count of zero is a no-op the sender cannot distinguish from success. This is the at-most-once promise — a message reaches a present subscriber zero or one times, never stored for a late one.

A Stream inverts every one of those defaults. XADD key * field value appends an entry, assigns it a monotonic id, and keeps it in the log until a retention policy removes it. A reader reads with XRANGE or XREADGROUP from any position — the start, a saved cursor, or only-new — so a reader that was absent reads what it missed, and a reader that crashed resumes where it stopped. The log remembers; the channel does not. Neither is the better one: a live metrics tick wants the channel's no-storage simplicity, and an audit trail wants the log's durability. The error is using the channel where the promise needed was the log's.

Pub/Sub delivers to whoever is listening now and forgets. A Stream stores every entry and lets any reader read it on its own schedule. The pattern is matching the promise to the need.

§2

EchoMQ.Events — the fire-and-forget side

events.ex · publish/5

EchoMQ.Events is EchoMQ's pub/sub surface, and it is honest about the promise. It subscribes once to the per-queue channel emq:{q}:events and dispatches each lifecycle event — a job completed, failed, scheduled, stalled — to the pids that registered with subscribe/2 and to an optional handler module, so a consumer reacts to work as it happens without polling the sets. The moduledoc states the cost in one sentence, and that honesty is the lesson.

echo/apps/echo_mq/lib/echo_mq/events.ex · the at-most-once promise, stated in the moduledocverbatim
The push channel is fire-and-forget (at-most-once): a PUBLISH with no live
subscriber, or one issued in the window between a socket drop and the
resubscribe, is lost. The emq.1 resubscribe is the mitigation (a reconnect
re-issues the SUBSCRIBE); the durable replayable receipt is emq3.2's
EchoMQ.Stream, not this. Stated, not papered over.

The mitigation is not a buffer — adding one would make it a different channel. It is the emq.1 resubscribe: a connector that reconnects re-issues its SUBSCRIBE, so the feed comes back live across a drop, narrowing the loss window to the reconnect itself. And where a receipt is actually required — replay, audit, a reader that must not miss an entry — the answer is the durable channel, not a harder push: EchoMQ.Stream, not EchoMQ.Events.

Notes on ValkeyPUBLISH returns the number of clients that received the message — an integer that is 0 when nobody is subscribed, indistinguishable at the protocol level from a delivery to one — valkey.io/commands/publish.

§3

EchoMQ.Stream — the durable contrast

stream.ex · append/4

EchoMQ.Stream.append(conn, queue, name, fields) is the other promise. It mints an EVT-branded record id host-side, appends XADD emq:{q}:stream:<name> <id> id <branded> <fields…>, and returns {:ok, branded} — the branded id is the receipt. Where a publish answers a subscriber count it cannot interpret, an append answers a durable handle the caller keeps. The entry stays in the log until retention removes it, readable by any reader at its own pace, replayable from the start.

XADD emq:{cm.feed}:stream:rounds * id EVT0Nh2k9pQ4rT event settled round RND0Nh2k7m
# → {:ok, "EVT0Nh2k9pQ4rT"}   # the branded id IS the receipt — durable, replayable

The same event can travel both channels, and often should: append it to the Stream so the record exists and can be replayed, and publish a lighter notification on Events so a live dashboard reacts without reading the log. The Stream is the source of truth; the publish is the doorbell. What a publish forgets the instant no one is listening, the log still holds for the reader who arrives a minute later.

the pattern

A PUBLISH is at-most-once — delivered to present subscribers, stored for nobody. An XADD is durable — the entry is retained with an ordered id, and a late or crashed reader can still read it.

its EchoMQ application

EchoMQ.Events.publish/5 is the fire-and-forget side on emq:{q}:events; EchoMQ.Stream.append/4 is the durable side returning {:ok, branded}. The system of record is the Stream; Events is for live reaction on top.

§4

The choosing rule

the decision · one question

The decision is not taste; it follows from the promise each channel makes, and it reduces to one question: does a reader who was not present need to read this later? If yes — replay, resume after a crash, an audit trail, time-travel to a past window — the answer is the Stream, every time. If the value of the message is genuinely now-or-never — a live progress tick, a presence ping, an ephemeral control signal — Events is enough and cheaper.

Which channel each need picks. Select a need to read which channel its promise calls for. Durability is the heavy side, and most needs lean onto it.
Streamdurable · replayable Eventsat-most-once replay from start resume after crash audit trail react now control signal
readout · Select a need. Replay, resume, and audit need a reader to read what it missed — the Stream. React-now and an ephemeral control signal expire the instant they are sent — Events.

The default is the Stream. The reasoning is asymmetric: a Stream can always feed a live reaction too — a reader tailing the log reacts within a round trip — but Events cannot become durable after the fact, since a message published with no subscriber is gone. Choosing the Stream keeps the durable option open and still serves the live case; choosing Events forecloses durability. EchoMQ made that choice for its own lifecycle: the system of record is the durable Stream, and Events exists for live reaction on top.

One question decides it: must an absent reader read this later? Yes is the Stream; genuinely now-or-never is Events. When unsure, choose the Stream — it keeps durability open and still reacts live.

§5

The dedicated connection

events.ex · :conn protocol 3 + push_to

The fire-and-forget side carries one operational cost the publish side hides: a subscriber monopolizes its connection. The moment a connection issues SUBSCRIBE, it enters subscriber mode and can no longer run ordinary commands — a GET on that connection is refused until it unsubscribes. A subscriber is therefore not a connection borrowed from the shared pool; it is a connection given over to listening.

EchoMQ.Events takes that as a requirement, not an accident. Its :conn must be a RESP3 connector (protocol: 3) with push_to set to the listener process, so pushed messages arrive as {:emq_push, …} mailbox frames on a connection dedicated to the subscription — the same dedicated-lane discipline the blocking stream read follows. A verb that holds a connection gets its own connection, so the single-owner socket the rest of the system shares is never stalled. The third dive reads this seam in full.

Notes on ValkeyA connection in subscriber mode accepts only the subscribe/unsubscribe and ping commands until it leaves — which is why a long-lived subscriber needs its own connection rather than one drawn from a shared pool — valkey.io/topics/pubsub.

§6

The three dives

forget → choose → the lane

The arc is see the two promises → learn the rule → pay the cost. Read them in order — the first sits the at-most-once publish against the durable receipt, the second turns the difference into a decision, and the third reads the one operational cost the fire-and-forget side carries.

built ready to read

Three dives, one decision: the two promises laid side by side, the rule that picks between them, and the dedicated lane the fire-and-forget channel demands.

§7

The pattern, applied

echo/apps/codemojex

codemojex (echo/apps/codemojex) is the worked consumer — a Telegram emoji-guessing game on the same stack. Its round lifecycle uses both channels for what each is good at. A round event — a round opened, a guess scored, a round settled — is appended to the durable activity Stream so the full history exists and a feed reader can replay it or resume after a restart. A lighter notification rides EchoMQ.Events so a live Telegram view reacts the instant something happens, without reading the log. If that publish reaches no live view, nothing is lost that matters: the Stream still holds the record. Durable where a record is needed, live where reaction is needed — one event, two channels, each making the promise it should.

Notes on ValkeyBoth the events channel and the stream carry the per-queue {q} hash tag, so a queue's emq:{q}:events and emq:{q}:stream:<name> hash to one of 16384 slots — co-located on one node — valkey.io/topics/cluster-spec.

§8

References

sources · related

Sources

Related in this course