jonnify·redis /redis-patterns/streams-events/custom-events/windowed-aggregation

R5.04.2 · dive 2

Windowed aggregation.

A projection is a fold scoped to a slice of the log: take the entries between two instants and reduce them to the one shape a view needs — a count, a total, a tally.

Because a stream entry id begins with the millisecond it was added, a DateTime becomes a range bound: "the events of the last hour" is an id range, and the projection is the fold over it. The floor id <ms>-0 and the ceiling id <ms>-<max-seq> make the edges exact. Select a window; the readout names which entries fall inside it.

A time window is an id range — the edges are exact. Select a window over the log; the readout names which entries fall inside it.
oldest newest e₀< t0 e₁at t0 e₂mid e₃at t1 e₄> t1 t0 floor ms-0 t1 ceil ms-max
readout · Select a window. A closed window keeps the entries at and between the two instants; a half-open window keeps everything from t0 onward — both lower edges exact at the floor id.
§1

Time is an id range

windowed-aggregation.md · the floor and the ceiling

A stream entry id is <milliseconds>-<sequence>. The leading milliseconds make the id a clock: an instant maps to an id, and an id range expresses a time range. To read everything in a closed window [t0, t1], form the smallest id at t0 — <t0-in-ms>-0 — and the largest id at t1 — <t1-in-ms>-<max-seq> — and range between them. To read everything from an instant onward, range from the floor id to +.

XRANGE guess-scored 1719312000000-0 1719315600000-0   # closed [t0, t1]
XRANGE guess-scored 1719312000000-0 +                 # half-open [t0, ∞)

The floor id <ms>-0 is the lowest id at that millisecond, so the lower edge is exact: an entry minted at the instant is in, one minted a millisecond earlier is out. The ceiling id <ms>-<max-seq> is the highest id at that millisecond, so the upper edge is exact too: an entry at t1 reads back, one a millisecond later does not.

A time window is an id range because the entry id is a clock; the floor id and the ceiling id make both edges exact to the millisecond.

§2

A projection is the fold over the window

windowed-aggregation.md · recompute, never maintain

With the window read, the projection is the same fold as any event-sourced rebuild — start with an empty accumulator, apply each entry in order — but scoped to the window and shaped for one question. A per-room count folds room-opened to a map; a running total folds guess-scored to a sum; the day's results fold game-settled to a list. The fold is deterministic in the entries, so the projection is recomputed from the log, never a separate table kept in sync.

The projection is the fold of the windowed entries. Step the fold over the guess-scored window; the accumulator is the running total after each entry.
total 480guess-scored total 320guess-scored total 600guess-scored projection (sum)0
readout · Step the fold. The accumulator starts at 0; applying 480, 320, 600 sums the window's totals to the projection — recomputed from the log, never a stored total kept in sync.

A projection is the deterministic fold of the windowed entries; many projections fold the same window into different shapes, each recomputed and never stale.

§3

Recompute, or snapshot

windowed-aggregation.md · the cost of a long window

A projection is recomputable, which is its strength: change the shape and re-fold, and the answer is right. The cost is that folding a long window is work proportional to its entries. Where a rollup is read far more often than its window changes, fold once and cache the result, or bound the window with retention (R5.02) so the replay stays short. Recomputing a stable projection on every read is the waste to avoid; recomputing it when the inputs change is exactly the point.

Recompute a projection when its inputs change; cache a snapshot or trim the window when a stable rollup is read far more often than it changes.

§4

In EchoMQ — read_window/6 and read_since/5

echo_mq/stream.ex · read_window/6 · minid_floor/1 · maxid_ceil/1

EchoMQ derives the window bounds host-side so the wire only ever sees an <ms>-<seq> id, never a raw integer. EchoMQ.Stream.read_window(conn, queue, name, t0, t1, count) reads the closed [t0, t1] window; EchoMQ.Stream.read_since(conn, queue, name, t0, count) reads the half-open [t0, ∞) window. Both delegate to the range read and return {branded, fields_map} tuples in mint order — the input to the fold.

# EchoMQ.Stream.read_window/6 — a closed mint-time window to fold (real)
EchoMQ.Stream.read_window(conn, "cm", "guess-scored",
  ~U[2024-06-25 12:00:00Z], ~U[2024-06-25 13:00:00Z])
#  from = minid_floor(t0)  ->  "<ms>-0"            (lowest id at t0)
#  to   = maxid_ceil(t1)   ->  "<ms>-<max-seq>"    (highest id at t1)
#  XRANGE emq:{cm}:stream:guess-scored <ms>-0 <ms>-<max-seq>
#  -> {:ok, [{branded, fields_map}, …]}            mint order, [t0, t1] — fold to a projection

# everything from an instant onward, half-open [t0, ∞)
EchoMQ.Stream.read_since(conn, "cm", "guess-scored", ~U[2024-06-25 12:00:00Z])
#  from = minid_floor(t0), to = "+"

minid_floor/1 turns t0 into "<ms>-0" and maxid_ceil/1 turns t1 into "<ms>-<max-seq>", where the milliseconds is the real Unix millisecond of the instant (DateTime.to_unix(dt, :millisecond)) and the sequence is the maximal 22-bit tail. So a read_window returns exactly the entries whose EVT mint instant falls in [t0, t1], which equals reading the whole stream and filtering by mint instant — the window is a server-side filter via the bounds, and read_window/6 raises before any wire on an inverted window (t1 before t0). The returned tuples fold to the projection: a per-window count of room openings, a per-player running total of scores, the settled games of the day.

In codemojex (echo/apps/codemojex), the score worker scores each guess with Codemojex.Scoring.score/2 and records the player's best with Codemojex.Board.record/3; the leaderboard Codemojex.Board.top/2 is the live rank. A windowed read of the guess-scored channel folds to the recent activity feed — the last hour's scores — while the board holds the all-time rank; the two are different projections of the same play, one a windowed fold of the log, one a running aggregate.

Notes on ValkeyAn XRANGE reads entries in id order between an inclusive start and end, and a bare <ms> bound is read as <ms>-0 — which is exactly how a millisecond instant becomes the floor id of a windowed read — valkey.io/commands/xrange.

the pattern

A projection is a fold scoped to a time window; time is an id range because the entry id carries the millisecond.

its EchoMQ application

EchoMQ.Stream.read_window/6 reads a closed [t0, t1] window and read_since/5 a half-open [t0, ∞) one, deriving exact bounds with minid_floor/1 and maxid_ceil/1; the mint-ordered tuples fold to the projection — codemojex's windowed activity feed beside the running board.

Redis ranges a stream by id; EchoMQ turns a DateTime window into exact id bounds and hands back the entries in mint order — the projection is the fold over them.

/echo-persistence

A door, not a depth — the persistence floor

A windowed history does not grow without bound: a log is trimmed under a retention policy, and what is trimmed folds into the durable page tier and, beyond it, to remote object storage behind a create-only commit fence. That floor — the single-writer engine, the lazy reader, and the durability dial a system turns from "hold nothing" to "commit-per-record + replicate off-box" — is the subject of the EchoStore persistence course.

/echomq/bus

The Bus pillar that owns the stream

The time-travel reads in full, the consumer-group read that resumes from a saved cursor, and the archive fold consumer are the subject of the dedicated EchoMQ course. From here, open onto the Bus pillar — the retained, replayable log and its readers.

§5

References

sources · related

Sources

Related in this course