jonnify·redis /redis-patterns/streams-events/streams-consumer-patterns/maxlen-trimming

R5.02.3 · Stream consumer patterns — dive

MAXLEN trimming — bound the log, keep the history.

A log that only grows is a leak, and acknowledgement does not shrink it: XACK marks an entry handled in group metadata, but the entry stays in the log. The right tool is XTRIM, not XDEL — and the ~ trims whole macro-nodes, so it may under-trim but can never over-trim.

Grounding: echo/apps/echo_mq/lib/echo_mq/stream.ex · trim/4 {:maxlen, count, approx?} / {:minid, dt, approx?} · stream_retention.ex (the opt-in driver) · echo/apps/echo_store/lib/echo_store/stream_archive.ex · fold/3 — the fold to the durable floor.

Three retention windows over one log. Select a policy to read what it keeps and what it removes. The newest entries are on the right; a retention policy bounds the log from the left.
oldernewest e1 e2 e3 e4 e5 e6 e7 t
readout · Select a policy. XACK does not shrink the log — retention is a separate XTRIM: MAXLEN keeps the newest N, MINID keeps entries at or after a time, and no policy means the log grows forever.
§1

XTRIM, not XDEL

streams-consumer-patterns.md.txt

The tempting cleanup is to XDEL an entry right after XACKing it. It is harmful. A stream stores entries in a radix tree of macro-nodes (listpacks); XDEL marks an entry deleted but does not free the memory until the entire macro-node is empty, so heavy XDEL use leaves "Swiss cheese" fragmentation — memory that is logically free but physically held.

The correct pattern keeps the two concerns apart:

  • XACK marks an entry handled — consumer-group state only, it does not touch the log.
  • XTRIM enforces a retention policy — it frees memory by removing whole macro-nodes from the tail.

XADD ... MAXLEN ~ N trims on every write; XTRIM ... MAXLEN ~ N trims periodically. Either way the unit of work is a macro-node, which is why approximate trimming is the cheap default.

Acknowledgement is group bookkeeping; retention is memory. Keep them apart: XACK to mark handled, XTRIM to bound the log, never XDEL per entry.

§2

Approximate trimming — why ~

stream.ex · approx_flag

MAXLEN ~ N does not promise exactly N entries. The ~ selects approximate trimming: Valkey removes whole macro-nodes from the tail and stops when removing the next one would drop the count below N. So it may keep a little more than N — it under-trims — but it never removes an entry it should have kept. The payoff is CPU: trimming a whole listpack at once is far cheaper than walking to an exact boundary entry. The exact form, MAXLEN = N, removes precisely to the edge and is the opt-in for when the count must be exact.

Approximate, exact, and the safe boundary. Select a mode to read where the cut lands. The dashed line is the declared window; neither mode ever deletes inside it.
declared window — always kept ~ removes whole nodes kept extra = removes to the edge exact cut blast radius bounded never inside window
readout · Select a mode. ~ trims whole macro-nodes and may keep a little extra; = trims to the exact edge; either way a trim can never delete an entry inside the declared window.

Notes on ValkeyBecause ~ works in whole macro-nodes, the removed-count XTRIM returns can be 0 even when entries are old — under-trimming is expected and safe, an over-trim is impossible — valkey.io/commands/xtrim.

§3

trim/4 — the verb, and the policy

stream.ex · trim/4 · stream_retention.ex

EchoMQ exposes both forms as one verb, EchoMQ.Stream.trim/4, issued direct over XTRIM:

echo/apps/echo_mq/lib/echo_mq/stream.ex · trim/4, the two window clausesverbatim
def trim(conn, queue, name, {:maxlen, count, approx?}) do
  key = stream_key(queue, name)
  xtrim(conn, ["XTRIM", key, "MAXLEN", approx_flag(approx?), Integer.to_string(count)])
end

def trim(conn, queue, name, {:minid, %DateTime{} = dt, approx?}) do
  key = stream_key(queue, name)
  xtrim(conn, ["XTRIM", key, "MINID", approx_flag(approx?), minid_floor(dt)])
end

{:maxlen, count, approx?} keeps the count newest entries; {:minid, dt, approx?} keeps entries minted at or after a DateTime, with the floor "<ms>-0" derived from the branded mint instant — never a raw snowflake integer handed to the wire. approx? true selects ~ (the safe default); false selects = (exact, the opt-in). Either way the blast radius is bounded by the window: a trim can never delete an entry inside it.

A trim is a destructive verb, so it is never coupled to the append and never folded into a consumer's beat. EchoMQ.StreamRetention is the named, opt-in driver that beats on a tick and re-applies a declared policy through the public trim/4: a stream nobody drains still trims, because bounded memory is a safety property and a consumer is up is a liveness fact — coupling the two is a silent-no-op the design refuses. A manual trim/4 call is the equally-supported cadence.

One verb, two windows, one safe default. Retention is a property of the stream, declared on its own beat — not a side effect of a consumer being alive.

§4

What is trimmed is not lost

echo_store/stream_archive.ex · fold/3

Trimming frees memory, but the history a trimmed entry held may still matter for an audit or a backtest. The two needs are reconciled by folding what is trimmed into a durable floor before it is removed. EchoStore.StreamArchive.fold/3 folds a mint-ordered slice of {branded, fields} records into the native Graft engine's page store — one page per record, in a reserved high page range disjoint from business data — and advances a watermark W to the branded EVT id of the highest-folded record:

EchoStore.StreamArchive.fold(volume_id, slice, db)
#=> {:ok, w}     # the new frontier — the highest folded branded id

The fold commits before the trim removes anything, so an entry is durable in the floor before it leaves the live log — no-gap, no-overlap is a consequence of fold-before-trim plus the order theorem, not a per-read check. A reader then merges the two: records with a branded id at or below W come from the archive, records above W come from the live tail, joined in mint order. Deep history stays readable without resident memory.

the pattern

Keep the log bounded with XTRIM MAXLEN ~ N (or MINID by time), keep acknowledgement separate from retention, and accept that trimming removes history from the live log.

its EchoMQ application

EchoMQ.Stream.trim/4 bounds the log and EchoMQ.StreamRetention re-applies the policy; EchoStore.StreamArchive.fold/3 folds what is trimmed into the Graft floor first — bounded memory, history preserved, the durability dial at /echo-persistence.

Notes on ValkeyThe archive frontier W is a branded EVT id, not the engine's integer page cursor — greater id means newer, so the merge-read splits archived from live on one comparison the leaderboard's first-claim also uses — tigrisdata.com/docs/objects/conditionals.

§5

The pattern, applied

echo/apps/codemojex · EchoStore

A codemojex game's activity stream (echo/apps/codemojex) grows as rounds are played. The operator declares a retention policy — keep the recent window live — and EchoMQ.StreamRetention trims it on its beat with MAXLEN ~, so memory stays bounded no matter how long the game runs. Before each trim, the trimmed segment is folded to the Graft floor, so the full round history of a long-running game is recoverable for a leaderboard audit even though only the recent window sits in the live log.

/echo-persistence

Where a trimmed segment lands

EchoStore.StreamArchive folds trimmed segments into the Graft floor on CubDB and on to Tigris behind a create-only fence — deep history without resident memory, the durability dial in full.

 /echomq/bus

Retention in the bus

The Bus pillar teaches the retained, replayable log: the trim driver, the archive fold, and time-travel over the merge-read of archived and live entries.

 /bcs/bus

The Stream Tier in the manuscript

Part B3: a log that only grows is a leak, so retention is a policy; what is trimmed folds into the durable Graft floor — deep history without resident memory.
§6

References

sources · related

Sources

Related in this course