jonnify·redis /redis-patterns/data-modeling/bitmap-patterns/daily-active-patterns

R7.04.3 · dive 3

One bitmap per day.

The most common bitmap pattern is one bitmap per day. Set a player's bit when they act, and the day's activity is a BITCOUNT; roll several days up with BITOP for weekly and monthly actives, and retention.

This is also where the encoding's one limitation bites hardest, because real ids are rarely a dense 0..N sequence. The dive ends on the sparse-offset rule, and on the forward-tense codemojex spike that solves it with a roaring bitmap. Select a window; the readout runs the BITOP OR over the seven daily rows below.

DAU Windows The limit codemojex (retired spike)
Seven daily bitmaps over offsets 0–9. Select a window; the result row is the BITOP OR over the days in it.
readout · Select a window. Each row is a day's active players over offsets 0–9; BITOP OR over a window is "active any day in it".
§1

A bitmap per day — DAU

bitmap-patterns.md.txt · SETBIT dau:<date>

Name the bitmap by date; set a player's bit on activity. The day's count is a BITCOUNT:

SETBIT dau:2024-01-30 12345 1      # player 12345 was active on this day
BITCOUNT dau:2024-01-30            # daily active users (DAU)
GETBIT  dau:2024-01-30 12345       # was this player active that day?

One bitmap per day, one bit per player. The day's DAU is BITCOUNT over its key. Memory: 1 million players = 125 KB per day — a year of daily-active bitmaps for a million players is about 46 MB, far less than a Set of ids per day.

A bitmap named per day makes DAU a BITCOUNT, at 125 KB per million players per day.

§2

BITOP over days — WAU, MAU, and retention

bitmap-patterns.md.txt · union and intersection

Windows are BITOP over the daily bitmaps:

# weekly active — set in ANY day this week
BITOP OR  wau:2024-W05 dau:2024-01-28 dau:2024-01-29 dau:2024-01-30 ...
BITCOUNT  wau:2024-W05

# active EVERY day this week — the most-engaged
BITOP AND wau:2024-W05:every dau:2024-01-28 dau:2024-01-29 ...
BITCOUNT  wau:2024-W05:every

BITOP OR over a week's daily bitmaps is weekly-active; over a month's, monthly-active. BITOP AND over them is "active every day in the window". The result is itself a bitmap, so it composes — a quarter is the OR of its months. Retention is the intersection of two days — select the cohort and watch the BITOP AND:

Retention is an intersection. Of those active on day 0, who returned later? Select the return day; the result is day0 AND dayN.
— the returned cohort, and the rate 
# of those active on day 0, who returned on day 7
BITOP AND retain:d0_d7 dau:2024-01-01 dau:2024-01-08
BITCOUNT retain:d0_d7
# day-7 retention rate = BITCOUNT(retain:d0_d7) / BITCOUNT(dau:2024-01-01)

A funnel is a chain of BITOP AND: registered-and-played-and-paid is the intersection of three attribute bitmaps. Every cohort question reduces to a union or an intersection followed by a count.

Windows and retention are BITOP over daily bitmaps: OR for "active any day", AND for "active every day" or "active on day 0 and day N". The result composes into the next question.

§3

The sparse-offset limitation

bitmap-patterns.md.txt · the highest-offset rule

The density holds only over a dense id space, because a bitmap allocates memory up to its highest set offset:

SETBIT users:active 999999999 1    # allocates ~125 MB to hold ONE bit

One bit at offset ~10&sup9; forces the String to ~125 MB of mostly zeros. The source's fixes are: use dense, sequential ids; hash ids to a bounded range and accept collisions; or use a Set for sparse data. A plain bitmap is the right structure only when the ids are packed near 0. This is the fact that decides the codemojex design below.

§4

The codemojex cohort note — a retired spike, and roaring

branded_id.ex · infra/cm-bitmapist (retired)

codemojex ships no bitmap code today: there is zero SETBIT/BITCOUNT/bitmapist in echo/apps/codemojex/lib/. What follows describes a retired spike that never shipped.

An analytics spike, cm-bitmapist (infra/cm-bitmapist/ until 2026-10-11 — a Go port of Doist's bitmapist4, branded-id-native, on its own Fly machine), would model daily-active players as bitmaps. The offset would be each player's branded-id placement — the hash32, the first half of MurmurHash3's fmix64 truncated to 32 bits, asserted at boot in branded_id.ex:

echo/apps/echo_data/lib/echo_data/branded_id.ex · self_check!frozen
-- the placement (hash32) is the bit offset a branded-native bitmap would use
placement("USR0KHTOWnGLuC")  →  234878118

A page-own codemojex example uses the live brand PLR (player):

# retired spike — cm-bitmapist, never in echo/apps/codemojex
Mark("active", "PLR0KHTOWnGLuC", today)   # offset = placement("PLR0KHTOWnGLuC")
Count("active", month)                      # BITCOUNT — monthly active players
AndCount(registered, played, paid)          # a registered → played → paid funnel

Here is where the sparse-offset rule decides the design. The branded placement is a hash, so the offset lands anywhere in the 32-bit space — sparse, exactly the §3 limitation. A flat bitmap over a sparse branded keyspace is enormous; the spike used roaring bitmaps instead (compressed bitmaps that store dense runs and sparse positions differently, not a flat allocation to the highest offset). The published result for a sparse branded keyspace is on the order of ~129 GB on a plain bitmap versus ~300 MB roaring — the §3 limitation, solved by the structure. The hash32 is also collision-bearing in 32 bits (on the order of N²/2³³ colliding pairs at N players — negligible into the millions), so distinct counts undercount slightly; a property to state, not assume.

the pattern

A bitmap per day; BITCOUNT = DAU; BITOP OR/AND over days = WAU/MAU and retention — exact cohort analytics over the bits.

its application (forward-tense)

codemojex's retired cm-bitmapist spike would have marked daily-active players at the offset = their branded-id placement (placement(…) → 234878118). The placement is sparse, so the spike used a roaring bitmap, not a flat one.

A bitmap per day makes DAU/WAU/MAU and retention a matter of BITCOUNT and BITOP — exact, cheap, composable. But only over a dense id space: a hashed (branded) id space is sparse, so a branded-native cohort store wants a roaring bitmap, the structure that does not allocate to the highest offset.

Notes on ValkeyBITOP OR|AND|XOR|NOT destkey key [key ...] writes a result bitmap the size of the longest input and is O(N) in that length; a flat String bitmap grows to its highest set offset, which is why a sparse id space wants a different structure — valkey.io/commands/bitop.

/bcs/overview

The branded-id placement vector

The 14-character branded id's placement is the 32-bit offset the retired cohort spike would have marked. Where the id vectors live and are asserted at boot is the Branded Component System overview — placement("USR0KHTOWnGLuC") → 234878118.

R7.04 · in this course

Back to the module

The three dives ground in the same standalone bitmap pattern — the flag, the aggregates, the daily rollup. Return to the Bitmaps hub, or on to the next module in the chapter.

§5

References

sources · related

Sources

Related in this course