jonnify·redis /redis-patterns/data-modeling/bitmap-patterns

R7.04 · Data Modeling & Memory — module hub

Bitmaps.

Store millions of boolean flags in minimal memory using Redis bitmaps — 1 bit per flag with O(1) access, plus fast aggregate operations across entire datasets.

A bitmap is a String where each bit is a boolean: SETBIT and GETBIT address a bit by offset, and one megabyte holds eight million flags. The whole pattern turns on one identity — the bit offset is the entity's integer id — and the whole limitation turns on the same one: a bitmap allocates up to its highest offset, so the id space must be dense. This is a standalone example (bitmaps are not an EchoMQ surface); a forward-tense codemojex note — the retired cm-bitmapist player-cohort spike — layers on top.

A flag is a bit, not a key. Select a store for one million entities; the readout reports its size and what it can answer.
bitmap ~125 KB · 1 bit/entity Set of ids ~50 MB · id per member key per entity largest · name + meta/entity bars not to scale · a bitmap is ~400× smaller than a Set of the same ids
readout · Select a store. A boolean per entity is one bit in a bitmap, one member in a Set, or a whole key per entity — and a bitmap is roughly 400× smaller than a Set of the same ids.
§1

Basic operations — one bit per entity

bitmap-patterns.md.txt · SETBIT / GETBIT / BITCOUNT

Three primitives address and count bits. SETBIT key offset value sets the single bit at offset to 0 or 1, growing the underlying String as needed; GETBIT key offset reads it; BITCOUNT key counts the set bits — a population count over the whole string, or over a range.

SETBIT users:active 1000 1     # user 1000 is active   (the offset IS the id)
GETBIT users:active 1000       # → 1                    (was this user active?)
BITCOUNT users:active          # → number of active users (a popcount over the string)

The cost model is the headline: a flag is one bit, and a count is a scan the engine does in hardware popcount steps. A bitmap is one boolean across many entities (online:users, a bit per user) or many booleans for one entity (user:5000:features, a bit per feature) — the same two commands, read either way.

A boolean per entity is a bit at the offset that is the id — no per-entity key, no map from id to slot. The 1-bit flags dive takes SETBIT/GETBIT and the feature grid apart.

§2

Daily active users (and weekly / monthly)

bitmap-patterns.md.txt · a bitmap per day

Name a bitmap per day and set a player's bit when they act. The day's count is a BITCOUNT; windows roll up with BITOP:

SETBIT dau:2024-01-30 12345 1      # user 12345 visited today
BITCOUNT dau:2024-01-30            # daily active users — 1M users = 125 KB/day

BITOP OR  wau:2024-W05 dau:2024-01-28 dau:2024-01-29 dau:2024-01-30 ...   # active ANY day (union)
BITOP AND wau:2024-W05:every dau:2024-01-28 dau:2024-01-29 ...            # active EVERY day (intersection)

BITOP OR over a week's daily bitmaps is weekly-active; BITOP AND over them is "active every day". Retention cohorts fall out of the same two operators: who was active on day 0 AND day 7.

A bitmap per day makes DAU, WAU, MAU, and retention a matter of BITCOUNT and BITOP. The daily-active patterns dive carves this slice and the codemojex note.

§3

Feature flags and online status

bitmap-patterns.md.txt · one entity's many booleans

A bitmap can hold one entity's many booleans, or one boolean across many entities. Feature flags per user — bit positions are features:

# 0 = dark_mode, 1 = notifications, 2 = beta_features, ...
SETBIT user:5000:features 0 1      # enable dark mode
SETBIT user:5000:features 2 1      # enable beta features
GETBIT user:5000:features 1        # notifications enabled?
GET    user:5000:features          # all flags at once — decode client-side

Online status is one bitmap, one bit per user: SETBIT online:users 12345 1 on, 0 off, BITCOUNT online:users for how many. For millions of users this costs a fraction of a key-per-user, and time-bucketed keys (online:minute:<ts> with an EXPIRE) give "active in the last five minutes" as a BITOP OR over the recent buckets.

Read a bitmap by the entity (a few feature bits in one key) or across entities (one online bit per user) — both are SETBIT/GETBIT at one bit each.

§4

Cohorts, BITFIELD, and BITPOS

bitmap-patterns.md.txt · combine, pack, locate

Cohort analysis intersects attribute bitmaps — every segment is one BITOP plus a BITCOUNT:

SETBIT users:premium 12345 1
SETBIT users:mobile  12345 1
BITOP AND cohort:premium_mobile users:premium users:mobile
BITCOUNT cohort:premium_mobile     # premium AND mobile
BITOP NOT users:not_mobile users:mobile          # complement, then
BITOP AND cohort:premium_desktop users:premium users:not_mobile

Two commands address bit ranges beyond a single bit. BITFIELD does atomic read-modify on arbitrary-width fields packed into one key — small counters with overflow control (BITFIELD counters INCRBY u8 0 1 OVERFLOW SAT). BITPOS finds the first set or unset bit (BITPOS users:active 1 — the first active id; BITPOS users:active 0 — the first free slot). And rate limiting with one bit per millisecond uses BITCOUNT over a per-second key, reclaimed by a short EXPIRE.

Beyond flags: BITOP composes cohorts, BITFIELD packs counters, BITPOS locates an edge — all over the bits. The bitcount aggregates dive takes BITCOUNT and BITOP apart, with the Bloom-filter-alternative contrast.

§5

Memory efficiency — and the sparse-offset limit

bitmap-patterns.md.txt · the density, and the caveat

The reason to reach for a bitmap is the memory table — and its one limitation is the same fact read the other way:

EntitiesBitmap sizeEquivalent Set
1 million125 KB~50 MB
10 million1.25 MB~500 MB
100 million12.5 MB~5 GB

Bitmaps are roughly 400× more memory-efficient than a Set of the same ids — but only over a dense integer id space, because a bitmap allocates memory up to its highest offset:

SETBIT users:active 999999999 1    # allocates ~125 MB to hold ONE bit

A sparse id space wastes memory. The source's fixes are: use dense, sequential ids; hash ids to a bounded range and accept collisions; or use a Set. BITOP is also O(N) in the longest input, so very large bitmaps are sharded by id range. This sparse-offset caveat is the whole story of the grounding below.

Notes on ValkeyBITCOUNT key [start end [BYTE | BIT]] counts set bits over the whole string or a range, and SETBIT/GETBIT/BITOP/BITPOS/BITFIELD are all core commands on the String type — no module required (the probabilistic BF.*/CF.* structures are the module exception R7.03 covers) — valkey.io/commands/bitcount.

§6

The grounding — the branded offset, in a retired spike

branded_id.ex · infra/cm-bitmapist (retired)

A bitmap's offset must be a non-negative integer. The Branded Component System already mints one for every entity: a 14-character branded id carries a placement — hash32, the first half of MurmurHash3's fmix64 truncated to 32 bits. The runtime asserts the vector at boot (branded_id.ex, self_check!):

echo/apps/echo_data/lib/echo_data/branded_id.ex · self_check!frozen
-- the placement (hash32) is a 32-bit integer — a bitmap bit offset
placement("USR0KHTOWnGLuC")  →  234878118

That 32-bit placement is exactly a bitmap bit offset. codemojex ships no bitmap code today — zero SETBIT/BITCOUNT/bitmapist in echo/apps/codemojex/lib/. What follows is a retired spike that never shipped: cm-bitmapist (infra/cm-bitmapist/ until 2026-10-11 — a Go port of Doist's bitmapist4, branded-id-native, on its own Fly machine) would mark each player's daily-active bit at Offset(id) = Hash32(Decode(id)). A page-own codemojex example uses the live brand PLR (player):

# retired spike — cm-bitmapist, never in echo/apps/codemojex
Mark("active", "PLR0KHTOWnGLuC", today)   # offset = placement("PLR0KHTOWnGLuC")
Count("active", month)                      # BITCOUNT — monthly active players
AndCount(registered, played, paid)          # a registered → played → paid funnel

Because the branded placement is a hash, the offset lands anywhere in the 32-bit space — sparse, exactly the limitation above. So the spike used roaring bitmaps (compressed bitmaps that store runs, not a flat allocation to the highest offset): the published result for a sparse branded keyspace is on the order of ~129 GB on a plain bitmap versus ~300 MB roaring. The hash32 is also collision-bearing in 32 bits (on the order of N²/2³³ colliding pairs), so distinct counts undercount slightly — a property to state, not assume. Select where a structure sits in the modeling family:

The modeling family — what each structure trades. Select one; the readout names what it stores, how exact it is, and its memory.
bitmap exact · dense 1 bit/entity roaring exact · sparse compressed Set exact · any id id per member HyperLogLog approximate 12 KB fixed exact & dense → exact & sparse → exact & large → approximate & tiny
readout · Select a structure. A plain bitmap is exact over a dense id space; a roaring bitmap stays compact over a sparse one (the branded placement); a Set is exact at any id but large; HyperLogLog (R7.03) trades exactness for 12 KB fixed.
the pattern

1 bit per flag; BITCOUNT/BITOP analytics over millions of entities at O(1) memory per flag.

its application (forward-tense)

codemojex's retired cm-bitmapist spike was designed to model daily-active players as bitmaps — the bit offset is the player's branded-id placement hash (placement(…) → 234878118), sparse → roaring.

Bitmaps buy O(1)-memory-per-flag answers to "how many" and "which set" — but only over a dense integer id space. The sparse-offset caveat is the whole limitation: a flat bitmap allocates to its highest offset, so a hashed (branded) id space wants a roaring bitmap, not a plain one.

/bcs/overview

The branded-id placement vector

The 14-character branded id's placement is the 32-bit offset a branded-native bitmap would use. Where the id vectors live and are asserted at boot is the Branded Component System overview — placement("USR0KHTOWnGLuC") → 234878118.

R7.03 · in this course

The approximate counterpart

Bitmaps are exact; when even one bit per entity is too much and a bounded error rate is acceptable, the probabilistic data structures of R7.03 (HyperLogLog, Bloom filters) are the road taken instead.

§7

The three dives

1-bit-flags → bitcount-aggregates → daily-active-patterns

Each dive takes one slice of the bitmap pattern, in order: the flag and its offset, the counting and combining of flags, then the daily-active rollup where the sparse-offset limit decides the structure. Read them in order — the flag first, the aggregates second, the cohorts third.

The three dives ground in the same standalone bitmap pattern: the flag, the aggregates, the daily rollup. Begin with 1-bit flags. Return to the Data Modeling & Memory chapter to continue.

§8

References

sources · related

Sources

Related in this course