R7.04 · Data Modeling & Memory — module hub
Bitmaps.
Store millions of boolean flags in minimal memory using Redis bitmaps — 1 bit per flag with O(1) access, plus fast aggregate operations across entire datasets.
A bitmap is a String where each bit is a boolean: SETBIT and GETBIT
address a bit by offset, and one megabyte holds eight million flags. The whole pattern turns on one identity
— the bit offset is the entity's integer id — and the whole limitation turns on the
same one: a bitmap allocates up to its highest offset, so the id space must be dense. This is a
standalone example (bitmaps are not an EchoMQ surface); a forward-tense
codemojex note — the retired cm-bitmapist player-cohort spike — layers on top.
Basic operations — one bit per entity
bitmap-patterns.md.txt · SETBIT / GETBIT / BITCOUNTThree primitives address and count bits. SETBIT key offset value sets the single bit at
offset to 0 or 1, growing the underlying String as needed; GETBIT key offset reads it;
BITCOUNT key counts the set bits — a population count over the whole string, or over a range.
SETBIT users:active 1000 1 # user 1000 is active (the offset IS the id) GETBIT users:active 1000 # → 1 (was this user active?) BITCOUNT users:active # → number of active users (a popcount over the string)
The cost model is the headline: a flag is one bit, and a count is a scan the engine does in hardware popcount
steps. A bitmap is one boolean across many entities (online:users, a bit per user)
or many booleans for one entity (user:5000:features, a bit per feature) — the
same two commands, read either way.
A boolean per entity is a bit at the offset that is the id — no per-entity key, no
map from id to slot. The 1-bit flags dive
takes SETBIT/GETBIT and the feature grid apart.
Daily active users (and weekly / monthly)
bitmap-patterns.md.txt · a bitmap per dayName a bitmap per day and set a player's bit when they act. The day's count is a BITCOUNT; windows
roll up with BITOP:
SETBIT dau:2024-01-30 12345 1 # user 12345 visited today BITCOUNT dau:2024-01-30 # daily active users — 1M users = 125 KB/day BITOP OR wau:2024-W05 dau:2024-01-28 dau:2024-01-29 dau:2024-01-30 ... # active ANY day (union) BITOP AND wau:2024-W05:every dau:2024-01-28 dau:2024-01-29 ... # active EVERY day (intersection)
BITOP OR over a week's daily bitmaps is weekly-active; BITOP AND over them is "active
every day". Retention cohorts fall out of the same two operators: who was active on day 0 AND
day 7.
A bitmap per day makes DAU, WAU, MAU, and retention a matter of BITCOUNT and
BITOP. The daily-active
patterns dive carves this slice and the codemojex note.
Feature flags and online status
bitmap-patterns.md.txt · one entity's many booleansA bitmap can hold one entity's many booleans, or one boolean across many entities. Feature flags per user — bit positions are features:
# 0 = dark_mode, 1 = notifications, 2 = beta_features, ... SETBIT user:5000:features 0 1 # enable dark mode SETBIT user:5000:features 2 1 # enable beta features GETBIT user:5000:features 1 # notifications enabled? GET user:5000:features # all flags at once — decode client-side
Online status is one bitmap, one bit per user: SETBIT online:users 12345 1 on,
0 off, BITCOUNT online:users for how many. For millions of users this costs a fraction
of a key-per-user, and time-bucketed keys (online:minute:<ts> with an EXPIRE)
give "active in the last five minutes" as a BITOP OR over the recent buckets.
Read a bitmap by the entity (a few feature bits in one key) or across entities (one online bit per
user) — both are SETBIT/GETBIT at one bit each.
Cohorts, BITFIELD, and BITPOS
bitmap-patterns.md.txt · combine, pack, locateCohort analysis intersects attribute bitmaps — every segment is one BITOP
plus a BITCOUNT:
SETBIT users:premium 12345 1 SETBIT users:mobile 12345 1 BITOP AND cohort:premium_mobile users:premium users:mobile BITCOUNT cohort:premium_mobile # premium AND mobile BITOP NOT users:not_mobile users:mobile # complement, then BITOP AND cohort:premium_desktop users:premium users:not_mobile
Two commands address bit ranges beyond a single bit. BITFIELD does atomic read-modify on
arbitrary-width fields packed into one key — small counters with overflow control
(BITFIELD counters INCRBY u8 0 1 OVERFLOW SAT). BITPOS finds the first set or unset bit
(BITPOS users:active 1 — the first active id; BITPOS users:active 0 — the
first free slot). And rate limiting with one bit per millisecond uses BITCOUNT
over a per-second key, reclaimed by a short EXPIRE.
Beyond flags: BITOP composes cohorts, BITFIELD packs counters,
BITPOS locates an edge — all over the bits. The
bitcount aggregates dive takes
BITCOUNT and BITOP apart, with the Bloom-filter-alternative contrast.
Memory efficiency — and the sparse-offset limit
bitmap-patterns.md.txt · the density, and the caveatThe reason to reach for a bitmap is the memory table — and its one limitation is the same fact read the other way:
| Entities | Bitmap size | Equivalent Set |
|---|---|---|
| 1 million | 125 KB | ~50 MB |
| 10 million | 1.25 MB | ~500 MB |
| 100 million | 12.5 MB | ~5 GB |
Bitmaps are roughly 400× more memory-efficient than a Set of the same ids — but only over a dense integer id space, because a bitmap allocates memory up to its highest offset:
SETBIT users:active 999999999 1 # allocates ~125 MB to hold ONE bit
A sparse id space wastes memory. The source's fixes are: use dense, sequential ids; hash ids to a
bounded range and accept collisions; or use a Set. BITOP is also O(N) in the longest
input, so very large bitmaps are sharded by id range. This sparse-offset caveat is the whole story of the
grounding below.
Notes on ValkeyBITCOUNT key [start end [BYTE | BIT]] counts
set bits over the whole string or a range, and SETBIT/GETBIT/BITOP/BITPOS/BITFIELD
are all core commands on the String type — no module required (the probabilistic
BF.*/CF.* structures are the module exception R7.03 covers) —
valkey.io/commands/bitcount.
The grounding — the branded offset, in a retired spike
branded_id.ex · infra/cm-bitmapist (retired)A bitmap's offset must be a non-negative integer. The Branded Component System already mints one for every
entity: a 14-character branded id carries a placement — hash32, the first
half of MurmurHash3's fmix64 truncated to 32 bits. The runtime asserts the vector at boot
(branded_id.ex, self_check!):
-- the placement (hash32) is a 32-bit integer — a bitmap bit offset
placement("USR0KHTOWnGLuC") → 234878118
That 32-bit placement is exactly a bitmap bit offset. codemojex ships no bitmap code today
— zero SETBIT/BITCOUNT/bitmapist in echo/apps/codemojex/lib/.
What follows is a retired spike that never shipped: cm-bitmapist
(infra/cm-bitmapist/ until 2026-10-11 — a Go port of Doist's bitmapist4, branded-id-native, on
its own Fly machine) would mark each player's daily-active bit at
Offset(id) = Hash32(Decode(id)). A page-own codemojex example uses the live brand
PLR (player):
# retired spike — cm-bitmapist, never in echo/apps/codemojex Mark("active", "PLR0KHTOWnGLuC", today) # offset = placement("PLR0KHTOWnGLuC") Count("active", month) # BITCOUNT — monthly active players AndCount(registered, played, paid) # a registered → played → paid funnel
Because the branded placement is a hash, the offset lands anywhere in the 32-bit space
— sparse, exactly the limitation above. So the spike used roaring bitmaps (compressed
bitmaps that store runs, not a flat allocation to the highest offset): the published result for a sparse
branded keyspace is on the order of ~129 GB on a plain bitmap versus ~300 MB roaring. The
hash32 is also collision-bearing in 32 bits (on the order of N²/2³³ colliding pairs),
so distinct counts undercount slightly — a property to state, not assume. Select where a structure sits in
the modeling family:
1 bit per flag; BITCOUNT/BITOP analytics over millions of entities at O(1) memory
per flag.
codemojex's retired cm-bitmapist spike was designed to model daily-active players
as bitmaps — the bit offset is the player's branded-id placement hash
(placement(…) → 234878118), sparse → roaring.
Bitmaps buy O(1)-memory-per-flag answers to "how many" and "which set" — but only over a dense integer id space. The sparse-offset caveat is the whole limitation: a flat bitmap allocates to its highest offset, so a hashed (branded) id space wants a roaring bitmap, not a plain one.
The branded-id placement vector
The 14-character branded id's placement is the 32-bit offset a branded-native bitmap would use.
Where the id vectors live and are asserted at boot is the Branded Component System
overview — placement("USR0KHTOWnGLuC") → 234878118.
The approximate counterpart
Bitmaps are exact; when even one bit per entity is too much and a bounded error rate is acceptable, the probabilistic data structures of R7.03 (HyperLogLog, Bloom filters) are the road taken instead.
The three dives
1-bit-flags → bitcount-aggregates → daily-active-patternsEach dive takes one slice of the bitmap pattern, in order: the flag and its offset, the counting and combining of flags, then the daily-active rollup where the sparse-offset limit decides the structure. Read them in order — the flag first, the aggregates second, the cohorts third.
1-bit flags
SETBIT/GETBIT, one bit per entity, the offset is the id; the
125 KB-per-1M-flags density; a feature-flag-per-user grid.
bitcount aggregates
BITCOUNT (popcount) + BITOP AND/OR/XOR/NOT (combine bitmaps) +
BITPOS; the Bloom-filter-alternative framing (a contrast to R7.03).
daily-active patterns
A bitmap per day, BITCOUNT = DAU, BITOP OR/AND = WAU/MAU and retention;
the codemojex forward-tense note + the sparse-offset → roaring caveat.
The three dives ground in the same standalone bitmap pattern: the flag, the aggregates, the daily rollup. Begin with 1-bit flags. Return to the Data Modeling & Memory chapter to continue.
References
sources · relatedSources
- Valkey — SETBIT — set the bit at an offset; the offset is the entity id.
- Valkey — BITCOUNT — population count over a String, whole or by range.
- Valkey — BITOP — AND / OR / XOR / NOT across bitmaps for cohorts and windows.
- Valkey — BITPOS — first set or unset bit; the first active id or free slot.
- Valkey — BITFIELD — atomic read-modify on packed bit-width counters.
- Redis — Bitmaps — the data-type overview the patterns build on.
Related in this course
- R7.04.1 · 1-bit flags
SETBIT/GETBIT, the offset is the id - R7.04.2 · bitcount aggregates
BITCOUNTandBITOPanalytics - R7.04.3 · daily-active patternsDAU/WAU/MAU and the codemojex note
- R7.03 · Probabilistic data structuresbitmaps as an exact alternative to an estimate
- R7.02 · Memory optimizationthe same memory family
- /bcs · Overviewthe branded-id placement vector