R7 · Data Modeling & Memory
How data lives
in RAM.
Redis is not only a cache. With the right persistence and eviction posture it is a system of record — and the modeling family is the set of techniques for spending the memory budget once a datum has earned its place there.
Every chapter before this used Redis structures as machinery. This one asks a different
question: how the data lives in RAM and whose truth it is. The grounding is EchoMQ's real job HASH
(echo/apps/echo_mq) — the @enqueue script writes state,
attempts, and payload at emq:{q}:job:<JOB> in one atomic step;
the committed valkey.conf keeps it with maxmemory-policy noeviction and makes it
durable with appendfsync everysec — worked through codemojex's per-datum split: game
state in Valkey, money in Postgres. R7 records what R8 then runs in production.
Why & when
echo/apps/echo_mqReach for Redis as a primary store when the data is hot, operational, and tolerant of about a second of
loss — and keep it out of Redis when it is money. Two threads run through the chapter. The first is the
store as a system of record: with noeviction and an append-only log, the row in Valkey
is not a copy of something durable elsewhere — it is the record, and there is no second home to fall back
on. The second is memory as a budget: a value in RAM costs more than a value on disk, so the
modeling family is the set of techniques that spend that budget well. Each demand below has one matching
answer, and the chapter is the set of those answers.
Sub-millisecond reads, high write throughput
Treat RAM as the primary storage layer and disk as a recovery mechanism — the enqueue/claim/complete cycle is point reads and writes, one key, server-side, one round-trip.
The truth has no second home
When the row in Redis is the authoritative state, it must survive a crash and must never be evicted — that is noeviction plus an append-only log, decided per datum.
The footprint must fit in RAM
A value in memory is the expensive value — compact listpack/intset encodings, short field names, and capped structures keep the resident set small.
An exact answer is not worth its memory
Counting uniques or testing membership exactly can cost more than the answer is worth — probabilistic structures trade a bounded error for a fixed, tiny footprint.
noeviction + AOF;
the durable floor replicates the record off-box. Durability is a dial a system turns per datum.Every technique in this chapter answers one demand: serve the hot value fast, hold the truth where it has no second home, spend the RAM budget well, and dial a datum toward durability only as far as it needs.
The patterns
seven modules · one builtThe first module is built; the rest are specified. Each module is a hub with its dives, grounded in the real echo data layer where it has an honest home and in a clean standalone example where it does not.
Redis as a primary database
The store as the system of record, not a cache. Grounded in EchoMQ's job HASH (emq:{q}:job:<JOB>) under noeviction + AOF.
Memory optimization
Compact encodings (listpack, intset) and short field names; capped structures with LTRIM and MAXLEN ~. EchoMQ's memory discipline.
Probabilistic data structures
HyperLogLog, Bloom/Cuckoo, Count-Min/T-Digest — trade a bounded error for a fixed footprint. The contrast with EchoMQ's exact dedup.
Bitmaps
Millions of boolean flags in minimal memory, aggregated with BITCOUNT — a standalone daily-active-cohort example.
Vectors & similarity search
Redis 8 native vector sets (HNSW) for semantic search; the Valkey engine reaches vectors via the valkey-search module — a standalone example, codemojex note forward-tense.
Geospatial
Locations and radius queries on a geohash sorted set — a standalone GEOADD / GEOSEARCH example.
Workshop
codemojex's dashboard read-models — the round leaderboard, the unique-players count, and the live game view, each a fold of operational state.
How to apply
name the datum · the home followsThe hard part is not the structure but the judgment: whose truth is this datum, and what does it cost in
RAM. Name the datum and its tolerances — how much loss it can survive, whether it needs a
transaction — and the home follows. Redis as a primary database is never a global switch
thrown once; it is a per-datum decision made against two axes, and codemojex makes it in the open. Select a
datum to see where it lives and why.
There is no global "Redis as a primary database" switch — only the per-datum judgment that sends a job to Valkey as the record of truth and a wallet balance to Postgres for the one-transaction coupling.
The workshop
echo/apps/codemojex · valkey.confThe chapter closes with R7.07: codemojex's dashboard read-models, built on the same
per-datum discipline R7.01 establishes. Game state — the round leaderboard (Codemojex.Board,
the sorted set cm:) and the per-player guess locks (Codemojex.Locks,
the hash cm:) — lives in Valkey because it is hot, operational, and
tolerant of about a second of loss. Money — the player balances (Codemojex.Wallet) and the
append-only transaction ledger (Codemojex.Ledger) — lives in Postgres because a balance must
be exact and must move with its ledger row in one transaction. The read-models are folds of the operational
side: the leaderboard a range read, the uniques a probabilistic count, the activity view a replay.
The contrast that frames the chapter is Oban: Oban keeps its jobs in the same Postgres as the application data, so a job and a business row commit in one transaction — one store, full coupling. Echo separates the bus (the job HASH in Valkey) from the store and buys an in-memory hot path plus a durability dial; it gives up that one-transaction coupling. Each design names its trade. EchoMQ chooses the in-memory record and the ~1s AOF bound; the next rungs of the dial carry the record further from loss.
@enqueue script
writes the three-field job HASH in one atomic step; claim, complete, and
retry mutate that same row. Select a field to see what it holds.state, attempts, and payload — written by the
@enqueue script in one atomic step and mutated in place, with no second canonical copy.The production config
The same noeviction + AOF posture, run on a Fly machine: the pinned image, the
maxmemory budget, appendfsync everysec, and why a queue's store must surface
memory pressure as write errors rather than drop work silently.
The durable floor
The ~1s AOF bound is one rung of the dial. Beneath it the record is committed per change and replicated
off-box — EchoStore.Graft into the page engine and out to Tigris, the durability dial taught
end to end.
Notes on Valkey. A small hash like the three-field job row stays in a compact listpack
layout while it is below the configured entry and value thresholds, so a record of truth costs little memory
per key; HSET writes the fields and HGETALL reads the whole row back.
valkey.io/topics/persistence
Up next
R8R7 establishes that Redis can be a system of record and how its data is modeled in RAM. The chapter that follows is the operations concern over that same record — running the tier at scale.
Production & Operations
Running the tier at scale — kernel tuning, persistence and failover, the case studies, and the capstone.
References
sources · relatedSources
- Valkey — Persistence — the
appendonlyeverysecposture and a roughly one-second loss bound after a crash, the durability mechanism this chapter is built on. - Valkey — Eviction — the
maxmemory-policymenu;noevictionrefuses the write rather than dropping a key, the posture a system of record needs. - Valkey — Cluster specification — the
{q}hash tag forces every key of one queue onto one of the 16384 hash slots. - Oban — jobs and data in one Postgres transaction; the trade EchoMQ makes the other way, an in-memory record plus the durability dial.
Related in this course
- R7.01 · Redis as a primary databasethe store as the system of record, with the system-of-record, noeviction, and persistence dives
- R1.01 · Cache-asidethe cache tier whose truth lives elsewhere — the opposite of a system of record
- The Branded Component System · Valkey on a Fly machinethe manuscript figure home (B8.2): the production posture of the same config
- The Branded Component System · persistencethe durability dial (B5) and the Oban trade in depth
- The persistence floorthe durable floor beneath the ~1s AOF bound — Graft to Tigris
- The Branded Component System · the storeEchoStore (B4): the near-cache that is explicitly not a system of record