jonnify·redis /redis-patterns/data-modeling

R7 · Data Modeling & Memory

How data lives
in RAM.

Redis is not only a cache. With the right persistence and eviction posture it is a system of record — and the modeling family is the set of techniques for spending the memory budget once a datum has earned its place there.

Every chapter before this used Redis structures as machinery. This one asks a different question: how the data lives in RAM and whose truth it is. The grounding is EchoMQ's real job HASH (echo/apps/echo_mq) — the @enqueue script writes state, attempts, and payload at emq:{q}:job:<JOB> in one atomic step; the committed valkey.conf keeps it with maxmemory-policy noeviction and makes it durable with appendfsync everysec — worked through codemojex's per-datum split: game state in Valkey, money in Postgres. R7 records what R8 then runs in production.

§1

Why & when

echo/apps/echo_mq

Reach for Redis as a primary store when the data is hot, operational, and tolerant of about a second of loss — and keep it out of Redis when it is money. Two threads run through the chapter. The first is the store as a system of record: with noeviction and an append-only log, the row in Valkey is not a copy of something durable elsewhere — it is the record, and there is no second home to fall back on. The second is memory as a budget: a value in RAM costs more than a value on disk, so the modeling family is the set of techniques that spend that budget well. Each demand below has one matching answer, and the chapter is the set of those answers.

Sub-millisecond reads, high write throughput

Treat RAM as the primary storage layer and disk as a recovery mechanism — the enqueue/claim/complete cycle is point reads and writes, one key, server-side, one round-trip.

The truth has no second home

When the row in Redis is the authoritative state, it must survive a crash and must never be evicted — that is noeviction plus an append-only log, decided per datum.

The footprint must fit in RAM

A value in memory is the expensive value — compact listpack/intset encodings, short field names, and capped structures keep the resident set small.

An exact answer is not worth its memory

Counting uniques or testing membership exactly can cost more than the answer is worth — probabilistic structures trade a bounded error for a fixed, tiny footprint.

The same record, dialled toward durability. A volatile tier serves it fast; a system of record holds the truth; the durable floor carries it off-box. Select a tier to see what it guarantees and where EchoMQ sits.
dial: hold nothing volatile · a cache copy truth lives elsewhere · droppable on eviction (EchoStore) ~1s loss bound system of record · emq:{q}:job:<JOB> noeviction + AOF everysec · the row IS the truth commit-per-record durable floor · off-box EchoStore.Graft → Tigris · replicated, far from loss
readout · Select a tier. The volatile tier serves a copy fast; the system of record holds the truth under noeviction + AOF; the durable floor replicates the record off-box. Durability is a dial a system turns per datum.

Every technique in this chapter answers one demand: serve the hot value fast, hold the truth where it has no second home, spend the RAM budget well, and dial a datum toward durability only as far as it needs.

§2

The patterns

seven modules · one built

The first module is built; the rest are specified. Each module is a hub with its dives, grounded in the real echo data layer where it has an honest home and in a clean standalone example where it does not.

built ready to read soon specified
§3

How to apply

name the datum · the home follows

The hard part is not the structure but the judgment: whose truth is this datum, and what does it cost in RAM. Name the datum and its tolerances — how much loss it can survive, whether it needs a transaction — and the home follows. Redis as a primary database is never a global switch thrown once; it is a per-datum decision made against two axes, and codemojex makes it in the open. Select a datum to see where it lives and why.

Where does this datum live? — a per-datum decision, not a global switch. codemojex keeps hot, loss-tolerant operational state in Valkey and exact, transactional money in Postgres. Select a datum to light its home.
Valkey · cm:* hot · ~1s-loss-ok noeviction + AOF Postgres exact · ACID one transaction a queued job round board guess lock wallet balance ledger row
readout · Select a datum. Each is placed against two axes — how much loss it can survive and whether it must move in one transaction. Hot, loss-tolerant state goes to Valkey; exact money goes to Postgres.

There is no global "Redis as a primary database" switch — only the per-datum judgment that sends a job to Valkey as the record of truth and a wallet balance to Postgres for the one-transaction coupling.

§4

The workshop

echo/apps/codemojex · valkey.conf

The chapter closes with R7.07: codemojex's dashboard read-models, built on the same per-datum discipline R7.01 establishes. Game state — the round leaderboard (Codemojex.Board, the sorted set cm::board) and the per-player guess locks (Codemojex.Locks, the hash cm::lock:) — lives in Valkey because it is hot, operational, and tolerant of about a second of loss. Money — the player balances (Codemojex.Wallet) and the append-only transaction ledger (Codemojex.Ledger) — lives in Postgres because a balance must be exact and must move with its ledger row in one transaction. The read-models are folds of the operational side: the leaderboard a range read, the uniques a probabilistic count, the activity view a replay.

The contrast that frames the chapter is Oban: Oban keeps its jobs in the same Postgres as the application data, so a job and a business row commit in one transaction — one store, full coupling. Echo separates the bus (the job HASH in Valkey) from the store and buys an in-memory hot path plus a durability dial; it gives up that one-transaction coupling. Each design names its trade. EchoMQ chooses the in-memory record and the ~1s AOF bound; the next rungs of the dial carry the record further from loss.

The row is the record — no second home. The @enqueue script writes the three-field job HASH in one atomic step; claim, complete, and retry mutate that same row. Select a field to see what it holds.
HSET emq:{q}:job:<JOB> · the record of truth, no shadow copy state pending / active scheduled / dead attempts the integer fence bumped by HINCRBY payload the job body written once at enqueue
readout · Select a field. The job HASH carries state, attempts, and payload — written by the @enqueue script in one atomic step and mutated in place, with no second canonical copy.

Notes on Valkey. A small hash like the three-field job row stays in a compact listpack layout while it is below the configured entry and value thresholds, so a record of truth costs little memory per key; HSET writes the fields and HGETALL reads the whole row back. valkey.io/topics/persistence

§5

Up next

R8

R7 establishes that Redis can be a system of record and how its data is modeled in RAM. The chapter that follows is the operations concern over that same record — running the tier at scale.

R8soon

Production & Operations

Running the tier at scale — kernel tuning, persistence and failover, the case studies, and the capstone.

§6

References

sources · related

Sources

  • Valkey — Persistence — the appendonly everysec posture and a roughly one-second loss bound after a crash, the durability mechanism this chapter is built on.
  • Valkey — Eviction — the maxmemory-policy menu; noeviction refuses the write rather than dropping a key, the posture a system of record needs.
  • Valkey — Cluster specification — the {q} hash tag forces every key of one queue onto one of the 16384 hash slots.
  • Oban — jobs and data in one Postgres transaction; the trade EchoMQ makes the other way, an in-memory record plus the durability dial.

Related in this course