jonnify·redis /redis-patterns/data-modeling/primary-database

R7.01 · Data Modeling & Memory — module hub

Redis as a primary database.

Use Redis as the authoritative data store for applications requiring sub-millisecond latency and high write throughput, treating disk as a recovery mechanism rather than the primary storage layer.

A cache holds a copy — lose it and the truth is still in the database behind it. A primary store holds the truth: there is no second home to fall back on, so eviction must be off, durability must be configured, and the relationships a relational engine used to enforce are modelled by hand. The grounding is EchoMQ's real applied choice (echo/apps/echo_mq): the job HASH at emq:{q}:job:<JOB> is the job's only canonical state — state/attempts/payload, written by the @enqueue script in one atomic step, kept by noeviction, made durable by AOF everysec. The three dives take it apart — the record, the eviction posture, the persistence.

The row is the record — there is no second canonical copy. Select a field of the job HASH to read what it holds; the whole HASH at the job key is the job's authoritative state.
key · emq:{q}:job:<JOB> (one HASH, the record) statepending attempts0 (the fence) payloadthe body written once by @enqueue · HSET KEYS[1] state pending attempts 0 payload … claim · complete · retry mutate this same row — no shadow copy elsewhere
readout · Select a field. The HASH at emq:{q}:job:<JOB> is the job's only canonical state — state, attempts, and payload, written by @enqueue in one step.
§1

When Redis works as primary

redis-as-primary-database.md.txt · the fit

Redis is a fit as a primary store for state that is hot, operational, and tolerant of roughly a second of loss on a crash. The source names four families:

  • Real-time application state — leaderboards, session stores, live dashboards.
  • High-velocity data — activity streams, gaming state, sensor feeds.
  • Precomputed views — timeline fan-out, materialized aggregations read far more often than written.
  • Microservices coordination — distributed locks, rate limiting, feature flags.

It is the wrong choice when the dataset exceeds RAM, when complex multi-key ACID transactions are required, when ad-hoc queries with joins are common, or when a regulator mandates disk-first storage. The decision is per datum, not global — the same application keeps one datum in Redis and another in a relational database, by what each datum needs.

the pattern

Redis as the authoritative data store: there is no truth elsewhere to fall back on, so the row is the record.

its EchoMQ application

The emq:{q}:job:<JOB> HASH is the job's only canonical state — state / attempts / payload, written by the @enqueue script in one atomic step, kept by noeviction, made durable by AOF everysec. No shadow copy in Postgres.

EchoMQ.Jobs states it directly: “Jobs are entities. A job's identity is a branded id under the JOB namespace; its row is a hash at the job key; the pending set is a same-score sorted set whose members are the ids themselves… Enqueue is one idempotent script: kind policy, duplicate refusal, row write, and pending insertion happen on the server in one atomic step.” The script is the whole write — admit by kind, refuse a duplicate, write the row and the pending entry:

-- @enqueue (EchoMQ.Script.new(:enqueue, …)) — the row IS the record
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

The HASH carries three fields — state (pending / active / scheduled / dead), attempts (the integer fence), payload (the body). A primary-database datum is one whose truth has no second home, and that is a judgment made per datum, not a global switch.

§2

Persistence configuration

redis-as-primary-database.md.txt · RDB / AOF / hybrid

If disk is recovery and not the source of truth, the persistence policy sets how much recent work a crash can take. The source enables both mechanisms for a general primary store, and tunes the loss bound with appendfsync:

appendonly yes
appendfsync everysec
save 900 1     # RDB: snapshot if ≥1 key changed in 900s
save 300 10
save 60 10000

RDB is point-in-time snapshotting via fork(): compact files and a fast restart, but everything since the last snapshot is lost on a crash — useful for backups and bootstrapping. AOF is a write log of every operation, and the appendfsync policy sets the loss bound:

policydurabilitylatencyuse
alwaysmaximumhighfinancial transactions
everysec~1 second losslowmost production workloads
noOS-dependentminimalcaching layer

For a primary store, everysec is the standard choice — about a second of worst-case loss for a far better write path — and aof-use-rdb-preamble yes starts the AOF with a compact RDB snapshot, combining a fast restart with the AOF's durability.

EchoMQ's real applied choice diverges from the source's “enable both”: it runs AOF-only with save "", so RDB snapshotting never competes for a second fork, and keeps aof-use-rdb-preamble yes so the rewrite still uses the compact format. The committed valkey.conf is exact: “AOF is the single source of durability. One fork source. AOF everysec bounds worst-case loss to about one second… RDB snapshotting is disabled so it does not compete for a second fork; the AOF still uses an RDB preamble for fast rewrite.” The result is a ~1s loss bound EchoMQ's checkpoints are designed against.

Notes on ValkeyHSET key field value … sets one or more fields on a Hash, creating the key if absent — the row write at the heart of the primary-store model, and the line the AOF records on every job admitted — valkey.io/commands/hset.

RDB snapshots are compact and fast to restart but lose the interval; the AOF logs every write and everysec bounds the loss to about a second — EchoMQ runs AOF-only with an RDB preamble, one fork source. The persistence dive carves this slice.

§3

Data modeling without SQL

redis-as-primary-database.md.txt · keys, indexes, integrity

A primary store gives up the relational engine: no joins, no foreign keys, no query planner. Denormalization and explicit indexing become the application's job:

  • Primary keys — an entity is a Hash under a namespaced key: HSET user:1001 username "alice" karma 42.
  • Secondary indexes — built by hand: SET user:email:alice@example.com 1001 for a lookup; ZADD users:by:age 25 1001 for a range.
  • One-to-many — a Set holds the relationship: SADD user:1001:followers 1002 1003.
  • Referential integrity — Redis enforces no foreign keys, so a delete that must stay consistent is a Lua script that walks the relationships and removes both ends atomically.

EchoMQ models exactly this way: the job is a Hash keyed by a branded id, and the pending set is a same-score Sorted Set whose members are the ids themselves — byte order is mint order, so the queue carries no second index. The key is built by hand, and the branded id is gated before it is used:

# EchoMQ.Keyspace — the key builder (real)
def queue_key(queue, type) when is_binary(queue) and is_binary(type),
  do: IO.iodata_to_binary(["emq:{", queue, "}:", type])

def job_key(queue, branded) when is_binary(branded) do
  if EchoData.BrandedId.valid?(branded) do
    queue_key(queue, "job:") <> branded
  else
    raise ArgumentError, "job_key requires a valid branded id"
  end
end

The key is emq:{q}:job:<JOB-branded-id>. The brace is the cluster hash tag, so every key of one queue lands on one slot — the referential-integrity script stays a single-slot, multi-key transaction.

§4

Case study: LamerNews

redis-as-primary-database.md.txt · a web app on Redis alone

LamerNews — a Hacker News clone by antirez — runs a complete web application on Redis alone, and is the canonical demonstration of the modeling discipline above.

  • User storage — user:1001 is a Hash of {username, password_hash, karma, created_at}, with username.to.id:alice and auth:sha1token as String indexes. Session validation is O(1) — no database round-trip on the common path.
  • News ranking — news.top is a Sorted Set scored by VOTES / (AGE_HOURS ^ GRAVITY). A vote recomputes the rank and re-ZADDs it; the homepage is one command, ZREVRANGE news.top 0 29.
  • Comment threading — all comments for an item live in one Hash; the application reads them with HGETALL, builds the parent-child tree in memory, and renders recursively.

The shape is EchoMQ's shape: an entity as a Hash, a Sorted Set as the ordered index, and the application holding the logic the relational engine used to hold.

A whole application can live on Redis when the data is modelled for it — entities as Hashes, ordered views as Sorted Sets, and relational logic moved into the application or a Lua script.

§5

Production patterns from scale

redis-as-primary-database.md.txt · Pinterest · Twitter · DoorDash

Three patterns recur where Redis is the primary store at scale:

  • Pinterest — virtual sharding. Billions of follower relationships are spread across 8192 logical shards mapped onto physical instances, giving predictable distribution and easier rebalancing across many single-threaded processes. EchoMQ's {q} hash tag is the same idea at queue granularity — every key of one queue lands on one of the 16384 cluster slots.
  • Twitter — timeline precomputation. The home timeline is a capped List per user (~800 entries); a tweet fans out to followers' timeline Lists, and a read is LRANGE timeline:user_id 0 199 — O(1), not a join. Write amplification is traded for read simplicity.
  • DoorDash — listpack-Hash memory. Switching a feature store from flat SET feature:1001:age 25 pairs to one HSET feature:1001 age 25 city "NYC" Hash cut memory ~40%: a small Hash uses Redis's compact listpack encoding. The job HASH benefits from the same encoding — three short fields under one key.

Notes on ValkeyA small Hash is stored in a compact listpack encoding until it crosses hash-max-listpack-entries / hash-max-listpack-value, when it converts to a hashtable — so grouping related fields under one key is the memory win DoorDash measured — valkey.io/topics/lru-cache.

At scale the modeling choices are memory and distribution choices: shard by a hash tag, precompute the read, and group small fields under one Hash so the compact encoding holds.

§6

Operational requirements

redis-as-primary-database.md.txt · HA · blocking · hot/big keys

Running a primary store has obligations a cache does not.

  • High availability. The source uses Redis Sentinel for automatic failover and min-replicas-to-write to prevent split-brain. EchoMQ's real posture is different — “auto-stop off, one machine always running” (it does not run Sentinel) — and the operating-at-scale depth (Sentinel, pooling, failover) is the subject of R8.02 · Persistence, pooling & failover, not this module.
  • Avoid blocking O(N) commands. In a single-threaded server, an O(N) command blocks everything: never KEYS * (use SCAN), avoid HGETALL on a Hash of millions of fields, prefer UNLINK to DEL for a large key.
  • Watch for hot keys and big keys. A single hot key saturates a CPU core; a big key spikes latency during operations. redis-cli --hotkeys and --bigkeys surface them.

EchoMQ pairs noeviction with propagation-error-behavior panic and a maxmemory set as a loud guardrail far above the working set, not a working ceiling. The bcs.8 manuscript puts the posture plainly: “Eviction is the wrong posture for a job store: a queue that silently drops keys is a queue that silently loses work, so memory pressure must surface as write errors and alerts.” The noeviction dive carves this slice.

Where does this datum live? — a per-datum decision. Select a datum to see which store holds it; codemojex keeps operational state in Valkey and money in Postgres.
Valkey · cm:* (hot, ~1s loss ok) Postgres (money, ACID) a queued job the round board (ZSET) a guess lock (HASH) a wallet balance the ledger (append-only)
readout · Select a datum. codemojex keeps the job, the round board (Codemojex.Board), and the guess lock (Codemojex.Locks) in Valkey; the wallet (Codemojex.Wallet) and ledger (Codemojex.Ledger) in Postgres.
the pattern

“Redis as a primary database” is a judgment made per datum, not a global switch: keep in Redis what is hot and loss-tolerant; keep elsewhere what needs full ACID.

its codemojex application

codemojex splits: the round board (Codemojex.Board, the cm:<game>:board ZSET) and the guess locks (Codemojex.Locks) live in Valkey; player balances (Codemojex.Wallet) and the transaction ledger (Codemojex.Ledger) live in Postgres, mutated in one database transaction.

The job's truth has no second home, so it lives in Valkey under noeviction and AOF; the wallet's truth wants the one-transaction coupling, so it lives in Postgres. Oban keeps the job and the business row in one Postgres transaction; EchoMQ gives up that coupling for an in-memory hot path and the durability dial — the trade stated beside the win.

/bcs/fly

The production posture — Valkey on a Fly machine

This module cites the committed valkey.conf — noeviction, AOF everysec, save "", propagation-error-behavior panic — as the applied choice. The production posture of that exact config is the subject of the Branded Component System production chapter: the release image, Valkey on a Fly machine, and the fly.toml beside it.

/bcs/store

What is not a system of record — EchoStore

A primary store holds the truth; a near-cache holds a copy. EchoStore is the cache tier — L1 ETS over L2 Valkey, coherence by mint time — the deliberate contrast to a datum whose truth has no second home. Lose the cache and the record is still behind it; lose the record and there is nothing to fall back on.

R7.1 grounds the primary-store posture in EchoMQ's job HASH and codemojex's per-datum split. The three dives below take the record, the eviction posture, and the persistence in turn. Return to the Data Modeling & Memory chapter to continue.

§7

The three dives

system of record → noeviction → persistence

Each dive takes one slice of the primary-store posture, in order: the row as the record and the per-datum decision, the eviction policy that refuses the write rather than drop the record, and the persistence that bounds the loss on a crash. Read them in order — the record first, the posture second, the durability third.

R7.01.1built

System of record

The row is the record: the @enqueue HSET writes it and claim/complete/retry mutate that same row; codemojex splits per datum. Closes with the Oban contrast.
R7.01.2built

noeviction

maxmemory-policy is a menu; the cache policies delete keys under pressure, noeviction refuses the write. A system of record runs noeviction so pressure is a loud error, not a vanished job.
R7.01.3built

Persistence (RDB / AOF)

Two durability mechanisms and the loss bound each sets; EchoMQ's AOF-only, everysec, RDB-preamble choice; and the durability frontier that doors to /echo-persistence.

The three dives ground in the same record and the same committed config: the row, the eviction posture, and the persistence. Begin with system of record.

§8

References

sources · related

Sources

Related in this course