jonnify·redis /redis-patterns/production-operations/persistence-pooling-failover/rdb-and-aof

R8.02.1 · Persistence, pooling & failover — dive 1

RDB and AOF, the durability dial.

A job store that loses acknowledged work on a restart is not a record of truth. Valkey offers two persistence mechanisms — a point-in-time snapshot and an append-only log — and the real valkey.conf chooses one on purpose.

Grounding: infra/valkey/conf/valkey.conf (the persistence block, verbatim) · the engine is Valkey 9. This is the server-level durability policy — distinct from R8.01’s kernel-level persistence-safe-settings (the save fork’s vm.overcommit_memory sysctl).

Pick a persistence directive — read what it guarantees. Select a line from the real valkey.conf to light the guarantee it buys; the readout names the trade.
appendonly yes a log on disk, write by write appendfsync everysec ~1 s crash-loss boundfsync off the main thread save "" one fork source (RDB off) rdb-preamble yes a fast, compact rewrite panic no false acknowledgement
readout · Select a directive. Together they set the durability dial for a volatile work tier: an append-only log, a one-second loss bound, one fork source, a fast rewrite, and no acknowledgement of data that may not survive.

Static map — appendonly yes → a log on disk · everysec → loss ≈ 1 s · save "" → one fork · rdb-preamble → fast rewrite · panic → no false OK.

§1

The two mechanisms, and the choice

RDB vs AOF

RDB writes a compact point-in-time snapshot of the whole dataset by forking a child that serializes memory to disk. It is fast to load and small on disk, but a crash loses everything written since the last snapshot — minutes of work, not seconds. AOF appends every keyspace-changing write to a log; on restart the log replays to rebuild the dataset. With a once-a-second fsync, a crash loses at most about a second.

For a cache, RDB’s snapshot-interval loss is fine. For a job store it is silent data loss: a job that was enqueued and acknowledged, then lost in the gap before the next snapshot. So valkey.conf makes AOF the single source of durability and turns RDB off.

RDB
a fork-and-serialize snapshot — fast load, small file, but loses a whole interval on a crash.
AOF
a write-by-write log replayed at startup — with everysec, a crash loses about one second.
§2

The persistence block, verbatim

valkey.conf · persistence

The real config, from infra/valkey/conf/valkey.conf — the block that fronts the codemojex queue:

infra/valkey/conf/valkey.conf · AOF is the single source of durabilityfrozen
# ---- Persistence: AOF is the single source of durability ----
appendonly yes
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-use-rdb-preamble yes
save ""
dir /data
propagation-error-behavior panic

Line by line:

  • appendonly yes turns the append-only file on — every write that changes the keyspace is appended to a log on disk.
  • appendfsync everysec flushes that log to disk once a second on a background thread. The worst-case loss bound after a crash is about one second of writes — not a whole snapshot interval, and the fsync stays off the command thread.
  • save "" disables RDB snapshotting. With AOF as the single durability mechanism there is no reason to keep a second snapshot fork competing for the machine. One fork source, not two.
  • aof-use-rdb-preamble yes lets the periodic rewrite start from a compact RDB-format preamble rather than replaying every command, so the rewrite is fast and the file stays small — RDB’s fast load with AOF’s low loss.
  • auto-aof-rewrite-percentage 100 / auto-aof-rewrite-min-size 64mb trigger that rewrite when the log has doubled and is at least 64 MB, keeping the file from growing without bound.
  • propagation-error-behavior panic fails a write rather than acknowledge data that may not survive. A false acknowledgement is worse than a visible error: the caller that hears OK stops worrying about the job.

Notes on ValkeyThe append-only file is replayed at startup to rebuild the in-memory dataset; appendfsync everysec is the balance Valkey recommends — durability close to always at a fraction of its fsync cost — valkey.io/topics/persistence.

§3

The single-threaded premise

valkey.conf · the sizing comment

The config’s own load-bearing comment names why the persistence work has to stay off the command thread: “command execution is single-threaded, so the second shared vCPU is reserved for background work — AOF fsync, fork copy-on-write.” The once-a-second appendfsync and the rewrite fork both run on that second vCPU; the one command thread keeps serving. This is why save "" matters beyond avoiding duplicate work: a second snapshot fork would contend for the same background budget the AOF fsync needs.

# valkey.conf — the sizing premise, verbatim
# command execution is single-threaded, so the second shared vCPU is
# reserved for background work (AOF fsync, fork copy-on-write).
io-threads 1

One command thread, one durability mechanism, one fork source. The everysec fsync and the rewrite run on the second vCPU, so the thread that answers callers never stops to flush.

§4

Reject, do not evict — the R7.01 tie

valkey.conf · the memory guardrail

Durability has a second failure mode besides a crash: memory pressure. maxmemory-policy noeviction makes a write that would exceed maxmemory return an error to the caller instead of evicting a key. For a queue, an evicted key is silently lost work — a job something already enqueued, gone. noeviction turns memory pressure into a rejected write and an alert, the loud failure R7.01 requires of a record of truth. (R8.01’s dive 3 takes noeviction apart at length; here it is the tie that makes the persistence policy a record-of-truth policy.)

the pattern

Choose the append-only log as the single source of durability; bound crash loss with everysec; turn snapshotting off so one fork serves; fail loudly on un-survivable data and on memory pressure.

its EchoMQ application

The committed valkey.conf fronts the codemojex queue’s emq:{q}: keyspace with appendonly yes + appendfsync everysec + save "" + propagation-error-behavior panic, so in-flight JOB state survives a restart.

the door · the durability dial, one tier down

Beneath the volatile tier

The AOF bounds the queue’s crash loss to about a second — the right floor for a volatile work tier, but not the whole durability story. Deep history and off-box durability live one tier down on the persistence floor (ETS → Valkey → CubDB → Tigris). The /echo-persistence course carries that dial; this dive stays on the volatile tier’s valkey.conf. The protocol depth — the lanes, the leases, the Lua — is the dedicated /echomq course.

Notes on ValkeyThe eviction policies (the allkeys-lru family and the rest) decide what to drop when a write would exceed maxmemory; noeviction is the one policy that drops nothing and returns an error instead — valkey.io/topics/lru-cache.

Durability is a dial, and this config sets it for a volatile work tier: keep the last second, loudly, with one fork. The append-only log bounds the crash; panic and noeviction refuse to lose or fake work.

§5

Recap — one fork, one second

next → R8.02.2 · pool-sizing

The real valkey.conf makes AOF the single source of durability: appendonly yes logs every write, appendfsync everysec bounds crash loss to about a second on a background thread, save "" retires RDB so one fork serves, aof-use-rdb-preamble yes keeps the rewrite fast, and propagation-error-behavior panic refuses a false acknowledgement. That is the engine-level half of operating the bus as a record of truth — distinct from R8.01’s kernel half. Next, the same record under load: how the connector pool is sized for a single-threaded server.

§6

References

sources · related

Sources

  • Valkey — Persistence — RDB and AOF, the fsync policies, and the durability trade the append-only file makes.
  • Valkey — BGREWRITEAOF — the rewrite the auto-aof-rewrite-* thresholds trigger, compacted via the RDB preamble.
  • Valkey — Eviction — the policies, and why noeviction returns an error rather than dropping a key.
  • Redis — Persistence — the canonical RDB-vs-AOF comparison and the everysec recommendation.

Related in this course