R8.01 · Production & Operations — module hub
Linux kernel tuning for Redis.
Configure Linux kernel parameters to prevent latency spikes, persistence failures, and connection drops in production Redis deployments — essential settings that override defaults hostile to in-memory databases.
Grounding: content/production/kernel-tuning.md.txt (the THP, overcommit, swappiness, and TCP figures) · infra/valkey/conf/valkey.conf (EchoMQ’s real Valkey posture, verbatim) · bcs.8.md B8.2 — the engine is Valkey 9; host sysctls and the config file are two halves of one posture.
Static verdicts — THP off → no 500x copy on a fork · overcommit on → the fork is not refused · swappiness 1 → no nanoseconds-to-milliseconds page · TCP backlog → connections are not dropped · FD limit → a client is not refused · activedefrag → empty pages return to the OS.
Transparent Huge Pages (THP)
kernel-tuning.md.txt · THPTHP is the most critical setting. When enabled, it causes severe latency spikes during persistence
operations. The engine uses fork() to create a child process for an AOF rewrite, and Linux uses
copy-on-write to share memory between parent and child. With THP enabled, the kernel hands out 2 MB pages
instead of 4 KB pages. When the parent modifies a single byte during the rewrite window, the kernel must
copy the entire 2 MB page — the source records this as a 500x amplification of the
copy.
The symptoms the source lists: latency spikes of 10–100x during a background save, memory usage doubling temporarily during persistence, and client timeouts while the save runs. THP is a kernel setting, not a server one — turn it off at the host:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# permanent, via systemd:
[Service]
ExecStartPre=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
The server logs a warning at startup if THP is enabled. The source’s instruction is to never ignore it. The first dive takes the fork and copy-on-write apart.
Memory Overcommit
kernel-tuning.md.txt · OvercommitThe fork() system call can fail if the kernel thinks there is insufficient memory. When the
server forks, the kernel may pessimistically assume the child needs as much memory as the parent. The
source’s example: on a 32 GB instance with 30 GB used, the fork can fail even though
copy-on-write means little real memory is actually needed.
Overcommit is a kernel sysctl. Set it so the kernel always allows the allocation request, trusting that copy-on-write keeps the real cost small:
sysctl vm.overcommit_memory=1
# persist in /etc/sysctl.conf:
vm.overcommit_memory=1
Swappiness
kernel-tuning.md.txt · SwappinessData in swap is a catastrophic failure mode. If any of the server’s memory pages are swapped to disk, access times go from nanoseconds to milliseconds — the source measures this as a 1,000,000x slowdown. A single swapped page can cause client timeouts.
Swappiness is a kernel sysctl. The source sets it to 1 — not 0, because 0 can cause OOM kills; 1 makes swapping extremely unlikely while still allowing the kernel to swap in a desperate situation:
sysctl vm.swappiness=1
# persist in /etc/sysctl.conf:
vm.swappiness=1
A swapped page trades the in-memory database’s entire reason for being. Keeping
swappiness at 1 makes that trade extremely unlikely without inviting an OOM kill.
TCP Stack Tuning
kernel-tuning.md.txt · TCPUnder high connection rates, default TCP settings cause connection failures. The listen queue for incoming
connections defaults to 128. During traffic spikes, new connections are dropped before the server ever sees
them. Two of these are kernel sysctls and one is a server setting, and they
interact: the effective backlog is min(tcp-backlog, somaxconn). Raise both, then set the
server-side backlog and the keepalive interval:
sysctl net.core.somaxconn=65536 sysctl net.ipv4.tcp_max_syn_backlog=65536 # in the server config (effective = min(tcp-backlog, somaxconn)): tcp-backlog 65536 tcp-keepalive 300 # keepalives every 300 s through a load balancer
File Descriptor Limits
kernel-tuning.md.txt · FD limitsEach connection consumes a file descriptor. The source raises the limit for the service account so a burst of clients is accepted rather than refused. In the systemd service file:
[Service] LimitNOFILE=65536
The server’s maxclients is bounded by the available file descriptors minus
roughly 32 reserved for internal use — raise the limit and the ceiling rises with it.
Memory Allocator
kernel-tuning.md.txt · jemallocThe engine uses jemalloc, which is well-suited to its allocation pattern. The source names two behaviours
to watch. After deleting a large amount of data, jemalloc retains memory to satisfy future allocations, so the
process RSS reflects peak usage rather than the current dataset size. And fragmentation shows up in
mem_fragmentation_ratio (read from INFO MEMORY):
- below 1.0 — the process is using swap (critical);
- 1.0–1.5 — normal;
- above 1.5 — significant fragmentation.
When fragmentation is high, active defragmentation relocates live objects so empty pages can be returned to the OS. It is a server setting:
activedefrag yes
On EchoMQ’s valkey.conf
infra/valkey/conf/valkey.conf · bcs.8.md B8.2The host sysctls are only half the posture. The other half is the server’s own config file — and
EchoMQ’s is real, committed, and applied. The kernel knobs (THP, overcommit, swappiness, somaxconn) live
in the host; the application-level knobs live in infra/valkey/conf/valkey.conf. That file never
carries a sysctl — it carries the threads, lazy-free, persistence, eviction, and defrag posture that the
sysctls leave room for.
The manuscript states the kernel half from the inside: an in-memory store needs the kernel tuned away
from its defaults — transparent huge pages off, overcommit set, swappiness low, and the connection
backlog and file-descriptor limits raised so a burst of clients is accepted rather than dropped
(bcs.8.md, B8.2).
Change the host sysctls — THP off, overcommit on, swappiness low, somaxconn and file
descriptors raised — so a fork stays cheap, a swapped page never appears, and a burst of clients is
accepted; and set the in-memory database’s own config away from defaults hostile to a job store.
valkey.conf sets io-threads 1, the full lazyfree-lazy-* family,
appendonly yes with appendfsync everysec, maxmemory-policy noeviction,
activedefrag yes, propagation-error-behavior panic, and
tcp-backlog 511 — the application half the kernel half leaves room for.
The production posture, committed and applied
These are not aspirations. The file pins one command thread so the second shared vCPU feeds fsync, fork copy-on-write, and jemalloc purge; durability is a single append-only log fsynced every second (valkey.io/topics/persistence); a runaway keyspace rejects writes loudly rather than losing work; and the write fails rather than acknowledge data that may not survive. The persistence and threads lines, verbatim:
io-threads 1 ← one command thread; helpers take the second vCPU appendonly yes appendfsync everysec ← ~1 s loss bound, fsync on a background thread auto-aof-rewrite-percentage 100 aof-use-rdb-preamble yes save "" ← no RDB; one fork source, not two dir /data propagation-error-behavior panic ← fail the write rather than ack lost data activedefrag yes ← relocate live objects so empty pages purge
Notes on ValkeyA background save forks a child and relies on
copy-on-write; the fork copy-on-write spike during an AOF rewrite is exactly what the kernel tuning is for, and
is why the config keeps maxmemory well under the machine’s RAM —
valkey.io/topics/latency.
Notes on ValkeyWith appendonly on and
appendfsync everysec, the volume holds an append-only log and the loss bound after a crash is
roughly one second of writes, not a whole snapshot interval —
valkey.io/topics/persistence.
The three dives
fork → command thread → durabilityEach dive takes one cluster of the source’s settings, in the arc the fork → the command thread → durability: the two settings that keep a fork cheap and allowed, the ones that keep the single command thread fast, and the ones that keep work durable on a crash. Read them in order — the fork starts.
The fork’s two settings
THP off and vm.overcommit_memory=1: the copy-on-write story behind the 500x
amplification and the fork the kernel would otherwise refuse.
Keeping the single command thread fast
Swappiness, the lazyfree-lazy-* frees, activedefrag, and the
observability that catches a spike before a client does.
Durable on a crash, never silently losing work
AOF everysec, the noeviction posture, and the connection backlog that
accepts a burst rather than dropping it.
Three dives, one host: keep the fork cheap and allowed, keep the command thread fast, and keep the work durable — the host sysctls and the config file working as one posture.
After R8.01, the chapter continues with persistence and failover ops, then the case studies
— Pinterest, Twitter/X, and Uber — before the capstone hands off to the dedicated EchoMQ course.
Return to the Production & Operations chapter to
continue, or open Codemojex on Fly for where this valkey.conf actually
runs.
References
sources · relatedSources
- Valkey — Diagnosing latency issues — the fork, fsync, and huge-page behaviour the kernel tuning is for.
- Valkey — Persistence — the appendonly everysec posture and the roughly one-second loss bound after a crash.
- Redis — Administration — the host-level checklist: THP, overcommit, somaxconn, and file-descriptor limits.
- Redis — Documentation — the persistence, memory, and connection topics the kernel tuning case study draws on.
Related in this course
- R8.01.1 · The fork’s two settingsTHP off & overcommit
- R8.01.2 · The command thread, fastswap, lazy frees, defrag
- R8.01.3 · Durable on a crashAOF, noeviction, backlog
- R8 · Production & Operationsthe chapter
- /echomqthe EchoMQ protocol whose Valkey this config tunes
- /bcs · FlyCodemojex on Fly, where this valkey.conf runs