jonnify·redis /redis-patterns/production-operations/kernel-tuning

R8.01 · Production & Operations — module hub

Linux kernel tuning for Redis.

Configure Linux kernel parameters to prevent latency spikes, persistence failures, and connection drops in production Redis deployments — essential settings that override defaults hostile to in-memory databases.

Grounding: content/production/kernel-tuning.md.txt (the THP, overcommit, swappiness, and TCP figures) · infra/valkey/conf/valkey.conf (EchoMQ’s real Valkey posture, verbatim) · bcs.8.md B8.2 — the engine is Valkey 9; host sysctls and the config file are two halves of one posture.

One host, one command thread, and a fork that must stay cheap. Select a setting to light where it acts and read the failure it prevents.
production host · Valkey 9 command threadsingle, hot path helper threadsfsync · purge fork childAOF rewrite · COW THP overcommit swap defrag backlog fd limit
readout · Select a setting. The fork must stay cheap (THP off, overcommit on), the command thread must never swap or fragment (swappiness low, activedefrag), and a burst of clients must be accepted (TCP backlog, file-descriptor limit).

Static verdicts — THP off → no 500x copy on a fork · overcommit on → the fork is not refused · swappiness 1 → no nanoseconds-to-milliseconds page · TCP backlog → connections are not dropped · FD limit → a client is not refused · activedefrag → empty pages return to the OS.

§1

Transparent Huge Pages (THP)

kernel-tuning.md.txt · THP

THP is the most critical setting. When enabled, it causes severe latency spikes during persistence operations. The engine uses fork() to create a child process for an AOF rewrite, and Linux uses copy-on-write to share memory between parent and child. With THP enabled, the kernel hands out 2 MB pages instead of 4 KB pages. When the parent modifies a single byte during the rewrite window, the kernel must copy the entire 2 MB page — the source records this as a 500x amplification of the copy.

The symptoms the source lists: latency spikes of 10–100x during a background save, memory usage doubling temporarily during persistence, and client timeouts while the save runs. THP is a kernel setting, not a server one — turn it off at the host:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
# permanent, via systemd:
[Service]
ExecStartPre=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'

The server logs a warning at startup if THP is enabled. The source’s instruction is to never ignore it. The first dive takes the fork and copy-on-write apart.

§2

Memory Overcommit

kernel-tuning.md.txt · Overcommit

The fork() system call can fail if the kernel thinks there is insufficient memory. When the server forks, the kernel may pessimistically assume the child needs as much memory as the parent. The source’s example: on a 32 GB instance with 30 GB used, the fork can fail even though copy-on-write means little real memory is actually needed.

Overcommit is a kernel sysctl. Set it so the kernel always allows the allocation request, trusting that copy-on-write keeps the real cost small:

sysctl vm.overcommit_memory=1
# persist in /etc/sysctl.conf:
vm.overcommit_memory=1
§3

Swappiness

kernel-tuning.md.txt · Swappiness

Data in swap is a catastrophic failure mode. If any of the server’s memory pages are swapped to disk, access times go from nanoseconds to milliseconds — the source measures this as a 1,000,000x slowdown. A single swapped page can cause client timeouts.

Swappiness is a kernel sysctl. The source sets it to 1 — not 0, because 0 can cause OOM kills; 1 makes swapping extremely unlikely while still allowing the kernel to swap in a desperate situation:

sysctl vm.swappiness=1
# persist in /etc/sysctl.conf:
vm.swappiness=1

A swapped page trades the in-memory database’s entire reason for being. Keeping swappiness at 1 makes that trade extremely unlikely without inviting an OOM kill.

§4

TCP Stack Tuning

kernel-tuning.md.txt · TCP

Under high connection rates, default TCP settings cause connection failures. The listen queue for incoming connections defaults to 128. During traffic spikes, new connections are dropped before the server ever sees them. Two of these are kernel sysctls and one is a server setting, and they interact: the effective backlog is min(tcp-backlog, somaxconn). Raise both, then set the server-side backlog and the keepalive interval:

sysctl net.core.somaxconn=65536
sysctl net.ipv4.tcp_max_syn_backlog=65536
# in the server config (effective = min(tcp-backlog, somaxconn)):
tcp-backlog 65536
tcp-keepalive 300                  # keepalives every 300 s through a load balancer
§5

File Descriptor Limits

kernel-tuning.md.txt · FD limits

Each connection consumes a file descriptor. The source raises the limit for the service account so a burst of clients is accepted rather than refused. In the systemd service file:

[Service]
LimitNOFILE=65536

The server’s maxclients is bounded by the available file descriptors minus roughly 32 reserved for internal use — raise the limit and the ceiling rises with it.

§6

Memory Allocator

kernel-tuning.md.txt · jemalloc

The engine uses jemalloc, which is well-suited to its allocation pattern. The source names two behaviours to watch. After deleting a large amount of data, jemalloc retains memory to satisfy future allocations, so the process RSS reflects peak usage rather than the current dataset size. And fragmentation shows up in mem_fragmentation_ratio (read from INFO MEMORY):

  • below 1.0 — the process is using swap (critical);
  • 1.0–1.5 — normal;
  • above 1.5 — significant fragmentation.

When fragmentation is high, active defragmentation relocates live objects so empty pages can be returned to the OS. It is a server setting:

activedefrag yes
§7

On EchoMQ’s valkey.conf

infra/valkey/conf/valkey.conf · bcs.8.md B8.2

The host sysctls are only half the posture. The other half is the server’s own config file — and EchoMQ’s is real, committed, and applied. The kernel knobs (THP, overcommit, swappiness, somaxconn) live in the host; the application-level knobs live in infra/valkey/conf/valkey.conf. That file never carries a sysctl — it carries the threads, lazy-free, persistence, eviction, and defrag posture that the sysctls leave room for.

The manuscript states the kernel half from the inside: an in-memory store needs the kernel tuned away from its defaults — transparent huge pages off, overcommit set, swappiness low, and the connection backlog and file-descriptor limits raised so a burst of clients is accepted rather than dropped (bcs.8.md, B8.2).

the pattern

Change the host sysctls — THP off, overcommit on, swappiness low, somaxconn and file descriptors raised — so a fork stays cheap, a swapped page never appears, and a burst of clients is accepted; and set the in-memory database’s own config away from defaults hostile to a job store.

its EchoMQ application

valkey.conf sets io-threads 1, the full lazyfree-lazy-* family, appendonly yes with appendfsync everysec, maxmemory-policy noeviction, activedefrag yes, propagation-error-behavior panic, and tcp-backlog 511 — the application half the kernel half leaves room for.

the application · EchoMQ’s Valkey on Fly

The production posture, committed and applied

These are not aspirations. The file pins one command thread so the second shared vCPU feeds fsync, fork copy-on-write, and jemalloc purge; durability is a single append-only log fsynced every second (valkey.io/topics/persistence); a runaway keyspace rejects writes loudly rather than losing work; and the write fails rather than acknowledge data that may not survive. The persistence and threads lines, verbatim:

infra/valkey/conf/valkey.conf · the production posturefrozen
io-threads 1            ← one command thread; helpers take the second vCPU

appendonly yes
appendfsync everysec      ← ~1 s loss bound, fsync on a background thread
auto-aof-rewrite-percentage 100
aof-use-rdb-preamble yes
save ""                   ← no RDB; one fork source, not two
dir /data

propagation-error-behavior panic   ← fail the write rather than ack lost data

activedefrag yes         ← relocate live objects so empty pages purge

Notes on ValkeyA background save forks a child and relies on copy-on-write; the fork copy-on-write spike during an AOF rewrite is exactly what the kernel tuning is for, and is why the config keeps maxmemory well under the machine’s RAM — valkey.io/topics/latency.

Notes on ValkeyWith appendonly on and appendfsync everysec, the volume holds an append-only log and the loss bound after a crash is roughly one second of writes, not a whole snapshot interval — valkey.io/topics/persistence.

§8

The three dives

fork → command thread → durability

Each dive takes one cluster of the source’s settings, in the arc the fork → the command thread → durability: the two settings that keep a fork cheap and allowed, the ones that keep the single command thread fast, and the ones that keep work durable on a crash. Read them in order — the fork starts.

Three dives, one host: keep the fork cheap and allowed, keep the command thread fast, and keep the work durable — the host sysctls and the config file working as one posture.

After R8.01, the chapter continues with persistence and failover ops, then the case studies — Pinterest, Twitter/X, and Uber — before the capstone hands off to the dedicated EchoMQ course. Return to the Production & Operations chapter to continue, or open Codemojex on Fly for where this valkey.conf actually runs.

§9

References

sources · related

Sources

Related in this course