jonnify·redis /redis-patterns/caching/workshop/harden-and-measure

R1.07.3 · Caching workshop — stage 3

Harden and measure.

The one emoji set every guess touches draws a herd when its row expires. EchoStore absorbs it by construction — the single-flight coalesces a herd onto one fill, the jittered TTL stops a cohort from re-herding — and stats/1 reads the hit rate off the counters.

Grounding: echo_store/table.ex (flights, launch_flight, expires_at, stats/1) · bcs.4.md (the single-flight gate, Valkey).

A herd of misses on one id. Select an actor to read what the owner does with it.
the first misslaunches the flight the rest of the herdcoalesce · waiters the loaderruns once launch join the flight one flight, one loader run, every waiter gets the answer
readout · Select an actor. Concurrent misses on one id coalesce onto a single flight; the loader runs once and every waiter reads the one answer.

Static verdicts — first miss → launch_flight/2 · the rest → coalesce onto the flight, bump coalesced · loader → runs once for the whole herd.

§1

One fill per herd, by construction

echo_store/table.ex:265 · bcs.4

A stampede — the thundering herd — happens when a popular key expires and every concurrent reader misses in the same instant. Each miss would read the source, so the loader is hit once per reader at once. EchoStore closes that by construction, not by a lock. Misses route through the table’s owner, and concurrent misses on one id coalesce onto a single flight: the first caller’s miss launches launch_flight/2; a concurrent miss on the same id finds the flight already in the flights map and joins its waiter list (put_in(state.flights[id], {ref, [from | waiters]}), bumping the coalesced counter). The flight checks L2, falls through to the loader, writes both layers, and the owner replies to every waiter with the one answer.

The moduledoc states the law: “Concurrent misses on a key coalesce onto a single in-flight load … every waiter reads the one answer.” The committed gate proves it: two hundred concurrent cold readers, the loader runs once, one hundred ninety-nine waiters coalesced, every reader holding the one answer. There is no SET NX PX lock here and none is needed — the herd is absorbed by the owner’s single-flight bookkeeping, not by a distributed mutex.

the flights map
id -> {ref, waiters} in the owner’s state; the first miss launches a flight, the rest join.
launch_flight/2
The single flight: GET ecc:{cm_emojisets}:<id> → loader → SET … PX, then reply to all waiters.
coalesced
The counter bumped for every miss that joins an in-flight load instead of launching its own.
one answer
Enum.each(waiters, &GenServer.reply(&1, reply)) — the herd costs one loader run.
§2

No synchronized expiry — the jittered clock

echo_store/table.ex:484 · bcs.4

Rows filled together must not expire together, or the TTL itself schedules the next herd. expires_at/1 draws each row’s expiry from ttl ± ttl·jitter: base + :rand.uniform(2 * spread + 1) - spread - 1, where spread = trunc(ttl * jitter). A cohort filled in the same instant spreads its expiries across a band, so the re-fill is a trickle, not a herd. The committed gate shows it: four hundred rows filled in twenty-four milliseconds expire one hundred thirty-eight milliseconds apart at jitter 0.2 — no synchronized re-herd. The sweeper then reclaims dead rows on a fixed tick, so memory is bounded by the declaration, not by luck.

Set the jitter to the herd a surface fears: 0.1–0.2 spreads a cohort across a fifth of its TTL, enough to turn a synchronized re-fill into a trickle. This is the honest mechanism EchoStore implements — expiry jitter plus single-flight — not a probabilistic early-expiration scheme; the early-refresh variant of stampede prevention is the R1.05 module’s territory, and EchoStore’s real answer is the jittered clock.

Notes on ValkeyThe L2 row is written with SET … PX carrying the jittered expiry, and the engine runs its own on-access and active expiration as a second reclamation path — two layers, each keeping its own house — valkey.io/commands/expire.

The herd is absorbed twice over: the single-flight coalesces a herd that has already formed, and the jittered TTL stops a cohort from forming one on expiry. Neither is a lock; both are construction.

§3

Make the hit rate a number

echo_store/table.ex:179 · stats/1

A cache you cannot measure is a cache you cannot tune. EchoStore.Table.stats/1 returns the counter snapshot plus the live size: hits, misses, fills, l2_hits, coalesced, swept, full_skips, sweeps, coh_applied, coh_stale, and size. Every counter is a :counters slot bumped on the hot path — atomic, lock-free, never a GenServer.call, which is why a hit stays 762 ns. The hit rate is a division: hits / (hits + misses) — the share of reads served without touching the loader.

Run a read load below and watch the counters fill. The readout reports the hits and misses tallies, the hit rate they compute, and the coalesced count under a herd. A hot emoji set kept warm drives the rate up; a short TTL that expires the row mid-load drives it down with extra misses; a herd shows up as coalesced waiters that cost no extra loader run.

Run a read load. The :counters fill; the readout computes the hit rate.
12

The counters turn the read path into a measured hit rate — hits / (hits + misses) — read off the cache’s own :counters, with coalesced proving the herd cost one load.

§4

Hardened and measured on EchoStore

echo_store/table.ex · codemojex/game.ex

Take one read of the hottest emoji set under load: many guesses hit the most-played set as its row expires. The owner receives a herd of misses on one id, coalesces them onto a single flight, and replies to every waiter with the one answer — the loader ran once. Every read, hot or not, bumps hits or misses, so the cache reports its own hit rate through stats/1. Codemojex.Guesses.submit/3 receives the emoji set or {:error, :bad_guess} for a code the round’s keyboard does not expose; the single-flight and the counters live in the cache.

# Harden by construction + measure — the owner coalesces a herd, the counters report.
# Inside the owner, on a herd of misses for one id (echo_store/table.ex:265):
case Map.fetch(state.flights, id) do
  {:ok, {ref, waiters}} ->                 # a flight is in progress: coalesce onto it
    :counters.add(state.spec.counters, counter(:coalesced), 1)
    {:noreply, put_in(state.flights[id], {ref, [from | waiters]})}

  :error ->                                # the first miss: launch the one flight
    ref = launch_flight(state, id)
    {:noreply, put_in(state.flights[id], {ref, [from]})}
end

# The hit rate, off the cache's own counters:
%{hits: h, misses: m, coalesced: c, size: n} = EchoStore.Table.stats(:cm_emojisets)
# hit_rate = h / (h + m)

The single-flight confines a hot regeneration to one loader run; the jittered TTL stops the cohort from re-herding; the counters make the cache’s effect a measured number. How the owner monitors a flight and replies to a crashed one is /elixir, not repeated here. With this stage the catalog cache is cached, consistent, hardened, and measured — the workshop’s read path is complete.

the application · one fill per herd, gated

EchoStore.Table against live Valkey

The herd discipline is measured, not assumed — the committed EchoStore gate against the live engine:

bcs.4.md · the single-flight and jitter gate, against live Valkeyfrozen
200 concurrent cold readers   loader runs 1     ← coalesced waiters 199, one answer
400 rows at jitter 0.2        expire 138 ms apart  ← no synchronized re-herd
L1 ETS hit                    762 ns         ← counters are :counters, never a call
hit rate                      hits / (hits + misses)
the pattern

Prevent a stampede and measure the result: coalesce a herd onto one fill, spread the expiry so a cohort never re-herds, and count every read.

its EchoStore application

The hottest emoji set’s herd coalesces onto one launch_flight, the jittered TTL spreads the cohort, and stats/1 reports hits / (hits + misses) with coalesced proving the herd cost one load.

Harden by construction, measure every read, and the catalog cache is done: cached, consistent, stampede-safe, and reporting its own hit rate.

§5

Recap — the read path is complete

back → R1.07

The workshop’s four layers are in place. Stage 1 cached the catalog with EchoStore.Table.fetch/3; stage 2 kept it consistent with Coherence.broadcast/enqueue and newer?; this stage hardened the hot emoji set with the single-flight flights map and the jittered TTL, and measured the cache with EchoStore.Table.stats/1. The emoji-set cache now reads through EchoStore, tracks changes, survives a herd on its busiest set by construction, and reports its own hit rate — built entirely from patterns taught in R1.01–R1.06.

§6

References

sources · related

Sources

  • Valkey — SET — SET … PX writes the L2 row with its jittered expiry in one atomic command.
  • Valkey — EXPIRE — the server’s own reclamation, the L2-side counterpart of the sweeper that bounds memory.
  • Redis — INCR — the atomic-counter primitive behind a measured hit rate; EchoStore keeps its tallies in lock-free :counters.
  • Sanfilippo, S. — antirez weblog — the Redis creator on expiry, atomic operations, and single-instance discipline.

Related in this course