R1.07.3 · Caching workshop — stage 3
Harden and measure.
The one emoji set every guess touches draws a herd when its row expires. EchoStore absorbs it
by construction — the single-flight coalesces a herd onto one fill, the jittered TTL stops a
cohort from re-herding — and stats/1 reads the hit rate off the counters.
Grounding: echo_store/table.ex (flights, launch_flight, expires_at, stats/1) · bcs.4.md (the single-flight gate, Valkey).
Static verdicts — first miss → launch_flight/2 · the rest
→ coalesce onto the flight, bump coalesced · loader → runs once for the whole
herd.
One fill per herd, by construction
echo_store/table.ex:265 · bcs.4A stampede — the thundering herd — happens when a popular key expires and every concurrent
reader misses in the same instant. Each miss would read the source, so the loader is hit once per reader at
once. EchoStore closes that by construction, not by a lock. Misses route through the table’s owner, and
concurrent misses on one id coalesce onto a single flight: the first caller’s miss
launches launch_flight/2; a concurrent miss on the same id finds the flight already in the
flights map and joins its waiter list
(put_in(state.flights[id], {ref, [from | waiters]}), bumping the coalesced
counter). The flight checks L2, falls through to the loader, writes both layers, and the owner replies to
every waiter with the one answer.
The moduledoc states the law: “Concurrent misses on a key coalesce onto a single in-flight load
… every waiter reads the one answer.” The committed gate proves it: two hundred concurrent
cold readers, the loader runs once, one hundred ninety-nine waiters coalesced, every reader holding the one
answer. There is no SET NX PX lock here and none is needed — the herd is absorbed by the
owner’s single-flight bookkeeping, not by a distributed mutex.
- the flights map
id -> {ref, waiters}in the owner’s state; the first miss launches a flight, the rest join.- launch_flight/2
- The single flight:
GET ecc:{cm_emojisets}:<id>→ loader →SET … PX, then reply to all waiters. - coalesced
- The counter bumped for every miss that joins an in-flight load instead of launching its own.
- one answer
Enum.each(waiters, &GenServer.reply(&1, reply))— the herd costs one loader run.
No synchronized expiry — the jittered clock
echo_store/table.ex:484 · bcs.4Rows filled together must not expire together, or the TTL itself schedules the next herd.
expires_at/1 draws each row’s expiry from ttl ± ttl·jitter:
base + :rand.uniform(2 * spread + 1) - spread - 1, where
spread = trunc(ttl * jitter). A cohort filled in the same instant spreads its expiries across a
band, so the re-fill is a trickle, not a herd. The committed gate shows it: four hundred rows filled in
twenty-four milliseconds expire one hundred thirty-eight milliseconds apart at jitter 0.2 — no
synchronized re-herd. The sweeper then reclaims dead rows on a fixed tick, so memory is bounded by the
declaration, not by luck.
Set the jitter to the herd a surface fears: 0.1–0.2 spreads a cohort across a fifth of its TTL, enough to turn a synchronized re-fill into a trickle. This is the honest mechanism EchoStore implements — expiry jitter plus single-flight — not a probabilistic early-expiration scheme; the early-refresh variant of stampede prevention is the R1.05 module’s territory, and EchoStore’s real answer is the jittered clock.
Notes on ValkeyThe L2 row is written with SET … PX
carrying the jittered expiry, and the engine runs its own on-access and active expiration as a second
reclamation path — two layers, each keeping its own house —
valkey.io/commands/expire.
The herd is absorbed twice over: the single-flight coalesces a herd that has already formed, and the jittered TTL stops a cohort from forming one on expiry. Neither is a lock; both are construction.
Make the hit rate a number
echo_store/table.ex:179 · stats/1A cache you cannot measure is a cache you cannot tune. EchoStore.Table.stats/1 returns the
counter snapshot plus the live size: hits, misses, fills, l2_hits, coalesced, swept, full_skips,
sweeps, coh_applied, coh_stale, and size. Every counter is a :counters slot
bumped on the hot path — atomic, lock-free, never a GenServer.call, which is why a hit
stays 762 ns. The hit rate is a division: hits / (hits + misses) — the share
of reads served without touching the loader.
Run a read load below and watch the counters fill. The readout reports the hits and
misses tallies, the hit rate they compute, and the coalesced count under a herd. A
hot emoji set kept warm drives the rate up; a short TTL that expires the row mid-load drives it down with
extra misses; a herd shows up as coalesced waiters that cost no extra loader run.
:counters fill; the readout computes the
hit rate.The counters turn the read path into a measured hit rate —
hits / (hits + misses) — read off the cache’s own :counters, with
coalesced proving the herd cost one load.
Hardened and measured on EchoStore
echo_store/table.ex · codemojex/game.exTake one read of the hottest emoji set under load: many guesses hit the most-played set as its row
expires. The owner receives a herd of misses on one id, coalesces them onto a single flight, and replies to every
waiter with the one answer — the loader ran once. Every read, hot or not, bumps hits or
misses, so the cache reports its own hit rate through stats/1.
Codemojex.Guesses.submit/3 receives the emoji set or {:error, :bad_guess} for a code
the round’s keyboard does not expose; the single-flight and the counters live in the cache.
# Harden by construction + measure — the owner coalesces a herd, the counters report. # Inside the owner, on a herd of misses for one id (echo_store/table.ex:265): case Map.fetch(state.flights, id) do {:ok, {ref, waiters}} -> # a flight is in progress: coalesce onto it :counters.add(state.spec.counters, counter(:coalesced), 1) {:noreply, put_in(state.flights[id], {ref, [from | waiters]})} :error -> # the first miss: launch the one flight ref = launch_flight(state, id) {:noreply, put_in(state.flights[id], {ref, [from]})} end # The hit rate, off the cache's own counters: %{hits: h, misses: m, coalesced: c, size: n} = EchoStore.Table.stats(:cm_emojisets) # hit_rate = h / (h + m)
The single-flight confines a hot regeneration to one loader run; the jittered TTL stops the cohort from re-herding; the counters make the cache’s effect a measured number. How the owner monitors a flight and replies to a crashed one is /elixir, not repeated here. With this stage the catalog cache is cached, consistent, hardened, and measured — the workshop’s read path is complete.
EchoStore.Table against live Valkey
The herd discipline is measured, not assumed — the committed EchoStore gate against the live engine:
200 concurrent cold readers loader runs 1 ← coalesced waiters 199, one answer 400 rows at jitter 0.2 expire 138 ms apart ← no synchronized re-herd L1 ETS hit 762 ns ← counters are :counters, never a call hit rate hits / (hits + misses)
Prevent a stampede and measure the result: coalesce a herd onto one fill, spread the expiry so a cohort never re-herds, and count every read.
The hottest emoji set’s herd coalesces onto one launch_flight, the jittered TTL
spreads the cohort, and stats/1 reports hits / (hits + misses) with
coalesced proving the herd cost one load.
Harden by construction, measure every read, and the catalog cache is done: cached, consistent, stampede-safe, and reporting its own hit rate.
Recap — the read path is complete
back → R1.07The workshop’s four layers are in place. Stage 1 cached the catalog with
EchoStore.Table.fetch/3; stage 2 kept it consistent with Coherence.broadcast/enqueue
and newer?; this stage hardened the hot emoji set with the single-flight flights
map and the jittered TTL, and measured the cache with EchoStore.Table.stats/1. The emoji-set cache
now reads through EchoStore, tracks changes, survives a herd on its busiest set by construction, and
reports its own hit rate — built entirely from patterns taught in R1.01–R1.06.
The Caching workshop
The assembled read path: cache-aside, coherence, single-flight and jitter, and the counters — four layers, one cache.
/bcs · Part IVCache-aside at ETS speed
The EchoStore manuscript chapter the single-flight, the jitter, and the 762 ns figure come
from — declared and gated.
Cache stampede prevention
The herd, the lock-on-miss, and probabilistic early refresh in full — the pattern this stage applies.
References
sources · relatedSources
- Valkey — SET —
SET … PXwrites the L2 row with its jittered expiry in one atomic command. - Valkey — EXPIRE — the server’s own reclamation, the L2-side counterpart of the sweeper that bounds memory.
- Redis — INCR — the atomic-counter primitive behind a measured hit rate; EchoStore keeps its tallies in lock-free
:counters. - Sanfilippo, S. — antirez weblog — the Redis creator on expiry, atomic operations, and single-instance discipline.
Related in this course
- R1.07 · Caching workshopthe workshop hub and the assembled read path
- R1.07.2 · Keep it consistentthe previous stage
- R1.05 · Cache stampede preventionthe herd, the lock, early refresh
- R1.01 · Cache-asidethe base pattern
- R1 · Cachingthe chapter
- /bcs · cache-asidethe EchoStore manuscript: single-flight, jitter, the gate