R8.05.3 · Uber: resilience & staggered sharding — dive 3

Graceful degradation: a worse answer beats no answer.

A cache exists to be faster than the store behind it. The resilience question is what happens when the cache is not there. Uber’s answer is graceful degradation: a miss or short-circuit falls through to Docstore — slower, but served.

Grounding: the Uber case study (How Uber Uses Integrated Redis Cache) for the degradation path and the results · the as-built echo/apps/echo_wire/lib/echo_mq/connector.ex for the applied half. This dive frames failover through Uber’s lens and cross-links R8.02. Valkey is the BCS engine; Uber’s Redis and Docstore are the case study.

Follow a read as the cache degrades. Select a condition to trace the request’s path — every degraded path still ends at a served value, on the durable store or a remote region.

readout · Select a condition. A hit returns from memory; a miss or a short-circuited request falls through to Docstore (slower but served); a downed region routes the read to a remote region. Degraded, not failed.

Static facts — hit → fast, from memory · miss / short-circuit → fall through to Docstore · region down → failover to the remote region — every path ends at a served value.

§1

Degrade, do not fail

ByteByteGo · fall through to Docstore

Graceful degradation is the discipline of having a worse-but-working answer ready for when the best answer is unavailable. For the integrated cache the best answer is a Redis hit; the worse-but-working answer is a Docstore read. A request that misses the cache, or that the circuit breaker short-circuits, or that hits a node which is down, does not error back to the caller — it falls through to Docstore. The read is slower than a cache hit because it pays the full database cost, but it returns a correct value. The cache is an optimisation in front of the truth; when the optimisation is gone, the truth is still reachable.

This is why the circuit breaker (the previous dive) is safe to be aggressive. Short-circuiting a request only helps if there is somewhere for that request to go; the fall-through to Docstore is that somewhere. Tripping the breaker does not drop the read — it reroutes it to the durable path. The two techniques compose: the breaker rules out Redis, and graceful degradation routes the read to Docstore.

A miss, a short-circuit, a dead node — each falls through to the durable store rather than erroring. The cache is the fast path; the database is the answer that is always there.

§2

The results, and remote-region failover

ByteByteGo · the measured effect

The combined effect of the integrated cache with its resilience layer was measured. Latency fell sharply at the tail as well as the median: P75 latency down about 75%, P99.9 latency down over 67%, with latency spikes limited rather than allowed to run away. The tail matters most here — P99.9 is where the dead-node timeouts and the hot-shard pile-ups would have shown up, and that is exactly where the circuit breaker and the staggered sharding hold the line.

Degradation also scales up a level, from a node to a region. A documented use case running at over 6 million reads per second with a 99% cache hit rate had its primary region run into trouble and failed over successfully to a remote region. The cache being unavailable in one region degraded to the remote path rather than taking the read down — the same fall-through principle, applied to a whole region’s worth of cache instead of a single node. The title figure for the whole integrated cache is 40 million reads per second; the resilience story is what let that number stay up under trouble.

P75: Down about 75% — the median read got much faster with the cache in front.
P99.9: Down over 67% — the tail, where dead-node timeouts hide, held the line.
region failover: A use case at over 6M reads/s failed over to a remote region rather than dropping the read.
aggregate: 40M reads/second served from the integrated cache, kept up under trouble.

The win is at the tail and at the region boundary — P99.9 down over 67%, and a region failure absorbed by failing over rather than failing the read.

§3

The BCS echo — recover by reconnecting, fail honestly

echo_wire/connector.ex · backoff + fail-not-replay

The EchoMQ connector’s graceful behaviour under a lost dependency is two properties: it fails the in-flight work honestly, and it recovers by reconnecting with backoff. Both are in the as-built connector.

Fail-not-replay. When the socket to Valkey drops, every in-flight caller is failed :disconnected and the in-flight FIFO is cleared. The connector does not silently retry those calls, because it cannot know which are idempotent — replaying a non-idempotent command after a partial failure could double-apply it. A fast, honest error is the safe degradation: the caller is told at once that the call did not complete, and the retry decision is left to the calling code.

echo/apps/echo_wire/lib/echo_mq/connector.ex · in-flight callers failed :disconnected, never replayedfrozen

Enum.each(:queue.to_list(s.pending), fn
  {:internal, _, _, _, _} -> :ok
  {from, _, _, _, _} -> GenServer.reply(from, {:error, :disconnected})
end)

Capped jittered backoff reconnect. After the socket drops, the connector schedules a reconnect, and each failed attempt backs off — doubling the delay up to a cap, with jitter so a fleet of connectors does not retry in lockstep:

echo/apps/echo_wire/lib/echo_mq/connector.ex · @backoff_min 100, @backoff_max 2_000frozen

defp schedule(%{sock: nil} = s) do
  jitter = :rand.uniform(div(s.backoff, 2) + 1)
  Process.send_after(self(), :reconnect, s.backoff + jitter)
  %{s | backoff: min(s.backoff * 2, s.backoff_max)}
end

Backoff is the recovery analogue of degradation: rather than spin on a node that is down (which would amplify the failure, the very thing the circuit breaker prevents), the connector waits a little longer between attempts, up to the @backoff_max of 2_000 ms, and re-fences on every reconnect. Its stats/1 reports status: :reconnecting and a reconnects count, so the recovery is observable.

Step the backoff. Each failed reconnect doubles the base delay up to the cap — select an attempt to compute its base delay from the real min(backoff * 2, 2000).

readout · Select an attempt. The base delay starts at @backoff_min = 100 ms and doubles each failed attempt, capped at @backoff_max = 2000 ms; a jitter of up to half the base is added on top.

Static facts — base doubles 100 → 200 → 400 … capped at 2000 · jitter :rand.uniform(div(backoff, 2) + 1) · the wait spreads a fleet’s retries so they do not reconnect in lockstep.

There is one guard. The connector has no special -READONLY handler. A Valkey primary failover surfaces to a client as a -READONLY error from a not-yet-promoted replica, but the connector does not branch on that string — its recovery to a topology change is the general one: the socket drops, the supervised connector reconnects. So -READONLY is taught here as the Valkey failover signal the connector recovers from by reconnecting, not as an echo surface. R8.02 (persistence, pooling & failover) taught the READONLY-reconnect failover and pool sizing in full; this dive frames the same recovery through Uber’s resilience lens rather than duplicating it.

door · the durability floor

/echo-persistence

A cache or a queue serves from memory; when the question is where does the value survive a crash, that is the durable substrate’s job. /echo-persistence is the dial from holding nothing to committing every record off-box, beneath the volatile tiers.

Notes on ValkeyWhen a Valkey primary fails over, a client that reconnects to a replica before it is promoted sees -READONLY You can’t write against a read only replica; once the replica is promoted the writes succeed. The owned wire has no -READONLY-specific branch — it recovers from the socket drop a failover causes by reconnecting with backoff, re-fencing each time. Valkey’s guidance on failover and the latency around it is at valkey.io/topics/latency.

the pattern

When the fast path is gone, serve the slow path (fall through to the durable store) and recover the fast path without hammering it — back off, retry, re-test.

its EchoMQ application

On socket loss the connector fails in-flight callers :disconnected (a fast honest error, never a silent replay) and reconnects with capped jittered backoff (@backoff_min 100 → @backoff_max 2_000), re-fencing on every reconnect.

Uber falls through to a durable store and fails over a region; the connector fails in-flight work honestly and reconnects with backoff. Both keep a worse-but-working path ready, and recover without amplifying the failure.

§4

Recap — a worse answer beats no answer

end of R8.05

Graceful degradation is the discipline of always having a worse-but-working path. Uber’s integrated cache falls through to Docstore on a miss or short-circuit, which is what makes the aggressive circuit breaker safe and what let a region failover keep serving — P75 down about 75%, P99.9 down over 67%, 40M reads/second held up under trouble. The EchoMQ connector reaches the same posture by failing in-flight work honestly (:disconnected, never replayed) and recovering with capped jittered backoff, treating a -READONLY failover as a socket drop to reconnect through. With this, the Uber module closes; return to the Production & Operations chapter to continue.

§5

References

sources · related

Sources

Uber Engineering — How Uber Serves Over 40 Million Reads Per Second Using an Integrated Cache — the primary engineering account of CacheFront: sharding Redis by partition key (distinct from the Docstore sharding) to avoid a hot shard, the sliding-window circuit breaker, and the 40M reads/second figures.
ByteByteGo — How Uber Uses Integrated Redis Cache to Serve 40M Reads/Second — graceful degradation falling through to Docstore, the P75 −75% / P99.9 −67% latency results, and the remote-region failover at over 6M reads/second.
Valkey — Diagnosing latency issues — failover and reconnect behaviour, and the -READONLY signal a client recovers from by reconnecting.
Microsoft Learn — Circuit Breaker pattern — the half-open probe that re-tests a recovering dependency, the recovery analogue of backoff-and-retry.

Related in this course

R8.05 · Uber: resilience & staggered shardingthe module hub
R8.05.2 · Circuit breakersthe previous dive — stop hammering a sick node
R8.02 · Persistence, pooling & failoverthe READONLY-reconnect failover and pool sizing in full
R8 · Production & Operationsthe chapter
/echo-persistencethe durability floor beneath the volatile tiers
/echomq/queuethe Queue pillar — the connector’s reconnect and fail-not-replay