jonnify·redis /redis-patterns/production-operations/uber-resilience/circuit-breakers

R8.05.2 · Uber: resilience & staggered sharding — dive 2

Circuit breakers: stop hammering a sick node.

When a Redis node is down, every call to it pays a timeout before it fails — and at millions of requests per second that penalty compounds into a service-wide slowdown. Uber’s sliding-window circuit breaker counts errors per node, short-circuits a fraction once a threshold is hit, and trips until the window passes.

Grounding: the Uber case study (How Uber Uses Integrated Redis Cache) and the canonical Circuit Breaker pattern for the technique · the as-built echo/apps/echo_wire/lib/echo_mq/connector.ex for the applied half. Valkey is the BCS engine; Uber’s Redis is the case study.

Walk the breaker’s state as errors accumulate. Select an error level in the window to see the breaker close, short-circuit a fraction, or trip — the window’s sum decides.
sliding window — errors per node, per time bucket 0bucket t−3 0bucket t−2 0bucket t−1 0bucket t (now) sum over the window determines the state closedall pass short-circuita fraction trippednone reach node old buckets age out as the window slides — recovered failures drop from the sum
readout · Select an error level. The breaker sums the errors in the window; below the threshold it stays closed, at the threshold it short-circuits a fraction, and as errors climb it trips so no request reaches the node until the window passes.

Static facts — closed → all requests pass to the node · short-circuit → a fraction skip straight to the fallback, paying no timeout · tripped → none reach the node until the window slides and the counts age out.

§1

The cost of calling a dead node

ByteByteGo · the latency penalty

A cache in front of a database trades a fast in-memory hit for a slow fall-through on a miss. That trade is good while the cache is healthy. It inverts when a node is down: a request routed to a dead node does not return quickly with a miss — it waits for a connection or a read to time out, then fails. Every such request pays the full timeout before it can fall through to the database.

At millions of requests per second, this is the dangerous case. The slow timeouts pile up, threads and connections stay blocked waiting on a node that will not answer, and the latency penalty for one bad node bleeds into the whole service. The naive behaviour — keep trying the node, keep timing out — is precisely what amplifies a single node’s failure into a system-wide latency spike.

A dead node is more expensive than a slow one, because the cost is paid as a timeout per request — and at scale, that compounds into a service-wide spike.

§2

The sliding-window circuit breaker

ByteByteGo · Circuit Breaker pattern

The circuit breaker breaks that loop. It is the canonical resilience pattern: stop calling a dependency that is known to be failing, and fail fast instead of waiting for each call to time out. Uber’s is a sliding-window breaker, per node:

  1. Count errors per node, per time bucket. Each node has a window divided into buckets; failed calls in the current bucket increment its count.
  2. Sum over the window’s width. The breaker’s view of a node’s health is the sum of the buckets in the window — recent errors weigh, old errors age out as the window slides forward.
  3. Short-circuit a fraction once the threshold is hit. When the error count crosses the threshold, the breaker short-circuits a fraction of requests to that node: those requests skip the doomed call and go straight to the fallback, paying no timeout.
  4. Trip until the window passes. If errors keep accumulating, the breaker trips fully — no requests reach the node at all — and stays tripped until the window passes and the counts age out, at which point traffic is allowed back to test whether the node has recovered.

The sliding window is the key refinement over a naive counter. A flat lifetime error count would trip on old, recovered failures; the old buckets age out of the sum, so the windowed count tracks how the node is doing now. Short-circuiting a fraction before tripping fully is the second refinement — it sheds the worst of the load while still probing whether the node is coming back, rather than going hard open and blind.

closed
Errors below the threshold. All requests pass through to the node.
short-circuit
Threshold crossed. A fraction of requests skip the node and go to the fallback.
tripped
Errors climbing. No requests reach the node until the window slides past the failures.
the window
Old buckets age out of the sum, so recovered failures drop and the count tracks health now.

Count errors in a moving window, shed a fraction at the threshold, trip if it worsens, and let the window forget — a breaker that reflects the node’s health now, not its whole history.

§3

The BCS echo — backpressure, not a circuit breaker

echo_wire/connector.ex · max_pending

The EchoMQ connector reaches the same goal — do not pile work onto a struggling path — but by a different mechanism, and the difference is the whole point of this section.

The connector bounds its in-flight depth. Every pipeline waiting for a reply sits in a FIFO; the connector caps how many can be outstanding at once with max_pending (default 10_000). When the cap is reached, the next call is refused immediately with {:error, :overloaded} instead of being queued without bound:

echo/apps/echo_wire/lib/echo_mq/connector.ex · the bounded in-flight depthfrozen
# the cap is in-flight DEPTH, not an error rate over a window
def handle_call({:pipeline, _}, _from, %{pending_n: n, max_pending: max} = s) when n >= max do
  emit([:emq, :connector, :overload], %{pending: n}, %{label: s.label})
  {:reply, {:error, :overloaded}, %{s | overloads: s.overloads + 1}}
end

This is backpressure, a bounded queue — and it is not a circuit breaker. The distinction is sharp and worth keeping:

  • A circuit breaker watches a dependency’s error rate over a sliding window and stops calling it once the errors cross a threshold. Its trigger is failures over time.
  • max_pending backpressure watches the connector’s in-flight depth and refuses new work once too many requests are outstanding. Its trigger is concurrency right now — it does not count errors or track a window.

They share a goal — refuse to amplify a struggling path — but a depth limit and an error-rate trip are different instruments. The connector has two more analogues that, together with the bound, cover the circuit breaker’s goals without being one: fail-fast errors (a call against a dead socket returns {:error, :disconnected} at once rather than hanging) and supervised reconnect with capped jittered backoff (the connector backs off between reconnect attempts rather than spinning on a node that is down — the next dive). Same goals — fail fast, do not hammer a sick dependency, recover — different mechanisms.

Which mechanism fires? Select a trigger to see which guard reacts — the two watch different signals, so they fire on different conditions.
readout · Select a trigger. A circuit breaker reacts to errors over time; the connector’s max_pending bound reacts to in-flight depth; fail-fast reacts to a dead socket. Different signals, different guards.

Static facts — errors-over-a-window → a circuit breaker (Uber’s concept) · in-flight depth → max_pending → :overloaded (backpressure) · dead socket → :disconnected (fail-fast).

the application · the in-flight bound

codemojex on a saturated wire

A burst of guesses during a popular round flows through one EchoMQ.Connector; if Valkey slows and replies back up past max_pending, the next guess gets a fast :overloaded rather than swelling an unbounded queue that would make the slowdown worse. The bound sheds load at the edge so a saturated wire degrades instead of collapsing.

Notes on ValkeyA circuit breaker is a client-side pattern — Valkey does not implement one for you; the connector’s bounded in-flight depth and fail-fast errors are the client’s responsibility, layered over the engine. Valkey’s own guidance on the latency a slow or unreachable instance imposes — the cost the circuit breaker and the bound exist to cap — is at valkey.io/topics/latency.

the pattern

A sliding-window circuit breaker counts a node’s errors over time, short-circuits a fraction at the threshold, and trips until the window passes — failing fast against a known-bad dependency.

its EchoMQ application (same goal, a different mechanism)

max_pending caps in-flight depth and answers :overloaded — backpressure, not an error-rate trip. With fail-fast (:disconnected) and reconnect-backoff, the connector meets the breaker’s goals by different means.

Same goal, different instrument: Uber trips on an error rate over a window; the connector refuses on in-flight depth. Calling the bound a circuit breaker would blur a real distinction — backpressure is a depth limit, a breaker is an error-rate trip.

§4

Recap — same goal, different instrument

next → R8.05.3

Calling a dead node is expensive: each request pays a timeout, and at scale those timeouts amplify one node’s failure into a service-wide spike. Uber’s sliding-window circuit breaker counts errors per node, short-circuits a fraction once a threshold is hit, and trips until the window passes — failing fast against a known-bad dependency. The EchoMQ connector reaches the same goal with a different instrument: max_pending backpressure caps in-flight depth and answers :overloaded, fail-fast returns :disconnected against a dead socket, and supervised backoff keeps it from spinning on a down node. The next dive follows the request that gets short-circuited — graceful degradation, falling through to the durable store.

§5

References

sources · related

Sources

Related in this course