jonnify·redis /redis-patterns/time-delay-priority/backoff-retry

R4.04 · Backoff & retry

Retry is a delayed job.

A failed job does not retry immediately. Re-schedule it into the future with a delay that grows on each attempt — a delayed job whose fire-time is computed by a backoff formula — so a struggling dependency is not hammered.

The delayed-queue pattern (R4.01) applied to retries: exponential backoff, jitter to break the thundering herd, and reuse of the schedule set. Grounded in EchoMQ.Backoff.delay_ms/2 (the host-side formula owner) and EchoMQ.Jobs.retry/7, which ZADDs the job onto emq:{queue}:schedule and lets EchoMQ.Jobs.promote/3 sweep it back when due.

§1

A retry is a delayed job

delayed-queue.md.txt · Retry with Backoff

The delayed-queue source puts the move plainly: “If a task fails, reschedule it with a delay … Track retry counts in the task data to implement exponential backoff or maximum retry limits.” A retry is not a new mechanism. It is a re-schedule: read the attempt count, compute a delay, and ZADD the job back onto the schedule set at now + delay. The promotion sweep that already moves due delayed jobs back to pending moves the retry too.

The attempt count is the input the formula needs. In the real bus it is the attempts field on the job row — the same fencing token a claim mints with HINCRBY — so the retry path reads the attempt that has failed and prices the next delay from it. One structure carries both the first-time delayed job and every retry.

attempts
The fencing token on the job row, incremented on each claim; the attempt that has failed.
delay_ms
The wait before the next attempt, computed host-side from (policy, attempts).
schedule set
The sorted set the retry re-enters, scored by its future fire-time — R4.01's machinery.
promote
The due-range sweep that brings a retry (and any delayed job) back to pending.

A retry adds no key and no script of its own. The only new piece is where the score comes from: a backoff formula instead of a caller-supplied time.

§2

The backoff formula (exponential)

EchoMQ.Backoff.delay_ms/2 · :exponential

Exponential backoff doubles the wait each attempt: delay = base × 2^(attempts-1). Attempt 1 waits base; attempt 2 waits 2 × base; attempt 3 waits 4 × base. The curve climbs fast, so a dependency that is down gets longer breathing room — and a clamp (cap) holds the curve at a ceiling so a stubborn job does not schedule itself years out.

Choose a base and an attempt below to compute the delay. The readout names the literal value EchoMQ.Backoff.delay_ms/2 returns and whether the cap was reached.

The retry curve. Hover or select an attempt to read its delay; the bars chart base × 2^(attempts-1) clamped at the cap.
100
10000
a1 a2 a3 a4 a5 delay
readout · Select an attempt to compute its delay. Each step doubles the wait, until the cap holds the curve flat.

Static verdicts — delay_ms({:exponential, 100, 10_000}, 1) = 100 · {:exponential, 100, 10_000}, 3 = 400 · {:exponential, 100, 10_000}, 20 = 10000 (the cap).

# EchoMQ.Backoff.delay_ms/2 — the :exponential clause (verbatim)
def delay_ms({:exponential, base, cap}, attempts) do
  raw = base * Bitwise.bsl(1, attempts - 1)   # base × 2^(attempts-1)
  min(raw, cap)                                # clamped at the ceiling
end

The formula lives above the wire, in a pure function. A consumer can compute, table, and test its retry cadence without a server — the wire takes the literal delay it is handed.

§3

Jitter — breaking the thundering herd

EchoMQ.Backoff.delay_ms/2 · :jitter

When many jobs fail at the same moment — a dependency blinks — a fixed backoff schedules every retry for the same future instant. They re-fire together: the thundering herd, a second spike on a service that is already struggling. Jitter spreads them. A jittered delay is a random value bounded by the backoff delay, so a cohort that failed together re-fires across a window instead of in one spike.

In the real bus jitter is a policy wrapper: {:jitter, inner}. It computes the inner policy's delay, then returns a uniform random value in 0..inner_delay — the full-jitter form. The randomness is the point; the inner delay is its bound. The second dive runs the herd both ways.

Notes on ValkeyA retry is one ZADD per job onto a sorted set scored by its fire-time; a jittered cohort writes scattered scores, so a single ZRANGEBYSCORE sweep releases the cohort across a window rather than in one pull — valkey.io/topics/data-types.

§4

The dives

three lessons

Three dives: the formula and its clamp, the herd and the jitter that breaks it, and the schedule-set reuse that adds no new structure.

built ready to read
§5

Backoff and retry on EchoMQ

echo/apps/echo_mq · jobs.ex

A retry adds no key and no script of its own. EchoMQ.Jobs.retry/7 is handed the literal delay_ms from EchoMQ.Backoff, reads the row's attempts, and — under the max-attempts ceiling — ZADDs the job onto the schedule set at now + delay_ms on the server clock. EchoMQ.Jobs.promote/3 then ZRANGEBYSCOREs the due range and moves those jobs back to pending. That is exactly the delayed-queue machinery from R4.01: the schedule set is the sorted set, the score is the fire-time, the promotion is the sweep.

A retry curve cannot climb forever. retry/7 carries a max_attempts argument; when the row's attempts has reached it, the job is not re-scheduled — its state is set to dead and it lands on the morgue set instead. A retry is bounded by both a ceiling on the delay (the cap) and a ceiling on the count (max attempts).

-- @retry (jobs.ex) — the non-terminal arm, verbatim: reschedule onto the schedule set
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('HSET', KEYS[4], 'state', 'scheduled')
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[3]), ARGV[1])   -- KEYS[2]=schedule, ARGV[3]=delay_ms
return 'scheduled'
the pattern

A delayed queue schedules a job for a future time. A retry is the same move with the future time computed by a backoff formula and bounded by a max-attempt count, so a failing job re-fires later and later, then gives up.

its EchoMQ application

EchoMQ.Backoff.delay_ms/2 sets the delay; EchoMQ.Jobs.retry/7 ZADDs the job onto emq:{queue}:schedule at now + delay_ms; EchoMQ.Jobs.promote/3 (R4.01's sweep) brings it back when due. One structure, two uses.

The pattern this module teaches is a property of the sorted set — the score, the backoff math, the sweep — not of any one key name. That is why the move carries across the queue, the scheduler, and the leaderboard the rest of this chapter builds.

§6

References

sources · related

Sources

Related in this course