R4.04 · Backoff & retry
Retry is a delayed job.
A failed job does not retry immediately. Re-schedule it into the future with a delay that grows on each attempt — a delayed job whose fire-time is computed by a backoff formula — so a struggling dependency is not hammered.
The delayed-queue pattern (R4.01) applied to retries: exponential backoff, jitter to break the
thundering herd, and reuse of the schedule set. Grounded in EchoMQ.Backoff.delay_ms/2 (the host-side
formula owner) and EchoMQ.Jobs.retry/7, which ZADDs the job onto emq:{queue}:schedule and lets
EchoMQ.Jobs.promote/3 sweep it back when due.
A retry is a delayed job
delayed-queue.md.txt · Retry with BackoffThe delayed-queue source puts the move plainly: “If a task fails, reschedule it with a delay …
Track retry counts in the task data to implement exponential backoff or maximum retry limits.” A retry
is not a new mechanism. It is a re-schedule: read the attempt count, compute a delay, and ZADD the job back onto
the schedule set at now + delay. The promotion sweep that already moves due delayed jobs back to
pending moves the retry too.
The attempt count is the input the formula needs. In the real bus it is the attempts field on the
job row — the same fencing token a claim mints with HINCRBY — so the retry path reads the
attempt that has failed and prices the next delay from it. One structure carries both the first-time delayed job
and every retry.
- attempts
- The fencing token on the job row, incremented on each claim; the attempt that has failed.
- delay_ms
- The wait before the next attempt, computed host-side from
(policy, attempts). - schedule set
- The sorted set the retry re-enters, scored by its future fire-time — R4.01's machinery.
- promote
- The due-range sweep that brings a retry (and any delayed job) back to pending.
A retry adds no key and no script of its own. The only new piece is where the score comes from: a backoff formula instead of a caller-supplied time.
The backoff formula (exponential)
EchoMQ.Backoff.delay_ms/2 · :exponentialExponential backoff doubles the wait each attempt: delay = base × 2^(attempts-1). Attempt 1
waits base; attempt 2 waits 2 × base; attempt 3 waits 4 × base.
The curve climbs fast, so a dependency that is down gets longer breathing room — and a clamp
(cap) holds the curve at a ceiling so a stubborn job does not schedule itself years out.
Choose a base and an attempt below to compute the delay. The readout names the literal value
EchoMQ.Backoff.delay_ms/2 returns and whether the cap was reached.
base × 2^(attempts-1) clamped at the cap.Static verdicts — delay_ms({:exponential, 100, 10_000}, 1) = 100 ·
{:exponential, 100, 10_000}, 3 = 400 · {:exponential, 100, 10_000}, 20 = 10000 (the
cap).
# EchoMQ.Backoff.delay_ms/2 — the :exponential clause (verbatim) def delay_ms({:exponential, base, cap}, attempts) do raw = base * Bitwise.bsl(1, attempts - 1) # base × 2^(attempts-1) min(raw, cap) # clamped at the ceiling end
The formula lives above the wire, in a pure function. A consumer can compute, table, and test its retry cadence without a server — the wire takes the literal delay it is handed.
Jitter — breaking the thundering herd
EchoMQ.Backoff.delay_ms/2 · :jitterWhen many jobs fail at the same moment — a dependency blinks — a fixed backoff schedules every retry for the same future instant. They re-fire together: the thundering herd, a second spike on a service that is already struggling. Jitter spreads them. A jittered delay is a random value bounded by the backoff delay, so a cohort that failed together re-fires across a window instead of in one spike.
In the real bus jitter is a policy wrapper: {:jitter, inner}. It computes the inner policy's delay,
then returns a uniform random value in 0..inner_delay — the full-jitter form. The randomness is
the point; the inner delay is its bound. The second dive runs the herd both ways.
Notes on ValkeyA retry is one ZADD per job onto a sorted set
scored by its fire-time; a jittered cohort writes scattered scores, so a single
ZRANGEBYSCORE sweep releases the cohort
across a window rather than in one pull —
valkey.io/topics/data-types.
The dives
three lessonsThree dives: the formula and its clamp, the herd and the jitter that breaks it, and the schedule-set reuse that adds no new structure.
Backoff and retry on EchoMQ
echo/apps/echo_mq · jobs.exA retry adds no key and no script of its own. EchoMQ.Jobs.retry/7 is handed the literal
delay_ms from EchoMQ.Backoff, reads the row's attempts, and — under
the max-attempts ceiling — ZADDs the job onto the schedule set at now + delay_ms on the server
clock. EchoMQ.Jobs.promote/3 then ZRANGEBYSCOREs the due range and moves those jobs back to pending.
That is exactly the delayed-queue machinery from R4.01: the schedule set is the sorted set, the score is the
fire-time, the promotion is the sweep.
A retry curve cannot climb forever. retry/7 carries a max_attempts argument; when the
row's attempts has reached it, the job is not re-scheduled — its state is set to
dead and it lands on the morgue set instead. A retry is bounded by both a ceiling on the delay (the
cap) and a ceiling on the count (max attempts).
-- @retry (jobs.ex) — the non-terminal arm, verbatim: reschedule onto the schedule set local t = redis.call('TIME') local now = t[1] * 1000 + math.floor(t[2] / 1000) redis.call('HSET', KEYS[4], 'state', 'scheduled') redis.call('ZADD', KEYS[2], now + tonumber(ARGV[3]), ARGV[1]) -- KEYS[2]=schedule, ARGV[3]=delay_ms return 'scheduled'
A delayed queue schedules a job for a future time. A retry is the same move with the future time computed by a backoff formula and bounded by a max-attempt count, so a failing job re-fires later and later, then gives up.
EchoMQ.Backoff.delay_ms/2 sets the delay; EchoMQ.Jobs.retry/7 ZADDs the job onto
emq:{queue}:schedule at now + delay_ms; EchoMQ.Jobs.promote/3 (R4.01's
sweep) brings it back when due. One structure, two uses.
The schedule set, in depth
The retry path, the claim that mints the attempts fencing token, and the promotion sweep are the
emq:{q}: queue protocol — taught in full in the Queue pillar.
When a retry reaches the floor
A dead-lettered job and a long-lived schedule reach the durable substrate — the persistence floor and its durability dial, where archived state is folded into the page engine and out to remote storage.
The pattern this module teaches is a property of the sorted set — the score, the backoff math, the sweep — not of any one key name. That is why the move carries across the queue, the scheduler, and the leaderboard the rest of this chapter builds.
References
sources · relatedSources
- Redis — ZADD — re-add the failed job to the schedule set scored by the backoff fire-time.
- Redis — ZRANGEBYSCORE — the due-sweep (
promote/3) that brings a retry back to pending. - Valkey — Data types — the sorted set the schedule set is, and the timer-wheel the retry reuses.
- Redis — Documentation — sorted sets, scoring, and scheduling in context.
Related in this course
- R4.04.1 · Exponential backoffthe doubling curve and its clamp
- R4.04.2 · Jitterbreaking the thundering herd
- R4.04.3 · Reusing the schedule setno new structure
- R4.01 · Delayed queuethe schedule-set machinery this reuses
- /echomq · the Queuethe schedule set and the retry path in depth