jonnify·redis /redis-patterns/time-delay-priority/backoff-retry/reuse-the-delayed-zset

R4.04.3 · Backoff & retry — dive 3

One sorted set, two uses.

A retry adds no new structure. It re-enters the same schedule set R4.01's delayed queue uses, scored by now + delay_ms, and the same promotion sweep brings it back when due.

Grounding: EchoMQ.Jobs.retry/7 ZADDs onto emq:{queue}:schedule; EchoMQ.Jobs.promote/3 ZRANGEBYSCOREs the due range — both verbatim from echo/apps/echo_mq/lib/echo_mq/jobs.ex.

Trace a retry's path. Hover or select a step to light the move it makes — failed, scheduled, swept, pending.
activefailed schedulenow + delay pendingdue ZADD promote
readout · Select a step. A retry ZADDs onto the schedule set; promote sweeps the due range back to pending.

Static verdicts — retry → one ZADD at score now + delay_ms · promote → one ZRANGEBYSCORE -inf now, then ZREM + ZADD pending.

§1

One sorted set, two callers

delayed-queue.md.txt · Retry with Backoff

The delayed queue (R4.01) schedules a job for a future time by ZADDing it onto a sorted set scored by its fire-time, then sweeping the due range back to pending. A retry is the same three moves with one difference: the score comes from a backoff formula instead of a caller-supplied time. Both write to the same key, emq:{<queue>}:schedule; both are swept by the same promotion.

The schedule set does not record which of its members is a first-time delayed job and which is a retry. The score is the only thing that matters — a member is due when the clock passes it, whatever put it there.

schedule set
The sorted set both the delayed queue and the retry write to, scored by fire-time.
retry/7
The reschedule: reads attempts, ZADDs at now + delay_ms under the max.
promote/3
The sweep: ZRANGEBYSCORE the due range, move each member back to pending.
dead
The terminal arm of the same script: at max attempts the job is set dead, not rescheduled.

A retry and a first-time delayed job are two callers of one structure. The sorted set carries both, keyed only by score.

§2

The reschedule

echo_mq/jobs.ex · @retry (the non-terminal arm)

EchoMQ.Jobs.retry/7 is token-fenced and reads the row's attempts before it does anything. Under the max-attempts ceiling, it reads the server clock, sets the row's state to scheduled, and ZADDs the job onto the schedule set at now + delay_ms — the literal delay the host computed with EchoMQ.Backoff:

-- @retry (jobs.ex) — the non-terminal arm, verbatim
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('HSET', KEYS[4], 'state', 'scheduled')
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[3]), ARGV[1])
return 'scheduled'

KEYS[2] is the schedule set, ARGV[3] is the literal delay_ms, ARGV[1] is the job id. The script reschedules at a delay it is given; it never computes a curve. At the max-attempts ceiling the other arm of the same script sets the row dead and ZADDs the job onto the morgue set instead — the retry gives up.

Notes on ValkeyThe reschedule is one ZADD with a score; the score is read from the server clock by TIME inside the script, so every member is timed on the same clock the sweep reads — valkey.io/commands/zadd.

The clock reaches the score. Advance the clock; the retry sits at now + delay until the clock passes it, then the sweep marks it due.
0
t=0 (failure) 400 1600 0

Static verdicts — clock 0 → 0 due (both scheduled) · clock 400 → the first member is due · clock 1600 → both due, the sweep promotes them.

A retry is a score on a sorted set. It waits until the clock passes it, then becomes due — nothing watches it in the meantime.

§3

The sweep

echo_mq/jobs.ex · @promote

The promotion that brings a retry back is the same one R4.01 teaches for delayed jobs. EchoMQ.Jobs.promote/3 reads the server clock and ZRANGEBYSCOREs the schedule set from -inf to now, then moves each due member back to pending:

-- @promote (jobs.ex) — the due-range sweep, the non-group path
local due = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, tonumber(ARGV[2]))
for _, id in ipairs(due) do
  redis.call('ZREM', KEYS[1], id)
  redis.call('ZADD', KEYS[2], 0, id)               -- a non-group job → the pending set
  redis.call('HSET', p .. 'job:' .. id, 'state', 'pending')
end
return #due

KEYS[1] is the schedule set, KEYS[2] is pending. A retry that was ZADDed at now + delay_ms falls into this range once the clock passes its score, and the sweep moves it back to pending exactly as it moves a first-time delayed job. No retry-specific path runs; the retry becomes due like any other member.

Reuse is the point, not a coincidence. The schedule set is a timer wheel keyed by score; anything with a future fire-time belongs on it. A delayed job's fire-time is a caller's clock; a retry's is a backoff curve; a scheduler's next run is an interval — all three are scores on the same sorted set, swept by the same promotion. A separate “retry set” would duplicate the sweep, the promotion, and the morgue handling for no gain.

the pattern

A retry is a delayed job whose score is a backoff delay; it rides the same sorted set, the same ZADD, and the same due-range sweep the delayed queue uses — no new structure.

its EchoMQ application

EchoMQ.Jobs.retry/7 ZADDs the job onto emq:{queue}:schedule at now + delay_ms; EchoMQ.Jobs.promote/3 ZRANGEBYSCOREs the due range and moves it back to pending — the same machinery R4.01 teaches.

One sorted set, scored by fire-time, is the whole delay machinery — and the retry is one more caller of it.

§4

Recap — no new structure

next → R4.04 hub

A retry re-enters the schedule set R4.01 built, scored by now + delay_ms, and the same promotion sweep brings it back when due. EchoMQ.Jobs.retry/7 ZADDs it; EchoMQ.Jobs.promote/3 ZRANGEBYSCOREs the due range. One sorted set carries the delayed job, the retry, and the scheduler — keyed only by score. That is the whole module: backoff is a value, and the structure it rides on already exists.

§5

References

sources · related

Sources

Related in this course