jonnify·redis /redis-patterns/time-delay-priority/delayed-queue/zrangebyscore-promotion

Redis Patterns/R4/The delayed queue/ZRANGEBYSCORE promotion

R4.01.2 · The delayed queue — dive 2

ZRANGEBYSCORE promotion.

Range the schedule set from -inf to the current server clock, ZREM each due member, and ZADD it onto the pending set. EchoMQ runs all three in one inline script.

Grounding: echo/apps/echo_mq/lib/echo_mq/jobs.ex · the @promote script · EchoMQ.Jobs.promote/3 — the real due-range sweep, on the server clock.

The sweep — range, claim, move. Select a step to light its path and read the command it runs against the schedule set.
schedule setscored by run-at @promoteone EVAL pending setscored 0 ZRANGEBYSCORE ZREM ZADD pending
readout · Select a step. The sweep ranges the due head, removes each id from the schedule set, and adds it onto the pending set — all in one atomic @promote script.

Static verdicts — range → ZRANGEBYSCORE schedule -inf now LIMIT 0 <batch> · claim → ZREM schedule <id> · move → ZADD pending 0 <id>, HSET … state pending.

§1

The due-range read

delayed-queue.md.txt · Polling

The read is a range query bounded above by the current clock. Every member from the start of the set up to now is due:

ZRANGEBYSCORE delayed_queue -inf <now> LIMIT 0 10   # the due head, up to ten

-inf is “no floor” — start from the lowest score. The upper bound is the current time. LIMIT 0 10 caps the batch to ten so one sweep claims a bounded slice and a backlog drains over several sweeps rather than one giant move. The result is the due head: the members whose run-at time has passed, earliest first. The range is a read only — it names the due jobs; it does not remove them.

-inf
The lower bound: start from the lowest score in the set.
now
The upper bound: the current clock, the cutoff for “due.”
LIMIT 0 N
Cap the batch at N so a backlog drains over several sweeps.

The due-range read names the head: ZRANGEBYSCORE -inf now, capped, returns the jobs whose run-at has passed — a read, not a claim.

§2

The atomic claim

delayed-queue.md.txt · Claiming

Each due member is claimed with a ZREM. The removal is the claim:

ZREM delayed_queue "task:abc123"   # 1 if this caller promoted it, 0 if already gone

ZREM returns 1 when this call removed the member — the caller owns the task — or 0 when another worker already removed it. With many workers sweeping the same set, the range may hand the same member to several of them, but only one ZREM returns 1; the rest get 0 and drop it. The removal establishes the owner, atomically, with no extra lock. Step the clock below to watch the sweep claim the due head off a fixed set of six jobs.

Step the clock — sweep the due head. Move the clock; the readout names the promoted ids and the set shrinks.
0
emq:{queue}:schedule — six jobs by run-at (ms) now

Static verdicts — a job whose run-at is at or below the clock is swept (ZREM from schedule, ZADD onto pending, state pending); the rest stay on the schedule set.

The ZREM is the claim: a 1 means this sweep promoted the job, a 0 means another already did — the removal is the only mark of ownership.

§3

Folding it into one script

delayed-queue.md.txt · Combining

The range and the ZREM are two commands, and between them a second worker can interleave. To make poll-and-claim a single atomic step, run both inside a Lua script:

local tasks = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, ARGV[2])
for i, task in ipairs(tasks) do
    redis.call('ZREM', KEYS[1], task)
end
return tasks

The script reads the due head and removes every member in one round trip; no other worker can claim a job between the range and the ZREM. This is the shape EchoMQ’s promote takes — a range, a ZREM per due id, and a move to the pending set, all inside one script, on the server clock.

A Lua script folds the range and the ZREM into one atomic poll-and-claim, so no worker interleaves between reading the due head and removing it.

§4

EchoMQ’s promote sweep

echo_mq/jobs.ex · @promote

In EchoMQ the sweep is the @promote script in echo/apps/echo_mq/lib/echo_mq/jobs.ex, driven by EchoMQ.Jobs.promote/3. It reads the due range against the server’s own clock, then moves each due id from the schedule set onto the pending set, all in one atomic EVAL:

-- @promote (jobs.ex) — the due-sweep on the server clock (real)
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)             # the server clock, in ms
local due = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, tonumber(ARGV[2]))
for _, id in ipairs(due) do
  redis.call('ZREM', KEYS[1], id)                          # claim: remove from schedule
  redis.call('ZADD', KEYS[2], 0, id)                        # pending, score 0 (mint-ordered)
  redis.call('HSET', p .. 'job:' .. id, 'state', 'pending')
end
return #due

KEYS[1] is the schedule set, KEYS[2] the pending set — both declared, both braced emq:{<queue>}: keys on one slot. The upper bound of the range is the server’s now, not a value the caller passed: the sweep promotes exactly the jobs whose run-at score is at or below the server clock the schedule was priced against. The LIMIT 0 <batch> caps each sweep — EchoMQ.Jobs.promote/3 passes the batch size — so a large schedule drains over several beats. Each promoted job lands on the pending set at score 0, where its branded JOB id is the sort key, and its row moves to state = pending. The script returns the count it promoted.

EchoMQ.Metronome calls promote/3 once per beat per queue, so promotion runs on the bus’s own pump rather than from each worker — a due job is released by the metronome, not by a separate poller per consumer.

Notes on ValkeyZRANGEBYSCORE with a LIMIT offset count reads a bounded slice of the score range, so one sweep claims at most a batch and a backlog drains over several — redis.io/commands/zrangebyscore.

the bus · the promote pump, in depth

The promote sweep and the metronome

This dive cites the @promote body and EchoMQ.Jobs.promote/3 as proof the due-sweep ships. The pump that beats it — the metronome, the consumer pool, the lifecycle controls — is the dedicated EchoMQ Queue course. The next dive in this module reads the metronome wake that triggers this sweep; the Queue pillar teaches the whole pump.

the pattern

Sweep the delayed set by reading the due head with a range query and claiming each member with an atomic ZREM, then move it to the queue workers fetch from.

its EchoMQ application

EchoMQ.Jobs.promote/3 runs @promote — ZRANGEBYSCORE schedule -inf now LIMIT 0 <batch> on the server clock, ZREM each due id, ZADD pending 0 id, HSET state pending — in one atomic EVAL, and the metronome drives it once per beat per queue.

The due-sweep is a range to read the head and a ZREM to claim it; EchoMQ runs it as one inline @promote script on the server clock, so a due job moves from the schedule set onto the pending set atomically, on the bus’s own pump.

§5

References

sources · related

Sources

  • Redis — ZRANGEBYSCORE — read the due head: every member from the start of the set up to the current clock.
  • Redis — ZREM — the atomic claim: 1 means this caller promoted the job, 0 means another already did.
  • Valkey — TIME — the server clock the sweep bounds the due range by, the same clock the schedule was priced against.
  • Redis — Sorted sets — the data type the sweep ranges over, ordered by run-at score.

Related in this course