jonnify·redis /redis-patterns/time-delay-priority/the-sorted-set-as-a-clock

R4 · Time, Delay & Priority — orientation dive

The sorted set as a clock.

One data structure, read two ways. Score a job by a future millisecond and the sorted set is a timer wheel; flatten the score to a constant and the member's own mint order serves as the queue.

Grounding: echo/apps/echo_mq · EchoMQ.Jobs.promote/3 · Keyspace.queue_key — the real emq:{q}:schedule and emq:{q}:pending sets, every command verbatim from the as-built inline Lua.

The delayed-queue simulator — move the clock, watch the due head. Six jobs are scored by a fire-time offset from now = 0; the slider sets the current clock.
now = 0s
JOB·a+10s JOB·b+25s JOB·c+40s JOB·d+60s JOB·e+85s JOB·f+110s now
readout · At now = 0s no job is due. The next wake is JOB·a at +10s.

Pure verdict — the due set is every job whose fire-time is at or below the clock (the ZRANGEBYSCORE -inf now semantics); the next wake is the smallest fire-time strictly above it. No timer fires; the slider only recomputes the range.

§1

The textbook delayed queue

delayed-queue · sorted-sets

A sorted set stores members ordered by a numeric score. That is the whole structure: a member, a score, and the ordering the score implies. The classic delayed-queue pattern scores each task by the millisecond at which it should run, and stores it in a sorted set. Tasks order themselves by execution time, earliest first, with no extra bookkeeping.

  • Member — the task identifier.
  • Score — the millisecond at which the task should run.
ZADD schedule 1706649000000 "JOB0Nb1VTbfnu4"            # schedule five minutes out
ZRANGEBYSCORE schedule -inf <now> LIMIT 0 10           # read the due head; -inf = no floor
ZREM schedule "JOB0Nb1VTbfnu4"                       # remove a promoted member

A pump reads the due head with ZRANGEBYSCORE bounded above by the current clock, removes each member with ZREM, and promotes it to the run queue. Constant polling of an empty set wastes work, so a pump reads the head and sleeps until it is due, or a bounded maximum interval — which cuts overhead when the set is empty or the next task is far out. A failed task reschedules itself by re-adding with a future score, so a retry is a delayed task like any other.

Score a job by its fire-time, and a range query bounded by the current clock returns exactly the jobs whose time has come; reading the head gives the next wake, so a pump sleeps instead of spinning.

§2

The second reading: an order

pending set · mint-ordered id

The same set shape carries a second pattern. Leave the member as the job id, but arrange the score so the smallest score is always the next job to serve. The textbook does this by packing two numbers into one score — a priority tier in the high bits, an arrival counter in the low bits — so a ZPOPMIN returns the next job in priority-then-FIFO order. EchoMQ reaches the same order through a simpler invariant: its pending set scores every member with the same score, 0, and lets the member itself — the branded job id — carry the order, because a branded id sorts as its mint instant. The smallest member is the oldest job; a ZPOPMIN serves it.

Two readings of one set. Toggle between fire-time and order; the same six jobs re-sort. Hover or select a row to trace where it lands.
JOB·ascore 10000 JOB·bscore 25000 JOB·cscore 40000 JOB·dscore 60000 JOB·escore 85000 JOB·fscore 110000
readout · fire-time — the jobs order by ascending fire-time score; the read is ZRANGEBYSCORE -inf now. The data structure is identical to the order reading; only the score's meaning changes.

The member is constant and the commands are a fixed vocabulary; the score is the semantic axis. Read it as a fire-time and the set is a timer wheel; flatten it and let the id order, and the set is a mint-ordered queue.

§3

What EchoMQ really stores

echo_mq/jobs.ex · promote/3

The textbook scores by a raw timestamp and splits delayed and prioritized into separate sets. EchoMQ keeps the structure and sharpens it to one set per reading. Its schedule set is Keyspace.queue_key(queue, "schedule") → emq:{q}:schedule, scored by the run-at millisecond — EchoMQ.Jobs.enqueue_at/6 writes the caller's absolute ms, enqueue_in/6 computes now + delay_ms from the server clock (redis.call('TIME')). No left-shift, no lossy recovery: the score is the millisecond.

Where the textbook re-polls, EchoMQ runs EchoMQ.Jobs.promote/3. Its inline Lua ranges the due head, removes each member, and adds it to the score-0 pending set, where the order reading takes over. The pending set is not a separate prioritized set; it is the one queue, scored 0, ordered by the branded JOB id, served oldest-first by the claim script's ZPOPMIN.

echo/apps/echo_mq · the promote pump

The due-range sweep, as the inline Lua stores it

The @promote script reads the due head with a range bounded by the server clock, then moves each member to the score-0 pending set. The body is verbatim from jobs.ex:

echo_mq/jobs.ex · @promote (the due-range sweep)frozen
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
local due = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, tonumber(ARGV[2]))
for _, id in ipairs(due) do
  redis.call('ZREM', KEYS[1], id)              ← off emq:{q}:schedule
  redis.call('ZADD', KEYS[2], 0, id)              ← onto emq:{q}:pending, score 0
  redis.call('HSET', jk, 'state', 'pending')
end
return #due
the pattern

One sorted set scores a job by a future millisecond to make a timer wheel, or flattens the score so the member's own mint order serves as the queue. The member is constant; the score is the meaning.

its EchoMQ application

emq:{q}:schedule scores by run-at ms and promote/3 ranges the due head with ZRANGEBYSCORE -inf now to pending; emq:{q}:pending scores every member 0 and is served oldest-first by the claim script's ZPOPMIN.

Notes on ValkeyA range query with an exclusive upper bound stops the sweep from re-promoting a member added at exactly now; ZRANGEBYSCORE with LIMIT caps the batch so one tick promotes a bounded slice — valkey.io/commands/zrangebyscore.

A delayed queue and a mint-ordered queue are the same sorted set under two readings of the score — ZADD to insert, a range query or a ZPOPMIN to read — and the chapter that follows takes each reading apart.

§4

References

sources · related

Sources

  • Valkey — Topics — the sorted-set type both readings stand on: members ordered by a numeric score, range and pop by score.
  • Redis — ZADD — insert or update a member's score; the single write both the timer wheel and the mint-ordered queue use.
  • Redis — ZRANGEBYSCORE — the range query that reads the due head of a schedule set, the read in promote/3.
  • Redis — ZPOPMIN — pop the smallest-score member; the read that pulls the oldest job from the pending set.

Related in this course