jonnify·redis /redis-patterns/time-delay-priority/schedulers

R4.02 · Schedulers

The delayed job that
re-schedules itself.

A repeatable job is a delayed job that re-schedules itself: a sorted set holds one entry per registration, scored by the next run time, and each time an occurrence fires the next timestamp is computed and the same entry's score is rewritten — so the recurrence survives restarts without ever duplicating.

Three dives' worth of clock arithmetic: naming the cadence, the upsert that keeps one entry per registration, and the start-to-start advance. The grounding is EchoMQ.Repeat (echo/apps/echo_mq) — the emq:{q}:repeat registry, the EXISTS-guarded register, and EchoMQ.Pump's per-tick sweep that mints a fresh branded JOB id per occurrence.

§1

What a scheduler adds over a one-shot delay

echo/apps/echo_mq · repeat.ex

A one-shot delayed job is a member in a sorted set scored by its fire-time. It runs once and is gone. A scheduler carries that one move forward: it keeps a registry of recurring schedules, one entry per registration, each scored by when it next runs. When an occurrence fires, the next run time is computed and the same entry's score is rewritten. The score is still a future timestamp, exactly as in the delayed queue — the new idea is that the entry is rewritten each cycle instead of consumed once.

A cadence

A rule for "when next" — a wall-clock cron slot or a fixed interval (every_ms), evaluated to a millisecond.

A registry

A separate sorted set, scored by the next run time, holding one entry per recurring registration.

An upsert

Registering a name that already exists leaves its entry alone rather than appending a duplicate.

A fresh job per fire

Each occurrence mints a new branded JOB id — a first-class, browsable, mint-ordered job, never a reused row.

One registration, rewritten each cycle. Hover or select a fire to light the registry entry whose score the advance rewrites — the registry stays at one row.
emq:{reports}:repeatZSET · 1 entry score = 09:15:00 score = 09:30:00 score = 09:45:00
readout · Select a fire. The registry holds one member; each advance rewrites its score to the next run, never appending a second row.

Every dive in this module is one answer to the same question: a single registry entry, scored by the next run — how is that score named, kept unique, and rewritten.

§2

The repeat registry & its upsert

repeat.ex · @register

Naming the cadence. There are two ways to name when next. A cron expression (0 9 * * *) names a wall-clock calendar position; a fixed interval (every_ms: 900000) names a gap. Both reduce to a single millisecond timestamp, which is the score the registry stores. EchoMQ.Repeat is the interval form: it carries an every_ms period and advances the score by it each cycle.

The registry. EchoMQ.Repeat declares two {q}-hashtagged keys, so a queue's repeat state lands on its own cluster slot: emq:{q}:repeat — a sorted set scored by the next-run millisecond, members the registration names — and emq:{q}:repeat:<name> — a hash carrying every_ms and the payload template. The registry carries only the next-run times; the period and template live in the companion hash.

The upsert (no duplicate on boot). A service that boots re-registers all its schedules. A naive add would push a second entry and the recurring job would fire twice. The @register script probes the record key first — if redis.call('EXISTS', KEYS[2]) == 1 then return 0. A present record returns 0 and writes nothing; register/6 answers {:ok, :exists}. Only a fresh name runs the HSET and the ZADD, answering {:ok, :registered}. One name, one record, one registry entry, across any number of boots.

Producing the next run (start-to-start). The registry holds only the next run. The cadence is EchoMQ.Pump's: each tick it reads the due registrations (ZRANGEBYSCORE emq:{q}:repeat -inf <now>), mints a fresh JOB id and enqueues each occurrence, then advances the score. advance/4 sets the new score to now_ms() + every_ms — start-to-start, measured from the release instant plus the period, not from a finish time. A slow occurrence does not push the cadence later.

Notes on Valkey. The pump's due read is ZRANGEBYSCORE emq:{q}:repeat -inf <now> LIMIT 0 <n> — the names scored at or before now, oldest-due first, bounded per tick. The score is the sort key, so "due" is a range query, not a scan. valkey.io/commands/zrangebyscore

§3

When to use, when to avoid

the position on the trade-off

Use — a calendar or fixed cadence

Periodic reports, cleanup sweeps, polling an upstream — recurring work named once.

Use — restart-safe recurrence

Schedules that must survive a restart without re-firing or duplicating; the registry is on the bus, not in a process.

Avoid — sub-second cadences

A sorted-set registry swept by a pump on a beat is millisecond-grained, not a real-time scheduler.

Avoid — millions of one-offs

Enormous fan-outs of distinct future runs are delayed jobs, not registrations — they belong in the schedule set.

Full calendar correctness across timezones and daylight-saving transitions is a parsing depth that belongs to EchoMQ's scheduler subsystem — the EchoMQ course's territory, not this module's.

§4

The three dives

name · keep unique · rewrite

Three dives, one per move on the single registry entry: how the next-run score is named, how it is kept unique across boots, and how it is rewritten each cycle.

built ready to read
§5

In EchoMQ — the registration that does not duplicate

repeat.ex · @register

A textbook delayed queue can already schedule one future run. A scheduler makes the run re-schedule itself. EchoMQ.Repeat keeps exactly one emq:{q}:repeat entry per registration name and makes the register idempotent with the EXISTS probe, so a restart re-registering the same name finds the record already present and changes nothing rather than spawning a duplicate. Registration is one inline Lua script; the guard is its first move, and only a fresh name runs the two writes:

echo/apps/echo_mq · repeat.ex · @register (verbatim)frozen
if redis.call('EXISTS', KEYS[2]) == 1 then
  return 0                                  -- already registered → {:ok, :exists}
end
redis.call('HSET', KEYS[2], 'every_ms', ARGV[2], 'template', ARGV[3])
redis.call('ZADD', KEYS[1], tonumber(ARGV[4]), ARGV[1])
return 1                                    -- fresh name → {:ok, :registered}

KEYS[1] is emq:{q}:repeat (the registry sorted set), KEYS[2] is emq:{q}:repeat:<name> (the record hash), both gated onto the queue's slot. The score is the next-run millisecond; the member is the registration name. EchoMQ.Pump then sweeps the set each tick, minting a fresh branded JOB id per occurrence and advancing the score by every_ms start-to-start. The full cron parser, the timezone handling, and the scheduler-vs-delayed reconciliation across a worker pool belong to the EchoMQ course's Queue pillar.

§6

References

sources · related

Sources

  • Redis — ZADDwrite the next-run millisecond as the registry score; re-adding the same member rewrites it
  • Valkey — ZRANGEBYSCOREthe pump's due read: the names scored at or before now, bounded per tick
  • Valkey — Sorted setsthe registry as a sorted set of next-run times, on the engine the connector is gated against
  • Redis — EXISTSthe idempotence probe: a present record makes the register a no-op

Related