Fair Lanes and the Metronome

A task queue supersession has two halves, and the second is the interesting one. The first half is the claim: a job is not taken, it is claimed — one atomic move leaves the ready set and enters a lease held by the claimant, with the attempts counter as a fencing token so a stalled worker cannot act on a job someone else now owns. Failure is not loss; the job retries with backoff, and a lease that expires is reclaimed. That much is a reliable queue.

The second half is fairness without polling. One queue serves many groups, and a busy group must not starve a quiet one, so each group's pending work is a lane and the serviceable lanes sit on a ring that rotates — weighted or deficit-scheduled so each group's share is honored. The trap most fair schedulers fall into is spinning: consumers polling the ring, burning the server on every empty check. The EchoMQ.Metronome closes it by parking — a blocking pop waits on readiness and fans it to a pool of consumers over BEAM messages, so a worker sleeps until there is work for its lane, then claims. Fair share, and not one wasted poll.

The fairness is not a policy bolted on; it is three concrete structures. A lane is a per-group pending set — a sorted set keyed by the group's branded name — so a group's backlog is isolated from every other's. The ring is a list holding exactly the lanes that are serviceable right now: nonempty, unpaused, under their limit. Rotating the ring, with a weight or a deficit counter per group, is what turns "many groups on one queue" into "each group gets its share," and the starvation drill exists to prove that even a flood in one lane cannot lock the others out.

The Metronome is the piece that makes fairness cheap. A naive scheduler asks every consumer to keep checking the ring, which spends the server on emptiness and wakes everyone at once when a job lands. The Metronome inverts it: a single blocking pop per queue holds the readiness signal, and when work arrives it fans that signal to a pool of consumers over BEAM messages — the runtime's own cheap mailboxes — so each parked worker is woken only when its lane has something to claim. Park-don't-poll is why the fair-lanes family scales to a multi-tenant bus without a busy loop anywhere in it.

The fairness rule — work is claimed under a lease, and lanes rotate on a ring so no group starves and no consumer polls. A claim is an atomic lease with the attempts fence and backoff on retry; the EchoMQ.Metronome parks a consumer on a blocking pop until its lane has work, then fans readiness over BEAM messages — fair share without a busy loop.