jonnify·redis /redis-patterns/flow-control/groups/round-robin-across-tenants

R6.03.1 · Groups — dive

Round-robin across tenants.

A tenant is a lane: work enqueued under a branded group id rides that lane, and the ring serves every lane in turn. A busy tenant cannot jump ahead, a quiet one is never skipped, and the order is a property of rotation — not a score, not a hash.

Grounding: echo/apps/echo_mq/lib/echo_mq/lanes.ex · EchoMQ.Lanes.enqueue/5 (the @genqueue admission) · EchoMQ.Lanes.claim/3 (the @gclaim rota: LMOVE ring ring LEFT RIGHT, then serve the head of the rotated lane).

Three tenants, one ring, served in turn. Select a lane to read what it holds and when the rota reaches it. The ring rotates one step per claim, so A, B, C each get one turn per cycle regardless of backlog depth.
claim → rotate one step → serve this lane Adeep Bshallow Cone job A → B → C → A … one turn per lane per cycle
readout · Select a lane. The ring holds the serviceable lanes; each claim rotates it one step and serves the lane it lands on. A deep backlog (A) does not earn more turns than a single job (C) — the rota gives one turn per lane per cycle, so no tenant starves another.
§1

Admission by group

lanes.ex · enqueue/5

A grouped queue admits work by tenant. The group id names the lane; the job goes onto that lane's pending set:

EchoMQ.Lanes.enqueue(conn, "orders", group, job_id, payload)
# group is a branded id; the job lands on emq:{orders}:g:<group>:pending

enqueue/5 is one idempotent script. It checks the kind of the job id, refuses a duplicate, writes the job row carrying its group field, adds the job to the lane's pending sorted set at score zero (mint order is the order), and — only if the group is not already on the ring and is below its ceiling — pushes the lane onto the ring with a wake for any parked consumer. The group is gated as a valid branded id at the key builder, so the lane is always named by a real identity; an ill-formed group raises before any command reaches the wire.

The lane is created by the first job that names the group. There is no provisioning step, no registry to update, no queue to declare per tenant — a new tenant exists the moment its first job is admitted, and disappears when its lane empties. The set of tenants is open by construction.

A tenant is named by a branded group id, and enqueue/5 puts its work on emq:{q}:g:<group>:pending and registers the lane. The lane is created by the first grouped job; ungrouped work rides no lane.

§2

The ring is the rota

lanes.ex · the @gclaim LMOVE

The ring emq:{q}:ring is a Valkey list, and it holds exactly the serviceable lanes: nonempty, unpaused, and below their concurrency ceiling. It is not the set of all tenants — it is the set of tenants that can be served right now. A lane joins the ring when it becomes serviceable and leaves the moment it does not.

Every claim rotates the ring one step before it serves. The rotation is a single atomic LMOVE:

EchoMQ.Lanes · echo/apps/echo_mq — the @gclaim script: rotate the ring, then serve the rotated laneverbatim
local g = redis.call('LMOVE', KEYS[1], KEYS[1], 'LEFT', 'RIGHT')   ← rotate one step, take the head
if not g then return {} end
local lane = ARGV[1] .. 'g:' .. g .. ':pending'
local popped = redis.call('ZPOPMIN', lane)              ← serve the oldest in that lane

LMOVE ring ring LEFT RIGHT pops the head of the ring and pushes it to the tail, returning the group it moved. That group is the tenant served this turn: the script derives the lane key from the group, pops the lowest-scored member (the oldest, mint order kept), leases it on the server clock, and returns it. The next claim moves the next head to the tail, and so on around the ring.

So the rota is the list order, advanced one step per claim. Three tenants on the ring are served A, B, C, A, B, C — each gets one turn per cycle, regardless of how deep its backlog is. Fairness between tenants is constructed by rotation, never hashed: there is no priority field to game, no consistent-hash bucket to fall into, only the ring advancing one position at a time.

Notes on ValkeyLMOVE source destination LEFT RIGHT is atomic: in one operation it removes the first element of the source list and pushes it to the end of the destination. With the same key for both, it rotates the list one position and returns the moved element — exactly the rota step the claim needs, with no read-modify-write window where two consumers could rotate the same head — valkey.io/commands/lmove.

The ring holds the serviceable lanes; claim/3 does LMOVE ring ring LEFT RIGHT then serves the head of the rotated lane. The schedule is the list order, advanced one step per claim — rotation, not a score.

§3

A lane that empties or fills leaves the ring

the @gclaim re-ring guard

The rota only holds serviceable lanes, so the script keeps the ring honest as it serves. After it pops a member, it checks two conditions:

EchoMQ.Lanes · echo/apps/echo_mq — the @gclaim post-serve guard: de-ring at the ceiling or when emptyverbatim
local act = redis.call('HINCRBY', ARGV[1] .. 'gactive', g, 1)
local lim = redis.call('HGET', ARGV[1] .. 'glimit', g)
if lim and act >= tonumber(lim) then
  redis.call('LREM', KEYS[1], 0, g)        ← at its ceiling: de-ring
elseif redis.call('ZCARD', lane) == 0 then
  redis.call('LREM', KEYS[1], 0, g)        ← lane now empty: de-ring
end

If serving that job pushed the tenant to its concurrency ceiling, the lane is removed from the ring so the rota skips it until a completion frees a permit. If the lane is now empty, it is removed because there is nothing left to serve. Either way, the next rotation lands on a lane that genuinely has serviceable work — the ring never wastes a turn on a tenant that cannot be served.

A lane returns to the ring the moment it becomes serviceable again: a fresh enqueue/5 onto an empty lane re-rings it, a completion that drops gactive below the ceiling re-rings it, a resume/3 re-rings a paused lane. The rota is self-maintaining.

The claim, step by step. Select a step to read what the script does and why. One claim rotates the ring once, serves the head of the rotated lane, and reconciles the ring.
LMOVErotate ring ZPOPMINserve oldest TIMElease deadline LREM?de-ring guard one atomic script: rotate, serve, lease on the server clock, reconcile the ring
readout · Select a step. The claim is one atomic script: LMOVE rotates the ring and returns the head group, ZPOPMIN serves the oldest job in that lane, the server clock sets the lease deadline, and a guard de-rings the lane if it is now at its ceiling or empty.

The ring never wastes a turn: a lane that reaches its ceiling or empties is de-ringed on the spot, and re-ringed the moment it is serviceable again. The rota always holds exactly the lanes that can be served.

§4

The pattern, applied

echo/apps/codemojex
the pattern

Round-robin scheduling serves a set of queues in strict rotation, one item per queue per turn, so no queue starves another. The order is the rotation, not a priority.

its EchoMQ application

The ring emq:{q}:ring holds the serviceable lanes; EchoMQ.Lanes.claim/3 does LMOVE ring ring LEFT RIGHT then serves the head of the rotated lane, so every tenant is served in turn and the order is rotation, not a score.

In codemojex — a Telegram emoji-guessing game on the same stack — a room is a tenant. Guesses from a room are enqueued under that room's branded id, so they ride one lane (emq:{cm}:g:<PLR…>:pending); the ring rotates through the active rooms, and a room flooding the engine with guesses takes only its turn in the rota — the quiet rooms are served on the very next steps. Work enqueued without a room id rides no lane and is never claimed by the grouped path: the lane is the unit of control.

/echomq/queue

The rota at the claim point

The Queue pillar wires the ring into the dequeue path: the consumer that drives claim/3, the wake that parks an idle consumer, and lease recovery per tenant.

 /bcs/bus/jobs-and-lanes

A queue is a set of lanes

Part B3.2: a queue is not one line but a set of lanes named by branded ids, drained fairly so no one lane starves the rest — the fair-lanes property.

 R6.03.2

The ceiling the rota respects

Round-robin keeps the turns fair; the per-group ceiling bounds how much one tenant holds at once, and de-rings a lane at its limit.
§5

References

sources · related

Sources

  • Valkey — LMOVE — the atomic head-to-tail move that rotates the ring one step and returns the served group.
  • Valkey — ZPOPMIN — pop the lowest-scored member of the lane — the oldest job, mint order kept.
  • Valkey — RPUSH — push a newly serviceable lane onto the tail of the ring, where the rota reaches it in turn.
  • Valkey — Cluster specification — the {q} hash tag pins the ring and every lane to one slot, so the claim script is slot-legal.

Related in this course