The Queue · module 05
Flows — orchestration over the queue
A parent job and a list of children. The children run; the parent runs only after they finish.
A flow composes dependent work: a parent job and a flat list of
children. The children are written claimable; the parent is held out of the pending
set with an outstanding-child counter at :dependencies and its row
state = awaiting_children. One atomic script lands a same-queue flow on one slot; the
fan-in inside @complete decrements the counter as children finish and, at
zero, releases the parent to pending. Composition is a counter and one atomic script —
not a coordinator. This module reads the shape, the three result reads, and the cross-queue path.
Three scripts carry a flow
A same-queue flow lands in one atomic script. A cross-queue flow lands parent-first across two more. The fan-in
that releases the parent is the single-queue branch already inside @complete. Pick a
script to read its handle, the keys it declares, and the transition it performs — the catalog is the protocol.
Flows run only as scripts: there is no native flow add, and EMQ.COMPLETE and
EMQ.FAIL refuse a flow child (a row carrying parent) and the
flow key layouts before any write. On a node that loads the native module, EMQ.COMPLETE
answers a flow child with EMQSTATE complete flow not native and
EMQ.FAIL with EMQSTATE fail flow not native;
the native module teaches the commands.
The flow script catalog · pick a script
declares: KEYS[1] parent row, KEYS[2] parent :dependencies, KEYS[3] pending set, KEYS[4..] child rows
values: ARGV[1] parent id, ARGV[2] parent payload, ARGV[3] N, ARGV[4..] (child id, payload) pairs
transition: gate every id, write each child claimable (state pending, carrying parent), hold the parent (state awaiting_children), SET the counter to N — one atomic step on one slot.
A flow is a held parent, claimable children, and a counter. Land it atomically; release it on the fan-in. There is no coordinator in the loop.
The dives
Three dives read flows in depth, each grounded in real echo/apps/echo_mq code: the
shape and the atomic same-queue add, the three pure reads the parent runs on, and the cross-queue path with its
failure policy.
Parent and children
The held parent (awaiting_children, :dependencies = N), the claimable children, the atomic @enqueue_flow in two beats, the fan-in hook inside @complete, and the native refusal of a flow child.
Reading the results
The parent runs ON its children's outcomes through three pure reads: children_values/3, dependencies/3, ignored_failures/3. All id-gated, no state change.
Cross-queue & the failure policy
A child on another slot: host-orchestrated, parent-first, fail-closed, eventually-consistent through the Pump sweep. Fail-parent vs ignore-dependency.
Pattern & implementation
Where Redis Patterns Applied teaches composing dependent work, this module is where the move lands. The flow is a counter and one atomic script.
Orchestrating dependent work — a parent step that waits on its legs — is the flow-control angle, R6 · Flow control. The near, resolvable door is R3 · Reliable Queues: each child is one reliable-queue job the flow composes.
EchoMQ.Flows.add/3 lands the held parent + the claimable children in one
@enqueue_flow; the parent's :dependencies counter holds it;
the fan-in branch inside EchoMQ.Jobs @complete decrements and at zero releases it.
Take the dives in order, or jump to the cross-queue path if you already know the same-queue shape. The module names R6 · Flow control as its pattern; the resolvable door is Reliable Queues.
References
Sources
- Valkey — SET — the
:dependenciesoutstanding-child counter the flow holds the parent with. - Valkey — DECR — the fan-in decrement inside
@complete. - Redis — EVALSHA — the load-once dispatch
@enqueue_flowruns by. - Valkey — Documentation — the substrate of record EchoMQ is backed by.
Related in this course
- The Queue — the chapter this module belongs to.
- Parent and children — the shape and the atomic same-queue add.
- Reading the results — the three pure reads the parent runs on.
- Cross-queue & the failure policy — the cross-queue add and the failure policy.
- The Protocol — the keyspace and the Lua layer the flow scripts are built on.
- redis-patterns · Reliable Queues — the reliable-queue pattern, the near side of the door.