Streams & pipelines

The dashboard’s activity feed wants the three most recent completions for a course. The transforms are the same either way — filter to that course, shape each into a feed entry, take three — but the engine matters. Enum runs each stage to completion and hands the next a full list; Stream fuses the stages into one pass that pulls elements on demand and stops the moment take/2 has its three.

The same pipeline over the same twelve completions. Switch the engine and watch how many it examines and how many lists it builds.

Examine only what you take Advanced: when eager still wins

Examine only what you take

Twelve completions; seven belong to this course, and the feed needs three. With Enum the filter walks all twelve and the map walks all seven before three are taken. With Stream the pull stops once the third match is shaped. Switch the engine.

Engine · select one

examined: 5 of 12

lists built: 0

Same transforms, same result — three feed entries. The lazy pipeline reaches them by examining five completions and allocating nothing in between; the eager one walks all twelve and builds two lists to throw most of away.

Advanced: when eager still wins

Building a Stream does no work — it composes a recipe of transforms; the work happens when an Enum function at the end (here Enum.take/2) starts pulling. Because the stages are fused, each element is filtered and mapped in one step, and take/2 halts the pull as soon as it is satisfied. That is the win on large or unbounded collections and on pipelines that take only a prefix: no intermediate lists, no work past the cutoff.

It is not free. Each stage is a closure, so a stream carries per-element overhead that an Enum pass does not. For a small list you will read to the end anyway, eager Enum is usually faster despite the extra lists — the constant factor beats the asymptotics until the collection is large or the early exit is real. The feed is exactly the case that favours a stream: a long activity log, and only three entries wanted.

# the activity feed — three most recent completions for a course
def recent(completions, course_id) do
  completions
  |> Stream.filter(&(&1.course_id == course_id))   # lazy
  |> Stream.map(&to_feed_entry/1)                  # lazy, fused with the filter
  |> Enum.take(3)                                # pulls until 3, then stops
end

Enum · eager

Each stage runs to completion and hands on a full list.

Stream · lazy

Stages fuse into one pull that stops at the cutoff, allocating nothing between.

Next: profiling & complexity — reading from the code why the store these pipelines pull from is a map and not a list.

References

Primary sources for this lesson, and where it connects in the course.

Sources

Elixir — Stream — lazy composition and fusion.
Elixir — Enum.take/2 — the terminal that halts the pull.
Elixir — Enumerable and Streams — eager versus lazy, and when each fits.

Related in this course

F4.10 · Practical recipes in Elixir — the module hub.
F4.09 · Branded CHAMP maps & GenServer — the store these pipelines read.
F4 · Algorithms & Data Structures