jonnify·redis /redis-patterns/production-operations/twitter-internals

R8.04 · Production & Operations — module hub

Twitter/X: internals & custom structures.

Twitter ran home timelines as native Redis lists at a scale few match — over 10,000 Redis servers and 100 TB of live memory per datacenter, roughly 30 billion Redis updates a day against more than 400 million tweets a day. Three things its scale taught the Redis world are worth taking apart.

Grounding: the Twitter source pack — How to Scale: Tweet Delivery at Twitter (HdM Stuttgart), 30 Billion Redis Updates Per Day (VMware Tanzu), and Redis Quicklist (Matt Stancliff) with redis/redis PR #2143 — for the case study; the as-built echo/apps/echo_mq (Lanes) and echo/apps/echo_data (BrandedId) for the applied half. The bus is Valkey-only; Twitter’s own Redis is the case study, quoted as theirs.

One timeline tier, three lessons. Select an idea to light where it sits and read the Twitter fact beside its BCS echo.
fanoutspread to followers a tweetid · author · bits timelinesRedis lists ×3 quicklistlistpack nodes fan-out memory win became core
readout · Select an idea. Twitter inserts a tweet id into every follower’s Redis list (fan-out on write), holds each list as a quicklist of packed nodes (the memory win), and that encoding became core Redis as quicklist (what became core).

Static facts — fan-out → O(1) read, O(n) write, 3× replication · memory win → 1.0 GB vs 11.86 GB for 200 lists of 1M integers · became core → quicklist, Redis 3.2, by Matt Stancliff (PR #2143) — the technique, not a code donation.

§1

The scale: timelines in Redis lists

VMware Tanzu · matt.sh

The home timeline is the feed a user reads. Twitter’s choice was to make that read cheap by paying at write time: when a tweet is posted, a fanout service inserts the tweet id into every follower’s home timeline, which is a Redis list. A read of a timeline is then a single range over one list — it does not touch the social graph and does not scan other users’ data.

The numbers report the scale of that design. “Twitter runs over 10,000 Redis servers per data center and their Redis deployment uses over 100 TB of live memory per datacenter.” The VMware Tanzu case study frames the home-timeline Redis tier at about 30 billion Redis updates per day, against more than 400 million tweets a day. Reads dominate: a reported “300K QPS … spent reading timelines and only 6000 requests per second … on writes.”

Three things follow from putting that much data in that many lists, and each is a lesson worth taking apart: the fan-out write path that fills a timeline, the memory a list of ids costs and the encoding that made it affordable, and the scale-driven technique that turned into a list encoding everyone now ships.

A timeline is a list of ids. Twitter held an enormous number of them, so the write path and the per-entry cost are not details — they are the design. Each idea below lands twice: the Twitter fact, and the move it becomes on the bus the reader is building.

§2

The three ideas it proves

HdM Stuttgart · matt.sh

Fan-out on write. “The fanout … is responsible for spreading out the tweet to all people following the author by inserting it to all their home timelines.” A tweet goes to the fanout service, which asks the social-graph service for the author’s follower set, then inserts the tweet id into each follower’s Redis list. The read is O(1); the write is O(n) in the follower count. Very-large accounts are the exception: delivery “can take up to 5 minutes … when celebrities like Lady Gaga or Kylie Jenner tweet,” which motivates a hybrid — fan-out-on-write for most accounts, fetch-and-merge-at-read for the few enormous ones.

The memory cost of a list, and quicklist. A naive Redis list is a doubly-linked list of single-element nodes, and “regular linked lists have overhead of 40+ bytes per entry.” A ziplist packs many entries into one contiguous block, with “overhead ranging from 1 byte to 10 bytes per entry” — far smaller, but a single block grows expensive to edit as it gets long. Quicklist combines them: a doubly-linked list of ziplists, “a quicklist looks like: [ziplist 0] <-> [ziplist 1] <-> … <-> [ziplist N].” The figure: storing 200 lists of 1 million integers used 1.0 GB at the optimal ziplist length versus 11.86 GB for traditional linked lists — roughly twelve times less.

A technique that became core. Quicklist did not arrive as a corporate code donation. Matt Stancliff is explicit: “Twitter started doing this years ago to store everybody’s timelines, but for various reasons they haven’t contributed their implementation back to the world.” The technique — ziplists in a linked list — was used internally at Twitter and elsewhere; it became core Redis as quicklist in Redis 3.2, independently re-implemented by Matt Stancliff (redis/redis PR #2143). The lesson is a scale-driven internal technique becoming a core encoding, not a transfer of source.

Three ideas, one tier: spread a tweet into every follower’s list to keep reads cheap; pack those lists so a billion entries fit in memory; and watch a private technique become the encoding everyone ships.

§3

The applied half — the BCS bus

echo_mq/lanes.ex · echo_data/branded_id.ex

The Twitter case is the lens; the bus the reader is building is EchoMQ, backed by Valkey. EchoMQ does not store timelines, but its queue is built on the same primitive Twitter chose — the native Redis list — and the connection is exact and verified. A lane is a per-group pending set; a ring lists the lanes that can be served right now; a second list signals a parked consumer when work appears. The relevant lines are verbatim from EchoMQ.Lanes:

# echo/apps/echo_mq/lib/echo_mq/lanes.ex — the enqueue script
redis.call('RPUSH', KEYS[3], ARGV[3])     # push the lane onto the ring
redis.call('LPUSH', KEYS[7], '1')         # signal a parked consumer
redis.call('LTRIM', KEYS[7], 0, 63)       # hold the wake list to 64 elements
# the claim side rotates the ring one step:
redis.call('LMOVE', KEYS[1], KEYS[1], 'LEFT', 'RIGHT')

RPUSH appends to the ring; LPUSH then LTRIM … 0, 63 writes a one-byte token to a wake list and trims it to 64 elements, so a consumer blocked on that list wakes when work appears (the module’s words: “pushing a wake for any parked consumer”). A list that short lives as one small listpack node of a quicklist — the precise case quicklist and listpack were built for.

the pattern

A feed is a list of ids, one id per entry; reads stay cheap because the id is the whole entry, and the list encoding (quicklist over ziplists/listpacks) keeps a billion such entries affordable.

its EchoMQ application

EchoMQ.Lanes pushes a branded JOB id onto a Redis list with RPUSH, rotates the ring with LMOVE, and signals readiness with LPUSH + LTRIM … 0, 63 — a wake list held to 64, so it stays one small listpack node.

Two honest framings keep the parallel straight:

  • The id is the thing in the list. Twitter stored a timeline entry as “its unique tweet id, the author’s id and some additional bits” — the id is the entry. EchoMQ’s lane entry is a branded JOB id, a compact printable 14-character name (Codemojex.Guesses.submit/3 mints one with EchoData.BrandedId.generate!(“JOB”)). Both keep the list cheap by storing an identity, not a payload — the payload lives elsewhere, keyed by the id.
  • All Valkey lists are quicklists — echo_mq does not choose it. Quicklist is the list encoding, not a per-list option a system turns on. EchoMQ’s wake list benefits from listpack-node packing because every Valkey list is a quicklist; the only thing the script controls is the cap (LTRIM … 0, 63), which keeps the wake list in a single small node. The historical core feature is quicklist (Redis 3.2); today’s node encoding is the listpack (which replaced the ziplist in 7.0). The list-encoding thresholds are the subject of R7.02 (memory optimization).

The worked consumer is codemojex. A guess is a job on a player’s lane: Codemojex.Guesses.submit/3 mints a branded JOB (EchoData.BrandedId.generate!(“JOB”)) and runs Lanes.enqueue(Bus.conn(), …), so the lane is a list of ids and the bus rotates service across players.

door · the protocol

/echomq

The EchoMQ protocol behind the lane and wake-list tie-backs — the owned wire, the braced keyspace, the Lua layer.

 door · the Queue pillar

/echomq/queue

Jobs, lanes, and the rotating claim in depth — the lists that carry the work and the wake that signals a parked consumer.

 door · the branded id

/bcs/overview

The branded id that is the thing in the list — a compact 14-character name, the analogue of Twitter’s tweet id.

Notes on ValkeyA Valkey list is stored as a quicklist — a doubly-linked list whose nodes are listpacks. A short list lives in a single listpack node; the list-max-listpack-size limit decides when a node splits. A 64-element wake list never crosses that threshold, so it stays one compact node — the same packing Twitter’s scale drove into core Redis — valkey.io/topics/data-types.

§4

The three dives

fan-out → quicklist → what became core

The module follows the arc — the write path, the memory it costs, and what came of it. Read them in order.

R8.04.1built

Timeline fan-out

Fan-out on write into per-follower Redis lists; O(1) read, O(n) write; 3× replication; the celebrity hybrid. The BCS echo — lanes are per-lane Redis lists, the wake list signals readiness.
R8.04.2built

Quicklist & memory

Linked-list 40+ B/entry vs ziplist 1–10 B/entry; quicklist = a linked list of ziplists/listpacks; the 1.0 GB vs 11.86 GB figure. The BCS echo — the 64-element wake list is one small listpack node.
R8.04.3built

What became core

The technique was internal at Twitter and never upstreamed; it became core as quicklist (Redis 3.2, by Matt Stancliff), then the listpack in 7.0. The BCS echo — echo_mq inherits the encoding for free.

Three dives, one timeline tier: how a tweet reaches a list, what a list of ids costs, and what the world took from it — the Twitter case study and the BCS bus read side by side.

After R8.04, the chapter continues to R8.05 — Uber: resilience & staggered sharding — before the capstone hands off to the dedicated EchoMQ course. Return to the Production & Operations chapter to continue, or open the Queue pillar for the depth behind the lanes and the wake.

§5

References

sources · related

Sources

Related in this course