jonnify·redis /redis-patterns/production-operations/twitter-internals/what-became-core

R8.04.3 · Twitter/X: internals & custom structures — dive 3

What became core: a technique, not a donation.

The technique of packing ziplists into a linked list was used internally at Twitter and never upstreamed. Quicklist became core Redis in 3.2 — independently re-implemented by Matt Stancliff — then evolved to listpack nodes in 7.0. A scale-driven technique became a core encoding.

Grounding: the Twitter case study (Matt Stancliff, Redis Quicklist; redis/redis PR #2143) · the BCS echo — EchoMQ.Lanes in echo_mq/lanes.ex. Valkey is the BCS engine; Twitter’s own Redis is the case study.

How the technique became the encoding. Select an era to light it and read what happened and who wrote it — Twitter’s code was never the core feature.
internal ziplists in a list private · not upstreamed quicklist Redis 3.2 · PR #2143 by Matt Stancliff listpack node encoding Redis 7.0
readout · Select an era. A private technique — ziplists in a linked list — became core Redis as quicklist (3.2, by Matt Stancliff), then evolved to listpack nodes (7.0). Twitter’s own implementation was never contributed.

The author (verbatim) — “Twitter started doing this years ago to store everybody’s timelines, but for various reasons they haven’t contributed their implementation back to the world.”

§1

The precise attribution

matt.sh · PR #2143

It is tempting to say Twitter contributed quicklist. That is wrong, and the encoding’s author says so directly: “Twitter started doing this years ago to store everybody’s timelines, but for various reasons they haven’t contributed their implementation back to the world.”

The honest version has three parts:

  1. The technique was internal at Twitter and elsewhere. Storing lists as ziplists inside a linked list was a known scale technique, used to hold timelines affordably. It lived in private codebases.
  2. It became core Redis as a separate, independent implementation. Quicklist shipped in Redis 3.2, authored by Matt Stancliff (redis/redis PR #2143) — written from the idea, not from Twitter’s source.
  3. Twitter’s own code was never upstreamed. The implementation that ran Twitter’s timelines did not become the core feature; the core feature was re-built in the open.

So the lesson is a scale-driven internal technique becoming a core encoding — not a corporate code donation. A pattern proven privately at scale was independently implemented as a primitive everyone ships. The credit for the core feature is the implementer’s; the motivation came from a problem several large deployments had already solved in private.

Check the three honest parts. Select a claim to light it and read the source that supports it — each is verified, none is the “Twitter donated the code” myth.
✓ the technique was internal at Twitter source · matt.sh — used to store timelines, in private code ✓ quicklist was authored independently source · redis/redis PR #2143 — Matt Stancliff, Redis 3.2 ✓ Twitter’s own code was never upstreamed source · matt.sh — “haven’t contributed… back to the world”
readout · Select a claim. All three are verified: the technique was internal, the core feature was written independently by Matt Stancliff, and Twitter’s own implementation was never contributed — a technique became core, not a code donation.

Together these say the same thing: credit the core encoding to its open re-implementation, not to the deployment that first used the technique.

the myth

“Twitter contributed quicklist to Redis.” Stated everywhere, and wrong — Twitter “haven’t contributed their implementation back to the world.”

the fact

Twitter used the technique internally; quicklist was authored independently by Matt Stancliff and shipped in Redis 3.2 (PR #2143). A technique became core, not a transfer of source.

The distinction matters: a private technique at scale can become everyone’s default, but the core feature’s credit belongs to whoever re-built it in the open. Here that is Matt Stancliff.

§2

The evolution to the listpack

matt.sh · Redis 7.0

Quicklist did not stop at ziplists. The node encoding evolved.

A ziplist was the original packed block — compact, but with a known sensitivity to cascading updates: changing one entry’s size could ripple through the encoding of following entries. The listpack was introduced to replace it, removing that cascade and simplifying the format. In Redis 7.0 the listpack became the node encoding for lists, so a modern quicklist is a doubly-linked list of listpacks, not of ziplists.

The shape is unchanged — a linked list of packed nodes — but the packed node is now a listpack. That is why this course says “ziplists/listpacks” when describing a quicklist node: the original is the ziplist, today’s is the listpack, and the principle is identical. Small lists are stored as a single packed node directly; only when a list grows past the node-size threshold does it become a multi-node quicklist. Those thresholds — the subject of R7.02 — are what decide when a single listpack becomes a quicklist of many.

ziplist
The original packed node. Compact, but a size change can cascade through following entries.
listpack
The replacement (Redis 7.0). No cascade; today’s quicklist node encoding.
quicklist
A doubly-linked list of those packed nodes — the shape is unchanged.

The encoding kept its shape and changed its node: ziplist, then listpack. A small list is one packed node; only crossing the size threshold makes it a quicklist of many.

§3

The BCS echo — inherited for free

echo_mq/lanes.ex

EchoMQ writes to Valkey lists, so it gets the modern encoding without asking. There is no setting to turn quicklist on and no code that chooses it; it is the list encoding, and all Valkey lists are quicklists. The lane ring and the wake list are native Redis lists:

echo/apps/echo_mq/lib/echo_mq/lanes.ex · native Redis listsfrozen
-- the lane ring (a Redis list), rotated one step per claim
redis.call('RPUSH', KEYS[3], ARGV[3])
redis.call('LMOVE', KEYS[1], KEYS[1], 'LEFT', 'RIGHT')

-- the wake list (a Redis list), held to 64 — one listpack node
redis.call('LPUSH', KEYS[7], '1')
redis.call('LTRIM', KEYS[7], 0, 63)

Each of these runs on a quicklist whose nodes are listpacks. The wake list, capped at 64, is a single listpack node; the ring holds the lanes serviceable right now and is short by construction. EchoMQ inherits, at no cost, the encoding that Twitter’s scale drove into core Redis a decade ago.

the pattern

A scale-driven technique (ziplists in a linked list), proven privately, became a core encoding (quicklist, Redis 3.2; listpack nodes, 7.0) that every Redis and Valkey list now uses.

its EchoMQ application

EchoMQ’s lists get that encoding automatically; the only choice the bus makes is to keep the wake list short with LTRIM … 0, 63, so it stays one compact listpack node. The bus does not select quicklist; it benefits because the encoding is universal.

The honest framing carries through to the end: the credit is Matt Stancliff’s, the motivation was a problem Twitter and others had at scale, and EchoMQ’s gain is only that the primitive is in the engine. The worked consumer, codemojex, never sees the encoding — it mints a branded JOB with EchoData.BrandedId.generate!(“JOB”) and enqueues it onto a lane, and the list it lands in is a quicklist because every Valkey list is.

the application · the encoding comes from the engine

echo_mq does not choose quicklist

Every Valkey list is a quicklist, so the bus inherits the encoding for free; its only lever is the cap that keeps the wake list one node:

echo/apps/echo_mq/lib/echo_mq/lanes.ex · the only lever is the capfrozen
# no encoding choice — all Valkey lists are quicklists
LPUSH wake '1'
LTRIM wake 0 63        # keep the wake list one listpack node

Notes on ValkeyValkey inherited quicklist and the listpack node from Redis: a list is a quicklist of listpack nodes, with list-max-listpack-size bounding a node by entries or bytes. A system that writes lists gets this encoding with no configuration; the only lever is keeping a list short enough to stay one node, which is what the wake list’s LTRIM … 0, 63 does — valkey.io/topics/data-types.

§4

Recap — a technique, not a donation

next → R8.05

Quicklist’s story is precise: the technique of packing ziplists into a linked list was used internally at Twitter and never upstreamed; the core feature was written independently by Matt Stancliff and shipped in Redis 3.2 (PR #2143); the node encoding later became the listpack in 7.0. A pattern proven privately at scale became a primitive everyone ships. EchoMQ inherits it for free — all Valkey lists are quicklists — and the only thing the bus controls is keeping the wake list short enough to stay one node. The chapter continues to R8.05, where Uber’s resilience techniques meet the connector’s recovery behaviour.

§5

References

sources · related

Sources

Related in this course