Quicklist & memory: pack the list, keep the nodes short.

A Redis list can be represented two ways, and each trades the other’s weakness.

A linked list holds one element per node, and each node carries pointers and allocator headers. Matt Stancliff puts the cost plainly: “Regular linked lists have overhead of 40+ bytes per entry.” For a list of small integers, the overhead dwarfs the data — forty-plus bytes of bookkeeping to hold an eight-byte value. The upside is cheap edits at either end: pushing or popping is a pointer update.

A ziplist packs many entries into one contiguous block of memory, with “overhead ranging from 1 byte to 10 bytes per entry.” That is an order of magnitude less than the linked list, because there are no per-element nodes — only lengths and the values, laid end to end. The downside is the cost of editing a long block: inserting into the middle, or growing past the allocation, can mean copying the whole block, so a single large ziplist becomes expensive as it grows.

So the two representations sit at opposite ends: the linked list is flexible but heavy per entry; the ziplist is compact but costly to edit when long. A list of a billion entries wants both — compact storage and cheap edits.

Quicklist is the combination. It is “a quicklist looks like: [ziplist 0] <-> [ziplist 1] <-> … <-> [ziplist N]” — a doubly-linked list whose nodes are ziplists (listpacks in modern Valkey). Each node packs many entries compactly; the linked list of nodes keeps edits local to one node and keeps any single node short enough to edit cheaply.

The two costs are bounded at once: per-entry overhead stays near the ziplist’s — a few bytes per entry inside a node, not the linked list’s 40+, because entries are packed — and edit cost stays bounded, because a push or a middle insert touches one node, and a node is kept short, so no operation copies a billion-entry block.

The figure makes the saving concrete. Storing 200 lists of 1 million integers used 1.0 GB of allocated memory at the optimal ziplist length versus 11.86 GB for traditional linked lists — roughly twelve times less memory for the same data. At Twitter’s reported scale — “over 10,000 Redis servers per data center” holding “over 100 TB of live memory per datacenter” — a twelve-times reduction in the cost of a list is the difference between a fleet that fits and one that does not.

EchoMQ does not store million-entry lists, but it has one list whose whole purpose is to stay in a single node — the wake list. When a job is admitted, the enqueue script signals a parked consumer and trims the signal list to 64 elements:

A 64-element list of one-byte tokens lives as one small listpack node of a quicklist. That is the precise case quicklist and listpack were built for — a short list that should be one compact, cheap-to-edit block. The LTRIM … 0, 63 is what keeps it there: no matter how often work arrives, the wake list never grows past one node, so the signal stays cheap.

The honest framing: EchoMQ does not configure quicklist — it cannot. Quicklist is the list encoding, and all Valkey lists are quicklists. The only thing the script controls is the cap, which keeps the wake list inside a single listpack node. The node-size thresholds that govern when any list splits across nodes are the subject of R7.02 (memory optimization) — the listpack and the limits that decide a node’s size.

The two early list representations each gave up what the other kept: the linked list was flexible but spent 40+ bytes per entry; the ziplist was compact at 1–10 bytes per entry but costly to edit when long. Quicklist combined them — a linked list of ziplist (now listpack) nodes — and the figure proves it: 200 million integers in 1.0 GB against 11.86 GB. EchoMQ’s wake list is the small-scale echo: held to 64 elements with LTRIM … 0, 63, it is one compact node by design. The next dive reads the precise story of how that technique became core Redis — and who wrote it.