jonnify·redis /redis-patterns/data-modeling/memory-optimization/listpack-and-intset

R7.02.1 · dive 1

A small collection is a byte array.

Small collections do not cost what they look like they cost. Redis stores a small Hash as a listpack — one sequential byte array, no pointers — and a small all-integer Set as an intset.

Both layouts avoid the pointers and buckets a general hash table needs, and both are chosen automatically below the configured thresholds. EchoMQ's three-field job HASH is the worked case: it is a listpack because the row is small, and the listpack is most of the memory win this chapter is about. Select a layout; the readout reports what it stores and what it costs.

Two layouts The thresholds The job HASH References
Three layouts for a small collection. Select one; the readout reports what it stores and what it costs.
listpack one array intset sorted ints hashtable + pointers
§1

Two layouts for the same data

memory-optimization.md.txt · listpack vs hashtable

A Hash can be stored two ways. The hashtable encoding is the general structure: a bucket array, a chain of entries per bucket, and a pointer for every key and value — fast at any size, but each entry carries pointer and allocation overhead that, for a small object, can be 80% of the bytes. The listpack encoding packs all the fields end to end into one contiguous byte array: length-prefixed entries, no per-entry pointer, no bucket array. A read walks the array; for a few fields that is cheaper than chasing pointers, and the memory is a fraction.

An all-integer Set has a third, tighter layout: the intset — a sorted array of fixed-width integers (16-, 32-, or 64-bit, widened only when a member needs it). No hashing, no pointers, only a packed sorted array a membership test binary-searches. The engine picks the layout by size and switches when a structure crosses a configured threshold.

A small Hash is a listpack and a small all-integer Set is an intset — a packed byte array with no pointer or bucket overhead, chosen for you below the thresholds.

§2

The thresholds, and the flip

memory-optimization.md.txt · the one-way conversion

These are Valkey engine defaults — read from the documentation, not set in EchoMQ's config:

hash-max-listpack-entries 512
hash-max-listpack-value 64
set-max-intset-entries 512
zset-max-listpack-entries 128

A Hash stays a listpack while it has fewer than 512 fields and every value is under 64 bytes; cross either bound and it converts to a hashtable. A Set stays an intset while it has fewer than 512 members and every member is an integer. The conversion is one-way — adding the element that crosses the line flips the encoding, and removing it does not flip it back. Step the worked Hash past a bound and watch the encoding change:

Cross a threshold, flip the encoding. Start at the 3-field job HASH, then add a 513th field or push a value to 65 bytes; the encoding flips to hashtable and stays there.
— the limits: 512 entries / 64 bytes
entries vs hash-max-listpack-entries 512 largest value vs hash-max-listpack-value 64 bytes encoding: listpack

Notes on ValkeyOBJECT ENCODING <key> reports a key's internal encoding — listpack, intset, or hashtable — so the conversion is observable; the thresholds hash-max-listpack-entries / hash-max-listpack-value / set-max-intset-entries are configurable but the flip is one-way — valkey.io/topics/memory-optimization.

Stay under both bounds and the structure is compact; cross either and it converts to a hashtable for good. The cheapest way to keep the listpack is to keep the structure small.

§3

The job HASH is a listpack

echo_mq/jobs.ex · the 3-field row

EchoMQ's job row is a Hash with exactly three fields, written by the @enqueue script in one atomic step:

echo/apps/echo_mq/lib/echo_mq/jobs.ex · @enqueuefrozen
-- @enqueue (EchoMQ.Script.new(:enqueue, …)) — the row write, verbatim
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

Three fields is far under 512; state is a short word, attempts is a bare integer, payload is a body that is small for the operational jobs EchoMQ runs — all under 64 bytes in the common case. So Valkey stores the row as a listpack. The committed infra/valkey/conf/valkey.conf sets no encoding thresholds, so this is the engine default at work on a deliberately minimal row — the win is the design, not a tuned knob.

the pattern

A small Hash is a listpack and a small all-integer Set is an intset — a sequential byte array with no pointer or bucket overhead, chosen automatically below the thresholds.

its EchoMQ application

The 3-field job HASH (state/attempts/payload) sits far under hash-max-listpack-entries 512 with values under hash-max-listpack-value 64, so Valkey keeps it a listpack — the compact encoding by construction, not configuration.

The cheapest memory optimization is to keep a structure small enough to stay in its compact encoding. EchoMQ never tunes the thresholds because its row is three short fields — a listpack the engine stores it as by default.

R7.02.2 · in this course

Short field names

The values stay under hash-max-listpack-value because the fields are short. The readable-vs-byte-tight trade, and why EchoMQ's state/attempts/payload are short but not abbreviated, is the next dive: short field names.

/bcs/store

EchoStore — the compact near-cache

The same compact encodings keep a near-cache's hot copies small. EchoStore — L1 ETS over L2 Valkey — is the Branded Component System's account of the tier that holds copies, where the memory budget is the live working set.

§4

References

sources · related

Sources

Related in this course