jonnify·redis /redis-patterns/coordination/hash-tag-colocation/crossslot-prevention

R2.05.2 · Hash-tag co-location — dive 2

CROSSSLOT prevention.

A multi-key command across two slots is refused. A shared tag lands the keys on one slot — and the command is legal again. Prevention, not recovery.

Grounding: echo/apps/echo_mq/lib/echo_mq/jobs.ex · @claim (jobs.ex:126) + claim/3 (jobs.ex:283) — KEYS = [emq:{q}:pending, emq:{q}:active], touches emq:{q}:job:<id>, all under the {q} hashtag.

The CROSSSLOT verdict — a multi-key command runs only when both keys hash to one slot. Pick a command and a tag state; hover or select a key row. The slots are computed by the real client-side slot rule.
MSET over two keys · cluster mode user:profileslot 1391 user:settingsslot 3618 slot 1391 ≠ 3618 → refused (error) CROSSSLOT Keys in request don't hash to the same slot
readout · Pick a command and toggle the shared {u42} tag. Without it the keys scatter; with it both hash u42 to one slot.
§1

The CROSSSLOT error

hash-tag-colocation.md.txt · Limitations

Valkey cluster splits the keyspace into 16384 hash slots and gives each slot to a node. A multi-key command can run atomically on one node only, so the cluster enforces a rule before it executes: every key the command names must hash to the same slot. When two keys in one request land on different slots they live on two different nodes, and no single node can run the command. The cluster refuses the whole request:

MSET user:profile "…" user:settings "…"
(error) CROSSSLOT Keys in request don't hash to the same slot

The trap is that a single Valkey node does not enforce this. On one node every key is reachable, so MSET, a MULTI/EXEC block, and a multi-key EVAL all run regardless of how the keys hash. The code passes every test on a single node, and the first cluster deploy is where the CROSSSLOT surfaces — a class of failure that stays invisible until the topology changes underneath working code. The hero figure runs the rule: pick a command, toggle the shared tag, and read back the slots and the verdict.

hash slot
One of 16384 partitions of the keyspace; each owned by one node. A key's slot is CRC16(key) % 16384 over the part the cluster hashes.
same-slot rule
Every key a single multi-key command names must hash to one slot, because the command runs atomically on the one node that owns it.
CROSSSLOT
The error the cluster returns when a multi-key request spans slots. A single Valkey node never raises it.
§2

Why colocation prevents it

hash-tag-colocation.md.txt · How hash tags work

A key's slot is CRC16(key) % 16384 over the whole key — unless the key carries a hash tag, the first non-empty {…} substring, in which case the cluster hashes only that substring. Two keys that share a tag therefore share a slot, whatever else differs between them. That is the prevention: arrange the related keys behind one tag and the multi-key command never spans a slot boundary.

# only the tag substring is hashed -> every {orders} key lands on one slot
slot("emq:{orders}:pending") == 105
emq:{orders}:pending   → slot 105
emq:{orders}:active    → slot 105
emq:{orders}:schedule  → slot 105   one slot → the multi-key claim stays legal

Drop the braces and each key hashes in full, so the keys scatter and the claim raises CROSSSLOT. Colocation is prevention — it removes the failure rather than recovering from it. That is the complement of the previous chapter's detection pattern: detection (R2.04) is the fallback for keys that genuinely cannot share a slot, such as a Valkey key paired with a database row; prevention is the first choice whenever the keys can be colocated.

A shared tag forces the related keys onto one slot, so a multi-key command that would span slots stays on a single node. Prevention removes the tear; detection only finds it.

§3

Commands that benefit from colocation

hash-tag-colocation.md.txt · Commands

Every command below requires same-slot keys in cluster mode, and a shared tag is what keeps them legal. A colocated key family turns each one back into a single atomic operation on one node:

CommandWhy it needs one slot
MULTI/EXECa transaction runs atomically on one node; every key in the block must share a slot.
WATCHthe optimistic-lock key and the keys the transaction touches must co-locate.
EVAL/EVALSHAa Lua script's multiple KEYS must all hash to one slot.
MGET/MSETa multi-key read or write spans one node only.
RENAMEsource and destination keys must share a slot.
COPYsource and destination keys must share a slot.

Notes on ValkeyIn cluster mode every KEYS element of a Lua script must hash to one slot, or the script is refused with CROSSSLOT — the same rule MSET and MULTI/EXEC obey — valkey.io/topics/cluster-spec.

§4

In EchoMQ — the multi-key claim

jobs.ex:126 · jobs.ex:283

EchoMQ's @claim script (echo/apps/echo_mq/lib/echo_mq/jobs.ex:126) is the worked case. The v2 law is that every key a script touches is declared in KEYS; claim/3 (jobs.ex:283) builds the key list and runs the script through EchoMQ.Connector.eval/5 (EVALSHA-first). The script touches three keys, and all three are built by Keyspace.queue_key/2, which always braces the queue name — so all three hash the substring q and answer one slot. Toggle a key in the figure to see which slot it answers.

The three keys of @claim — all under the {orders} hashtag, all on slot 105. Hover or select a key to read its declared role and the slot it answers.
@claim · declared keys, all hashing "orders" KEYS[1] emq:{orders}:pending ZPOPMINslot 105 KEYS[2] emq:{orders}:active ZADD leaseslot 105 job row emq:{orders}:job:<id> HINCRBYslot 105 3 keys → 1 slot (105) → one atomic EVALSHA
readout · Pick a key. All three are built by queue_key/2, so all three hash orders and answer slot 105.
# claim/3 (jobs.ex:283) — declared keys, all under the {q} hashtag
keys = [Keyspace.queue_key(queue, "pending"), Keyspace.queue_key(queue, "active")]
argv = [Keyspace.queue_key(queue, "job:"), Integer.to_string(lease_ms)]
Connector.eval(conn, @claim, keys, argv)

# @claim (jobs.ex:126) — ZPOPMIN pending, HINCRBY attempts (the fencing token), ZADD active on the server clock
local popped = redis.call('ZPOPMIN', KEYS[1])    -- emq:{q}:pending
if #popped == 0 then return {} end
local id = popped[1]
local jk = ARGV[1] .. id                          -- emq:{q}:job:<id>
local att = redis.call('HINCRBY', jk, 'attempts', 1)
redis.call('HSET', jk, 'state', 'active')
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)   -- emq:{q}:active
return {id, redis.call('HGET', jk, 'payload'), att}

The script applies in full or not at all: a job leaves pending (ZPOPMIN), its attempts fencing token is incremented (HINCRBY), and it enters active with a server-clock lease (ZADD … TIME) in one indivisible EVALSHA. Strip the {q} tag and the three keys would scatter, and the claim would raise CROSSSLOT. The braced keyspace is precisely what keeps the multi-key claim legal.

the pattern

A multi-key command whose keys span slots raises CROSSSLOT. Colocation via a shared {q} tag lands the keys on one slot, so the command is legal again — prevention, not recovery.

in EchoMQ

The multi-key @claim stays one atomic EVALSHA because Keyspace.queue_key/2 colocates emq:{q}:pending, emq:{q}:active, and the job row on one slot.

CROSSSLOT is the cluster refusing a multi-key command whose keys span slots. A shared hash tag is the prevention: it lands the keys on one slot, which is exactly why the multi-key @claim can be one atomic EVALSHA and never tears.

the application · the keyspace, gated against live Valkey

Three keys, one slot — the claim never tears

The colocation is not assumed — queue_key/2 always braces the queue name, and slot/1 computes the slot client-side over CRC16 % 16384 (valkey.io/topics/cluster-spec). The committed manuscript record froze the slot-locality:

content/bcs3.1.md · the co-location law, slot-locality against live Valkeyfrozen
emq:{orders}:* — pending, active, meta, job row   all answer slot 105   ← one queue, one slot
emq:{fills}:*                                  slot 4165
the hashtag IS the queue name — per-queue scripts are single-slot legal by grammar

The full Lua bundle, the EVALSHA / NOSCRIPT dispatch, and the version fence the connector negotiates before the first command are the subject of the dedicated EchoMQ course and the BCS state-machine chapter. This dive cites one script as proof of the colocation; it does not teach the bundle.

Next, R2.05.3 shows how slot/1 computes that slot client-side, so the connector routes and partitions without a server round trip — the same arithmetic the cluster runs, run ahead of time on the client.

§5

References

sources · related

Sources

Related in this course