jonnify·redis /redis-patterns/coordination/cross-shard-consistency/torn-writes

R2.04.1 · Cross-shard consistency — dive 1

The torn write.

Two keys on two shards are two independent writes. One lands, one fails — and there is no transaction to undo the half that took. Pattern 1 catches it: stamp both values with one shared token, then compare the tokens on read.

Grounding: echo/apps/echo_mq · EchoMQ.Keyspace.queue_key (keyspace.ex:13) · EchoMQ.Jobs.claim → @claim (jobs.ex:126) — one slot, one Lua EVAL over co-located keys, so the queue cannot tear. The detector is the fallback for keys that cannot share a slot.

The tear — one logical write to two shards, stamped with token t7. Select whether the second shard's write succeeds; hover a write to read its result.
client · mint t7 shard 1 · user:profile t7:{profile} shard 2 · user:settings t7:{settings} t7 == t7 — consistent same logical write

Static verdicts — both succeed → profile and settings both carry t7, tokens match, consistent · shard-2 fails → settings keeps the previous t6, tokens disagree (t7 ≠ t6), torn write detected, repair triggered.

§1

The problem — torn writes

cross-shard-consistency.md.txt · The Problem

When two related keys A and B live on different shards, the pair of writes is not one atomic step. The client writes A to shard 1 and it succeeds; it writes B to shard 2 and it fails on a network error or a timeout. A now holds new data, B holds old data, and there is no transaction to roll back. A read that fetches both returns inconsistent state.

On a single primary a MULTI/EXEC block or a Lua script would make the pair atomic; across shards that guarantee is gone, so the application has to detect the tear after the fact. The cause is structural, not a bug in the write path. Two keys on two shards are two independent commands sent to two independent servers; an error on the second leaves the first already committed. Nothing links the two writes, so nothing can undo the first when the second fails. The damage is bounded only by detection — catch the mismatch on the next read, then repair.

Two keys on two shards are two independent writes. One can succeed while the other fails, and the pair is left inconsistent with nothing to undo it.

§2

Pattern 1 — the shared token

cross-shard-consistency.md.txt · Transaction Stamp

Generate one unique token per logical transaction and include it in every related write. A token is a UUID, a ULID, or random bytes — any value unique to this one logical update. Prefix each value with it, so the token travels inside the value rather than in any engine feature:

token = generate_random_token()            # UUID, ULID, or random bytes
SET user:123:profile  "{token}:{profile_json}"
SET user:123:settings "{token}:{settings_json}"

On read, fetch both values, parse their tokens, and compare. Equal tokens prove both values came from the same logical write; unequal tokens flag a tear. If the second SET failed, user:123:settings still carries the previous transaction's token, so the two disagree and the comparison catches it. The check is cheap read-time proof — one string compare — that needs no distributed transaction and no cluster feature, and the token doubles as the operation ID for idempotency.

token
A value unique to one logical write. It doubles as the operation ID, so a retried write is recognisable for idempotency.
{token}:{data}
The stored value: the token prefixed onto the payload. Parsing splits on the first colon to recover the token and the data.
independent SET
No transaction, no Lua — each write stands alone, which is exactly why the pair can tear and why no cluster feature is needed to compare them.

Two costs ride along. It is detection only, not prevention: the comparison flags the tear, but a recovery policy still has to follow. And a bare token cannot say which side is newer — equal or not is all it returns. The next dive adds a version to name the write to keep. Because the token travels in the value, the same check works whether the two keys sit on two cluster slots, two standalone primaries, or two different storage systems — a Valkey read-model and a relational database, say. Step through the comparison:

The shared-token check — parse both tokens, compare, name the stale side. Select whether settings is current or stale; hover a value to read its parsed token.
GET user:profile t7:{profile} GET user:settings t7:{settings} parse_token: t7 == t7 — match consistent · both payloads safe to use

Static verdicts — both written → parse_token(profile) = t7, parse_token(settings) = t7, match, return both payloads · settings stale → t7 ≠ t6, mismatch, settings is the older token, trigger_repair().

Parse both tokens and compare: equal proves the pair is whole, unequal flags the tear and names the value carrying the older token as the side to repair.

§3

Applied — why EchoMQ cannot tear

echo_mq/keyspace.ex:13 · jobs.ex:126

EchoMQ does not run this detector on its own queue keys, because it removes the tear before it can happen. Every per-queue key is built by EchoMQ.Keyspace.queue_key(q, type), which is "emq:{" <> q <> "}:" <> type — the queue name always lands inside the braces. Valkey Cluster hashes only the substring inside the first non-empty {...}, so emq:{orders}:pending and emq:{orders}:active (and the row key emq:{orders}:job:<id>) all hash to the same slot.

That colocation lets the whole transition be one inline Lua script. EchoMQ.Jobs.claim/3 (jobs.ex:283) hands Connector.eval/5 two declared keys — KEYS[1] the pending set, KEYS[2] the active set — and the @claim script (jobs.ex:126) touches the row key built from ARGV[1]. All three share the {orders} hashtag, so the move applies in full or not at all:

echo/apps/echo_mq/lib/echo_mq/jobs.ex:126 · the @claim script, verbatimfrozen
local popped = redis.call('ZPOPMIN', KEYS[1])      ← emq:{orders}:pending
if #popped == 0 then return {} end
local id = popped[1]
local jk = ARGV[1] .. id                           ← emq:{orders}:job:<id>
local att = redis.call('HINCRBY', jk, 'attempts', 1)
redis.call('HSET', jk, 'state', 'active')
redis.call('ZADD', KEYS[2], now + tonumber(ARGV[2]), id)  ← emq:{orders}:active
return {id, redis.call('HGET', jk, 'payload'), att}

The v2 law that makes this safe is in Connector.eval/5 (echo_wire/connector.ex:63): parts = ["EVALSHA", s.sha, Integer.to_string(length(keys))] ++ keys ++ argv — every key a script touches is declared in KEYS; ARGV carries values only. Hand that script a key on a different slot and the engine raises CROSSSLOT. There is no partial write to detect, because the script is the serialization point.

Notes on ValkeyValkey Cluster hashes only the substring inside the first non-empty {...}, modulo 16384, so co-located keys share a slot and a multi-key command stays legal — valkey.io/topics/cluster-spec.

the protocol · the colocation that removes the tear

One slot, no torn write — the next module, in depth

The colocation mechanic — the {tag} hash, CRC16 mod 16384, and the CROSSSLOT prevention — is the chapter's next module, R2.05 hash-tag colocation. The full v2 script bundle and the EVALSHA/NOSCRIPT dispatch are taught in depth in the dedicated EchoMQ course. So the token detector is the fallback for the case EchoMQ engineers away: keys on different shards, or a Valkey read-model paired with codemojex's database of record.

the pattern

Across shards there is no atomic multi-key write, so a torn pair is caught after the fact: stamp both values with one shared token and compare them on read. Portable across shards, standalone primaries, and unrelated systems.

its EchoMQ application

Colocation removes the tear. The {orders} hashtag puts every queue key on one slot (Keyspace.slot/1), so @claim stays atomic and never tears. A cross-slot EVAL raises CROSSSLOT.

Without one slot, a multi-key write can tear; a shared token is the cheapest way to catch it on read. EchoMQ keeps the one slot, so the token detector is its fallback only where keys genuinely cannot share one.

§4

Recap — detect, then repair

next → R2.04.2

A multi-key write off one slot can tear: one side lands, one fails, with nothing to roll back. The cheapest catch is a shared token in every related value, compared on read — equal means whole, unequal means torn. It is detection only, so a recovery policy follows. EchoMQ sidesteps it by keeping a queue's keys on one slot and the move in one Lua EVAL. What a bare token cannot do is name which side is newer — the next dive adds a version for that.

§5

References

sources · related

Sources

Related in this course