Module 6 · Dive 6.3
Replica catch-up & the feed.
A replica behind at some LSN pulls the segments committed after it from object storage and applies them — the same fold as Dive 6.1, with bytes from the wire — until its lag is zero. And the change feed is not a separate thing: it is this ordered stream of commits, handed to any subscriber that asks for “everything after LSN k.” Commit on the primary, pull on the replica, watch the lag close.
Push segments, pull to catch up
interactiveThe feed is catch-up, productized
graft.design.mdA replica and a feed subscriber are the same client wearing different hats. Both say “I’m at LSN k, give me what’s after”; both receive the ordered segments and apply — or react to — each one. The only differences are intent and lifetime: a replica wants a full copy and runs forever; a subscriber may want just the deltas and may stop. That is why the engine ships one stream and exposes it as both replication and the change feed. The bytes are the Zstd-framed page segments uploaded behind the conditional-write fence from Module 3, so a replica reads exactly what the primary committed, in order — deterministic catch-up, no divergence. With this, Chapter II is complete: a durable, versioned, replicated local store. Chapter III opens the two engines that implement it.
References & sources
- Echo records
- graft.design.mdreplica catch-up, segments, change feed
- graft.engine-split.design.mdsegment format, the feed in both engines
- External
- orbitinghail/graftpull-based catch-up, the change feed
- Change data capturethe log as a stream of changes
- Designing Data-Intensive Applications, Kleppmann 2017replication logs, CDC (Ch. 5, 11)