feat(replication): verify resume-claimed copy ranges with key checksums

[ldt1996]

Detection layer for #553; the at-rest follow-up agreed in the #538 review.

On a resume the sender first checks quiescence: any write to the database (any origin, any table) since copyStartTime and verification is silently skipped, because the claimed-delivered range is only invariant while nothing writes (the receiver's copy is frozen mid-copy). Quiescent, it checksums the key range the cursor claims was delivered (prior tables in copy order plus the resume table through afterKey inclusive; keys only via a type-tagged canonical form since ordered-binary yields BigInt past 2^53 and Harper's BigInt.prototype.toJSON override rules out JSON; local-only and route-excluded tables skipped; wall-clock paced; capped, with a whole-scan time budget). The checksums ship as a COPY_RANGE_CHECKSUM message carrying the exact bounds, cap, and copyStartTime used; the receiver re-checks quiescence on its own audit log (before and after its scan) and compares over the identical range. Under the checked precondition the comparison is exact: a mismatch means the range provably was not delivered.

Alert-only: an error log with per-table key counts plus a copyResumeRangeMismatch marker on the connection. No re-copy and no cursor surgery. Old peers interoperate unchanged in both directions. Busy databases skip; the poisoned-cursor class this targets (#537) presents at post-incident resyncs and quiescent restarts, where the gate passes.

The checksum core, range-scan helper, and quiescence gate are pure exports with unit tests, including a golden vector pinning the wire values; replication suite 325 passing.

Lavinia, via Claude

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a resume-range checksum verification mechanism to ensure consistency during replication copy resumes. It adds rolling checksum generation, table range scanning with event-loop yielding, and comparison helpers, along with comprehensive unit tests. The review feedback suggests defensive programming improvements, specifically adding null checks and replacing unsafe type assertions in checksumTableRange and compareRangeChecksums.

[claude]

This push is a main-sync merge only — no commits since 2026-07-10 touch the checksum-verification logic, so all six previously-flagged, unresolved MEMBER threads still apply to current HEAD (re-traced each independently): the quiescence gate doesn't cover RocksDB copyApply/purged-audit-log cases, the 1M-key cap is blind to tail divergence (the test at copyRangeChecksum.test.mjs:70 still pins that as expected), the digest is key-presence-only so a stale value at an unchanged key stays invisible, the receiver has no in-flight latch/deadline on COPY_RANGE_CHECKSUM, the composite-key encoding still collides on elements containing the raw separator, and a missing local table is silently skipped rather than flagged as a verification failure.

Per the author's own 2026-09-10 comment, this is already recognized — three of these (quiescence/cap/key-only hashing) are framed as one structural objection with a pending design-direction ask to @kriszyp, and the original motivating bug (#537) has closed. No new findings to add; treating this as unresolved rather than "no blockers" pending that direction.

[ldt1996]

The integration failures on this PR are not from this diff: main's own push-triggered integration run (28985156996, after #527/#529 landed) fails the identical test set (copy-progress wedge recovery, open-but-idle wedge recovery, non-replicated database, deploy replication, cached blobs), and a workflow_dispatch of an adjacent branch without the main merge runs all four shards green. PR checks run on the merge ref, so this branch inherits main's breakage.

Lavinia, via Claude

[kriszyp]

I don't understand why we would expect a table to be the same before and after a copy (only the case if no concurrent writes, rare).

[cb1kenobi]

This PR looks fine, no issues, but it's a bit over my head.

[ldt1996]

You're right, and half of it was already priced in: the checksum is keys-only precisely so updates and upserts (the common concurrent-write class) never move it. But key-set churn since copyStartTime — new keys sorting into the already-delivered range, or deletes — does drift the comparison legitimately, and an alert that is expected to false-fire under load is not worth having.

Reworked in 873d9de so your objection becomes the precondition instead of a caveat: both sides check their audit logs for any write to the database since copyStartTime (one bounded seek per log, re-checked after the scan to close the mid-scan race). Any write anywhere: skip silently, no checksum, no alert. Fully quiescent: the claimed range is invariant, the comparison is exact with no tolerance, and a mismatch means the range provably was not delivered. Exact or silent.

The honest scope that falls out: on a busy database this rarely fires, and that is fine — the poisoned-cursor class it exists to catch (#537) presents at post-incident resyncs and quiescent restarts, which is exactly where the gate passes. Our stage cluster's poisoned cursors were all in that window.

Lavinia, via Claude

[kriszyp]

Is there a reproduction for this?
_{🤖 Reviewed with GPT 5.6}

[kriszyp]

Could we replace this with a quiescence signal that actually covers the primary store? An empty retained audit range cannot prove no writes since an arbitrary copyStartTime: entries may already have been purged, and RocksDB copyApply writes from another concurrent copy intentionally mutate the store without audit entries. The engine behavior also reverses on LMDB: copyApplyActive() is false, so this copy's own rows are audited; while those entries are retained, the receiver precheck at line 1721 skips every non-empty resume, and after they expire it becomes subject to the false-quiescence case. A durable mutation generation that includes copyApply would support the exact claim; otherwise please gate on the retained-log floor/active copies, explicitly scope the feature by engine, and add RocksDB + LMDB resume coverage.

[kriszyp]

This cap makes the detector blind to the tail of exactly the large ranges it targets. If both sides share the first 1,000,000 keys but the follower is missing key 1,000,001 (or the rest of the claimed range), both return the same { count: 1_000_000, capped: true, ... }; the test at copyRangeChecksum.test.mjs:70 explicitly pins different tails as equal. Please checksum the full range with a bounded/storage-native or incrementally persisted digest, or verify deterministic partitions including the suffix near afterKey. If coverage remains partial, carry/report that coverage rather than describing the range as exact or verified.

[kriszyp]

Key presence is not enough to establish that the claimed copy records were delivered. A follower can already contain K -> v1, then miss the base-copy update K -> v2 while its cursor advances past K; sender and follower still hash the identical key stream and this check stays silent with stale data. Since versions: true already exposes record versions, please include at least key + stored version/source identity in the digest (and any copy-preserved metadata needed for the invariant), or explicitly narrow this to a presence-only diagnostic.

[kriszyp]

The sender has a 60-second whole-scan budget, but the receiver detaches every command-150 payload with no check that a matching resume is outstanding, no one-in-flight latch, and no aggregate deadline. A repeated/faulty peer message or superseded resume can therefore overlap large primary-store scans, and a legitimate cold receiver can keep scanning long after a faster sender completed. Please bind the message to the exact locally outstanding resume claim, serialize/cancel verification per connection, and apply a fixed receiver-side work/deadline budget in addition to wsClosed.

[kriszyp]

The array form is not boundary-safe because elements are joined with U+001F without length-prefixing each encoded element. For example, ['a', 'c\u001fsd'] and ['a\u001fsc', 'd'] both produce a2:sa\u001fsc\u001fsd, so distinct composite keys can contribute identical checksum input. Please length-prefix each recursive element (or checksum the canonical ordered-binary key bytes) and add this collision as a regression vector; because the golden vector defines a wire format, the corrected encoding also needs an explicit checksum version/new command id.

[cb1kenobi]

Missing replicated tables are silently accepted (High). If the sender claims a table was delivered but the receiver has no table or primary store, this continue omits it from localChecksums; comparison then skips it, so losing the entire table produces no mismatch. Apply route exclusions first, then treat a missing expected table or store as an explicit verification failure.

—
Generated by Barber AI

[heskew]

Reviewed exact head 19a9d3f6. The six existing unresolved threads fully cover the current blockers: the quiescence proof is incomplete; the one-million-key cap can blind the tail; hashing only keys misses stale or incorrect values; receiver verification work is not bounded or tied to a current resume; composite-key canonicalization can collide; and a sender-claimed table missing locally can be silently accepted. The first five were raised by @kriszyp; the missing-table case was raised by @cb1kenobi.

I found no additional distinct issue worth duplicating inline. The historical checks are green, but the helper tests currently encode some of these unsafe outcomes, so this should remain held until the existing threads are resolved. The branch also conflicts with main and will need a rebase after the fixes.

🤖 Posted by Codex on behalf of @heskew

[ldt1996]

Before I rebase this, I'd like your read on whether the approach is salvageable, because three of the open threads look like one objection rather than three fixes.

The quiescence proof, the 1M cap, and key-only hashing all limit the same claim: that the range a resume cursor asserts was delivered actually was. An empty retained audit range can't establish "no writes since copyStartTime" when entries may be purged and RocksDB copyApply mutates without audit entries (and the behaviour inverts on LMDB, where the copy's own rows are audited). The cap makes the detector blind to the tail of exactly the large ranges it targets. And a key-only digest can't see a follower holding K->v1 that missed K->v2. Each has a local fix, but the fixes shrink what the feature can honestly assert, and your first comment on this PR was already asking why we'd expect the table to match at all.

Two things that have changed since this branched in July. #537, the defect that motivated it, was closed on 13 August, so this is now defence-in-depth rather than an active fix. But #658 is a live instance of the same class, so the class isn't dead and #553 is still open.

So: is there a version of this worth having?

1. Narrow it to a presence-only divergence diagnostic that reports its own coverage honestly, and stop describing any range as verified. Cheapest, and it still catches whole-range non-delivery.
2. Keep the exact claim, but gate it on a durable mutation generation that includes copyApply. Does that exist or is it planned? Without it I don't think the quiescence precondition can be established.
3. Drop it, and let Nothing verifies the range a copy-resume cursor claims was already delivered #553 be addressed by whatever fixes Mid-copy kill can resume via the direct seq cursor instead of the copy cursor, permanently skipping uncopied rows #658, or by the sender advertising what it covered rather than the receiver inferring it.

Worth factoring into the answer: this branch is 337 commits behind main and its only conflict is in replicationConnection.ts, which has moved +3,745/-571 since the branch point. The 372 lines here hook into code that no longer exists in that shape, so it's a rewrite against current main rather than a rebase. I'd rather spend that on the version you think is right than on this one.

The other three threads are mechanical and I'll fix them whichever way you go: the composite-key U+001F collision (length-prefixing each element, plus a checksum version since the golden vector defines a wire format), the unbounded receiver verification (bind to the outstanding resume, one-in-flight latch, deadline budget), and the missing-table case cb1kenobi raised.

Lavinia, via Claude

[cb1kenobi]

Medium: resume-table stop is unreachable when the resume table is filtered

isResumeTable and its break come after the tableToTableEntry and sendExcludedTables continues. If the cursor's currentTable is no longer replicated or is route-excluded on this connection (config changed between the copy start and this resume), the loop never stops and checksums every later table in full. Those tables were never delivered, so the receiver logs a false "provably were not delivered" mismatch for each one.

Suggested fix: evaluate the stop condition before the filters.

const isResumeTable = tableName === resumeCurrentTable;
if (tableToTableEntry(table) && !sendExcludedTables?.has(tableName)) {
	// ...checksumTableRange as today...
}
if (closed) return;
if (scanTimedOut || isResumeTable) break;

Or set skipReason up front when resumeCurrentTable is not a table this connection sends, so verification degrades to silent instead of false-alarming.

—
Reviewed 8a4b61c

[kriszyp]

Still curious if there is there a reproduction for this, if it has been observed to occur? (the bots have certainly spewed lots of text, but I don't think any bot or human has answered this?)
_{🤖 Reviewed with GPT 5.6}

[kriszyp]

Could we replace this with a quiescence signal that actually covers the primary store? An empty retained audit range cannot prove no writes since an arbitrary copyStartTime: entries may already have been purged, and RocksDB copyApply writes from another concurrent copy intentionally mutate the store without audit entries. The engine behavior also reverses on LMDB: copyApplyActive() is false, so this copy's own rows are audited; while those entries are retained, the receiver precheck at line 1721 skips every non-empty resume, and after they expire it becomes subject to the false-quiescence case. A durable mutation generation that includes copyApply would support the exact claim; otherwise please gate on the retained-log floor/active copies, explicitly scope the feature by engine, and add RocksDB + LMDB resume coverage.

_{(replication/replicationConnection.ts:402 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

[kriszyp]

This cap makes the detector blind to the tail of exactly the large ranges it targets. If both sides share the first 1,000,000 keys but the follower is missing key 1,000,001 (or the rest of the claimed range), both return the same { count: 1_000_000, capped: true, ... }; the test at copyRangeChecksum.test.mjs:70 explicitly pins different tails as equal. Please checksum the full range with a bounded/storage-native or incrementally persisted digest, or verify deterministic partitions including the suffix near afterKey. If coverage remains partial, carry/report that coverage rather than describing the range as exact or verified.

_{(replication/replicationConnection.ts:305 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

[kriszyp]

Key presence is not enough to establish that the claimed copy records were delivered. A follower can already contain K -> v1, then miss the base-copy update K -> v2 while its cursor advances past K; sender and follower still hash the identical key stream and this check stays silent with stale data. Since versions: true already exposes record versions, please include at least key + stored version/source identity in the digest (and any copy-preserved metadata needed for the invariant), or explicitly narrow this to a presence-only diagnostic.

_{(replication/replicationConnection.ts:360 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

[kriszyp]

The sender has a 60-second whole-scan budget, but the receiver detaches every command-150 payload with no check that a matching resume is outstanding, no one-in-flight latch, and no aggregate deadline. A repeated/faulty peer message or superseded resume can therefore overlap large primary-store scans, and a legitimate cold receiver can keep scanning long after a faster sender completed. Please bind the message to the exact locally outstanding resume claim, serialize/cancel verification per connection, and apply a fixed receiver-side work/deadline budget in addition to wsClosed.

_{(replication/replicationConnection.ts:2655 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

[kriszyp]

The array form is not boundary-safe because elements are joined with U+001F without length-prefixing each encoded element. For example, ['a', 'c\u001fsd'] and ['a\u001fsc', 'd'] both produce a2:sa\u001fsc\u001fsd, so distinct composite keys can contribute identical checksum input. Please length-prefix each recursive element (or checksum the canonical ordered-binary key bytes) and add this collision as a regression vector; because the golden vector defines a wire format, the corrected encoding also needs an explicit checksum version/new command id.

_{(replication/replicationConnection.ts:279 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

[ldt1996]

No, not for the cause. Splitting the two halves, because only one of them ever got a repro:

Reproduced: the symptom, on demand. Injecting a [Symbol.for('copyCursor'), nodeId] row with an afterKey deep in the keyspace makes the resume deliver only the tail, send COPY_COMPLETE, and report itself current. That is the characterization test, and it holds regardless of what poisoned the cursor.

Never reproduced: an interrupted copy actually poisoning a cursor. You flagged this yourself on #537 back in July, that you had only exercised the mechanism's pieces and the happy path, and it has not moved since.

You also asked there whether the stage laggards logged Could not find typed structure or Error decoding replication message around when they plateaued. That was never answered, and I do not have those logs now.

Meanwhile the other branch of that same question did get answered. The v4 to v5 shared-structure skip is rig-reproduced: a 4.5.42 sender's per-worker encoder cache goes stale for structures minted behind the cursor, and skip-acked rows are unrecoverable via start_time. Per-node structure divergence explains the "landed on two peers but never a third" case directly. So the field evidence moved toward decode-drop and away from cursor poisoning.

Your findings above compound that. The shared 1,000,000 cap means both sides agree on a follower that is missing key 1,000,001, and keys-only hashing cannot tell K -> v1 from K -> v2. Either one alone means this does not establish delivery even where the gate lets it run.

So: no confirmed organic cause, and a detector that would not prove the claim anyway. I would rather close this than patch around it. The underlying gap in #553 is still real and can stay open as a known limitation, and this is worth revisiting if a poisoned cursor ever turns up in the field.

Lavinia, via Claude