From 0e5a5234b9fe4a26244a0e52d3862916a7f4ce7e Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 30 Oct 2026 17:05:11 -0600 Subject: [PATCH 1/2] fix(db): retain failed demand during sibling replay --- .changeset/retain-loading-sibling-replay.md | 5 + docs/contributing/oracle-coverage.md | 20 ++ .../2026-10-30-release-replay-retention.md | 154 ++++++++ .../tests/multi-tab-electric-oracle.test.ts | 59 +++- packages/db/src/collection/subscription.ts | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 7 +- .../query/live/collection-config-builder.ts | 6 +- ...ad-subset-replay-refinement-oracle.test.ts | 330 +++++++++++++++++- 8 files changed, 555 insertions(+), 28 deletions(-) create mode 100644 .changeset/retain-loading-sibling-replay.md create mode 100644 docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md diff --git a/.changeset/retain-loading-sibling-replay.md b/.changeset/retain-loading-sibling-replay.md new file mode 100644 index 0000000000..64368eee0d --- /dev/null +++ b/.changeset/retain-loading-sibling-replay.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Keep the last complete live-query result when a new include demand fails while a sibling subscription is still replaying after truncate. A successful sibling load cannot publish an incomplete replacement. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 6e5ba7182e..0821c9962e 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -6304,3 +6304,23 @@ useful extension: the hash identity oracle owns those semantic distinctions, but its helper-level evidence does not by itself exercise distinct storage. The separate Index structural-collision obligation above remains open. See the appended [review evidence](oracle-reviews/2026-10-30-distinct-collisions.md). + +## Loading sibling replay and final catch-up reach (2026-10-30) + +The primary load-subset replay owner now constructs 48 finite histories where +an include with no demand at truncate starts under its sibling's loading replay. +It measures held failure (44 histories), failure retirement (33), and later own +coverage (12), through the existing public retention and recovery comparisons. +The original guard fails the quiet retention check. Subscription-scoped verdict +and successful-coverage calibrations remain in that primary owner. + +The browser multi-tab Electric owner preserves its stop prefix and arbitrary +tail and constructs an abandoned reservation immediately before final catch-up. +This removes dependence on sampling that boundary without changing its model, +80-run budgets, comparisons or reach minima. + +See [the versioned experiment record](oracle-reviews/2026-10-30-release-replay-retention.md) +for authority, RED, reach, mutant, grammar controls, historical gate limits and +receiving-path limits. This is core Collection publication evidence and a Node +SQLite/coordinator receiving witness with controlled Electric streams. It does +not claim Effect, provider, framework, deeper include, or browser-process closure. diff --git a/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md b/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md new file mode 100644 index 0000000000..7917412d03 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md @@ -0,0 +1,154 @@ +# Release replay retention and generated final catch-up + +The release workflow for `e75f8455a78bc33c43376288218e9912dc2a3f4c` +failed before publication. Its two failures have different causes. +The DB property exposed a production retention defect. The browser property +passed its behavior comparisons but missed a required state. Neither failure +establishes a regression in the callback-metadata size refactor: the DB +counterexample also fails on the exact release source. + +## Law and the first divergent observation + +Authority: live-query `ARCHITECTURE.md`, **Replay participants and failure**, +and normative law 9. A query's subscriptions to one source share publication. +Every acquisition started under that replay participates. A failed held demand +keeps the last complete result. Sibling success cannot decide its verdict. +Demand retirement or successful own replacement are the neighboring controls. + +The original `replay.ordered-failed-replay` counterexample uses seed +`-533188721`, path `428:3:3:6`. Its commands hold loads, truncate markers, +settle the two oldest loads successfully, fail the newest load, then remove +marker 1. The scenario has descending order, offset 1, limit 2, no ties or +peer, an anti-join and an inline include over on-demand markers. At quiet +loading, the expected retained snapshot is empty; production publishes row 1. + +The adapter receipts distinguish production failure from demand retirement. +The include held no demand at truncate. It later acquires marker key 1 while +the sibling join replay is loading. That include acquisition fails and remains +held and un-aborted. Its sibling later succeeds. Only cleanup unloads the +failed acquisition, after the mismatching observation. The old guard looked +only for a failed sibling replay. The repair also recognizes a loading replay +that still withholds publication. Ordinary failures outside replay retain their +existing request-error behavior. + +## Primary owner and experiment order + +`load-subset-replay-refinement-oracle.test.ts` remains the primary owner. +The change retains the subscription-scoped verdict and successful-coverage +corrections previously reviewed on PR 2312. Those corrections concern the +model, not a new production promise: shared publication does not merge replay +verdicts. Their two finite calibrations remain in the same owner. + +Before the runtime edit, a new constructive finite grammar produced the +intended RED on unchanged main. The failing assertion reports one measured new +sibling failure and the same empty-snapshot/row-1 mismatch. The first reach +recorder incorrectly excluded retired historical acquisitions; the corrected +pre-edit run measures acquisitions actually held at truncate. Both logs remain. + +The final grammar has 48 histories: two order directions, three widths, one or +two successful pre-failure settlements, and four later actions (none, marker +change, rank move, or another marker replay). Every history uses the existing +model, driver, retention comparison after each command, quiet checkpoint, and +final authoritative recovery. No expected row set was added to the histories. + +The initial repair trial passed all retention comparisons but failed the newly +declared own-coverage reach. The final recorder observes same-subscription key +coverage, rather than predicate object identity: a later demand may construct +new request data. The grammar also includes an explicit later replay control. +This post-trial reach work does not retroactively establish a pre-edit recovery +reach gate. The original retention RED and the receiving path were obtained +before production changed. + +Final measured reach: 44 histories with a first-demand failure under a loading +sibling; 33 with retirement of such failed demand; 12 with later successful +coverage by its own subscription. The three states belong to this constructive +grammar. Adding them to the shared result type initially made the unconstrained +campaign demand them by accident; the explicit owner assignment corrects that +without removing any pre-existing reach obligation. + +The original-guard mutant fails at the intended quiet retention comparison, +with the same one measured new sibling failure. It was restored in a `finally` +block and the repaired source hash was checked. Setup errors are recorded +separately and are not RED or mutant rejection evidence. + +## Browser grammar repair + +`multi-tab-electric-oracle.test.ts` owns the existing five browser laws. +The failed 80-history random campaign missed **reset-abandoned source recovered +during final catch-up**. Its fixed campaign reached that state four times. +A previous stop/peer prefix constructs other required states, but this state +still depended on an unconstrained 8–24-step tail. + +The repair keeps that prefix, tail, 80-run budgets, model, comparisons and reach +minimum. A suffix creates a fresh handle, recovers and catches up every tab, +delivers a new change to one tab without committing it, then resets its +reservation. The existing final delivery must observe the abandoned commit's +failure before restarting and checking convergence. Two/three tabs, source tab, +key and recovery method vary. Removing the partial delivery or reset loses the +premise; committing before final catch-up moves the failure to a different cut. +The original named final-delivery replays remain unchanged. + +## Review and limits + +This is an author self-review. The earlier independent diagnosis supplied the +receipt trace; it is not an independent review of the final implementation. + +- ORC-001/002/003/005/009: accepted architecture law, existing independent + row/retention model, literate grammar and real Collection sync boundary; + subscription identities and requested keys are adapter receipts. +- ORC-004/013/015: finite direction/width/settlement/change grammar reconstructs + the reported witness and adjacent controls. Absent replay is the ordinary + failure neighbor; obsolete or retired requests cannot count as held failures. + Browser suffix ablations lose its stated premise. Reach is measured. +- ORC-006/010: original guard rejected at the retention assertion; original + failure, initial reach-only failures and setup errors remain distinguishable. + Existing fixture cleanup remains in place. +- ORC-007: existing fixed/random properties and direct seed/path interface are + unchanged. The new 48-cell grammar is finite enumeration, not a random + property. Browser fixed/random campaigns retain the same 80-run grammar. +- ORC-008: no new semantic model state. New receipt fields serve reach only. + Retained subscription distinction is required because sibling success and + own success permit different verdicts. +- ORC-011: no new shared semantic fault hypothesis. Independent adapter trace + rejected the alternative retirement explanation of this failure. +- ORC-012/014/016: this record states the bounded claim and the unchanged + controlled premises. Histories use the shared comparison, with no unit-test + expected answers. + +Core evidence is limited to live-query Collection publication with one ordered +source, a one-level inline include, and the controlled core sync adapter. This +change does not claim Effect failure disposal, provider delivery, framework +scheduling, arbitrary joins or deeper recursion. Those paths have separate +owners; the modified builder failure path serves live-query Collections. +Browser evidence uses real SQLite, BroadcastChannel and coordinator code in one +Node process with controlled Electric streams. It does not establish installed +SDK/HTTP, real browser multi-process timing, on-demand row contents or existing +known-gap closure. No laws, assertions, run budgets or reach minima are weakened. + +## Final commands, results and source binding + +Local runs use saved configs under `/private/tmp/callback-metadata/release-replay-repair` to resolve the exact worktree source with cached dependencies. Every command sets `pnpm_config_verify_deps_before_run=false`; Vitest uses one worker. + +| Check | Command or selector | Result | +| --- | --- | --- | +| Pre-edit primary RED | `pnpm exec vitest run --config /vitest.config.mjs -t 'keeps new include failures behind a loading sibling replay'` | Intended quiet retention mismatch; measured first-demand sibling failure 1. | +| Final finite grammar | Same command | PASS, 48 histories, failure/retirement/own-recovery reach 44/33/12. | +| Original guard mutant | Same command with only the old guard restored | Intended quiet retention mismatch, measured failure 1. Repaired source restored and hash verified. | +| Full primary owner | `pnpm exec vitest run --config /vitest.config.mjs` | 43 passed, two accidental new-reach obligations failed. After explicit ownership correction, both affected fixed/random campaigns passed. All 45 tests pass across those runs. | +| Core neighbors | Saved `neighbors.config.mjs` | 21 tests in three files pass, covering replay refinement, failure boundaries and request errors. | +| Browser fixed/random campaigns | `pnpm exec vitest run --config /browser.config.mjs -t 'keeps reserved commits lawful'` | Both pass. Final catch-up recovery reached 80/80 histories in each campaign. | +| Browser final-delivery controls | Same config with the two existing final-delivery selectors | Both existing shared-model replays pass. | +| Focused TypeScript | `pnpm exec tsc --noEmit -p /db-types.json` and `browser-types.json` | Both pass. | + +The browser campaign retains 80 runs for each seed mode. Its expected injected stream errors are diagnostics, not comparison failures. The first two browser setup failures, the initial reach-only failures, and the intermediate TypeScript literal-inference error remain separate from RED/GREEN evidence. + +The following SHA256 fingerprints bind the executable evidence to this change. The commit containing this record supplies its immutable Git identity. + +| File | SHA256 | +| --- | --- | +| `packages/db/src/query/live/collection-config-builder.ts` | `bb1b62aaf00aed1dc5eaa5e537671d92bb99def7a53a6e951ef38f73b00f5088` | +| `packages/db/src/collection/subscription.ts` | `bd171bdc97b38a79ebbc81d7fb605267787308559ad4c0646dff1e072a43798a` | +| `packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts` | `e5ce4a77fe8d16f4b0d26f4c71fdadb53b2779c8e7e8313a7eb32d2de3ebdd38` | +| `packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts` | `67c16d284522417f1a77e0a36efcc552b718f858ea8ec00771f001f001160782` | + +Focused ESLint reports zero errors and two existing `no-shadow` warnings. Prettier, whitespace, and `pnpm lint:changed` pass. No run budget decreased or disappeared. diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index f109c45c08..341715569a 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -989,24 +989,53 @@ describe(`multi-tab Electric over one persisted database`, () => { }), steps: fc.array(historyStep, { minLength: 8, maxLength: 24 }), }) - .map(({ tabCount, source, key, recovery, stopOrder, steps }) => { - const tab = source % tabCount - const peer = (tab + 1) % tabCount - return { + .map( + ({ tabCount, - steps: [ - ...stopOrder.flatMap((pick): Array => [ + source, + key, + recovery, + stopOrder, + steps, + }): GeneratedHistory => { + const tab = source % tabCount + const peer = (tab + 1) % tabCount + return { + tabCount, + steps: [ + ...stopOrder.flatMap((pick): Array => [ + { kind: `server change`, key, pick: 0 }, + { kind: `advance`, tab, pick: 0 }, + { kind: `stop`, tab, pick }, + { kind: `catch up`, tab: peer, pick: 0 }, + { kind: `recover`, tab, pick: recovery }, + { kind: `catch up`, tab, pick: 0 }, + ]), + ...steps, + // Final catch-up must encounter an abandoned reservation in every + // generated history. Start a fresh handle, recover every tab, and + // deliver one new change without committing it. Reset then abandons + // that reservation. No later history step commits or recovers it; + // the existing final-delivery comparison must observe its failure + // before the existing recovery establishes convergence (Laws 2/5). + // The source tab, key and two/three-tab host vary with this grammar. + // Removing advance or reset loses that premise; catching up here + // would move the failure out of the final observation cut. + { kind: `deploy` }, + ...Array.from( + { length: tabCount }, + (_, index): Array => [ + { kind: `recover`, tab: index, pick: recovery }, + { kind: `catch up`, tab: index, pick: 0 }, + ], + ).flat(), { kind: `server change`, key, pick: 0 }, { kind: `advance`, tab, pick: 0 }, - { kind: `stop`, tab, pick }, - { kind: `catch up`, tab: peer, pick: 0 }, - { kind: `recover`, tab, pick: recovery }, - { kind: `catch up`, tab, pick: 0 }, - ]), - ...steps, - ], - } - }) + { kind: `peer reset` }, + ], + } + }, + ) type GeneratedHistory = { tabCount: number; steps: Array } type ServerLog = { diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 428dd2fac8..949e0eb4d4 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -871,7 +871,7 @@ export class CollectionSubscription /** * A lazy demand that failed outside this subscription's replay, while a - * failed replay of another of the graph's subscriptions to this collection + * replay of another of the graph's subscriptions to this collection still * holds publication, fails this subscription's replacement too. It keeps * the last publication until every demand load that joins this replay * loads, a later replay succeeds, or this subscription's last demand diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 6dc031ae33..a3a07c71d7 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1532,10 +1532,11 @@ A live query's subscriptions to one collection, such as a join and an include of the same source, share one publication barrier. None of their replays publishes, and none whose last demand retired closes, while another of them still has replay work, so demand that any of them starts meanwhile joins its -own open replay. While a failed replay of any of them holds publication, a +own open replay. While a replay of any of them still holds publication, a lazy demand of any of them that fails after initial-query readiness fails its -own subscription's replacement too, so retiring the other replay's failed -demand cannot publish the window without this demand's rows. It holds until +own subscription's replacement too. This includes a subscription that held no +demand at truncate. Settling a sibling's replay or retiring its failed demand +cannot publish the window without this demand's rows. It holds until every demand load that joins it loads, a later replay of the source succeeds, or the subscription's last demand retires. The query retries the failed demand at once and the retry joins the hold, so the hold ends only after the failed diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index ee60295643..0089c5a7be 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -596,8 +596,8 @@ export class CollectionConfigBuilder< if (this.liveQueryCollection?._lifecycle.status === `ready`) { // The failure settles the demand: the request errors of its keys show // it, and ordered work must not wait for a load that already ended. - // Output held for this demand publishes now, unless a failed replay of - // this source holds publication: the failure then holds this + // Output held for this demand publishes now, unless a replay of this + // source still holds publication: the failure then holds this // subscription's publication too, so retiring the other replay's // failed demand cannot publish without this demand's rows. demand.settled = true @@ -605,7 +605,7 @@ export class CollectionConfigBuilder< this.collectionSources.some( (source) => source.collection === collection && - this.subscriptions[source.sourceId]?.hasFailedTruncateReplacement, + this.subscriptions[source.sourceId]?.hasPendingTruncateReplacement, ) ) subscription.failReplacement(normalized) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 8a11da8ce3..6f94d276e3 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -165,7 +165,8 @@ import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' * node may list a failure, and the replay-error identity law skips peer * histories. The replay-error identity law judges only rejections with a failure of a replay * the move waited on; a move that resumes into another failed replay is not - * judged. + * judged. Each subscription owns its replay verdict, even when publication + * is shared with a sibling. Error-order comparisons keep that distinction. */ type OrderedFailureState = `initiating-keyed-request-failed` | `keyed-reacquisition-failed` @@ -316,6 +317,12 @@ type OrderedReach = { // Deferred window moves that rejected with a failure of a replay they // waited on. deferredMoveRejections: number + ownReplayFailureComparisons: number + siblingReplayFailureComparisons: number + siblingSuccessfulCoverage: number + newSiblingReplayFailures: number + newSiblingReplayRetirements: number + newSiblingReplayRecoveries: number replayLoads: number abortRejections: number preloadChecks: number @@ -463,6 +470,11 @@ async function checkOrderedReplay( error?: Error // Its source, the keys it asked for, and whether it is an ordered read. source?: BarrierSource + // Publication is shared, but each subscription owns its replay verdict. + subscription: LoadSubsetOptions[`subscription`] + // This subscription held no acquisition at truncate, then started demand + // during its sibling replay. Measured only from adapter receipts. + newSiblingReplay?: boolean keys?: Set ordered?: boolean } @@ -485,6 +497,8 @@ async function checkOrderedReplay( ambiguous: boolean pending: Map own: Set + // Acquisition receipts held at truncate, used only for declared reach. + subscriptionsAtTruncate: Set } const barriers = new Map() // Every barrier this history raised, in raise order. @@ -514,6 +528,18 @@ async function checkOrderedReplay( ambiguous: [...barriers.values()].some(({ ambiguous }) => ambiguous), pending: new Map(barriers.get(source)?.pending), own: new Set(), + subscriptionsAtTruncate: new Set( + [...acquired].flatMap(([options, count]) => + count > 0 && + loads.some( + (load) => + load.source === source && + load.subscription === options.subscription, + ) + ? [options.subscription] + : [], + ), + ), } barriers.set(source, barrier) raisedBarriers.push(barrier) @@ -540,6 +566,7 @@ async function checkOrderedReplay( barrier.own.add(load) return true } + let replayScopeViolation: string | undefined const settleParticipant = ( source: BarrierSource, load: Load, @@ -561,6 +588,13 @@ async function checkOrderedReplay( barrier.own.has(load) && !options.signal?.aborted ) { + if ( + load.newSiblingReplay && + [...barrier.pending.values()].some( + (pending) => pending.subscription !== options.subscription, + ) + ) + reach.newSiblingReplayFailures++ barrier.failures.set(load, options) barrier.history.push(load) } @@ -569,15 +603,33 @@ async function checkOrderedReplay( // failure no longer fails the replay (ARCHITECTURE.md law 17: a failure // counts for a key no later held load loaded). if (ok && barrier?.pending.has(load) && !options.signal?.aborted) { + // Calibration of the replay account: a successful sibling can supply the + // same source keys without deciding another subscription's verdict. + // Preserve those failure identities independently of the mutation below. + const siblingFailures = [...barrier.failures.keys()].filter( + (failed) => failed.subscription !== load.subscription, + ) for (const failed of barrier.failures.keys()) { if ( + failed.subscription !== load.subscription && loads.indexOf(failed) < loads.indexOf(load) && failed.ordered === load.ordered && failed.keys !== undefined && [...failed.keys].every((key) => load.keys?.has(key)) ) + reach.siblingSuccessfulCoverage++ + if ( + loads.indexOf(failed) < loads.indexOf(load) && + failed.subscription === load.subscription && + failed.ordered === load.ordered && + failed.keys !== undefined && + [...failed.keys].every((key) => load.keys?.has(key)) + ) { barrier.failures.delete(failed) + } } + if (siblingFailures.some((failed) => !barrier.failures.has(failed))) + replayScopeViolation ??= `successful sibling removed another subscription's replay failure` } barrier?.pending.delete(load) // A replay completes when its last participant settles, so demand that @@ -594,7 +646,11 @@ async function checkOrderedReplay( const failures = barrier.failures.size for (const owned of [barrier.pending, barrier.failures]) { for (const [load, held] of owned) { - if (held === options) owned.delete(load) + if (held === options) { + if (owned === barrier.failures && load.newSiblingReplay) + reach.newSiblingReplayRetirements++ + owned.delete(load) + } } } // One failed demand retires while another still holds the barrier. @@ -758,6 +814,19 @@ async function checkOrderedReplay( ) ) reach.successWhileSubscriptionOwes++ + // Later successful coverage by this subscription is a recovery control. + // Coverage by a sibling cannot decide this subscription's failure. + if ( + loads.some( + (earlier) => + earlier.newSiblingReplay && + earlier.outcome === `failed` && + earlier.source === source && + earlier.subscription === options.subscription && + [...(earlier.keys ?? [])].every((key) => keys.has(key)), + ) + ) + reach.newSiblingReplayRecoveries++ loadedKeys.push({ source, keys, at: ++requestClock }) } const recordRequestFailure = ( @@ -890,6 +959,12 @@ async function checkOrderedReplay( requestErrorChecks: 0, successWhileSubscriptionOwes: 0, deferredMoveRejections: 0, + ownReplayFailureComparisons: 0, + siblingReplayFailureComparisons: 0, + siblingSuccessfulCoverage: 0, + newSiblingReplayFailures: 0, + newSiblingReplayRetirements: 0, + newSiblingReplayRecoveries: 0, replayLoads: 0, abortRejections: 0, preloadChecks: 0, @@ -942,11 +1017,20 @@ async function checkOrderedReplay( // owner, whose release unloads it. acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { + subscription: options.subscription, epoch: truncates, outcome: `failed`, source: `markers`, keys: requestedKeys(`markers`, options), } + load.newSiblingReplay = + barriers.get(`markers`) !== undefined && + !barriers + .get(`markers`)! + .subscriptionsAtTruncate.has(options.subscription) && + [...(barriers.get(`markers`)?.pending.values() ?? [])].some( + (pending) => pending.subscription !== options.subscription, + ) loads.push(load) reach.markerLoads++ // The throw happens while the subscription registers the @@ -990,11 +1074,20 @@ async function checkOrderedReplay( const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { + subscription: options.subscription, epoch: truncates, outcome: `pending`, source: `markers`, keys: requestedKeys(`markers`, options), } + load.newSiblingReplay = + barriers.get(`markers`) !== undefined && + !barriers + .get(`markers`)! + .subscriptionsAtTruncate.has(options.subscription) && + [...(barriers.get(`markers`)?.pending.values() ?? [])].some( + (pending) => pending.subscription !== options.subscription, + ) loads.push(load) reach.markerLoads++ const participant = joinBarrier(`markers`, load, options) @@ -1146,6 +1239,7 @@ async function checkOrderedReplay( const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { + subscription: options.subscription, epoch: truncates, outcome: `pending`, source: `rows`, @@ -1367,7 +1461,12 @@ async function checkOrderedReplay( // failing, as a demand's replacement does, no longer count. A failure of a // load in no waited replay is the move's own window failure after it // resumed, which this law does not judge. + // The source-wide barrier combines subscriptions only for publication. + // Their verdicts remain separate: a sibling can fail while another replay + // still loads. Only failures of the rejecting subscription compete for its + // earliest error. Subscription identity comes from adapter request data. const covers = (later: Load, earlier: Load) => + later.subscription === earlier.subscription && later.ordered === earlier.ordered && later.keys !== undefined && earlier.keys !== undefined && @@ -1392,13 +1491,32 @@ async function checkOrderedReplay( ) if (candidates.length === 0) return reach.deferredMoveRejections++ + const rejected = loads.find((load) => load.error === error)! + for (const barrier of candidates) { + if ( + barrier.history.filter( + (load) => load.subscription === rejected.subscription, + ).length > 1 + ) + reach.ownReplayFailureComparisons++ + if ( + barrier.history.some( + (load) => load.subscription !== rejected.subscription, + ) + ) + reach.siblingReplayFailureComparisons++ + } const earliest = candidates.some((barrier) => barrier.history .slice( 0, barrier.history.findIndex((load) => load.error === error), ) - .every((earlier, at) => !stillDecides(barrier, earlier, at)), + .every( + (earlier, at) => + earlier.subscription !== rejected.subscription || + !stillDecides(barrier, earlier, at), + ), ) if (!earliest) moveErrorViolation ??= `move ${move} rejected with "${error.message}", not its replay's earliest failure that still decides it` @@ -1441,9 +1559,10 @@ async function checkOrderedReplay( const raised = [...barriers.values()] if (raised.some(({ ambiguous }) => ambiguous)) return if (live.status === `error`) return - expect(JSON.parse(JSON.stringify(live.toArray)), label).toEqual( - JSON.parse(retainedRows), - ) + expect( + JSON.parse(JSON.stringify(live.toArray)), + `${label}; new sibling replay failures: ${reach.newSiblingReplayFailures}`, + ).toEqual(JSON.parse(retainedRows)) if (raised.some(({ failures }) => failures.size > 0)) { reach.retainedFailedReplays++ } else reach.heldPublicationChecks++ @@ -1697,6 +1816,8 @@ async function checkOrderedReplay( if (moveOrderViolation) { throw new Error(`newest move wins: ${moveOrderViolation}`) } + if (replayScopeViolation) + throw new Error(`replay account scope: ${replayScopeViolation}`) if (moveErrorViolation) { throw new Error(`replay error identity: ${moveErrorViolation}`) } @@ -2429,6 +2550,190 @@ describe(`Ordered windows across truncate replays`, () => { ) }) + // A window with no initial include demand can activate its first include + // acquisition while a sibling join replay is still loading. Failure of that + // demand belongs to the shared publication barrier (law 9), even though its + // subscription had no replay of its own at the truncate. Adapter receipts + // prove both premises. This bounded grammar varies order, width, prior + // settlements and later demand changes; it reuses every ordinary observation + // and the independent retention account above. A move can retire the failed + // demand; a later truncate can replace it with successful own coverage. + it(`keeps new include failures behind a loading sibling replay`, async () => { + const states = [ + `newSiblingReplayFailures`, + `newSiblingReplayRetirements`, + `newSiblingReplayRecoveries`, + ] as const + const declared = createOracleReach(`replay.new-sibling-demand`, states) + for (const direction of [`desc`, `asc`] as const) { + for (const limit of [2, 1, 3] as const) { + for (const settles of [2, 1]) { + for (const change of [`mark`, `none`, `move`, `replay`] as const) { + declared.beginHistory() + const observed = await checkOrderedReplay( + [ + { type: `hold` }, + { type: `truncateMarkers` }, + ...Array.from({ length: settles }, () => ({ + type: `settle` as const, + ok: true, + keepHolding: true, + newest: false, + })), + { type: `settle`, ok: false, newest: true, keepHolding: true }, + ...(change === `none` + ? [] + : change === `replay` + ? [{ type: `truncateMarkers` as const }] + : change === `mark` + ? [{ type: `mark` as const, id: 1, on: false }] + : [ + { + type: `move` as const, + id: 1, + rank: direction === `desc` ? 60 : 1, + }, + ]), + ], + { + ...ordinaryOrderedScenario, + direction, + offset: 1, + limit, + joined: `none`, + includes: true, + markerMode: `on-demand`, + }, + ) + for (const state of states) { + if (observed[state] > 0) declared.reach(state) + } + } + } + } + } + console.info( + `replay.new-sibling-demand`, + Object.fromEntries(declared.counts()), + ) + declared.assertReached() + }, 120_000) + + // This finite history grammar varies which source replays, the order of + // two load failures, join form, direction and window width. Join plus include + // gives sibling subscriptions; the root replay gives same-subscription + // failures. The same comparison checks both, without an expected error + // copied from production or a source-wide error order. + it(`keeps replay error order within each subscription`, async () => { + const declared = createOracleReach( + `replay.subscription-error-order`, + [ + `ownReplayFailureComparisons`, + `siblingReplayFailureComparisons`, + ] as const, + { minimum: 5 }, + ) + for (const direction of [`asc`, `desc`] as const) { + for (const joined of [false, `some`, `none`, `inner`] as const) { + for (const limit of [1, 2, 3] as const) { + for (const markers of [false, true]) { + for (const newest of [false, true]) { + declared.beginHistory() + const observed = await checkOrderedReplay( + [ + { type: `hold` }, + markers + ? { type: `truncateMarkers` } + : { type: `truncate`, armAfterStart: false }, + { type: `window`, offset: 0, limit }, + { type: `settle`, ok: false, newest }, + { + type: `settle`, + ok: false, + newest: !newest, + keepHolding: false, + }, + ], + { + ...ordinaryOrderedScenario, + direction, + joined, + limit, + markerMode: `on-demand`, + includes: true, + }, + ) + for (const state of [ + `ownReplayFailureComparisons`, + `siblingReplayFailureComparisons`, + ] as const) { + if (observed[state] > 0) declared.reach(state) + } + } + } + } + } + } + console.info( + `replay.subscription-error-order`, + Object.fromEntries(declared.counts()), + ) + declared.assertReached() + }, 120_000) + + // Failure followed by successful coverage is the neighboring regime of + // the two-failure grammar above. Real adapter subscriptions supply identity; + // the shared account comparison requires a sibling's failure to survive. + it(`retains sibling replay failures through successful coverage`, async () => { + const declared = createOracleReach( + `replay.subscription-success-scope`, + [`siblingSuccessfulCoverage`] as const, + { minimum: 1 }, + ) + for (const direction of [`asc`, `desc`] as const) { + for (const joined of [false, `some`, `none`, `inner`] as const) { + for (const limit of [1, 2, 3] as const) { + for (const markers of [false, true]) { + for (const newest of [false, true]) { + declared.beginHistory() + const observed = await checkOrderedReplay( + [ + { type: `hold` }, + markers + ? { type: `truncateMarkers` } + : { type: `truncate`, armAfterStart: false }, + { type: `window`, offset: 0, limit }, + { type: `settle`, ok: false, newest, keepHolding: true }, + { + type: `settle`, + ok: true, + newest: !newest, + keepHolding: false, + }, + ], + { + ...ordinaryOrderedScenario, + direction, + joined, + limit, + markerMode: `on-demand`, + includes: true, + }, + ) + if (observed.siblingSuccessfulCoverage > 0) + declared.reach(`siblingSuccessfulCoverage`) + } + } + } + } + } + console.info( + `replay.subscription-success-scope`, + Object.fromEntries(declared.counts()), + ) + declared.assertReached() + }, 120_000) + it.each(orderedReplayCampaigns)( `keeps the ordered window under load and release faults (%s campaign)`, async (campaign) => { @@ -2455,6 +2760,12 @@ describe(`Ordered windows across truncate replays`, () => { requestErrorChecks: 0, successWhileSubscriptionOwes: 0, deferredMoveRejections: 0, + ownReplayFailureComparisons: 0, + siblingReplayFailureComparisons: 0, + siblingSuccessfulCoverage: 0, + newSiblingReplayFailures: 0, + newSiblingReplayRetirements: 0, + newSiblingReplayRecoveries: 0, replayLoads: 0, abortRejections: 0, preloadChecks: 0, @@ -2493,6 +2804,13 @@ describe(`Ordered windows across truncate replays`, () => { `discardedReplayMoves`, `requestErrorChecks`, `deferredMoveRejections`, + `ownReplayFailureComparisons`, + `siblingReplayFailureComparisons`, + `siblingSuccessfulCoverage`, + // The constructive new-sibling grammar declares these rare premises. + `newSiblingReplayFailures`, + `newSiblingReplayRetirements`, + `newSiblingReplayRecoveries`, `successWhileSubscriptionOwes`, ]) // Counts are histories, rather than repeated events in one trace. From ec25fc4f117c0b20fa0827f33b2a400a6e77a36a Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 30 Oct 2026 17:32:06 -0600 Subject: [PATCH 2/2] Scope replay request-error oracle to each subscription --- docs/contributing/oracle-coverage.md | 14 +++++ .../2026-10-30-release-replay-retention.md | 63 +++++++++++++++++++ ...ad-subset-replay-refinement-oracle.test.ts | 45 ++++++++++--- 3 files changed, 115 insertions(+), 7 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 0821c9962e..648c33badc 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -6324,3 +6324,17 @@ for authority, RED, reach, mutant, grammar controls, historical gate limits and receiving-path limits. This is core Collection publication evidence and a Node SQLite/coordinator receiving witness with controlled Electric streams. It does not claim Effect, provider, framework, deeper include, or browser-process closure. + + +### Subscription-scoped request-error assertion (2026-10-30) + +The primary load-subset replay owner now retains subscription identity in its +later-request and successful-coverage account. A sibling requesting overlapping +source keys cannot exempt the failed demand's public health error. Its existing +96-history sibling-success grammar measures the actual owed-error assertion in +four histories, alongside four successful-sibling-coverage histories. A targeted +omitted-error observation passes the former source-wide account and fails the +scoped health comparison. All existing comparisons, histories, budgets and +reach minima remain. The controlled core, one-level include and peer limits +remain those of this owner; no provider or framework claim follows. See the +[versioned repair record](oracle-reviews/2026-10-30-release-replay-retention.md#subscription-scoped-request-error-checker-repair). diff --git a/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md b/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md index 7917412d03..9e1b624ba0 100644 --- a/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md +++ b/docs/contributing/oracle-reviews/2026-10-30-release-replay-retention.md @@ -152,3 +152,66 @@ The following SHA256 fingerprints bind the executable evidence to this change. T | `packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts` | `67c16d284522417f1a77e0a36efcc552b718f858ea8ec00771f001f001160782` | Focused ESLint reports zero errors and two existing `no-shadow` warnings. Prettier, whitespace, and `pnpm lint:changed` pass. No run budget decreased or disappeared. + +## Subscription-scoped request-error checker repair + +CodeRabbit review 5719714297 covered repair head +`0e5a5234b9fe4a26244a0e52d3862916a7f4ce7e`. Its one behavioral +finding identified a false-green request-error classification. Architecture +law 16 and the glossary's request-error rule already scope a lazy demand's +error to that demand. A sibling subscription asking for or loading the same +source keys cannot decide it. Shared publication does not merge verdicts. +The existing complete authoritative source-recovery control still decides the +held demands together; an individual sibling verdict does not supply it. + +The old account stored successful keys by source alone. Both its later-request +and later-coverage exemptions omitted subscription identity. The repair keeps +that identity in the recorded successful acquisition and requires it in both +exemptions. This is an oracle enforcement repair. Production is unchanged. + +The existing 96-history sibling-success grammar varies direction, join form, +width, replay source and settlement order. A receipt probe found four histories +with eight observation cuts where the old account exempted an error the scoped +account owed. Production listed the exact injected error at all eight cuts. +The cut concerns a later sibling request; the experiment did not establish a +later successful sibling-coverage cut with an otherwise owed request error. +An initial reach counter for that stronger premise correctly failed at zero. +It remains in the scratch evidence. The final declared reach names the actual +sibling-request error comparison and reaches four histories. The existing +successful-sibling-coverage reach also remains four. + +Calibration supplies a wrong public observation to the same health comparison: +omit the injected error only at those distinguishing cuts. The old account +accepts it. The scoped account rejects it at `after settle: request error`, +with expected true and actual false. A broader temporary production mutant +that suppressed every ready-query replay error already failed the old checker; +it is not evidence for this newly repaired gap. Both temporary sources were +restored. The real production owner passes all 45 tests, including fixed and +random campaigns, with the stronger classifier. The reported seeds and paths +from issues 2294 and 2315 each pass the direct replay interface as well. These +are bounded replay results, not closure of either whole issue or provider +adapter conformance from issue 2118. + +ORC review (author self-review): 001/002/003 use the accepted demand-scoped +law, independently evaluated request keys and explanatory local prose. +004/013 retain all 96 legal histories and their direction/join/width/source/ +settlement neighbors; collapsing subscription identity is the wrong rule. +005/006/015 exercise the real Collection sync path and public health error +identity after each command, measure four distinguishing histories and reject +the targeted omission. 007 retains all fixed/random campaigns, budgets and +seed/path replays. 008/009 retain subscription identity because equal source +keys can have different demand verdicts. 010 retains primary mismatch and +cleanup separation. 011 has no new shared production/model semantic-fault +hypothesis: the defect was an exemption before the assertion. 012 is this +versioned bounded record. 014 keeps the controlled core boundary and makes no +provider/framework handoff claim. 016 adds no unit expectation or standalone +fixture. No law, comparison, budget or prior reach minimum was weakened. + +The checker evidence is stored under +`/private/tmp/callback-metadata/release-replay-repair/review-5719714297`. +Commands use the saved `vitest.config.mjs` for the full primary owner and +`-t 'retains sibling replay failures through successful coverage'` for the +finite calibration, with one worker and dependency verification disabled. +The amended primary owner SHA256 is `a2ab454ff5a77aa47bf57cbd590b67a2a5602d3dd6897af051fb1819b9482ce7` on both branches. The earlier +table binds the original repair evidence. Its runtime and browser hashes remain +unchanged; callback runtime hashes remain bound by its separate port record. diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 6f94d276e3..7ad2fdfd13 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -320,6 +320,8 @@ type OrderedReach = { ownReplayFailureComparisons: number siblingReplayFailureComparisons: number siblingSuccessfulCoverage: number + // Owed request-error comparisons after another subscription asked for its keys. + siblingRequestErrorChecks: number newSiblingReplayFailures: number newSiblingReplayRetirements: number newSiblingReplayRecoveries: number @@ -749,18 +751,19 @@ async function checkOrderedReplay( // request error on the query's own node. The error stays until one of these // meets it: // - its demand retires, which aborts its request and unloads it; - // - a later load of the source asks for some of its keys: that newer + // - a later load of the same subscription asks for some of its keys: that newer // request of the demand decides them, and keys it no longer asks for // left the demand (ARCHITECTURE.md law 16: a lazy demand's error clears // when that demand settles or is released; a key's latest load decides // it); - // - a later load of the same source covers its keys; + // - a later load of the same subscription covers its keys; // - a replay of its source that rose after the failure completes, since a // replay asks again for every held demand. // A failure that joins a replay waits for its subscription's verdict, // which comes once that subscription's participants settled, so it owes // no error before then. The barrier that holds publication stays shared - // across the query's subscriptions to the source. Keys come from evaluating the request's + // across the query's subscriptions to the source. A sibling's successful + // load cannot decide this demand's error. Keys come from evaluating the request's // `where` over a small key domain, independent of production's bookkeeping. // The account combines the production concepts of a failed request and the // replay verdict that decides it; it names no production structure. @@ -790,6 +793,7 @@ async function checkOrderedReplay( const requestFailures: Array = [] const loadedKeys: Array<{ source: BarrierSource + subscription: LoadSubsetOptions[`subscription`] keys: Set at: number }> = [] @@ -827,7 +831,12 @@ async function checkOrderedReplay( ) ) reach.newSiblingReplayRecoveries++ - loadedKeys.push({ source, keys, at: ++requestClock }) + loadedKeys.push({ + source, + subscription: options.subscription, + keys, + at: ++requestClock, + }) } const recordRequestFailure = ( load: Load, @@ -868,6 +877,7 @@ async function checkOrderedReplay( (later, index) => index > at && later.source === failure.source && + later.subscription === failure.options.subscription && !later.ordered && [...failure.keys].some((key) => later.keys?.has(key)), ) @@ -875,6 +885,7 @@ async function checkOrderedReplay( const covered = loadedKeys.some( (loaded) => loaded.source === failure.source && + loaded.subscription === failure.options.subscription && loaded.at > failure.at && [...failure.keys].every((key) => loaded.keys.has(key)), ) @@ -962,6 +973,7 @@ async function checkOrderedReplay( ownReplayFailureComparisons: 0, siblingReplayFailureComparisons: 0, siblingSuccessfulCoverage: 0, + siblingRequestErrorChecks: 0, newSiblingReplayFailures: 0, newSiblingReplayRetirements: 0, newSiblingReplayRecoveries: 0, @@ -1442,6 +1454,19 @@ async function checkOrderedReplay( if (owed.length === 0) return reach.requestErrorChecks++ for (const failure of owed) { + // Reach is measured at the health assertion, after a sibling asked + // for overlapping keys that cannot decide this subscription's error. + if ( + loads.some( + (later, index) => + index > loads.indexOf(failure.load) && + later.source === failure.source && + later.subscription !== failure.options.subscription && + !later.ordered && + [...failure.keys].some((key) => later.keys?.has(key)), + ) + ) + reach.siblingRequestErrorChecks++ expect( [live, peer].some((query) => query?.health.error?.errors.some( @@ -2687,7 +2712,7 @@ describe(`Ordered windows across truncate replays`, () => { it(`retains sibling replay failures through successful coverage`, async () => { const declared = createOracleReach( `replay.subscription-success-scope`, - [`siblingSuccessfulCoverage`] as const, + [`siblingSuccessfulCoverage`, `siblingRequestErrorChecks`] as const, { minimum: 1 }, ) for (const direction of [`asc`, `desc`] as const) { @@ -2720,8 +2745,12 @@ describe(`Ordered windows across truncate replays`, () => { includes: true, }, ) - if (observed.siblingSuccessfulCoverage > 0) - declared.reach(`siblingSuccessfulCoverage`) + for (const state of [ + `siblingSuccessfulCoverage`, + `siblingRequestErrorChecks`, + ] as const) { + if (observed[state] > 0) declared.reach(state) + } } } } @@ -2763,6 +2792,7 @@ describe(`Ordered windows across truncate replays`, () => { ownReplayFailureComparisons: 0, siblingReplayFailureComparisons: 0, siblingSuccessfulCoverage: 0, + siblingRequestErrorChecks: 0, newSiblingReplayFailures: 0, newSiblingReplayRetirements: 0, newSiblingReplayRecoveries: 0, @@ -2807,6 +2837,7 @@ describe(`Ordered windows across truncate replays`, () => { `ownReplayFailureComparisons`, `siblingReplayFailureComparisons`, `siblingSuccessfulCoverage`, + `siblingRequestErrorChecks`, // The constructive new-sibling grammar declares these rare premises. `newSiblingReplayFailures`, `newSiblingReplayRetirements`,