From 2099cb238a0495832e643ecca02c22b87f646131 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 09:36:50 -0600 Subject: [PATCH 01/40] fix(db): keep a lazy demand that fails a truncate replay from ending its live query A marker truncate privately moved an ordered window, so an include's demand changed keys. The demand controller released the old segment first; that was the subscription's last demand, so its truncate replay ended, and the new demand started outside it. Its failure then ended the query with error 228. - Start new demand coverage before releasing retired coverage. - Open a replay at every truncate for a query subscription that acquires, even without demand, so demand the commit starts belongs to it. - A demand failure that fails its subscription's replay records the error and keeps the last publication instead of ending the query. The replay-refinement oracle now classifies recovery loads by request data (replay loads ask again with the same `where`) and by the commit window, and requires the campaigns to reach replay loads. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/collection/subscription.ts | 13 ++- packages/db/src/query/live/ARCHITECTURE.md | 19 +++- .../query/live/collection-config-builder.ts | 27 +++-- .../src/query/live/collection-subscriber.ts | 5 +- .../query/live/subset-demand-controller.ts | 64 ++++++----- ...ad-subset-replay-refinement-oracle.test.ts | 101 +++++++++++++++--- 7 files changed, 176 insertions(+), 55 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 5afacbd874..fabf473470 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a replay; named replays reach that branch and the aborted-move branch. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand, and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query. Named replays keep the trace and a variant that starts from no include demand; reverting each of the three production changes fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. Limit: the generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 9027af7a4f..1c8e19166e 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -401,11 +401,18 @@ export class CollectionSubscription this.collection._sync.syncLoadSubsetFn !== null // Deferred demand never acquired, so a truncate gives it nothing to // reload; it acquires the replacement rows when acquisition resumes. - const demandsToReload = - reloadsSubsets && !this.defersAcquisition ? [...this.subsetDemands] : [] + const acquires = reloadsSubsets && !this.defersAcquisition + const demandsToReload = acquires ? [...this.subsetDemands] : [] // Retained rows still need the committed replacement even without demand. - if (demandsToReload.length === 0 && this.stalePublishedRows.size === 0) { + // A query subscription that acquires opens the replay without demand too: + // the commit's private deletes can make its graph start demand, and that + // demand belongs to this replay. + if ( + demandsToReload.length === 0 && + this.stalePublishedRows.size === 0 && + !(acquires && this.options.truncateReplayPublication) + ) { this.resetSnapshotTracking() return } diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 86eae32fb9..8520a71a50 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -739,6 +739,9 @@ old constant, even after its first load settles: deduplication and query state may retain its identity. Request data uses stable data properties, not stateful getters. The signal and subscription references do not change, but their lifecycle remains live. Cancellation and release are not data mutations. +A truncate replay asks again for each retained demand with that demand's +request data, so an adapter sees the same `where` expression it received +before the truncate. The immutable-demand boundary matrix checks direct and deferred sync startup, adapter return and asynchronous settlement, cancellation, and release identity. @@ -884,7 +887,10 @@ applied settlement. A failed or obsolete replacement cannot retire established coverage, and a retry requests the same union again. A partial shrink may keep one intersecting acquisition without replacement. This avoids repeatedly loading the growing prefix while allowing a changing bounded window to return -to one acquisition. +to one acquisition. The controller starts new coverage before it releases an +acquisition that no longer intersects current demand. A demand change +therefore never leaves its subscription without logical demand partway +through, which would end that subscription's truncate replay. Every sync `commit()` returns an applied receipt: `true` when that transaction's writes and events are already visible, or a promise when the @@ -1432,6 +1438,17 @@ normalized once by the subscription. The `loadSubset:error` event, `lastSubsetError`, and any window move waiting on that replay expose the same `Error` object. +A truncate's commit deletes the source's rows privately, so the graph can +demand keys that the query never published. A query subscription that acquires +from an on-demand source therefore opens a replay at every truncate, even when +it holds no demand. Lazy demand that the graph starts while that replay is open +belongs to it. A lazy demand load whose failure fails its subscription's replay +does not end the live query. The query records the error and keeps its last +complete publication until a later truncate replay of that source succeeds, +which reacquires the retained demand. A lazy demand load that fails outside a +replay ends the query (error 228): the parent rows it serves may already be +public, and no barrier keeps their incomplete join side or bucket private. + ### Mutation boundaries and initial readiness A transaction `mutationFn` must not start or await collection or live-query diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index a8bf06e336..e5b1b446de 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -487,17 +487,30 @@ export class CollectionConfigBuilder< ) } - failDemand(planId: string, generation: number, error: unknown): void { + /** + * A lazy demand load failure ends the query unless it failed a truncate + * replay: that replay keeps the last publication until a later replay of + * the source reacquires the demand. A failed demand is settled either way, + * so it holds no ordered loading. + */ + failDemand( + planId: string, + generation: number, + error: unknown, + fatal = true, + ): void { const demand = this.activeDemands.get(planId) if (!demand || demand.generation !== generation) return + demand.settled = true const normalized = this.recordSubsetError(error) demand.participant?.reject(normalized) - this.transitionToError( - devBuild() && process.env.NODE_ENV !== `production` - ? `Subset demand '${planId}' failed: ${normalized.message}` - : codedMessage(228, { planId, cause: normalized.message }), - normalized, - ) + if (fatal) + this.transitionToError( + devBuild() && process.env.NODE_ENV !== `production` + ? `Subset demand '${planId}' failed: ${normalized.message}` + : codedMessage(228, { planId, cause: normalized.message }), + normalized, + ) } recordSubsetError(error: unknown, fatalBeforeReady = false): Error { diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index 36e1c68d43..6c01a41f29 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -225,7 +225,9 @@ export class CollectionSubscriber< () => this.collectionConfigBuilder.settleDemand(plan.id, generation), (error) => { // A truncate replay aborts the acquisition it supersedes. Its - // replacement decides the demand; a real failure rejects it too. + // replacement decides the demand. A failure that fails the + // replay keeps the last publication until a later replay of this + // source succeeds, so it does not end the query. const replacement = subscription.pendingTruncateReplacement if (replacement) settleWith(replacement) else @@ -233,6 +235,7 @@ export class CollectionSubscriber< plan.id, generation, error, + !subscription.hasFailedTruncateReplacement, ) }, ) diff --git a/packages/db/src/query/live/subset-demand-controller.ts b/packages/db/src/query/live/subset-demand-controller.ts index 0b64a2585d..e4077f433b 100644 --- a/packages/db/src/query/live/subset-demand-controller.ts +++ b/packages/db/src/query/live/subset-demand-controller.ts @@ -79,20 +79,6 @@ export class SubsetDemandController { const state: DemandState = { keys: nextKeys, segments } this.states.set(plan.id, state) - // Release every retired segment even if subscriber code that a release - // runs throws, then report the first error. - runAllCallbacks( - retired.map((segment) => () => releaseSegment(subscription, segment)), - ) - if (this.states.get(plan.id) !== state) { - return { changed: false, empty: false, ready: true } - } - - if (nextKeys.size === 0) { - this.states.delete(plan.id) - return { changed: true, empty: true, ready: true } - } - const coveredKeys = new Set( segments.flatMap((segment) => [...segment.keys.keys()]), ) @@ -105,16 +91,38 @@ export class SubsetDemandController { const replace = retryReplacement || (removed && (added.size > 0 || segments.length > 1)) const requestedKeys = replace ? nextKeys : added - if (requestedKeys.size > 0) { - const segment = createSegment(plan, requestedKeys) - if (replace) segment.replaces = [...segments] - segments.push(segment) - const started = startSegment(subscription, segment, () => - this.warnUnoptimized(plan), - ) - if (this.states.get(plan.id) !== state) { - return { changed: false, empty: false, ready: true } - } + const newSegment = + requestedKeys.size > 0 ? createSegment(plan, requestedKeys) : undefined + // The start callback assigns this; the cast keeps it a plain boolean. + let started = true as boolean + // Start the new segment before releasing retired ones. Releasing a + // subscription's last demand ends its truncate replay, so demand started + // after that release would fall outside the replay that owns it. Release + // every retired segment even if the start or subscriber code that a + // release runs throws, then report the first error. + runAllCallbacks([ + () => { + if (!newSegment) return + if (replace) newSegment.replaces = [...segments] + segments.push(newSegment) + started = startSegment(subscription, newSegment, () => + this.warnUnoptimized(plan), + ) + }, + ...retired.map( + (retiredSegment) => () => releaseSegment(subscription, retiredSegment), + ), + ]) + if (this.states.get(plan.id) !== state) { + return { changed: false, empty: false, ready: true } + } + + if (nextKeys.size === 0) { + this.states.delete(plan.id) + return { changed: true, empty: true, ready: true } + } + + if (newSegment) { if (!started) throw new Error( devBuild() && process.env.NODE_ENV !== `production` @@ -122,13 +130,13 @@ export class SubsetDemandController { : codedMessage(159), ) if (replace) { - if (segment.ready instanceof Promise) { - void segment.ready.then( - () => this.finishReplacement(subscription, plan.id, segment), + if (newSegment.ready instanceof Promise) { + void newSegment.ready.then( + () => this.finishReplacement(subscription, plan.id, newSegment), () => {}, ) } else { - this.finishReplacement(subscription, plan.id, segment) + this.finishReplacement(subscription, plan.id, newSegment) } } } diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 32ec955ad6..22cf8bcfee 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1095,12 +1095,19 @@ describe(`loadSubset replay refinement`, () => { * convergence does not apply. Either side can be the lazy one: the planner * loads an inner join's root by key when the marker source is eager. A * demand load asks for keys (a `where` with no `orderBy`). A failed load - * that a truncate's replay issued is not fatal: it keeps the last - * publication until the next replay. A load that fails before the query - * first becomes ready is fatal too (an initial load failure), except on - * a lazy join source, which owns its demand. So a query may end in - * `error` only after a demand load issued outside a replay failed, or a - * load failed while the query was still loading. + * that belongs to a truncate's recovery is not fatal: the replay fails, and + * the query keeps its last publication until a later replay of that source + * (law 9). Two kinds of load belong to a recovery: + * - a replay load, which asks again for a demand the source held before + * the truncate; and + * - a demand load that the graph starts while the truncate commits. The + * commit deletes the source's rows privately, so the query can demand + * keys that it never published, such as a row that a LEFT join filter + * now keeps. + * A load that fails before the query first becomes ready is fatal too (an + * initial load failure), except on a lazy join source, which owns its + * demand. So a query may end in `error` only after a demand load outside a + * recovery failed, or a load failed while the query was still loading. * - **Status:** with no failed load, a quiet live query and its peer are * `ready`; a release that throws does not change status. * - **Reported faults:** an injected fault may surface as a rejected load, a @@ -1244,6 +1251,7 @@ type OrderedReach = { markerTruncates: number abortedMoves: number terminalErrors: number + replayLoads: number } const ordinaryOrderedScenario: OrderedScenario = { @@ -1341,12 +1349,13 @@ async function checkOrderedReplay( let throwNextUnload = false // Window moves whose settlement waits on held loads. const pendingMoves: Array> = [] - // The truncate starts its replay while it commits. + // The source truncate is committing. let truncating = false let anyLoadFailed = false let earlyThrowSinceTruncate = false - // A marker truncate's commit starts its replay loads synchronously. - let markerReplaying = false + // The marker truncate is committing, and how many marker truncates ran. + let markerCommitting = false + let markerTruncates = 0 let demandFailed = false // A failed initial load is fatal too (error 229). The queries are created // below; a load can only fail after they start. @@ -1357,6 +1366,29 @@ async function checkOrderedReplay( // Lazy join demand asks for keys; ordered and full-source loads do not. const isDemand = (options: LoadSubsetOptions) => options.where !== undefined && options.orderBy === undefined + // The truncate count of its source when each request was first issued. A + // replay asks again for a held demand with the same request data, because + // request data is immutable (ARCHITECTURE.md), and a new demand builds a + // new `where`. So a load whose `where` was first issued before its source's + // latest truncate is a replay load. + const firstIssued = new WeakMap() + // A load belongs to a recovery when it is a replay load, or when the graph + // starts it while its source's truncate commits. + const belongsToRecovery = ( + options: LoadSubsetOptions, + sourceTruncates: number, + committing: boolean, + ) => { + const where = options.where + if (!where) return committing + if (!firstIssued.has(where)) firstIssued.set(where, sourceTruncates) + const replay = firstIssued.get(where)! < sourceTruncates + // The campaigns require replay loads, so production that rebuilt request + // data for a replay fails this witness instead of making every replay + // load look like new demand. + if (replay) reach.replayLoads++ + return committing || replay + } let violation: string | undefined // The faults this history injects. They may surface as rejected loads, as // thrown releases, or as errors the subscription rethrows from a microtask, @@ -1395,6 +1427,7 @@ async function checkOrderedReplay( markerTruncates: 0, abortedMoves: 0, terminalErrors: 0, + replayLoads: 0, } type Marker = { id: number; rowId: number } let markerSync!: Parameters[`sync`]>[0] @@ -1430,7 +1463,9 @@ async function checkOrderedReplay( const load: Load = { epoch: truncates, outcome: `pending` } loads.push(load) reach.markerLoads++ - const demand = !markerReplaying && isDemand(options) + const demand = + !belongsToRecovery(options, markerTruncates, markerCommitting) && + isDemand(options) if (holding) { const ok = await new Promise((resolve) => held.push(resolve), @@ -1494,7 +1529,9 @@ async function checkOrderedReplay( acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { epoch: truncates, outcome: `pending` } loads.push(load) - const demand = !truncating && isDemand(options) + const demand = + !belongsToRecovery(options, truncates, truncating) && + isDemand(options) if (holding) { const ok = await new Promise((resolve) => held.push(resolve), @@ -1688,12 +1725,13 @@ async function checkOrderedReplay( installedMarkers.add(id) } } - markerReplaying = true + markerTruncates++ + markerCommitting = true let receipt: true | Promise try { receipt = markerSync.commit() } finally { - markerReplaying = false + markerCommitting = false } if (receipt !== true) await receipt } @@ -2081,6 +2119,39 @@ describe(`Ordered windows across truncate replays`, () => { markerMode: `on-demand`, }, }, + { + name: `a failed include demand that a marker truncate starts`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `none`, + markerMode: `on-demand`, + includes: true, + }, + reaches: `failedLoads`, + }, + { + name: `a failed include demand started from no prior demand`, + history: [ + { type: `mark`, id: 2, on: true }, + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `none`, + markerMode: `on-demand`, + includes: true, + }, + reaches: `failedLoads`, + }, ] satisfies Array<{ name: string history: Array @@ -2119,6 +2190,7 @@ describe(`Ordered windows across truncate replays`, () => { markerTruncates: 0, abortedMoves: 0, terminalErrors: 0, + replayLoads: 0, } await fc.assert( fc.asyncProperty( @@ -2140,7 +2212,8 @@ describe(`Ordered windows across truncate replays`, () => { ) // Reach witnesses: releases throw without a truncate, while a replay // starts, and while it settles; loads fail and stay held across a - // truncate; and the settled-window law is checked. + // truncate; truncates ask again for held demand; and the settled-window + // law is checked. // An aborted move needs a truncate to land inside a held move, and a // terminal error needs a held marker demand load to fail; the named // replays above reach both. From ed89a31e526f9a0dc6b590d60115d1c8240fbca5 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 14:05:44 -0600 Subject: [PATCH 02/40] fix(db): treat aborted lazy demand as obsolete and keep initial failures fatal Review fixes on top of the replay-failure change: - A recovery failure before the live query first becomes ready still ends it, so preload no longer waits for a later truncate. - A replay opened without demand starts no recovery work and reports no loading, so a zero-limit ordered window issues no full-source read. - A rejected aborted lazy demand request is obsolete demand: its truncate replay decides it even when the replay settled first, and a segment that newer coverage replaced settles. Both cases ended the query on main when the adapter rejected an aborted request with AbortError. The replay-refinement oracle now generates histories during the initial preload, aborted marker loads that reject, and newest-first settles, with an initial-readiness law; the ordered-lifecycle oracle checks truncate work. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/collection/subscription.ts | 16 +- packages/db/src/query/live/ARCHITECTURE.md | 24 ++- .../query/live/collection-config-builder.ts | 10 +- .../query/live/subset-demand-controller.ts | 19 ++- ...ad-subset-replay-refinement-oracle.test.ts | 145 ++++++++++++++++-- .../ordered-lifecycle-oracle.property.test.ts | 81 ++++++++++ 7 files changed, 265 insertions(+), 32 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index fabf473470..8dbf9d94bf 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand, and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query. Named replays keep the trace and a variant that starts from no include demand; reverting each of the three production changes fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. Limit: the generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 1c8e19166e..47330f26b6 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -405,14 +405,12 @@ export class CollectionSubscription const demandsToReload = acquires ? [...this.subsetDemands] : [] // Retained rows still need the committed replacement even without demand. - // A query subscription that acquires opens the replay without demand too: + const recovers = + demandsToReload.length > 0 || this.stalePublishedRows.size > 0 + // A query subscription that acquires opens a replay without demand too: // the commit's private deletes can make its graph start demand, and that - // demand belongs to this replay. - if ( - demandsToReload.length === 0 && - this.stalePublishedRows.size === 0 && - !(acquires && this.options.truncateReplayPublication) - ) { + // demand belongs to the replay. Such a replay starts no recovery work. + if (!recovers && !(acquires && this.options.truncateReplayPublication)) { this.resetSnapshotTracking() return } @@ -437,13 +435,13 @@ export class CollectionSubscription this.truncateReplayState = replayState } const attempt = replayState.currentAttempt - this.setStatus(`loadingSubset`) + if (recovers) this.setStatus(`loadingSubset`) if (this.truncateReplayState !== replayState) return if (this.options.truncateReplayPublication) { this.truncateReplacementPending = true - this.options.truncateReplayPublication.start() + if (recovers) this.options.truncateReplayPublication.start() } // A newer replay replaces every prior acquisition for these demands. Abort diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 8520a71a50..1964155d69 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1441,13 +1441,23 @@ that replay expose the same `Error` object. A truncate's commit deletes the source's rows privately, so the graph can demand keys that the query never published. A query subscription that acquires from an on-demand source therefore opens a replay at every truncate, even when -it holds no demand. Lazy demand that the graph starts while that replay is open -belongs to it. A lazy demand load whose failure fails its subscription's replay -does not end the live query. The query records the error and keeps its last -complete publication until a later truncate replay of that source succeeds, -which reacquires the retained demand. A lazy demand load that fails outside a -replay ends the query (error 228): the parent rows it serves may already be -public, and no barrier keeps their incomplete join side or bucket private. +it holds no demand. A replay without demand starts no recovery work and reports +no loading state. Lazy demand that the graph starts while that replay is open +belongs to it, including demand started after the replay failed while it still +holds publication. A rejection of an aborted lazy demand request is obsolete demand, +not a load failure, as for the subscription. When a truncate supersedes the +acquisition, its replay decides the demand, even if the replay settles before +the aborted request rejects. A segment that newer coverage replaced settles. A +lazy demand load whose failure fails its subscription's replay +does not end the live query once it has first become ready. The query records +the error and keeps its last complete publication until a later truncate +replay of that source succeeds, which reacquires the retained demand. Before +the first ready there is no complete publication to keep, so the failure is an +initial load failure and ends the query. A lazy demand load that fails outside +a replay ends the query (error 228): the parent rows it serves may already be +public, and no barrier keeps their incomplete join side or bucket private. A +synchronous `loadSubset` throw rolls its logical owner back, so it fails no +replay that could reacquire the demand, and it ends the query too. ### Mutation boundaries and initial readiness diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index e5b1b446de..8e0633821e 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -489,9 +489,11 @@ export class CollectionConfigBuilder< /** * A lazy demand load failure ends the query unless it failed a truncate - * replay: that replay keeps the last publication until a later replay of - * the source reacquires the demand. A failed demand is settled either way, - * so it holds no ordered loading. + * replay after the query first became ready: that replay keeps the last + * publication until a later replay of the source reacquires the demand. + * Before the first ready there is no publication to keep, so the failure + * is an initial load failure. A failed demand is settled either way, so it + * holds no ordered loading. */ failDemand( planId: string, @@ -504,7 +506,7 @@ export class CollectionConfigBuilder< demand.settled = true const normalized = this.recordSubsetError(error) demand.participant?.reject(normalized) - if (fatal) + if (fatal || this.liveQueryCollection?.status === `loading`) this.transitionToError( devBuild() && process.env.NODE_ENV !== `production` ? `Subset demand '${planId}' failed: ${normalized.message}` diff --git a/packages/db/src/query/live/subset-demand-controller.ts b/packages/db/src/query/live/subset-demand-controller.ts index e4077f433b..0918b31aea 100644 --- a/packages/db/src/query/live/subset-demand-controller.ts +++ b/packages/db/src/query/live/subset-demand-controller.ts @@ -273,9 +273,24 @@ function startSegment( signal: segment.abortController.signal, trackLoadSubsetPromise: false, onUnoptimized, - onLoadSubsetResult: (result) => { + onLoadSubsetResult: (result, options) => { observed = true - segment.ready = result + // An aborted request is obsolete demand, not a load failure, as for the + // subscription. A truncate replay decides a segment whose acquisition + // it superseded, even when the replay settled first. Otherwise newer + // coverage replaced the segment, and it settles. + segment.ready = + result instanceof Promise + ? result.catch((error: unknown) => { + if (!options.signal?.aborted) throw error + return ( + subscription.pendingTruncateReplacement ?? + (subscription.hasFailedTruncateReplacement + ? Promise.reject(subscription.lastError) + : undefined) + ) + }) + : result segment.state = result instanceof Promise ? `pending` : `settled` }, }) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 22cf8bcfee..5b067202aa 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1110,6 +1110,10 @@ describe(`loadSubset replay refinement`, () => { * recovery failed, or a load failed while the query was still loading. * - **Status:** with no failed load, a quiet live query and its peer are * `ready`; a release that throws does not change status. + * - **Initial readiness:** when the history starts during the initial + * preload, that preload has resolved or rejected once loading is quiet. A + * failure before the first ready ends the query, so the preload cannot wait + * for a later truncate. * - **Reported faults:** an injected fault may surface as a rejected load, a * thrown release, a window move rejected by a load failure, or an error the * subscription rethrows from a microtask. A window move whose load a @@ -1117,8 +1121,13 @@ describe(`loadSubset replay refinement`, () => { * window move. No other error may escape. * * History grammar: rank moves, holding the next loads, settling the oldest - * held load with success or failure, truncates (optionally arming the next - * unload once the replay has started), and arming the next unload to throw. The driver uses a real on-demand Collection with an index and an + * or newest held load with success or failure (optionally keeping later + * loads held), + * truncates (optionally arming the next unload once the replay has started), + * and arming the next unload to throw. A history may start while the initial + * preload's loads are held. A marker load whose request was aborted either + * resolves without writing or rejects with `AbortError`, as the source + * contract asks; an `AbortError` is not a failed load. The driver uses a real on-demand Collection with an index and an * ordered live query. At the end, every held load succeeds in order. * * The joined scenarios cross a LEFT join that keeps marked or unmarked rows @@ -1137,7 +1146,11 @@ type OrderedRow = { id: number; rank: number } type OrderedCommand = | { type: `move`; id: number; rank: number } | { type: `hold` } - | { type: `settle`; ok: boolean } + // `keepHolding` leaves later loads held, so a history can settle one load + // while the next stays pending. `newest` settles the most recently held + // load instead of the oldest, so a later load can settle before an earlier + // one, as independent provider requests may. + | { type: `settle`; ok: boolean; keepHolding?: boolean; newest?: boolean } | { type: `truncate`; armAfterStart: boolean } | { type: `unloadThrows` } | { type: `window`; offset: 0 | 1; limit: 1 | 2 | 3 } @@ -1161,6 +1174,12 @@ type OrderedScenario = { markerMode: `eager` | `on-demand` // Whether each shown row includes its markers as a `toArray` child. includes: boolean + // How a marker load whose request was aborted settles. The source contract + // asks a source to reject it with `AbortError` once its accepted rows apply; + // a source that cannot cancel may also resolve it without writing. + abortedMarkerLoads: `resolve` | `reject` + // Whether the history starts while the initial preload is still held. + historyDuringPreload: boolean } const orderedScenario: fc.Arbitrary = fc.record({ @@ -1177,6 +1196,8 @@ const orderedScenario: fc.Arbitrary = fc.record({ ), markerMode: fc.constantFrom(`eager` as const, `on-demand` as const), includes: fc.boolean(), + abortedMarkerLoads: fc.constantFrom(`resolve` as const, `reject` as const), + historyDuringPreload: fc.boolean(), }) const orderedCommand: fc.Arbitrary = fc.oneof( @@ -1197,6 +1218,8 @@ const orderedCommand: fc.Arbitrary = fc.oneof( { weight: 3, arbitrary: fc.constant(true) }, { weight: 1, arbitrary: fc.constant(false) }, ), + keepHolding: fc.boolean(), + newest: fc.boolean(), }), }, { @@ -1252,6 +1275,8 @@ type OrderedReach = { abortedMoves: number terminalErrors: number replayLoads: number + abortRejections: number + preloadChecks: number } const ordinaryOrderedScenario: OrderedScenario = { @@ -1263,6 +1288,8 @@ const ordinaryOrderedScenario: OrderedScenario = { joined: false, markerMode: `eager`, includes: false, + abortedMarkerLoads: `resolve`, + historyDuringPreload: false, } async function checkOrderedReplay( @@ -1386,7 +1413,7 @@ async function checkOrderedReplay( // The campaigns require replay loads, so production that rebuilt request // data for a replay fails this witness instead of making every replay // load look like new demand. - if (replay) reach.replayLoads++ + if (replay && isDemand(options)) reach.replayLoads++ return committing || replay } let violation: string | undefined @@ -1428,6 +1455,8 @@ async function checkOrderedReplay( abortedMoves: 0, terminalErrors: 0, replayLoads: 0, + abortRejections: 0, + preloadChecks: 0, } type Marker = { id: number; rowId: number } let markerSync!: Parameters[`sync`]>[0] @@ -1483,7 +1512,13 @@ async function checkOrderedReplay( } else await Promise.resolve() if (options.signal?.aborted) { load.outcome = `ok` - return + if (scenario.abortedMarkerLoads === `resolve`) return + reach.abortRejections++ + throw inject( + Object.assign(new Error(`marker load aborted`), { + name: `AbortError`, + }), + ) } operations.begin() for (const id of marked) { @@ -1666,6 +1701,9 @@ async function checkOrderedReplay( if (peer) queries.push(peer) // A subscriber keeps each live query from garbage collection while a // history waits on held loads. + // A history during preload starts with every load held, including the + // loads that the first subscriber starts. + holding = scenario.historyDuringPreload const subscriptions = [live, peer].flatMap((collection) => collection ? [collection.subscribeChanges(() => {})] : [], ) @@ -1735,9 +1773,13 @@ async function checkOrderedReplay( } if (receipt !== true) await receipt } + // The history's commands can interleave with the initial loads. + let preloadSettled = false + const preload = Promise.allSettled([live.preload(), peer?.preload()]).then( + () => (preloadSettled = true), + ) try { - await live.preload() - await peer?.preload() + if (!scenario.historyDuringPreload) await preload for (const command of history) { try { switch (command.type) { @@ -1769,8 +1811,8 @@ async function checkOrderedReplay( holding = true break case `settle`: { - holding = false - held.shift()?.(command.ok) + holding = command.keepHolding === true && holding + ;(command.newest === true ? held.pop() : held.shift())?.(command.ok) break } case `truncate`: @@ -1864,6 +1906,13 @@ async function checkOrderedReplay( await Promise.all(pendingMoves.splice(0)) await settleAll() await flushPromises() + // Initial readiness: once loading is quiet, the initial preload has + // settled. It resolves when the first window is covered and rejects when + // an initial load fails; it does not wait for a later truncate. + if (scenario.historyDuringPreload) { + reach.preloadChecks++ + expect(preloadSettled, `initial preload settles`).toBe(true) + } if (violation) throw new Error(`adapter release: ${violation}`) if (moveOrderViolation) { throw new Error(`newest move wins: ${moveOrderViolation}`) @@ -2152,11 +2201,84 @@ describe(`Ordered windows across truncate replays`, () => { }, reaches: `failedLoads`, }, + { + name: `a failed include replay before the first ready`, + history: [ + { type: `settle`, ok: true, keepHolding: true }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + markerMode: `on-demand`, + includes: true, + abortedMarkerLoads: `reject`, + historyDuringPreload: true, + }, + reaches: `abortRejections`, + }, + { + name: `an aborted marker load that rejects after its replay succeeds`, + history: [ + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: true, keepHolding: false }, + { type: `truncateMarkers` }, + { type: `unloadThrows` }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + peer: true, + joined: `some`, + markerMode: `on-demand`, + abortedMarkerLoads: `reject`, + historyDuringPreload: true, + }, + reaches: `abortRejections`, + }, + { + name: `a replaced include segment whose aborted load rejects`, + history: [ + { type: `hold` }, + { type: `truncate`, armAfterStart: false }, + { type: `move`, id: 1, rank: 1 }, + { type: `move`, id: 2, rank: 31 }, + ], + scenario: { + ...ordinaryOrderedScenario, + markerMode: `on-demand`, + includes: true, + abortedMarkerLoads: `reject`, + }, + reaches: `abortRejections`, + }, + { + name: `an include demand that fails while its replay still loads`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `move`, id: 3, rank: 1 }, + { type: `settle`, ok: false, newest: true }, + ], + scenario: { + ...ordinaryOrderedScenario, + markerMode: `on-demand`, + includes: true, + }, + reaches: `failedLoads`, + recovers: true, + }, ] satisfies Array<{ name: string history: Array scenario?: OrderedScenario reaches?: keyof OrderedReach + // A replay whose history the law places in a recovery, where the model's + // own classification is permissive (see the terminal-error law). + recovers?: true }>)( `keeps the ordered window for $name (generated replay)`, async ({ history, scenario, ...replay }) => { @@ -2164,6 +2286,9 @@ describe(`Ordered windows across truncate replays`, () => { const reaches: keyof OrderedReach | undefined = `reaches` in replay ? replay.reaches : undefined if (reaches) expect(reach[reaches], reaches).toBeGreaterThan(0) + if (`recovers` in replay) { + expect(reach.terminalErrors, `recovery failure ends the query`).toBe(0) + } }, ) @@ -2191,6 +2316,8 @@ describe(`Ordered windows across truncate replays`, () => { abortedMoves: 0, terminalErrors: 0, replayLoads: 0, + abortRejections: 0, + preloadChecks: 0, } await fc.assert( fc.asyncProperty( diff --git a/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts index 281ffd4788..b32a5b55ea 100644 --- a/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts @@ -3353,3 +3353,84 @@ describe(`synchronous initial full-source settlement refinement`, () => { }, ) }) + +describe(`truncate replay work for an ordered window`, () => { + // A truncate reacquires the demand its subscriptions hold (ARCHITECTURE.md, + // "Replay participants and failure"). A window that holds no demand, such + // as a zero-limit window, owes no provider request and no loading state at + // a truncate. A query subscription still opens a replay then, so demand + // its graph starts during the commit belongs to that replay; the replay + // itself starts no recovery work. The one-row window is the neighbour: its + // truncate must reacquire, so a law that issued nothing would fail there. + it.each([0, 1])( + `reacquires only held demand when a %s-row window truncates`, + async (limit) => { + let sync!: Parameters[`sync`]>[0] + const requests: Array = [] + const source = createCollection({ + id: `ordered-truncate-work-${limit}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + sync.begin() + for (const value of fullSourceRows) { + sync.write({ + type: source.has(value.id) ? `update` : `insert`, + value, + }) + } + sync.commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(limit), + ) + let primaryFailure: CapturedOracleFailure | undefined + let observation: + { replayRequests: number; loadingAtCommit: boolean } | undefined + try { + await live.preload() + const before = requests.length + sync.begin() + sync.truncate() + sync.commit() + const loadingAtCommit = live.isLoadingSubset + await flushPromises() + observation = { + replayRequests: requests.length - before, + loadingAtCommit, + } + } catch (error) { + primaryFailure = { error } + } + await finishOracleCleanup( + primaryFailure, + [() => live.cleanup(), () => source.cleanup()], + `Ordered truncate work oracle cleanup failed`, + ) + if (limit === 0) { + expect(observation).toEqual({ + replayRequests: 0, + loadingAtCommit: false, + }) + } else { + expect(observation!.replayRequests).toBeGreaterThan(0) + } + }, + ) +}) From 4afe71eb3d563fa1c540ffe88d14bb721878ddc1 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 14:10:27 -0600 Subject: [PATCH 03/40] chore: add changeset for live-query replay failure fix Co-authored-by: Isaac --- .changeset/keep-live-query-through-failed-replay.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 .changeset/keep-live-query-through-failed-replay.md diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md new file mode 100644 index 0000000000..872b21e17d --- /dev/null +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -0,0 +1,7 @@ +--- +'@tanstack/db': patch +--- + +Keep a live query running when a lazy join or include load fails as part of a source truncate. The truncate's replay fails instead, and the query keeps its last complete result until a later replay of that source succeeds. Before, the query could end in error 228, for example when a truncate made an ordered window demand a row that it never showed. A load that fails before the query first becomes ready still ends it. + +An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. From 21fbd0394c00ca79240e38e5b4a2bfc9a4f08365 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 14:52:27 -0600 Subject: [PATCH 04/40] fix(db): repair an ordered window whose refetched page moved before its tie A signalling refetch can move the first page's end earlier, to a rank that a row the provider never sent ties with. The loader ignored the report because an installed row past the page sorted after the new end, but that row was there only because an older tie request read it. A later window move then read local rows and skipped the unsent row. The loader now takes an installed row past the page as evidence only when its own page read it or when it read the tie of the page's current end. The ordered-work oracle keeps the trace as a named replay; the request-authority oracle gains an unproven-report law. Both fail on main. Co-authored-by: Isaac --- .../keep-live-query-through-failed-replay.md | 2 + docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 9 ++- .../src/query/live/ordered-source-loader.ts | 6 ++ ...ordered-source-loader-state-oracle.test.ts | 75 ++++++++++++++++--- .../ordered-work-oracle.property.test.ts | 38 ++++++++++ 6 files changed, 121 insertions(+), 11 deletions(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index 872b21e17d..e4692b5bbc 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -5,3 +5,5 @@ Keep a live query running when a lazy join or include load fails as part of a source truncate. The truncate's replay fails instead, and the query keeps its last complete result until a later replay of that source succeeds. Before, the query could end in error 228, for example when a truncate made an ordered window demand a row that it never showed. A load that fails before the query first becomes ready still ends it. An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. + +An ordered window over a source that reports shifted pages after a refetch no longer skips a row tied with the refetched first page's new end. When the report arrives, the window repairs instead of reading a row that an older tie request installed. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 8dbf9d94bf..3036d6e35c 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair), whose replay and fixed campaign fail on main. Limit: the fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable, and the ordered-work oracle reaches the narrowed branch only when another consumer installs rows past the page, which it does not generate with a refetching provider (#2155); a mutant that always repairs on a report survives both oracles. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 1964155d69..b993930810 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1006,7 +1006,14 @@ when its window is inside that page: every window row sorts at or before the `offset + limit`-th source row, and the next source row sorts strictly after it in the leading term. The refetched page holds every row from the start through its end, and any row it left out sorts after that end, so the window -is proven. A window that needs a row past the page, such as one whose page +is proven. The loader checks the next source row through an installed row, +which is evidence only when the page read it, or when the loader read the tie +of the page's current end: the refetch re-read that tie, so every row that ties +with the end is installed. A row that another request installed proves +nothing. Neither does a tie that the loader read for an older end, because a +refetch can move the end earlier, so a row the provider never sent can tie +with the new end. The loader repairs in those cases. A window that needs a row +past the page, such as one whose page holds rows a joined filter rejects, still repairs. So does any loader while its source holds accepted sync rows back, as behind a persisting optimistic transaction: the visible rows then predate the refetch. Core reissues the diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 806c5176f8..2f0e95e3e8 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -695,7 +695,13 @@ export class OrderedSourceLoader { // Optimistic rows are left out, so a short read proves nothing. if (rows.length <= size) return false const end = rows[size - 1]!.value + // A row past the page proves the source's next row only when the page + // read it or it ties with the page's end. A refetch can move the end + // before the tie that the loader read for an older end, so that tie's + // rows prove nothing about the rows that now tie with the end. return ( + ((this.ownPrefixCount ?? 0) > size || + Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary)) && this.info.comparator(rows[size]!.value, end) > 0 && this.info.dataNeeded(end) === 0 ) diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 8deecfb889..7eee4b00f4 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -34,6 +34,14 @@ type RequestOptions = LoadSubsetOptions & { ) => void } +// The leading order value that a tie request asks for: its `where` is +// `eq(rank, value)`. Other requests have no tie value. +function tieValue(options: RequestOptions): unknown { + const where = options.where as + { name?: string; args?: Array<{ value?: unknown }> } | undefined + return where?.name === `eq` ? where.args?.[1]?.value : undefined +} + function createDeferred() { let resolve!: () => void let reject!: (error: unknown) => void @@ -1816,10 +1824,20 @@ describe(`Successful finite demand retirement`, () => { * exact result for its own options), or while the loader's coverage is * its own first ordered prefix of at least `offset + limit` rows, with * only that prefix's tie after it, and a row past the page sorts strictly - * after its end with no more rows needed (ARCHITECTURE.md). The law judges - * only quiet histories: every earlier step settled a request successfully, - * ran a pass, inserted a row, reported, or changed joined work, and no - * release callback has side effects, so no earlier event owes a repair. + * after its end with no more rows needed (ARCHITECTURE.md). That row is + * evidence only when the page read it, or when the tie the loader read is + * the tie of the page's end. The law judges only quiet histories: every + * earlier step settled a request successfully, ran a pass, inserted a row, + * reported, or changed joined work, and no release callback has side + * effects, so no earlier event owes a repair. + * - **Unproven report:** in such a quiet history, a report over the loader's + * own first prefix that no row it read proves starts a repair on its pass. + * An installed row past the page can come from another request, and a + * refetch can move the page's end before the tie the loader read, so the + * loader cannot use either as proof. This fake's installed rows ignore a + * read's limit, so its tie is always for the last installed row; the + * ignored-report branch for a first page is therefore unreachable here, + * and only the unproven-report law runs on such a page. * - **One request at a time:** a request issued while another is in flight * starts authority, or an authority start lies between them. A repair * chain that a newer start replaced issues no more steps. @@ -2174,6 +2192,7 @@ type AuthorityObservation = { joinedDeferrals: number ignoredReportsBeforeSettle: number ignoredReportsInsideFirstPage: number + repairedReportsOnOwnPage: number truncates: number replayFailures: number optimisticDepartures: number @@ -2540,6 +2559,7 @@ async function checkAuthorityCase( let joinedDeferrals = 0 let ignoredReportsBeforeSettle = 0 let ignoredReportsInsideFirstPage = 0 + let repairedReportsOnOwnPage = 0 let truncates = 0 let replayFailures = 0 let optimisticDepartures = 0 @@ -2835,7 +2855,9 @@ async function checkAuthorityCase( const windowSize = testCase.offset + info.limit const [first, ...later] = ledger const beforeSettle = ledger.every(({ outcome }) => outcome !== `ok`) - const insideFirstPage = + // The loader's coverage is its own first page with only tie + // requests after it. + const ownFirstPage = !testCase.requiresFullSource && !testCase.needsMore && testCase.snapshotRows > windowSize && @@ -2855,10 +2877,38 @@ async function checkAuthorityCase( options.cursor === undefined && options.orderBy === undefined, ) + // A row past the page proves the source's next row only when the + // page read it, or when it ties with the page's end and the loader + // read that tie. Any other installed row can come from another + // request, so it proves nothing. The installed ranks are 1..n, so + // the end's rank is the window size. + const insideFirstPage = + ownFirstPage && + ((first.options.limit ?? 0) > windowSize || + later.some(({ options }) => tieValue(options) === windowSize)) const before = ledger.length loader.revalidateLoadedCoverage() if (command.react) graphPass() await flush() + // A report the loader cannot prove covers nothing it could change + // owes a repair, which its next pass starts. + if ( + command.react && + quiet && + !beforeSettle && + ownFirstPage && + !insideFirstPage + ) { + const issued = ledger + .slice(before) + .some(({ authoritative }) => authoritative) + if (!issued) { + throw new Error( + `unproven report: a report over a first page that no row it read proves issued no repair`, + ) + } + repairedReportsOnOwnPage++ + } if (quiet && (beforeSettle || insideFirstPage)) { const issued = ledger .slice(before) @@ -3196,6 +3246,7 @@ async function checkAuthorityCase( joinedDeferrals, ignoredReportsBeforeSettle, ignoredReportsInsideFirstPage, + repairedReportsOnOwnPage, truncates, replayFailures, optimisticDepartures, @@ -3612,9 +3663,12 @@ describe(`Ordered source request authority`, () => { }, }, { - // Without the first-page check, the report repaired a window that the - // loader's own first prefix already proves. - name: `a report while the window is inside the loader's first page`, + // The first page read only the window, and the loader's tie request + // read a value past it, not the page's end. An installed row past the + // page can come from another request, so it proves nothing, and the + // report must repair. Ignoring it kept a window that missed a row tied + // with a refetched page's end. + name: `a report over a first page that read only the window`, testCase: { indexed: false, limit: 1, @@ -3625,6 +3679,7 @@ describe(`Ordered source request authority`, () => { snapshotRows: 3, needsMore: false, history: [ + { type: `settle`, pick: 0, ok: true }, { type: `settle`, pick: 0, ok: true }, { type: `report`, react: true }, ], @@ -3667,6 +3722,7 @@ describe(`Ordered source request authority`, () => { joinedDeferrals: 0, ignoredReportsBeforeSettle: 0, ignoredReportsInsideFirstPage: 0, + repairedReportsOnOwnPage: 0, truncates: 0, replayFailures: 0, optimisticDepartures: 0, @@ -3731,7 +3787,8 @@ describe(`Ordered source request authority`, () => { for (const [dimension, count] of Object.entries(later)) { if ( dimension === `startThrowsDuringFailure` || - dimension === `ignoredReportsInsideFirstPage` + dimension === `ignoredReportsInsideFirstPage` || + dimension === `repairedReportsOnOwnPage` ) continue expect(count, dimension).toBeGreaterThan(0) diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 8ef2cd6d1d..996c5b2a15 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -7131,6 +7131,44 @@ describe(`eager indexed ordered windows`, () => { }) }) + // A signalling refetch moves the first page's end earlier, to a rank that + // a row the provider never sent ties with. The window is still inside the + // page, but the rank-2 row past it is local only because the old tie + // request read it, so it proves nothing about the rows that now tie with + // the end. The report must repair; ignoring it made a later window move + // read the local rows and skip row 11. + it(`repairs a signalling refetch before a window move reads past its page`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `joined`, + via: `marker`, + indexed: true, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 2, + offset: 0, + initial: [ + { id: 0, rank: null as unknown as number, eligible: true }, + { id: 1, rank: 2, eligible: true }, + { id: 2, rank: 3, eligible: true }, + { id: 3, rank: Number.NaN, eligible: true }, + ], + history: [ + { type: `upsert`, id: 10, rank: 0, eligible: true }, + { type: `flip`, id: 2 }, + { type: `upsert`, id: 11, rank: 0, eligible: true }, + { type: `refetch` }, + { type: `move`, offset: 0, limit: 3 }, + { type: `delete`, id: 0 }, + ], + }) + }) + // A page with no later page whose end moves earlier. Two rank-0 rows // arrive while the provider lags; the refetched page ends at rank 0, but // the tie request still holds the rank-1 row, so the loader sees no From 26bff5f4faba554ad774cb0dbad1c030f3cfa3fa Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 14:52:55 -0600 Subject: [PATCH 05/40] docs: split a long changeset sentence Co-authored-by: Isaac --- .changeset/keep-live-query-through-failed-replay.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index e4692b5bbc..45ef1c583a 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -6,4 +6,4 @@ Keep a live query running when a lazy join or include load fails as part of a so An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. -An ordered window over a source that reports shifted pages after a refetch no longer skips a row tied with the refetched first page's new end. When the report arrives, the window repairs instead of reading a row that an older tie request installed. +Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report arrives, the window repairs instead of reading a row that an older tie request installed. From ce1299ded3afbd1c5d8d663cb6ad8aed3a778411 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 16:39:29 -0600 Subject: [PATCH 06/40] fix(db): read only the moved end's tie after a refetch, and apply moves deferred behind a discarded replay Ordered loading after a signalling refetch: - When a report shows that the first page's end moved, the loader now reads only the new end's tie on the next pass instead of a bounded repair, and releases the older tie without treating its deletes as an order change. Query DB's top-N insert costs one tie request and keeps its active Query count. The loader repairs instead when it already owes a repair, has work in flight, a replay is pending, or no cursor can express the end. Window moves: - A window move deferred behind a replay that a newer truncate discards now waits for every replay that holds publication and then applies. Before, it rejected with that replay's AbortError and an older move's window stayed. Oracles: - The ordered-work refetch law now allows the documented full-source refill of an unindexed plan whose bounded prefix leaves the window short. - The replay-refinement oracle keeps the discarded-replay trace; one older replay now checks the moved window instead of an AbortError. - Query DB's first-page witness requires exactly the end's tie request and an unchanged active Query count; the request-authority unproven-report law accepts the end's tie as the proof step. Co-authored-by: Isaac --- .../keep-live-query-through-failed-replay.md | 4 +- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 13 ++- .../query/live/collection-config-builder.ts | 23 +++- .../src/query/live/ordered-source-loader.ts | 101 ++++++++++++++---- ...ad-subset-replay-refinement-oracle.test.ts | 27 ++++- ...ordered-source-loader-state-oracle.test.ts | 13 ++- .../ordered-work-oracle.property.test.ts | 68 +++++++++++- .../refetch-ordered-prefix-oracle.test.ts | 25 ++++- 9 files changed, 232 insertions(+), 44 deletions(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index 45ef1c583a..ef19984f1f 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -6,4 +6,6 @@ Keep a live query running when a lazy join or include load fails as part of a so An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. -Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report arrives, the window repairs instead of reading a row that an older tie request installed. +Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window reads the new end's tie group. It no longer trusts a row that an older tie request installed. + +A window move made while a truncate replay holds a live query's results now applies after that replay, even when a newer truncate discards it. Before, the move could reject, and an older move's window stayed. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 3036d6e35c..b8338fc3fb 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair), whose replay and fixed campaign fail on main. Limit: the fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable, and the ordered-work oracle reaches the narrowed branch only when another consumer installs rows past the page, which it does not generate with a refetching provider (#2155); a mutant that always repairs on a report survives both oracles. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. Limit: the fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable, and the ordered-work oracle reaches the narrowed branch only when another consumer installs rows past the page, which it does not generate with a refetching provider (#2155); a mutant that always repairs on a report survives both oracles. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index b993930810..001cc6809e 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1012,7 +1012,13 @@ of the page's current end: the refetch re-read that tie, so every row that ties with the end is installed. A row that another request installed proves nothing. Neither does a tie that the loader read for an older end, because a refetch can move the end earlier, so a row the provider never sent can tie -with the new end. The loader repairs in those cases. A window that needs a row +with the new end. When the end moved, the loader reads only the new end's tie +on the next pass: the refetched page holds every row before the end, so that +tie completes the proof without a repair. It then releases the older tie, +whose rows sort after the end and outside the window, and the deletes that +release causes are not an order change. A loader that already owes a repair, +has other work in flight, or has an end that no cursor can express repairs +instead. A window that needs a row past the page, such as one whose page holds rows a joined filter rejects, still repairs. So does any loader while its source holds accepted sync rows back, as behind a persisting optimistic @@ -1322,7 +1328,10 @@ late rejection from an abandoned sync run cannot reset the replacement sync run's requested window. A window move started during an active source replay waits for that replay and applies only after its replacement is complete. A failed replay rejects the -move without advancing the reported window. Replay completion callbacks carry +move without advancing the reported window. A replay that a newer truncate or +the retirement of its last demand discards did not fail, even though its +completion rejects with `AbortError`: the move waits for any replay that still +holds publication and then applies. Replay completion callbacks carry their sync-run identity and become no-ops after cleanup or restart. Cleanup rejects the replay barrier, and therefore every window move waiting on it, with `AbortError`; no waiter may outlive the discarded subscription. diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 8e0633821e..624bfa9fc9 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -340,17 +340,36 @@ export class CollectionConfigBuilder< const request = ++this.windowRequestGeneration const sourceRecovery = this.pendingSourceRecovery() if (sourceRecovery) { - const deferred: Promise = sourceRecovery.then(async () => { + const syncRunGeneration = this.syncRunGeneration + const resume = async (): Promise => { + // Wait for every replay that holds publication, including one a + // newer truncate starts meanwhile. A replay that a newer truncate or + // a retired demand discards rejects its completion with an + // AbortError, but it did not fail. Cleanup also rejects it, and then + // the move rejects with that error. + for ( + let recovery: Promise | undefined = sourceRecovery; + recovery; + recovery = this.pendingSourceRecovery() + ) { + try { + await recovery + } catch (error) { + if (syncRunGeneration !== this.syncRunGeneration) throw error + } + } // A newer call wins, even if this deferred one resumes after it. if (request !== this.windowRequestGeneration) { await this.latestWindowRequest return } + // A failed replay rejects the call. Otherwise it applies the window. const settlement = this.setWindow(requestedWindow) // The resumed call settles on behalf of this one. this.latestWindowRequest = deferred if (settlement !== true) await settlement - }) + } + const deferred = resume() return (this.latestWindowRequest = deferred) } if (this.hasFailedSourceRecovery()) { diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 2f0e95e3e8..85924ae7fb 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -90,11 +90,15 @@ export class OrderedSourceLoader { private work: AuthoritativeWork = { kind: `idle` } // Keep callbacks, not copied requests or rows. Successful full-source work // subsumes these logical owners; unfinished transports remain observed. - // Each records its prefix count and, for a repair's own acquisitions, the - // epoch of the repair chain it belongs to. + // Each records its prefix count, for a repair's own acquisitions the epoch + // of the repair chain it belongs to, and for a tie request its boundary row. private settledFiniteAcquisitions = new Map< ReleaseLoadSubset, - { count: number | undefined; chain: number | undefined } + { + count: number | undefined + chain: number | undefined + tie: Record | undefined + } >() // The record's presence blocks automatic retry, including initial requests // that have no explicit window-operation generation. @@ -661,22 +665,81 @@ export class OrderedSourceLoader { revalidateLoadedCoverage(): void { // A `requiresFullSource` plan's coverage is its full-source read: nothing // has settled before it, and it covers the source after. + if (!this.active || !this.hasSettledSourceRequest || this.coversSource()) + return + const end = this.ownPrefixEnd() + if (end === undefined) { + this.invalidateSourceOrdering() + return + } + // An installed row past the page proves the source's next row only when + // the page read it, or when it ties with an end whose tie the loader + // read. A refetch can move the end before that tie; reading the new + // end's tie then proves the window. if ( - !this.active || - !this.hasSettledSourceRequest || - this.coversSource() || - this.windowIsInsideOwnPrefix() + (this.ownPrefixCount ?? 0) > this.info.offset + this.info.limit || + Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary) ) return - this.invalidateSourceOrdering() + this.loadMovedEndTie(end) + } + + // The refetched page still holds every row before its end, so the end's + // tie completes the proof without a repair. The older tie's rows sort + // after the end and outside the window, and they prove nothing now. + private loadMovedEndTie(end: Record): void { + // Other work owns the boundary, a repair is already owed, a replay owns + // the leases, or no cursor can express the end's tie, so only the + // repair's full-source fallback can read it. + if ( + !canExpressCursorOrder( + normalizeOrderByPaths(this.info.orderBy, this.alias).slice(0, 1), + [this.info.valueExtractorForRawRow(end)], + ) || + this.debt !== `none` || + this.work.kind !== `idle` || + this.pending || + this.stagedContinuation || + this.subscription.hasPendingTruncateReplacement + ) { + this.invalidateSourceOrdering() + return + } + const orderedLoadGeneration = this.orderedLoadGeneration + // The release can delete rows only the older tie held. Those deletes + // leave rows past the end, so they are not an order change. + this.retiringSupersededAcquisitions = true + try { + this.releaseOwned( + this.settledFiniteAcquisitions, + (_, { tie }) => tie !== undefined && this.info.comparator(tie, end) > 0, + ) + } finally { + this.retiringSupersededAcquisitions = false + } + // A release that resets the cursor hands the window to the replay. + if (!this.active || orderedLoadGeneration !== this.orderedLoadGeneration) + return + this.settledSourceBoundary = end + // The pass that follows the report reads the tie, as a repair's first + // request would start then, so a start that throws fails that pass. + this.stagedContinuation = { + orderedLoadGeneration, + authorityGeneration: this.authorityGeneration, + orderingInvalidationGeneration: this.orderingInvalidationGeneration, + role: `finite`, + kind: `boundary`, + continuesOrderedPrefixRepair: false, + } } // The refetched first page holds every row from the start of the order // through its end, and any row it left out sorts after that end. So a // window whose rows all sort at or before the end is right. A row after // the page that ties with its end could sort before a window row; then - // the window is not proven. - private windowIsInsideOwnPrefix(): boolean { + // the window is not proven. Returns the end when the window is inside + // the page. + private ownPrefixEnd(): Record | undefined { const size = this.info.offset + this.info.limit // A report can arrive while the refetched rows are accepted but held // back, as behind a persisting optimistic transaction. The visible rows @@ -686,25 +749,19 @@ export class OrderedSourceLoader { !this.info.dataNeeded || this.subscription.hasHeldSyncTransaction ) { - return false + return undefined } const rows = this.subscription.readOrderedSnapshot({ orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), limit: size + 1, }) // Optimistic rows are left out, so a short read proves nothing. - if (rows.length <= size) return false + if (rows.length <= size) return undefined const end = rows[size - 1]!.value - // A row past the page proves the source's next row only when the page - // read it or it ties with the page's end. A refetch can move the end - // before the tie that the loader read for an older end, so that tie's - // rows prove nothing about the rows that now tie with the end. - return ( - ((this.ownPrefixCount ?? 0) > size || - Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary)) && - this.info.comparator(rows[size]!.value, end) > 0 && + return this.info.comparator(rows[size]!.value, end) > 0 && this.info.dataNeeded(end) === 0 - ) + ? end + : undefined } private finishOrderedPrefixRepair(windowOperationGeneration?: number): void { @@ -1020,6 +1077,7 @@ export class OrderedSourceLoader { ): Promise | undefined { const isFullSource = kind === `full-source` const isOrderedRepair = kind === `ordered-repair` + const tie = kind === `boundary` ? this.settledSourceBoundary : undefined const isAuthoritativeRequest = isFullSource || isOrderedRepair || continuesOrderedPrefixRepair let observed: @@ -1136,6 +1194,7 @@ export class OrderedSourceLoader { this.settledFiniteAcquisitions.set(release, { count: prefixCount, chain, + tie, }) // Full-source coverage also retires a repair request that // settles after it. diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 5b067202aa..5207a7fa90 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -2001,8 +2001,8 @@ describe(`Ordered windows across truncate replays`, () => { // graph that the pending joined work had held, so the window stayed on // old rows until an unrelated write); // - a window move whose joined load a truncate aborts (the model counted - // the move's AbortError as an escaped fault; it is a legal rejection, and - // this replay keeps the branch reached); + // the move's AbortError as an escaped fault; the move now waits for the + // newer replay and applies, so this replay checks the moved window); // - a window move that retires lazy joined demand whose release throws (the // subscription reported the release failure as a load error, so the move // rejected with it; core now reports it uncaught); @@ -2078,7 +2078,7 @@ describe(`Ordered windows across truncate replays`, () => { joined: `none`, markerMode: `on-demand`, }, - reaches: `abortedMoves`, + reaches: `settledWindowChecks`, }, { name: `a window move that retires lazy joined demand whose release throws`, @@ -2271,6 +2271,27 @@ describe(`Ordered windows across truncate replays`, () => { reaches: `failedLoads`, recovers: true, }, + { + // A move during the initial load waits behind a marker replay that the + // source truncate then discards. The discarded replay rejected the + // move, so the older move's window won. + name: `a newer window move deferred behind a discarded replay`, + history: [ + { type: `move`, id: 1, rank: 1 }, + { type: `window`, offset: 0, limit: 2 }, + { type: `truncateMarkers` }, + { type: `window`, offset: 0, limit: 1 }, + { type: `truncate`, armAfterStart: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + markerMode: `on-demand`, + includes: true, + historyDuringPreload: true, + }, + reaches: `settledWindowChecks`, + }, ] satisfies Array<{ name: string history: Array diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 7eee4b00f4..94cb097a81 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -1831,7 +1831,8 @@ describe(`Successful finite demand retirement`, () => { * reported, or changed joined work, and no release callback has side * effects, so no earlier event owes a repair. * - **Unproven report:** in such a quiet history, a report over the loader's - * own first prefix that no row it read proves starts a repair on its pass. + * own first prefix that no row it read proves starts a repair on its pass, + * or reads the tie of the page's end, which completes the proof. * An installed row past the page can come from another request, and a * refetch can move the page's end before the tie the loader read, so the * loader cannot use either as proof. This fake's installed rows ignore a @@ -2891,7 +2892,8 @@ async function checkAuthorityCase( if (command.react) graphPass() await flush() // A report the loader cannot prove covers nothing it could change - // owes a repair, which its next pass starts. + // owes a repair, which its next pass starts, or a tie request for + // the page's end, which completes the proof. if ( command.react && quiet && @@ -2901,10 +2903,13 @@ async function checkAuthorityCase( ) { const issued = ledger .slice(before) - .some(({ authoritative }) => authoritative) + .some( + ({ authoritative, options }) => + authoritative || tieValue(options) === windowSize, + ) if (!issued) { throw new Error( - `unproven report: a report over a first page that no row it read proves issued no repair`, + `unproven report: a report over a first page that no row it read proves issued no repair and no tie of its end`, ) } repairedReportsOnOwnPage++ diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 996c5b2a15..394cf3978b 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -4970,6 +4970,28 @@ function expectEagerWindow( expect(new Set(actual).size, `${label}: distinct rows`).toBe(actual.length) } +/** + * Whether a bounded prefix of the source leaves the window short. The prefix + * is the first `offset + limit` source rows in the query's order. A source + * filter runs inside the request, so its prefix holds only eligible rows. A + * joined or function filter drops rows only after the source's limit, so its + * prefix can hold fewer eligible rows than the window needs. + */ +function prefixLeavesWindowShort( + rows: ReadonlyMap, + testCase: EagerCase, +): boolean { + const size = testCase.offset + testCase.limit + const candidates = + testCase.filter === `source` + ? rows + : new Map( + [...rows].map(([rowId, row]) => [rowId, { ...row, eligible: true }]), + ) + const prefix = eagerOrder(candidates, testCase).slice(0, size) + return prefix.filter(({ id }) => rows.get(id)!.eligible).length < size +} + /** Ranks of the last window row and the first row after the window. */ function boundaryRanks( rows: ReadonlyMap, @@ -5890,9 +5912,16 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ await refetch() // The repair a report starts reads each window's bounded prefix, // tie, and refill; a whole-source read would be the cost the report - // exists to avoid. A page that ends on a null or NaN rank takes the - // documented full-source fallback instead. - if (signalsRefetch && unexpressiblePageEnds === unexpressibleBefore) { + // exists to avoid. Two documented fallbacks read the full source + // instead: a page that ends on a null or NaN rank, and the refill of + // an unindexed plan whose bounded prefix leaves the window short + // (ARCHITECTURE.md, "an underfilled unindexed prefix retains the + // established full-source fallback"). + if ( + signalsRefetch && + unexpressiblePageEnds === unexpressibleBefore && + (testCase.indexed || !prefixLeavesWindowShort(rows, shown)) + ) { expect( fullSourceRequests, `full-source requests after a signalling refetch at step ${step}`, @@ -7131,12 +7160,41 @@ describe(`eager indexed ordered windows`, () => { }) }) + // An unindexed joined window whose only domain row is ineligible. After a + // signalling refetch the repair's bounded prefix holds just that row, so + // the window is short and the refill takes the documented full-source + // fallback. The refetch law allows that read for an unindexed plan only + // when its prefix leaves the window short. + it(`allows the full-source refill of a short unindexed window after a signalling refetch`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `joined`, + via: `marker`, + indexed: false, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 1, + offset: 0, + initial: [], + history: [ + { type: `upsert`, id: 0, rank: 0, eligible: false }, + { type: `dup`, id: 0 }, + { type: `refetch` }, + ], + }) + }) + // A signalling refetch moves the first page's end earlier, to a rank that // a row the provider never sent ties with. The window is still inside the // page, but the rank-2 row past it is local only because the old tie // request read it, so it proves nothing about the rows that now tie with - // the end. The report must repair; ignoring it made a later window move - // read the local rows and skip row 11. + // the end. The loader must read the new end's tie; ignoring the report + // made a later window move read the local rows and skip row 11. it(`repairs a signalling refetch before a window move reads past its page`, async () => { await checkEagerWindow({ path: `on-demand`, diff --git a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts index f4d4ad4e90..0ed0cbbf6e 100644 --- a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts +++ b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts @@ -452,6 +452,9 @@ type Run = { failAfter: (count: number | undefined) => void // The Query cache entries this Collection owns: one per tracked request. cachedQueries: () => number + // Queries an observer still holds. A refetch re-runs only these; a released + // Query stays cached until its gcTime. + activeQueries: () => number // Reports that served pages may have shifted (`loadedSubsets:invalidated` // events on the source Collection) so far. reports: () => number @@ -741,6 +744,9 @@ async function startRun(testCase: Case): Promise { }, cachedQueries: () => queryClient.getQueryCache().findAll({ queryKey: [id] }).length, + activeQueries: () => + queryClient.getQueryCache().findAll({ queryKey: [id], type: `active` }) + .length, reports: () => reports, changedReanswers: () => changedReanswers, failNextApplication: () => { @@ -1298,8 +1304,12 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => }) // The ordinary insert into a top-N list. The page's end moves earlier, so - // Query DB reports, but every window row is in the window's own first page: - // the refetch must send only the Queries it refetches, with no repair. + // Query DB reports, but every window row is in the window's own first page. + // The tie the loader read belongs to the old end, so its rows prove + // nothing about rows that tie with the new end. The refetch sends the + // Queries it refetches and one tie request for the new end, with no + // repair. The loader releases the old end's tie, so the Query count stays + // the same and later refetches do not re-run a tie that proves nothing. it.each([1, 2] as const)( `a refetch of a window inside its first page starts no repair (%i order terms)`, async (terms) => { @@ -1322,14 +1332,19 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => await run.apply({ type: `upsert`, id: 5, rank: 5 }) await run.apply({ type: `flip`, id: 5 }) const queries = run.cachedQueries() + const activeBefore = run.activeQueries() const requestsBefore = run.requests.length await run.apply({ type: `refetch` }) expect(run.windowIds()).toEqual( expectedWindow(run.backend, run.markers, scenario), ) - expect(run.requests.length - requestsBefore).toBeLessThanOrEqual( - queries, - ) + const extra = run.requests.slice(requestsBefore + queries) + expect(extra).toHaveLength(1) + // A tie request reads equal leading values, with no order or cursor. + expect(extra[0]!.orderBy).toBeUndefined() + expect(extra[0]!.cursor).toBeUndefined() + expect(extra[0]!.where).toBeDefined() + expect(run.activeQueries()).toBe(activeBefore) } catch (error) { primary = error } From de45e1405016d823116a6c2bb19de850184f7283 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 16:43:26 -0600 Subject: [PATCH 07/40] chore(db): update the mangle cache for renamed loader members Co-authored-by: Isaac --- packages/db/mangle-cache.json | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index acc9210bbf..d53feec048 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -403,7 +403,6 @@ "repairNeedsReplacement": "gK", "coversSource": "gL", "standing": "gM", - "windowIsInsideOwnPrefix": "gN", "debt": "gO", "releaseOwned": "gP", "moveWork": "ae", @@ -415,5 +414,7 @@ "cancelRepairRetry": "cG", "coverSource": "fN", "windowRequestGeneration": "av", - "latestWindowRequest": "gE" + "latestWindowRequest": "gE", + "loadMovedEndTie": "gN", + "ownPrefixEnd": "gT" } From 58ea61052ff141c9c59d860fd98bd3b56a974869 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 18:19:37 -0600 Subject: [PATCH 08/40] fix(db): keep an undefined page end from matching the no-tie sentinel Address CodeRabbit review: an undefined order value at the page's end matched lastBoundary's no-tie sentinel, so a report was ignored without a proof. Also reword a named replay's comment to state the fixed behavior. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ordered-source-loader.ts | 4 +++- .../tests/query/load-subset-replay-refinement-oracle.test.ts | 5 +++-- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b8338fc3fb..d052b8c57f 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. Limit: the fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable, and the ordered-work oracle reaches the narrowed branch only when another consumer installs rows past the page, which it does not generate with a refetching provider (#2155); a mutant that always repairs on a report survives both oracles. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. No generated history ends a page on an `undefined` order value, and the ordered-work model sorts `undefined` with NaN, so the guard that keeps an `undefined` end from matching the loader's no-tie sentinel has no witness. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. Limit: the fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable, and the ordered-work oracle reaches the narrowed branch only when another consumer installs rows past the page, which it does not generate with a refetching provider (#2155); a mutant that always repairs on a report survives both oracles. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 85924ae7fb..b5d52d55cc 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -678,7 +678,9 @@ export class OrderedSourceLoader { // end's tie then proves the window. if ( (this.ownPrefixCount ?? 0) > this.info.offset + this.info.limit || - Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary) + // Undefined means that no tie request has been attempted. + (this.lastBoundary !== undefined && + Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary)) ) return this.loadMovedEndTie(end) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 5207a7fa90..d80d8aa59f 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -2273,8 +2273,9 @@ describe(`Ordered windows across truncate replays`, () => { }, { // A move during the initial load waits behind a marker replay that the - // source truncate then discards. The discarded replay rejected the - // move, so the older move's window won. + // source truncate then discards. The newer move must wait for the + // replay that still holds publication and then apply. Before the fix, + // the discarded replay rejected it, so the older move's window won. name: `a newer window move deferred behind a discarded replay`, history: [ { type: `move`, id: 1, rank: 1 }, From 9b65273f8b3f94c6e2fe8735c28a64f9514d9797 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 18:29:47 -0600 Subject: [PATCH 09/40] test(db): judge an unproven report after joined work defers its pass The request-authority unproven-report law judged the report's own pass, but pending joined work defers ordinary loading, so the owed repair or tie runs on the pass after that work finishes. test:dist found the case. The law now judges that later pass, and a named replay fails under the old law. Co-authored-by: Isaac --- ...ordered-source-loader-state-oracle.test.ts | 66 +++++++++++++++---- 1 file changed, 54 insertions(+), 12 deletions(-) diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 94cb097a81..6d9b74f9d2 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -2561,6 +2561,22 @@ async function checkAuthorityCase( let ignoredReportsBeforeSettle = 0 let ignoredReportsInsideFirstPage = 0 let repairedReportsOnOwnPage = 0 + // An unproven report whose pass pending joined work deferred: the ledger + // index at the report and the window size it judged. + let owedProof: { from: number; windowSize: number } | undefined + const expectProofRequest = (from: number, windowSize: number) => { + const issued = ledger + .slice(from) + .some( + ({ authoritative, options }) => + authoritative || tieValue(options) === windowSize, + ) + if (!issued) { + throw new Error( + `unproven report: a report over a first page that no row it read proves issued no repair and no tie of its end`, + ) + } + } let truncates = 0 let replayFailures = 0 let optimisticDepartures = 0 @@ -2893,7 +2909,9 @@ async function checkAuthorityCase( await flush() // A report the loader cannot prove covers nothing it could change // owes a repair, which its next pass starts, or a tie request for - // the page's end, which completes the proof. + // the page's end, which completes the proof. Pending joined work + // defers ordinary loading, so the pass that runs when that work + // finishes owes the request instead. if ( command.react && quiet && @@ -2901,18 +2919,12 @@ async function checkAuthorityCase( ownFirstPage && !insideFirstPage ) { - const issued = ledger - .slice(before) - .some( - ({ authoritative, options }) => - authoritative || tieValue(options) === windowSize, - ) - if (!issued) { - throw new Error( - `unproven report: a report over a first page that no row it read proves issued no repair and no tie of its end`, - ) + if (joinedPending && testCase.joined) { + owedProof = { from: before, windowSize } + } else { + expectProofRequest(before, windowSize) + repairedReportsOnOwnPage++ } - repairedReportsOnOwnPage++ } if (quiet && (beforeSettle || insideFirstPage)) { const issued = ledger @@ -2965,6 +2977,11 @@ async function checkAuthorityCase( joinedPending = false graphPass() await flush() + if (owedProof) { + expectProofRequest(owedProof.from, owedProof.windowSize) + owedProof = undefined + repairedReportsOnOwnPage++ + } break case `truncate`: truncate() @@ -3667,6 +3684,31 @@ describe(`Ordered source request authority`, () => { ], }, }, + { + // Pending joined work defers the pass after an unproven report, so the + // repair or tie that the report owes waits for that work. The pass that + // runs when it finishes must start it. A random campaign found the law + // judging the deferred pass instead. + name: `an unproven report while joined work defers loading`, + testCase: { + indexed: false, + limit: 1, + settledFirst: true, + releaseFault: false, + syncRepair: false, + ...ordinaryScenario, + joined: true, + snapshotRows: 3, + needsMore: false, + history: [ + { type: `joinedBusy` }, + { type: `settle`, pick: 0, ok: true }, + { type: `settleNewest` }, + { type: `report`, react: true }, + { type: `joinedIdle` }, + ], + }, + }, { // The first page read only the window, and the loader's tie request // read a value past it, not the page's end. An installed row past the From b79e4a9882bc2b87785cf7f5a1a6e82aa87c698f Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 18:45:39 -0600 Subject: [PATCH 10/40] test(db): void a deferred report proof when a later step changes what it owes Address CodeRabbit review: a truncate, disposal, failure, or move between a deferred unproven report and the pass after joined work can change or end what the report owes. Only quiet steps now keep the obligation; a named replay with a truncate failed under the old law. Co-authored-by: Isaac --- ...ordered-source-loader-state-oracle.test.ts | 47 ++++++++++++++----- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 6d9b74f9d2..0b34636e92 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -2561,6 +2561,15 @@ async function checkAuthorityCase( let ignoredReportsBeforeSettle = 0 let ignoredReportsInsideFirstPage = 0 let repairedReportsOnOwnPage = 0 + // Steps that settle a request successfully, run a pass, report, or change + // joined work. None of them owes a repair. + const quietCommand = (command: AuthorityCommand) => + (command.type === `settle` && command.ok) || + command.type === `settleNewest` || + command.type === `pass` || + command.type === `report` || + command.type === `joinedBusy` || + command.type === `joinedIdle` // An unproven report whose pass pending joined work deferred: the ledger // index at the report and the window size it judged. let owedProof: { from: number; windowSize: number } | undefined @@ -2796,6 +2805,9 @@ async function checkAuthorityCase( await flush() if (testCase.settledFirst && ledger[0]) await settle(ledger[0], true) for (const [step, command] of testCase.history.entries()) { + // An event outside the quiet set, such as a truncate, a failure, a + // move, or a disposal, can change or end what a deferred report owes. + if (owedProof && !quietCommand(command)) owedProof = undefined switch (command.type) { case `settle`: { const open = inflight() @@ -2857,17 +2869,7 @@ async function checkAuthorityCase( !testCase.releaseTruncates && !testCase.releaseDisposes && !boundaryUnexpressible && - testCase.history - .slice(0, step) - .every( - (earlier) => - (earlier.type === `settle` && earlier.ok) || - earlier.type === `settleNewest` || - earlier.type === `pass` || - earlier.type === `report` || - earlier.type === `joinedBusy` || - earlier.type === `joinedIdle`, - ) && + testCase.history.slice(0, step).every(quietCommand) && ledger.every(({ outcome }) => outcome !== `failed`) const windowSize = testCase.offset + info.limit const [first, ...later] = ledger @@ -3684,6 +3686,29 @@ describe(`Ordered source request authority`, () => { ], }, }, + { + // A truncate after a deferred unproven report hands the window to its + // replay, so the pass after the joined work owes no repair or tie. + name: `an unproven report whose deferred pass a truncate replaces`, + testCase: { + indexed: false, + limit: 1, + settledFirst: true, + releaseFault: false, + syncRepair: false, + ...ordinaryScenario, + joined: true, + snapshotRows: 3, + needsMore: false, + history: [ + { type: `joinedBusy` }, + { type: `settle`, pick: 0, ok: true }, + { type: `report`, react: true }, + { type: `truncate` }, + { type: `joinedIdle` }, + ], + }, + }, { // Pending joined work defers the pass after an unproven report, so the // repair or tie that the report owes waits for that work. The pass that From 40fea3f258f2e7093697b25d98c21593d76b6bb8 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 22 Oct 2026 21:58:12 -0600 Subject: [PATCH 11/40] test(db): cover nullish ranks, staged end ties, and proven reports in the ordered oracles - The ordered-work model now sorts undefined ranks with null ones, as the comparator does, and its histories generate both. A random campaign then found that its in-flight order check treated null and undefined as different ranks; nullish ranks now tie there too. - Peers over a refetching provider stay in the grammar for source filters (#2155 excludes only joined filters). A proven-report law requires that a report which changes nothing the first query's own page and end tie prove starts no request for it and leaves it owing no repair. A campaign fixed to that premise must reach it; a loader that repairs every report fails it. - A request-authority named replay reaches a staged end tie that pending joined work defers; a loader that drops the staged tie fails it. - The held fixed campaign moves to a seed that keeps its reach after the row generator changed. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/tests/oracle-config.ts | 1 + ...ordered-source-loader-state-oracle.test.ts | 25 ++ .../ordered-work-oracle.property.test.ts | 317 ++++++++++++++++-- 4 files changed, 322 insertions(+), 23 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d052b8c57f..b5ee5791ff 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. No generated history ends a page on an `undefined` order value, and the ordered-work model sorts `undefined` with NaN, so the guard that keeps an `undefined` end from matching the loader's no-tie sentinel has no witness. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. Limit: the fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable, and the ordered-work oracle reaches the narrowed branch only when another consumer installs rows past the page, which it does not generate with a refetching provider (#2155); a mutant that always repairs on a report survives both oracles. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. The guard that keeps an `undefined` end from matching the loader's no-tie sentinel is equivalent in reachable states: a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way, and a mutant without the guard survives the generated campaigns. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 8fe37b9d2e..9722cde44b 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -167,6 +167,7 @@ const staticOracleProperties = [ `ordered-work.eager-joined-gap`, `ordered-work.eager-full-join`, `ordered-work.eager-held-load`, + `ordered-work.eager-proven-report`, `ordered-work.lifecycle`, `ordered-work.nullable-lifecycle`, `pagination.async-cursor`, diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 0b34636e92..7a69bf283c 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -3686,6 +3686,31 @@ describe(`Ordered source request authority`, () => { ], }, }, + { + // The first page and its tie settle, then joined work starts. A report + // over the moved end stages only that end's tie, and pending joined + // work defers it. The pass after the joined work finishes must read it; + // a loader that dropped the staged tie read nothing. + name: `a staged end tie that waits for joined work`, + testCase: { + indexed: false, + limit: 1, + settledFirst: true, + releaseFault: false, + syncRepair: false, + ...ordinaryScenario, + joined: true, + snapshotRows: 3, + needsMore: false, + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settleNewest` }, + { type: `joinedBusy` }, + { type: `report`, react: true }, + { type: `joinedIdle` }, + ], + }, + }, { // A truncate after a deferred unproven report hands the window to its // replay, so the pass after the joined work owes no repair or tie. diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 394cf3978b..bb621a79ab 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -4373,7 +4373,7 @@ describe(`ordered source work oracle`, () => { * source's limit), the index (present or absent), the direction, one or two * order terms (rank, then ascending id), limit 1..3, offset 0..1, and a history * of upserts, deletes, deletes of the first visible row, and label-only - * touches over ids 0..29 and ranks 0..15, NaN, or null, so ties, deletes + * touches over ids 0..29 and ranks 0..15, NaN, null, or undefined, so ties, deletes * inside and at the edge of the window, rank changes across its boundary, and * distant rows the query has already received all occur. A second campaign * fixes the premise of the joined law: a joined filter, an index and one order @@ -4472,9 +4472,25 @@ describe(`ordered source work oracle`, () => { * One case in four adds a peer with its own window, the first * `offset + limit + 1` rows. Its rows law holds beside the first query's at * every checkpoint; the work laws count one consumer, so they do not apply - * to those cases. A peer over a refetching provider is excluded from the - * grammar until #2155 is fixed: an indexed trace is fixed and pinned, but an - * unindexed peer still keeps a short joined window after a refetch. + * to those cases. A peer over a refetching provider with a joined filter is + * excluded from the grammar until #2155 is fixed: an indexed trace is fixed + * and pinned, but an unindexed peer still keeps a short joined window after + * a refetch. A peer over a source filter stays in the grammar. + * + * `null` and `undefined` ranks are both nullish, so they tie: either may + * come first, and neither is an expressible cursor value. + * + * - **Proven report:** a signalling refetch can report a shift that changes + * nothing the first query's own first page proves. Its page and the tie + * of the page's end answer the same rows, the page ends on a plain number, + * and the model's next row sorts strictly after that end and is + * installed, as a peer's longer page installs it. The refetch also deletes + * and reorders no installed row the first query received, because such a + * change starts the repair path by itself. Then the report starts no + * request for the first query, and until a step that can change source + * ordering, the first query issues no repair (`refetch: true`) request. + * A campaign fixed to that premise must reach it, and a loader that + * repairs every report fails it. * Joined filters can match a row through several markers, by `dup`, so a * LEFT-joined filter and a full join keep one result row per matched * (row, marker) pair, and an anti-join keeps a row once. @@ -4587,6 +4603,7 @@ function eagerRowOf(maxId: number) { { weight: 6, arbitrary: fc.integer({ min: 0, max: 15 }) }, { weight: 1, arbitrary: fc.constant(Number.NaN) }, { weight: 1, arbitrary: fc.constant(null as unknown as number) }, + { weight: 1, arbitrary: fc.constant(undefined as unknown as number) }, ), eligible: fc.boolean(), }) @@ -4678,13 +4695,15 @@ const eagerCaseBase: fc.Arbitrary = fc.record({ // A peer over a refetching on-demand provider can keep a short joined window // after a refetch (#2155). An indexed plan's trace is fixed and pinned below, // but an unindexed one still fails (a peer shows [100] where the model shows -// [100, 101]). Each campaign excludes only that pairing, as its last map, -// until the fix. +// [100, 101]). Each campaign excludes only a joined, anti-joined, or full-joined +// filter in that pairing, as its last map, until the fix; a source filter +// keeps its peer. function withoutPeerRefetch(testCase: EagerCase): EagerCase { return testCase.path === `on-demand` && (testCase.provider === `refetch` || testCase.provider === `refetch-signal`) && - testCase.peer + testCase.peer && + testCase.filter !== `source` ? { ...testCase, peer: false } : testCase } @@ -4776,6 +4795,7 @@ const gapRank: fc.Arbitrary = fc.oneof( { weight: 6, arbitrary: fc.integer({ min: 0, max: 15 }) }, { weight: 1, arbitrary: fc.constant(Number.NaN) }, { weight: 1, arbitrary: fc.constant(null as unknown as number) }, + { weight: 1, arbitrary: fc.constant(undefined as unknown as number) }, ) const eagerJoinedGapCase: fc.Arbitrary = fc @@ -4846,6 +4866,54 @@ const eagerFullJoinCase: fc.Arbitrary = fc // first deletes a visible row, so the refill request is in flight while the // later commands change rows. Releases answer it from the rows at that // moment, between those changes. +// The proven-report premise: a peer over a signalling refetch provider and a +// source filter, so the peer installs the row after the first query's page. +// Histories upsert rows that tie with the peer's end or sort past both +// windows, then refetch, so a report fires while the first query's own page +// and tie stay the same; later moves check that it owes no repair. +const eagerProvenReportCase: fc.Arbitrary = fc + .record({ + base: eagerCaseBase, + ranks: fc.uniqueArray(fc.integer({ min: 0, max: 15 }), { + minLength: 3, + maxLength: 6, + }), + upserts: fc.array( + fc.record({ id: fc.integer({ min: 10, max: 19 }), offset: fc.nat(3) }), + { minLength: 1, maxLength: 3 }, + ), + rest: eagerJoinedHistory, + }) + .map(({ base, ranks, upserts, rest }) => { + const sorted = [...ranks].sort((left, right) => left - right) + const size = base.offset + base.limit + // Ranks at or after the peer's end, so the first query's page keeps its + // rows while the peer's answers change. + const peerEnd = sorted[Math.min(size, sorted.length - 1)]! + return { + ...base, + path: `on-demand` as const, + provider: `refetch-signal` as const, + latency: `immediate` as const, + peer: true, + filter: `source` as const, + indexed: true, + direction: `asc` as const, + terms: 1 as const, + initial: sorted.map((rank, id) => ({ id, rank, eligible: true })), + history: [ + ...upserts.map(({ id, offset }) => ({ + type: `upsert` as const, + id, + rank: peerEnd + offset, + eligible: true, + })), + { type: `refetch` as const }, + ...rest, + ], + } + }) + const eagerHeldCase: fc.Arbitrary = fc .tuple( eagerCaseBase, @@ -4917,9 +4985,10 @@ function eagerOrder( ): Array { // Documented order (live-queries guide): nulls first in either direction // unless the query asks for them last; NaN is greater than every other - // non-null value; ties by ascending id. - const position = (rank: number | null): [number, number] => { - if (rank === null) return [nulls === `last` ? 3 : 0, 0] + // non-null value; ties by ascending id. `null` and `undefined` are both + // nullish, so they tie with each other. + const position = (rank: number | null | undefined): [number, number] => { + if (rank == null) return [nulls === `last` ? 3 : 0, 0] if (Number.isNaN(rank)) return [direction === `asc` ? 2 : 1, 0] return [direction === `asc` ? 1 : 2, direction === `asc` ? rank : -rank] } @@ -4962,9 +5031,12 @@ function expectEagerWindow( expect(actual, label).toEqual(expected) return } + // `null` and `undefined` tie, so either may come first among nullish rows. const rank = (id: number) => { const row = rows.get(id) - return row?.eligible ? row.rank : `ineligible ${id}` + return row?.eligible + ? ((row.rank as number | null | undefined) ?? null) + : `ineligible ${id}` } expect(actual.map(rank), label).toEqual(expected.map(rank)) expect(new Set(actual).size, `${label}: distinct rows`).toBe(actual.length) @@ -5016,6 +5088,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ heldRequests: number cooperativeCancels: number lateAnswers: number + ignoredProvenReports: number }> { const joined = isJoinedFilter(testCase) const fullJoin = testCase.filter === `full-joined` @@ -5110,6 +5183,10 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // The requests the provider has served. A request without a predicate // covers every row. const servedRequests: Array = [] + // Loader requests per consumer subscription. The first query starts its + // sync run before any peer, so the first request is its own. + const requestsBy = new Map() + let firstSubscription: unknown // A refetching provider has skipped a change since its last refetch, so // the Collection may lag the rows and the window laws wait for a refetch. let stale = false @@ -5213,11 +5290,34 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // The rows each served request last returned, at its answer or at the // latest refetch. A refetching provider keeps exactly these rows. const lastAnswered = new Map>() + let refetchedAnswers = new Map>() // Bounded pages that ended on a null or NaN rank. Neither can be // expressed as a cursor (canExpressCursorOrder), so the loader then takes // the documented full-source fallback. let unexpressiblePageEnds = 0 + // Reports that the first query's own page proved, so it owed no request, + // and the first query's request count before the latest such report. + let ignoredProvenReports = 0 + let provenReportRequestsBefore: number | undefined + // Set by such a report until a step that can change source ordering, and + // the first repair request the first query issued meanwhile. + let provenSinceOrderingChange = false + let repairAfterProvenReport: string | undefined function servePage(options: LoadSubsetOptions): true | Promise { + firstSubscription ??= options.subscription + // A repair marks its requests `refetch: true`. After a report the first + // query's own page proved, it owes none until ordering changes again. + if ( + provenSinceOrderingChange && + options.subscription === firstSubscription && + options.refetch === true + ) { + repairAfterProvenReport ??= `request ${providerRequests + 1}` + } + requestsBy.set( + options.subscription, + (requestsBy.get(options.subscription) ?? 0) + 1, + ) providerRequests++ if (providerRequests > 4 * rows.size + 100) { // A rejected load is a provider failure the loader must handle, so it @@ -5268,7 +5368,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ rangeEnds.set(options, { ...page.at(-1)! }) // Generated rows may carry a null rank behind the `number` row type. const end = page.at(-1)!.rank as number | null - if (end === null || Number.isNaN(end)) unexpressiblePageEnds++ + if (end == null || Number.isNaN(end)) unexpressiblePageEnds++ } served += page.length servedBy.set( @@ -5332,7 +5432,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // the rows apply, and only when some answer changed; the contract lets a // sync adapter report whenever its pages may have shifted, so this // provider judges the loader's response, not query-db's gate. - async function refetch(): Promise { + async function refetch(): Promise { const current = new Map() let changed = false for (const request of servedRequests) { @@ -5349,6 +5449,11 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ lastAnswered.set(request, ids) for (const row of covered) current.set(row.id, row) } + // The answers as this refetch produced them, before a report's repair + // can retire the requests that gave them. + refetchedAnswers = new Map( + [...lastAnswered].map(([request, ids]) => [request, new Set(ids)]), + ) const applied = send( [...current.values()], [...sent.keys()].filter((rowId) => !current.has(rowId)), @@ -5360,6 +5465,74 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // Core reports in a microtask; the repair then runs its requests. await flushPromises() } + return signalsRefetch && changed + } + // The first query's own served requests, when they are only its first + // page and one tie request: the answers a report could change for it. + function firstPageAnswers(): + | { + page: LoadSubsetOptions + tie: LoadSubsetOptions + pageIds: Set + tieIds: Set + } + | undefined { + if (testCase.filter !== `source` || inFlight()) return + const own = servedRequests.filter( + (options) => options.subscription === firstSubscription, + ) + const size = shown.offset + shown.limit + const page = own.find( + (options) => + options.orderBy !== undefined && + options.cursor === undefined && + options.limit === size, + ) + const tie = own.find( + (options) => options.where !== undefined && options.orderBy === undefined, + ) + const pageIds = page && lastAnswered.get(page) + const tieIds = tie && lastAnswered.get(tie) + if (own.length !== 2 || !page || !tie || !pageIds || !tieIds) return + return { page, tie, pageIds: new Set(pageIds), tieIds: new Set(tieIds) } + } + // After a refetch, the first query's window is proven by its own page + // alone when the refetch left its page and tie answers unchanged, the page + // holds the model's first rows through a plain numeric end, the tie reads + // that end, and the model's next row sorts strictly after the end and is + // installed. Then a report has nothing to repair for it. + function provenFirstPage( + before: NonNullable>, + ): boolean { + // Read the answers of the requests judged before the refetch. Requests + // the step added, such as a repair, must not hide the premise. + const pageIds = refetchedAnswers.get(before.page) + const tieIds = refetchedAnswers.get(before.tie) + if (!pageIds || !tieIds) return false + const after = { ...before, pageIds, tieIds } + const sameIds = (left: Set, right: Set) => + left.size === right.size && [...left].every((key) => right.has(key)) + const size = shown.offset + shown.limit + const ordered = eagerOrder(rows, shown) + const end = ordered[size - 1] + const next = ordered[size] + const endRank = end?.rank as number | null | undefined + return ( + sameIds(before.pageIds, after.pageIds) && + sameIds(before.tieIds, after.tieIds) && + end !== undefined && + next !== undefined && + endRank != null && + !Number.isNaN(endRank) && + !Object.is(next.rank, endRank) && + sent.has(next.id) && + after.pageIds.size === size && + ordered.slice(0, size).every((row) => after.pageIds.has(row.id)) && + // The tie admits the end's rank and nothing on either side of it. + matches(after.tie.where, end) && + !matches(after.tie.where, { ...end, rank: endRank + 0.5 }) && + !matches(after.tie.where, { ...end, rank: endRank - 0.5 }) + ) } function pageFor(options: LoadSubsetOptions): Array { let page = [...rows.values()].filter( @@ -5721,7 +5894,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // A full join reads its full source too: a rootless row cannot be shown // unmatched until every root row has loaded. if (testCase.filter !== `source`) return undefined - if (boundary.some((rank) => rank === null || Number.isNaN(rank))) return + if (boundary.some((rank) => rank == null || Number.isNaN(rank))) return return prefixBound(boundary, shown.offset + shown.limit) } // Publication: a change publishes its window once. Each batch a subscriber @@ -5782,7 +5955,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // Generated rows may carry a null rank behind the `number` row type. prefix.every( (row) => - (row.rank as number | null) !== null && !Number.isNaN(row.rank), + (row.rank as number | null) != null && !Number.isNaN(row.rank), ) if (proven) { expect( @@ -5853,9 +6026,10 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ ) for (const [index, rowId] of shownIds.slice(1).entries()) { const predecessor = shownIds[index]! + // `null` and `undefined` ranks tie, so either may come first. const sameRank = Object.is( - ledger.get(predecessor)?.rank, - ledger.get(rowId)?.rank, + ledger.get(predecessor)?.rank ?? null, + ledger.get(rowId)?.rank ?? null, ) expect( sameRank || position.get(predecessor)! < position.get(rowId)!, @@ -5906,10 +6080,39 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ const requestsBefore = providerRequests const visibleBefore = JSON.stringify(visibleIds()) stepChangedInFlight = false + // Only a window move keeps a proven report's no-repair premise; any + // other step can change source ordering. + if (command.type !== `move`) provenSinceOrderingChange = false if (command.type === `refetch`) { const fullSourceBefore = fullSourceRequests const unexpressibleBefore = unexpressiblePageEnds - await refetch() + const firstAnswers = firstPageAnswers() + const firstRequestsBefore = requestsBy.get(firstSubscription) ?? 0 + const installedBefore = new Map(sent) + const reported = await refetch() + // A delete or rank change of an installed row that the first query's + // subscription received starts the repair path by itself + // (ARCHITECTURE.md), so only a refetch that changes neither is judged. + // A source-filter subscription receives every eligible row. + const keptInstalledRows = [...installedBefore.values()] + .filter((row) => row.eligible) + .every((row) => { + const now = sent.get(row.id) + return now !== undefined && Object.is(now.rank, row.rank) + }) + // A report that changes nothing the first query's own page proves + // starts no request for it (ARCHITECTURE.md: a window inside the + // loader's own first page needs no repair). + // The report's repair would start on a later pass, so the step's + // checkpoint judges it. + if ( + reported && + keptInstalledRows && + firstAnswers && + provenFirstPage(firstAnswers) + ) { + provenReportRequestsBefore = firstRequestsBefore + } // The repair a report starts reads each window's bounded prefix, // tie, and refill; a whole-source read would be the cost the report // exists to avoid. Two documented fallbacks read the full source @@ -6047,6 +6250,19 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ ) if (changedInFlight()) caseHeldChanges++ expectProviderLedger(`after step ${step}`) + if (provenReportRequestsBefore !== undefined) { + ignoredProvenReports++ + expect( + (requestsBy.get(firstSubscription) ?? 0) - provenReportRequestsBefore, + `requests the first query issued for a report its own page proves, at step ${step}`, + ).toBe(0) + provenReportRequestsBefore = undefined + provenSinceOrderingChange = true + } + expect( + repairAfterProvenReport, + `a repair the first query issued after a report its own page proved, by step ${step}`, + ).toBeUndefined() if (inFlight()) { expectInFlightWindow( visibleIds(), @@ -6175,6 +6391,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ heldRequests: caseHeldRequests, cooperativeCancels, lateAnswers, + ignoredProvenReports, } } finally { unsubscribe?.() @@ -7189,6 +7406,39 @@ describe(`eager indexed ordered windows`, () => { }) }) + // A peer's tie answer changes, so the provider reports, but the first + // query's page and tie answers stay the same and the peer installed the + // row after its page. That page proves the window, so the report must + // start no request for the first query and leave it owing no repair: the + // window move that follows continues from its boundary instead. + it(`starts no request for a window its own first page still proves after a report`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: true, + filter: `source`, + via: `marker`, + indexed: true, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 1, + offset: 0, + initial: [ + { id: 0, rank: 1, eligible: true }, + { id: 1, rank: 2, eligible: true }, + { id: 2, rank: 3, eligible: true }, + ], + history: [ + { type: `upsert`, id: 10, rank: 2, eligible: true }, + { type: `refetch` }, + { type: `move`, offset: 0, limit: 3 }, + ], + }) + }) + // A signalling refetch moves the first page's end earlier, to a rank that // a row the provider never sent ties with. The window is still inside the // page, but the rank-2 row past it is local only because the old tie @@ -8051,7 +8301,7 @@ describe(`eager indexed ordered windows`, () => { if ( hasRootless && window.some( - (id) => id >= 0 && (model.get(id)!.rank as number | null) === null, + (id) => id >= 0 && (model.get(id)!.rank as number | null) == null, ) ) { mixedNullGroup++ @@ -8079,8 +8329,31 @@ describe(`eager indexed ordered windows`, () => { // the reach check always reads the cases that test just executed. Reach is // keyed by case, so rerunning an identical case overwrites its entry; // shrinking runs only after a failure, which ends the test first. + // Reconstruction: the fixed campaign must reach the proven-report law, so + // a loader that repairs every report fails it. + it(`starts no repair for reports a window's own first page proves for a fixed seed`, async () => { + const PROVEN_SEED = 2091 + let proven = 0 + await fc.assert( + fc.asyncProperty(eagerProvenReportCase, async (testCase) => { + proven += (await runEagerWindow(testCase)).ignoredProvenReports + }), + { numRuns: runs, seed: PROVEN_SEED }, + ) + expect(proven, `reports a first page proved`).toBeGreaterThan(0) + }) + + fcTest.prop( + [eagerProvenReportCase], + oracleRandomParameters(runs, replay, `ordered-work.eager-proven-report`), + )( + `starts no repair for reports a window's own first page proves for a random or replayed seed`, + checkEagerWindow, + ) + it(`keeps a window whose loads answer between live changes for a fixed seed`, async () => { - const HELD_SEED = 2081 + // Chosen for reach after nullish ranks joined the row generator. + const HELD_SEED = 2083 const heldReach = new Map< string, Awaited> @@ -8104,7 +8377,7 @@ describe(`eager indexed ordered windows`, () => { ).toEqual([]) const reach = [...executed].map((key) => heldReach.get(key)!) // A case whose deletes refill from rows the query already holds issues - // no held request; at one times the run count, 35 of 40 cases issue one. + // no held request; at one times the run count, 33 of 40 cases issue one. expect( reach.filter((count) => count.heldRequests > 0).length, `held cases that issue a held request`, @@ -8116,7 +8389,7 @@ describe(`eager indexed ordered windows`, () => { `held cases with a held request but no in-flight change`, ).toEqual([]) // `release-peer` aborts held requests: at one times the run count, a - // cooperative provider cancels 4 and a late provider answers 1 anyway. + // cooperative provider cancels 6 and a late provider answers 4 anyway. expect( reach.reduce((sum, count) => sum + count.cooperativeCancels, 0), `held requests a cooperative provider cancelled`, From 7907d4e8434615843deb93f07d42d8750e61cbda Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 09:00:09 -0600 Subject: [PATCH 12/40] refactor(db): drop the equivalent undefined-end guard from report revalidation An `undefined` page end matched the loader's no-tie value, and the guard kept that match from ignoring a report. No cursor expresses a nullish end, so the loader has started or will start a full-source read, which covers the window either way. A mutant without the guard survived every ordered oracle. Saves 18 B gzip in @tanstack/db. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ordered-source-loader.ts | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b5ee5791ff..35b7247803 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. The guard that keeps an `undefined` end from matching the loader's no-tie sentinel is equivalent in reachable states: a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way, and a mutant without the guard survives the generated campaigns. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index b5d52d55cc..4ad9ecbf17 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -675,12 +675,12 @@ export class OrderedSourceLoader { // An installed row past the page proves the source's next row only when // the page read it, or when it ties with an end whose tie the loader // read. A refetch can move the end before that tie; reading the new - // end's tie then proves the window. + // end's tie then proves the window. An `undefined` end also matches a + // loader that attempted no tie. No cursor expresses a nullish end, so the + // loader has started or will start a full-source read, which covers it. if ( (this.ownPrefixCount ?? 0) > this.info.offset + this.info.limit || - // Undefined means that no tie request has been attempted. - (this.lastBoundary !== undefined && - Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary)) + Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary) ) return this.loadMovedEndTie(end) From 9f6cd4ab115531970872c931b7177b5789b17df0 Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 09:48:25 -0600 Subject: [PATCH 13/40] test(db): judge law 9 retention in generated replay histories The replay refinement oracle's generated section excused the window whenever a load failed, so only fixed tests judged that a failed replay keeps the last publication, and one named replay pinned a demand that fails while its replay is open. A retention law and a failed-replay campaign now judge both. - Law: ARCHITECTURE.md law 9 and "Replay participants and failure". A truncate raises a publication barrier; every load its source starts while it is up joins it; a participant's failure fails the replay while its logical demand is held; while any barrier is up the query shows the rows it showed when the first one rose. A participant counts as recovery for the terminal-error law. - Reach (createOracleReach, minimum 5, 200 histories each): judged failed replay fixed 72 / random 78; judged held replay 75 / 79; demand started after the commit failing while its replay loads 33 / 46; the include form 18 / 22. - RED: the branch's production files reverted to origin/main fail both failed-replay campaigns (convergence: terminal error). - GREEN: load-subset-replay-refinement-oracle.test.ts, 34 tests. - Mutants: a failed replay that releases the publication gate fails both failed-replay campaigns ("retained publication"); a subscriber that ends the query when a demand fails while its replay still loads fails both ("convergence: terminal error"). - Model corrections found while building it, each from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and its replacement publish; an ordered tie request is not lazy demand, which had let a failed tie request excuse a terminal error. - Limits: sources with two subscriptions in the query (a peer, or a query that both joins and includes markers) are not judged; barriers that rise before the first ready are not judged; a barrier where a release threw during the replay's start is skipped. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/tests/oracle-config.ts | 1 + ...ad-subset-replay-refinement-oracle.test.ts | 408 +++++++++++++++++- 3 files changed, 395 insertions(+), 16 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 35b7247803..693b79b765 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. Limits: the model judges recovery ownership only at the commit and for replay loads, because the adapter boundary cannot tell which subscription issued a load; a demand that starts after the commit while its subscription's replay is still open is non-fatal in production but only permitted by the model, so one named replay pins that case and a mutant that ends the query there fails only that replay. The generated section excuses the window whenever a load failed, so it does not judge that a failed recovery keeps the last publication; the file's fixed replay-failure tests kill a mutant that lets a failed replay stop withholding publication. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. Building the law found two model errors, both corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. Limits: the adapter boundary cannot tell which subscription issued a load, and each subscription has its own replay, so the retention law and the participant rule apply only where each recovering source has one subscription in the query (no peer, and not a query that both joins and includes the marker source). The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 9722cde44b..bf373471ab 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -21,6 +21,7 @@ const staticOracleProperties = [ `query-db.refetch-ordered-prefix`, `ordered-loader.request-authority`, `replay.ordered-window-faults`, + `replay.ordered-failed-replay`, `oracle-replay.calibration`, `structural-identity.equal-representations`, `structural-identity.independent-values`, diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index d80d8aa59f..b174562a09 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -12,6 +12,7 @@ import { import { BasicIndex } from '../../src/indexes/basic-index.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' import { evaluateReferenceExpression } from '../reference-expression-oracle.js' +import { createOracleReach } from '../oracle-reach.js' import { flushPromises } from '../utils.js' import { oraclePropertyOptions, @@ -1079,6 +1080,22 @@ describe(`loadSubset replay refinement`, () => { * it is a settlement-phase release, which must not hold back publication. * An unload that throws while the replay starts can fail the replay, which * law 9 allows, so the law does not judge that case. + * - **Retention:** a truncate replay is one publication barrier (law 9 and + * "Replay participants and failure"). It rises when a source truncates and + * falls when its replay succeeds. Every load the source starts while its + * replay is open joins the barrier; a load that a newer truncate aborts or + * whose demand retires stops participating, and work an older replay + * started still holds the newer one. A participant that fails fails the + * replay while its logical demand is held. Once that demand retires, its + * failure no longer counts, and demand started meanwhile joins the + * replacement, which publishes when it succeeds; otherwise only a later + * truncate of that source can replace the failed replay. While + * any barrier is up, the query shows exactly the rows it showed when the + * first one rose: partly replayed rows, rank moves, and marker changes stay + * private, and a failed replay keeps that result instead of ending the + * query. The law is checked after every command and once loading is quiet, + * for barriers that rise after the query first became ready; before that + * there is no complete publication to keep, and initial readiness governs. * - **Convergence:** after a final truncate with no faults, and an explicit * window move that revalidates a window whose own move failed, the window * equals the model. @@ -1104,6 +1121,10 @@ describe(`loadSubset replay refinement`, () => { * commit deletes the source's rows privately, so the query can demand * keys that it never published, such as a row that a LEFT join filter * now keeps. + * A load that joins an open replay belongs to it as well, which covers a + * demand that the graph starts after the commit while the replay still + * loads. The model judges that only where the source has one subscription + * in the query (see Limits). * A load that fails before the query first becomes ready is fatal too (an * initial load failure), except on a lazy join source, which owns its * demand. So a query may end in `error` only after a demand load outside a @@ -1140,7 +1161,13 @@ describe(`loadSubset replay refinement`, () => { * * Limits: one ordered source; no full joins (the ordered-work oracle owns * them); includes one level deep; and a provider that reads its rows at the time it - * answers. + * answers. The adapter boundary cannot tell which subscription issued a load, + * and each subscription has its own replay. So the retention law, and the + * rule that a replay participant counts as recovery, apply only where each + * recovering source has one subscription in the query: the order source + * without a peer, and the marker source unless the query both joins and + * includes it (or a peer reads it too). A release that throws while a replay starts may + * fail that replay, so the retention law skips a barrier where one did. */ type OrderedRow = { id: number; rank: number } type OrderedCommand = @@ -1277,6 +1304,10 @@ type OrderedReach = { replayLoads: number abortRejections: number preloadChecks: number + heldPublicationChecks: number + retainedFailedReplays: number + participantDemandFailures: number + includeDemandFailures: number } const ordinaryOrderedScenario: OrderedScenario = { @@ -1369,6 +1400,103 @@ async function checkOrderedReplay( // be placed before or after the replay's loads settle. type Load = { epoch: number; outcome: `pending` | `ok` | `failed` } const loads: Array = [] + // Retention model. Each source with an open or failed replay has a + // barrier. `pending` holds the loads that still hold its publication: the + // loads started while it is up, and work an older replay started. `own` + // holds the loads this replay started, whose failure fails it. A failure + // is scoped to its logical demand: once that demand retires, it no longer + // fails the replay, and demand started meanwhile joins the replacement. + type BarrierSource = `rows` | `markers` + type Barrier = { + failures: Map + // A release threw while this replay started, which may fail it. + ambiguous: boolean + pending: Map + own: Set + } + const barriers = new Map() + // The rows the query showed when the first barrier rose, if it had + // published by then. Before the first ready there is no complete + // publication to keep, and the initial-readiness law governs. + let retainedRows: string | undefined + let retainedAfterReady = false + const raiseBarrier = (source: BarrierSource) => { + lowerSettledBarriers() + if (barriers.size === 0) { + retainedRows = JSON.stringify(live.toArray) + retainedAfterReady = live.status === `ready` + } + barriers.set(source, { + failures: new Map(), + ambiguous: false, + pending: new Map(barriers.get(source)?.pending), + own: new Set(), + }) + } + // A replay whose participants all settled without a failure publishes. + const lowerSettledBarriers = () => { + for (const [source, barrier] of barriers) { + if (barrier.failures.size === 0 && barrier.pending.size === 0) { + barriers.delete(source) + } + } + if (barriers.size === 0) retainedRows = undefined + } + // Returns whether the load joined a replay. + const joinBarrier = ( + source: BarrierSource, + load: Load, + options: LoadSubsetOptions, + ) => { + const barrier = barriers.get(source) + if (!barrier) return false + barrier.pending.set(load, options) + barrier.own.add(load) + return true + } + const settleParticipant = ( + source: BarrierSource, + load: Load, + options: LoadSubsetOptions, + ok: boolean, + ) => { + const barrier = barriers.get(source) + // The replay has not failed, and other participants still load. + const replayLoading = + barrier !== undefined && + barrier.failures.size === 0 && + barrier.pending.has(load) && + barrier.pending.size > 1 + // An aborted request is obsolete demand, and a released one no longer + // participates, so neither fails the replay. + if ( + !ok && + barrier?.pending.has(load) && + barrier.own.has(load) && + !options.signal?.aborted + ) { + barrier.failures.set(load, options) + } + barrier?.pending.delete(load) + return replayLoading + } + const retireParticipants = ( + source: BarrierSource, + options: LoadSubsetOptions, + ) => { + const barrier = barriers.get(source) + if (!barrier) return + for (const owned of [barrier.pending, barrier.failures]) { + for (const [load, held] of owned) { + if (held === options) owned.delete(load) + } + } + } + // Each subscription has its own replay, and the adapter boundary cannot + // tell which subscription issued a load. + const oneSubscription = (source: BarrierSource) => + !scenario.peer && + (source === `rows` || !(scenario.joined && scenario.includes)) const acquired = new Map() const held: Array<(ok: boolean) => void> = [] let holding = false @@ -1391,8 +1519,22 @@ async function checkOrderedReplay( // Adapter callbacks set these flags, so read them through a call. const fatalLoadFailed = () => demandFailed || failedWhileLoading // Lazy join demand asks for keys; ordered and full-source loads do not. + // An ordered tie request also has a `where` and no `orderBy`, but it asks + // for rows by rank, so it is not demand. + const readsRank = (expression: unknown): boolean => + Array.isArray(expression) + ? expression.some(readsRank) + : typeof expression === `object` && expression !== null + ? Object.entries(expression).some(([field, value]) => + field === `path` && Array.isArray(value) + ? value.includes(`rank`) + : readsRank(value), + ) + : false const isDemand = (options: LoadSubsetOptions) => - options.where !== undefined && options.orderBy === undefined + options.where !== undefined && + options.orderBy === undefined && + !readsRank(options.where) // The truncate count of its source when each request was first issued. A // replay asks again for a held demand with the same request data, because // request data is immutable (ARCHITECTURE.md), and a new demand builds a @@ -1457,6 +1599,10 @@ async function checkOrderedReplay( replayLoads: 0, abortRejections: 0, preloadChecks: 0, + heldPublicationChecks: 0, + retainedFailedReplays: 0, + participantDemandFailures: 0, + includeDemandFailures: 0, } type Marker = { id: number; rowId: number } let markerSync!: Parameters[`sync`]>[0] @@ -1492,24 +1638,48 @@ async function checkOrderedReplay( const load: Load = { epoch: truncates, outcome: `pending` } loads.push(load) reach.markerLoads++ - const demand = - !belongsToRecovery(options, markerTruncates, markerCommitting) && - isDemand(options) + const participant = joinBarrier(`markers`, load, options) + const recovery = + belongsToRecovery(options, markerTruncates, markerCommitting) || + (participant && oneSubscription(`markers`)) + const demand = !recovery && isDemand(options) if (holding) { const ok = await new Promise((resolve) => held.push(resolve), ) + const replayLoading = settleParticipant( + `markers`, + load, + options, + ok, + ) if (!ok) { load.outcome = `failed` anyLoadFailed = true if (demand) demandFailed = true + // A demand the graph started after the commit, failing + // while its replay still loads. + if ( + participant && + replayLoading && + !options.signal?.aborted && + oneSubscription(`markers`) && + !markerCommitting && + isDemand(options) + ) { + reach.participantDemandFailures++ + if (scenario.includes) reach.includeDemandFailures++ + } if (queries.some(({ status }) => status === `loading`)) { failedWhileLoading = true } reach.failedLoads++ throw inject(new Error(`marker load failed`)) } - } else await Promise.resolve() + } else { + await Promise.resolve() + settleParticipant(`markers`, load, options, true) + } if (options.signal?.aborted) { load.outcome = `ok` if (scenario.abortedMarkerLoads === `resolve`) return @@ -1539,6 +1709,7 @@ async function checkOrderedReplay( load.outcome = `ok` }, unloadSubset: (options) => { + retireParticipants(`markers`, options) const count = acquired.get(options) ?? 0 if (count === 0) { violation ??= `marker unloadSubset released a demand it did not hold` @@ -1564,24 +1735,42 @@ async function checkOrderedReplay( acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { epoch: truncates, outcome: `pending` } loads.push(load) - const demand = - !belongsToRecovery(options, truncates, truncating) && - isDemand(options) + const participant = joinBarrier(`rows`, load, options) + const recovery = + belongsToRecovery(options, truncates, truncating) || + (participant && oneSubscription(`rows`)) + const demand = !recovery && isDemand(options) if (holding) { const ok = await new Promise((resolve) => held.push(resolve), ) + const replayLoading = settleParticipant(`rows`, load, options, ok) if (!ok) { load.outcome = `failed` anyLoadFailed = true if (demand) demandFailed = true + // A demand the graph started after the commit, failing + // while its replay still loads. + if ( + participant && + replayLoading && + !options.signal?.aborted && + oneSubscription(`rows`) && + !truncating && + isDemand(options) + ) { + reach.participantDemandFailures++ + } if (queries.some(({ status }) => status === `loading`)) { failedWhileLoading = true } reach.failedLoads++ throw inject(new Error(`load failed`)) } - } else await Promise.resolve() + } else { + await Promise.resolve() + settleParticipant(`rows`, load, options, true) + } if (options.signal?.aborted) { load.outcome = `ok` return @@ -1625,6 +1814,7 @@ async function checkOrderedReplay( load.outcome = `ok` }, unloadSubset: (options) => { + retireParticipants(`rows`, options) const count = acquired.get(options) ?? 0 if (count === 0) { violation ??= `unloadSubset released a demand it did not hold` @@ -1643,6 +1833,8 @@ async function checkOrderedReplay( )) if (replayUnsettled) { earlyThrowSinceTruncate = true + const barrier = barriers.get(`rows`) + if (barrier) barrier.ambiguous = true reach.replayStartThrows++ } else if (truncates > 0) reach.throwsAfterReplay++ else reach.settlementThrows++ @@ -1723,6 +1915,27 @@ async function checkOrderedReplay( } }), ) + // Retention: while a barrier is up, the query shows the rows it showed when + // the first one rose. A query in `error` is judged by the terminal-error + // law instead. + const checkRetention = (label: string) => { + lowerSettledBarriers() + if (retainedRows === undefined || !retainedAfterReady) return + // A source with two subscriptions has two replays, which the adapter + // boundary cannot tell apart. + if ([...barriers.keys()].some((barrier) => !oneSubscription(barrier))) { + return + } + const raised = [...barriers.values()] + if (raised.some(({ ambiguous }) => ambiguous)) return + if (live.status === `error`) return + expect(JSON.parse(JSON.stringify(live.toArray)), label).toEqual( + JSON.parse(retainedRows), + ) + if (raised.some(({ failures }) => failures.size > 0)) { + reach.retainedFailedReplays++ + } else reach.heldPublicationChecks++ + } const settleAll = async () => { // Each truncate replays every retained demand of each query, so a long // held history can hold hundreds of loads. A load that starts another @@ -1740,6 +1953,7 @@ async function checkOrderedReplay( earlyThrowSinceTruncate = false if (held.length > 0) reach.heldAcrossTruncate++ installed.clear() + raiseBarrier(`rows`) truncating = true try { sync.begin() @@ -1755,6 +1969,7 @@ async function checkOrderedReplay( // demand it retains. const truncateMarkers = async () => { installedMarkers.clear() + raiseBarrier(`markers`) markerSync.begin() markerSync.truncate() if (!onDemandMarkers) { @@ -1899,6 +2114,7 @@ async function checkOrderedReplay( swallow(error) } await flushPromises() + checkRetention(`retained publication after ${command.type}`) } holding = false throwNextUnload = false @@ -1922,6 +2138,7 @@ async function checkOrderedReplay( cause: escapedError(), }) } + checkRetention(`retained publication once loading is quiet`) // Status: a release that throws is reported, but it fails no load, so a // quiet query with no failed load stays ready. if (!anyLoadFailedNow()) { @@ -2298,8 +2515,9 @@ describe(`Ordered windows across truncate replays`, () => { history: Array scenario?: OrderedScenario reaches?: keyof OrderedReach - // A replay whose history the law places in a recovery, where the model's - // own classification is permissive (see the terminal-error law). + // A replay whose history the law places in a recovery. The pin also + // holds where the model cannot tell which subscription issued a load + // (see Limits). recovers?: true }>)( `keeps the ordered window for $name (generated replay)`, @@ -2340,6 +2558,10 @@ describe(`Ordered windows across truncate replays`, () => { replayLoads: 0, abortRejections: 0, preloadChecks: 0, + heldPublicationChecks: 0, + retainedFailedReplays: 0, + participantDemandFailures: 0, + includeDemandFailures: 0, } await fc.assert( fc.asyncProperty( @@ -2365,15 +2587,171 @@ describe(`Ordered windows across truncate replays`, () => { // law is checked. // An aborted move needs a truncate to land inside a held move, and a // terminal error needs a held marker demand load to fail; the named - // replays above reach both. + // replays above reach both. A failed replay that the retention law can + // judge, and a demand that fails while its replay is open, are rare + // here; the failed-replay campaigns below declare both. + const ownedElsewhere = new Set([ + `abortedMoves`, + `terminalErrors`, + `retainedFailedReplays`, + `participantDemandFailures`, + `includeDemandFailures`, + ]) if (campaign !== `replay`) { for (const [dimension, count] of Object.entries(total)) { - if (dimension === `abortedMoves` || dimension === `terminalErrors`) - continue + if (ownedElsewhere.has(dimension)) continue expect(count, dimension).toBeGreaterThan(0) } } }, 120_000, ) + + // The retention law needs a replay that fails while the query holds its + // last publication. Each history here starts by holding the next loads and + // truncating a source, so the replay's loads stay held. It may then change + // a rank, a marker, or the window, which starts demand while the replay is + // open. It then fails a held load: the oldest is a replay load, and the + // newest is the demand just started. Random commands follow, so later + // moves, marks, window moves, and truncates act on a failed replay. Peers + // stay out because the retention law does not judge them. + const failedReplayTruncate = fc.constantFrom( + { type: `truncate` as const, armAfterStart: false }, + { type: `truncateMarkers` as const }, + ) + const failedReplayHistory: fc.Arbitrary> = fc + .tuple( + failedReplayTruncate, + fc.option( + fc.oneof( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + fc.record({ + type: fc.constant(`mark` as const), + id: fc.integer({ min: 1, max: 5 }), + on: fc.boolean(), + }), + fc.record({ + type: fc.constant(`window` as const), + offset: fc.constantFrom(0 as const, 1 as const), + limit: fc.constantFrom(1 as const, 2 as const, 3 as const), + }), + ), + { nil: undefined }, + ), + fc.boolean(), + fc.array(orderedCommand, { maxLength: 8 }), + ) + .map(([truncate, change, keepHolding, rest]) => [ + { type: `hold` as const }, + truncate, + ...(change + ? [change, { type: `settle` as const, ok: false, newest: true }] + : [{ type: `settle` as const, ok: false, keepHolding }]), + ...rest, + ]) + const failedReplayScenario = orderedScenario.map((scenario) => ({ + ...scenario, + peer: false, + })) + // An include demand that fails while its replay still loads needs an + // include over on-demand markers and a move that brings a row into the + // window during the marker replay; this branch builds that start. The + // window starts at offset 0 without ties, and the moved row starts past + // it, so moving it before every other row shows it. + const includeDemandCase = failedReplayScenario.chain((scenario) => { + const included = { + ...scenario, + offset: 0 as const, + ties: false, + joined: false as const, + includes: true, + markerMode: `on-demand` as const, + } + const ascending = included.direction === `asc` + return fc + .tuple( + fc.integer({ min: 1, max: 9 }), + fc.array(orderedCommand, { maxLength: 8 }), + ) + .map(([rank, rest]): [Array, OrderedScenario] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { + type: `move`, + id: ascending ? 5 : 1, + rank: ascending ? rank : 61 - rank, + }, + { type: `settle`, ok: false, newest: true }, + ...rest, + ], + included, + ]) + }) + const failedReplayCase = fc.oneof( + { + weight: 3, + arbitrary: fc.tuple(failedReplayHistory, failedReplayScenario), + }, + { weight: 1, arbitrary: includeDemandCase }, + ) + const failedReplayProperty = `replay.ordered-failed-replay` + const failedReplayCampaigns = + orderedReplayConfig.replayPath === undefined + ? ([`fixed`, `random`] as const) + : orderedReplayConfig.replayProperty === failedReplayProperty + ? ([`replay`] as const) + : ([] as const) + + it.each(failedReplayCampaigns)( + `keeps the last publication through a failed replay (%s campaign)`, + async (campaign) => { + // Declared reach: histories where the retention law judged a failed + // replay, judged a replay that still held publication, and saw a + // demand that the graph started after the commit fail while its replay + // still loaded, on either source and for an include. An include + // demand's rejection arrives before its replay records the failure, so + // only there does the demand wait for the replay to decide it. + const reach = createOracleReach( + failedReplayProperty, + [ + `retainedFailedReplays`, + `heldPublicationChecks`, + `participantDemandFailures`, + `includeDemandFailures`, + ] as const, + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(failedReplayCase, async ([history, scenario]) => { + reach.beginHistory() + const observed = await checkOrderedReplay(history, scenario) + if (observed.retainedFailedReplays > 0) { + reach.reach(`retainedFailedReplays`) + } + if (observed.heldPublicationChecks > 0) { + reach.reach(`heldPublicationChecks`) + } + if (observed.participantDemandFailures > 0) { + reach.reach(`participantDemandFailures`) + } + if (observed.includeDemandFailures > 0) { + reach.reach(`includeDemandFailures`) + } + }), + campaign === `fixed` + ? { seed: 20261023, numRuns: oracleRuns(200) } + : { + ...oraclePropertyOptions(200, failedReplayProperty), + ...(campaign === `replay` ? { numRuns: 1 } : {}), + }, + ) + reach.assertReached() + }, + 120_000, + ) }) From 4614d39689ff88d85ef1af3bcc6091ecf0bd4996 Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 10:12:16 -0600 Subject: [PATCH 14/40] test(db): generate replay recovery during the initial preload Two production repairs in this branch were killed only by named replays: the deferred window move that a discarded replay rejected, and the guard that keeps a failure before the first ready fatal. A preload-recovery campaign now generates both. - Law: ARCHITECTURE.md "Replay participants and failure" (a deferred move waits for every replay that holds publication and applies unless one failed) and the initial-readiness law (a failure before the first ready ends the query, so the preload settles). - Reach (createOracleReach, minimum 5, 300 histories each): lazy demand in a replay failing before the first ready, fixed 37 / random 32-40; a window move behind a replay a newer truncate discards, 41 / 39-49. - RED: the builder from 21fbd0394 fails the fixed campaign ("settled window"); the builder without the initial-failure guard fails it ("initial preload settles"). The fixed seed kills both; each dies in about half of 200-run seeds. - GREEN: load-subset-replay-refinement-oracle.test.ts, 36 tests, three runs. - Model correction: a replay completes when its last participant's rows apply, so the retention model now settles a participant after its commit and lowers the barrier then. Demand that a deferred move starts afterwards no longer joins a finished replay. The gate-release and pending-replacement mutants still fail both failed-replay campaigns. - Limits: the random campaign kills each named mutant about half the time. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/tests/oracle-config.ts | 1 + ...ad-subset-replay-refinement-oracle.test.ts | 209 ++++++++++++++++-- 3 files changed, 194 insertions(+), 18 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 693b79b765..68100559aa 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. Building the law found two model errors, both corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. Limits: the adapter boundary cannot tell which subscription issued a load, and each subscription has its own replay, so the retention law and the participant rule apply only where each recovering source has one subscription in the query (no peer, and not a query that both joins and includes the marker source). The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. Limits: the adapter boundary cannot tell which subscription issued a load, and each subscription has its own replay, so the retention law and the participant rule apply only where each recovering source has one subscription in the query (no peer, and not a query that both joins and includes the marker source). The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index bf373471ab..c5b6d464d0 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -22,6 +22,7 @@ const staticOracleProperties = [ `ordered-loader.request-authority`, `replay.ordered-window-faults`, `replay.ordered-failed-replay`, + `replay.ordered-preload-recovery`, `oracle-replay.calibration`, `structural-identity.equal-representations`, `structural-identity.independent-values`, diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index b174562a09..8fcce9082d 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1308,6 +1308,8 @@ type OrderedReach = { retainedFailedReplays: number participantDemandFailures: number includeDemandFailures: number + preloadReplayFailures: number + discardedReplayMoves: number } const ordinaryOrderedScenario: OrderedScenario = { @@ -1422,6 +1424,11 @@ async function checkOrderedReplay( let retainedAfterReady = false const raiseBarrier = (source: BarrierSource) => { lowerSettledBarriers() + // A newer truncate discards the replay that a pending window move waits + // behind. + if (barriers.size > 0 && !settledMoves.has(latestMove)) { + reach.discardedReplayMoves++ + } if (barriers.size === 0) { retainedRows = JSON.stringify(live.toArray) retainedAfterReady = live.status === `ready` @@ -1478,6 +1485,9 @@ async function checkOrderedReplay( barrier.failures.set(load, options) } barrier?.pending.delete(load) + // A replay completes when its last participant settles, so demand that + // starts after it does not join it. + lowerSettledBarriers() return replayLoading } const retireParticipants = ( @@ -1603,6 +1613,8 @@ async function checkOrderedReplay( retainedFailedReplays: 0, participantDemandFailures: 0, includeDemandFailures: 0, + preloadReplayFailures: 0, + discardedReplayMoves: 0, } type Marker = { id: number; rowId: number } let markerSync!: Parameters[`sync`]>[0] @@ -1647,13 +1659,13 @@ async function checkOrderedReplay( const ok = await new Promise((resolve) => held.push(resolve), ) - const replayLoading = settleParticipant( - `markers`, - load, - options, - ok, - ) if (!ok) { + const replayLoading = settleParticipant( + `markers`, + load, + options, + false, + ) load.outcome = `failed` anyLoadFailed = true if (demand) demandFailed = true @@ -1672,16 +1684,18 @@ async function checkOrderedReplay( } if (queries.some(({ status }) => status === `loading`)) { failedWhileLoading = true + // Lazy demand in a replay failed before the first ready. + if (participant && isDemand(options)) { + reach.preloadReplayFailures++ + } } reach.failedLoads++ throw inject(new Error(`marker load failed`)) } - } else { - await Promise.resolve() - settleParticipant(`markers`, load, options, true) - } + } else await Promise.resolve() if (options.signal?.aborted) { load.outcome = `ok` + settleParticipant(`markers`, load, options, true) if (scenario.abortedMarkerLoads === `resolve`) return reach.abortRejections++ throw inject( @@ -1707,6 +1721,8 @@ async function checkOrderedReplay( } await operations.commit() load.outcome = `ok` + // The load settles once its rows apply. + settleParticipant(`markers`, load, options, true) }, unloadSubset: (options) => { retireParticipants(`markers`, options) @@ -1744,8 +1760,13 @@ async function checkOrderedReplay( const ok = await new Promise((resolve) => held.push(resolve), ) - const replayLoading = settleParticipant(`rows`, load, options, ok) if (!ok) { + const replayLoading = settleParticipant( + `rows`, + load, + options, + false, + ) load.outcome = `failed` anyLoadFailed = true if (demand) demandFailed = true @@ -1763,16 +1784,18 @@ async function checkOrderedReplay( } if (queries.some(({ status }) => status === `loading`)) { failedWhileLoading = true + // Lazy demand in a replay failed before the first ready. + if (participant && isDemand(options)) { + reach.preloadReplayFailures++ + } } reach.failedLoads++ throw inject(new Error(`load failed`)) } - } else { - await Promise.resolve() - settleParticipant(`rows`, load, options, true) - } + } else await Promise.resolve() if (options.signal?.aborted) { load.outcome = `ok` + settleParticipant(`rows`, load, options, true) return } const rows = [...truth.values()] @@ -1812,6 +1835,8 @@ async function checkOrderedReplay( } await sync.commit() load.outcome = `ok` + // The load settles once its rows apply. + settleParticipant(`rows`, load, options, true) }, unloadSubset: (options) => { retireParticipants(`rows`, options) @@ -2562,6 +2587,8 @@ describe(`Ordered windows across truncate replays`, () => { retainedFailedReplays: 0, participantDemandFailures: 0, includeDemandFailures: 0, + preloadReplayFailures: 0, + discardedReplayMoves: 0, } await fc.assert( fc.asyncProperty( @@ -2588,14 +2615,17 @@ describe(`Ordered windows across truncate replays`, () => { // An aborted move needs a truncate to land inside a held move, and a // terminal error needs a held marker demand load to fail; the named // replays above reach both. A failed replay that the retention law can - // judge, and a demand that fails while its replay is open, are rare - // here; the failed-replay campaigns below declare both. + // judge, a demand that fails while its replay is open, a replay that + // fails before the first ready, and a window move behind a discarded + // replay are rare here; the campaigns below declare them. const ownedElsewhere = new Set([ `abortedMoves`, `terminalErrors`, `retainedFailedReplays`, `participantDemandFailures`, `includeDemandFailures`, + `preloadReplayFailures`, + `discardedReplayMoves`, ]) if (campaign !== `replay`) { for (const [dimension, count] of Object.entries(total)) { @@ -2707,6 +2737,151 @@ describe(`Ordered windows across truncate replays`, () => { ? ([`replay`] as const) : ([] as const) + // Recovery during the initial preload. A history starts while the first + // loads are held, over on-demand markers that the query includes or joins. + // Its grammar settles held loads one at a time, truncates either source, + // moves the window, and fails a held load, so replays start, fail, and are + // discarded before the query first becomes ready, with window moves + // waiting behind them. + const preloadCommand: fc.Arbitrary = fc.oneof( + { + weight: 4, + arbitrary: fc.record({ + type: fc.constant(`settle` as const), + ok: fc.oneof( + { weight: 3, arbitrary: fc.constant(true) }, + { weight: 1, arbitrary: fc.constant(false) }, + ), + keepHolding: fc.oneof( + { weight: 3, arbitrary: fc.constant(true) }, + { weight: 1, arbitrary: fc.constant(false) }, + ), + newest: fc.boolean(), + }), + }, + { weight: 2, arbitrary: fc.constant({ type: `truncateMarkers` as const }) }, + { + weight: 1, + arbitrary: fc.constant({ + type: `truncate` as const, + armAfterStart: false, + }), + }, + { + weight: 2, + arbitrary: fc.record({ + type: fc.constant(`window` as const), + offset: fc.constantFrom(0 as const, 1 as const), + limit: fc.constantFrom(1 as const, 2 as const, 3 as const), + }), + }, + { + weight: 1, + arbitrary: fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + }, + { weight: 1, arbitrary: fc.constant({ type: `hold` as const }) }, + ) + const preloadScenario = orderedScenario.map((scenario) => ({ + ...scenario, + markerMode: `on-demand` as const, + includes: scenario.includes || !scenario.joined, + historyDuringPreload: true, + })) + // A replay of include demand that fails before the first ready: initial + // loads settle one at a time, so the order source delivers rows and the + // include demands them; the marker source truncates; more held loads + // settle; and the oldest or newest held load fails. The oldest is often + // a load the truncate aborted, and the newest a replay load. + const settleHeld = { type: `settle` as const, ok: true, keepHolding: true } + const preloadDemandFailure = fc + .tuple( + preloadScenario, + fc.integer({ min: 1, max: 2 }), + fc.integer({ min: 0, max: 3 }), + fc.boolean(), + fc.array(preloadCommand, { maxLength: 6 }), + ) + .map( + ([scenario, before, after, newest, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + ...Array.from({ length: before }, () => settleHeld), + { type: `truncateMarkers` }, + ...Array.from({ length: after }, () => settleHeld), + { type: `settle`, ok: false, newest }, + ...rest, + ], + { ...scenario, joined: false, includes: true }, + ], + ) + const preloadCase = fc.oneof( + { + weight: 2, + arbitrary: fc.tuple( + fc.array(preloadCommand, { minLength: 1, maxLength: 12 }), + preloadScenario, + ), + }, + { weight: 1, arbitrary: preloadDemandFailure }, + ) + const preloadProperty = `replay.ordered-preload-recovery` + const preloadCampaigns = + orderedReplayConfig.replayPath === undefined + ? ([`fixed`, `random`] as const) + : orderedReplayConfig.replayProperty === preloadProperty + ? ([`replay`] as const) + : ([] as const) + + it.each(preloadCampaigns)( + `recovers replays that start during the initial preload (%s campaign)`, + async (campaign) => { + // Declared reach: a replay participant that fails before the first + // ready, which must end the query or let the preload settle, and a + // window move that waits behind a replay a newer truncate discards, + // which must still apply. + const reach = createOracleReach( + preloadProperty, + [ + `preloadReplayFailures`, + `discardedReplayMoves`, + `preloadChecks`, + ] as const, + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(preloadCase, async ([history, scenario]) => { + reach.beginHistory() + const observed = await checkOrderedReplay(history, scenario) + for (const state of [ + `preloadReplayFailures`, + `discardedReplayMoves`, + `preloadChecks`, + ] as const) { + if (observed[state] > 0) reach.reach(state) + } + }), + // The fixed seed kills both mutants that the evidence names: a + // builder that lets a discarded replay reject a deferred window move, + // and one that keeps a failure before the first ready non-fatal. Each + // dies in about half of 200-run seeds. + campaign === `fixed` + ? { seed: 20261027, numRuns: oracleRuns(300) } + : { + ...oraclePropertyOptions(300, preloadProperty), + ...(campaign === `replay` ? { numRuns: 1 } : {}), + }, + ) + reach.assertReached() + }, + 120_000, + ) + it.each(failedReplayCampaigns)( `keeps the last publication through a failed replay (%s campaign)`, async (campaign) => { From a45ebcc6bcbe913b0fd45ca20b9c984b43aed496 Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 10:18:53 -0600 Subject: [PATCH 15/40] test(db): declare reach for the ordered report laws this branch added The unproven-report law was reached in about one broad-campaign history in 10,000, the proven-report random campaign counted no reach, and the refetch law's unindexed short-refill exception had no reach on either side. - Law: ARCHITECTURE.md ordered recovery. A report over a first page that no row the loader read proves starts a repair or reads the end's tie; a report the first page proves starts nothing; a signalling refetch's repair reads no full source except the refill of an unindexed plan whose bounded prefix leaves the window short. - Reach (createOracleReach): unproven-report campaign, 300 histories, minimum 20: fixed 104, random 84. Unindexed-refetch campaign, minimum 5: law applied fixed 16 / random 18, short refill excused 26 / 24. Proven-report random campaign: at least one history. - RED: main's loader fails both unproven-report campaigns. - GREEN: ordered-source-loader-state and ordered-work oracles, 354 tests. - Mutant: a loader that answers every report with a full-source read fails the unindexed-refetch fixed seed and random runs; about half of fixed seeds kill it, and this one does. - Limits: the unproven-report campaign starts every history with the same three commands; the fake's installed rows ignore a read's limit, so the first-page ignore branch stays with the ordered-work oracle. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/tests/oracle-config.ts | 2 + ...ordered-source-loader-state-oracle.test.ts | 62 +++++++++++++ .../ordered-work-oracle.property.test.ts | 93 +++++++++++++++++-- 4 files changed, 150 insertions(+), 9 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 68100559aa..b1b927ec06 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. Limits: the adapter boundary cannot tell which subscription issued a load, and each subscription has its own replay, so the retention law and the participant rule apply only where each recovering source has one subscription in the query (no peer, and not a query that both joins and includes the marker source). The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. An unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random) over joined windows behind a signalling provider declares reach on both sides of that exception, where the law applies and where the short refill excuses it; the full-source mutant fails its fixed seed and random runs. The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. Limits: the adapter boundary cannot tell which subscription issued a load, and each subscription has its own replay, so the retention law and the participant rule apply only where each recovering source has one subscription in the query (no peer, and not a query that both joins and includes the marker source). The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index c5b6d464d0..e812032369 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -20,6 +20,7 @@ const staticOracleProperties = [ `cursor-pagination.failure`, `query-db.refetch-ordered-prefix`, `ordered-loader.request-authority`, + `ordered-loader.unproven-report`, `replay.ordered-window-faults`, `replay.ordered-failed-replay`, `replay.ordered-preload-recovery`, @@ -170,6 +171,7 @@ const staticOracleProperties = [ `ordered-work.eager-full-join`, `ordered-work.eager-held-load`, `ordered-work.eager-proven-report`, + `ordered-work.eager-unindexed-refetch`, `ordered-work.lifecycle`, `ordered-work.nullable-lifecycle`, `pagination.async-cursor`, diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 7a69bf283c..8967b3cc49 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -9,6 +9,7 @@ import { } from '../../src/query/live/ordered-source-loader.js' import { PropRef } from '../../src/query/ir.js' import { evaluateReferenceExpression } from '../reference-expression-oracle.js' +import { createOracleReach } from '../oracle-reach.js' import { captureMicrotaskErrors, flushPromises } from '../utils.js' import { oraclePropertyOptions, @@ -3894,4 +3895,65 @@ describe(`Ordered source request authority`, () => { }, 120_000, ) + + // The unproven-report law needs a quiet history whose coverage is the + // loader's own first page, longer than the window, with no more rows to + // read, and then a report whose pass reacts. The broad grammar rarely lines + // these up, so each history here settles the first page, settles the read + // after it, and reports, over random case parameters, before a random + // suffix. A release that truncates or disposes would end the quiet history. + const unprovenReportProperty = `ordered-loader.unproven-report` + const unprovenReportCase: fc.Arbitrary = fc + .tuple(authorityCaseArbitrary, fc.array(authorityCommand, { maxLength: 8 })) + .map(([testCase, rest]) => ({ + ...testCase, + settledFirst: true, + requiresFullSource: false, + snapshotRows: 3, + needsMore: false, + releaseTruncates: false, + releaseDisposes: false, + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `report`, react: true }, + ...rest, + ], + })) + const unprovenReportCampaigns = + authorityReplay.replayPath === undefined + ? ([`fixed`, `random`] as const) + : authorityReplay.replayProperty === unprovenReportProperty + ? ([`replay`] as const) + : ([] as const) + + it.each(unprovenReportCampaigns)( + `reads the end's tie or repairs after an unproven report (%s campaign)`, + async (campaign) => { + // Declared reach: histories where the unproven-report law judged a + // report over the loader's own first page. + const reach = createOracleReach( + unprovenReportProperty, + [`repairedReportsOnOwnPage`] as const, + { minimum: 20 }, + ) + await fc.assert( + fc.asyncProperty(unprovenReportCase, async (testCase) => { + reach.beginHistory() + const observation = await checkAuthorityCase(testCase) + if (observation.repairedReportsOnOwnPage > 0) { + reach.reach(`repairedReportsOnOwnPage`) + } + }), + campaign === `fixed` + ? { seed: 20261023, numRuns: oracleRuns(300) } + : { + ...oraclePropertyOptions(300, unprovenReportProperty), + ...(campaign === `replay` ? { numRuns: 1 } : {}), + }, + ) + reach.assertReached() + }, + 120_000, + ) }) diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index bb621a79ab..af5345ecd6 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -18,6 +18,7 @@ import { readOracleRunConfig, } from '../oracle-config.js' import { evaluateReferenceExpression } from '../reference-expression-oracle.js' +import { createOracleReach } from '../oracle-reach.js' import { flushPromises } from '../utils.js' import { withHistoryCleanup } from '../optimistic-history-oracle.js' import type { InitialQueryBuilder } from '../../src/query/builder/index.js' @@ -5089,6 +5090,8 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ cooperativeCancels: number lateAnswers: number ignoredProvenReports: number + unindexedRefetchChecks: number + shortUnindexedRefills: number }> { const joined = isJoinedFilter(testCase) const fullJoin = testCase.filter === `full-joined` @@ -5298,6 +5301,10 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // Reports that the first query's own page proved, so it owed no request, // and the first query's request count before the latest such report. let ignoredProvenReports = 0 + // Signalling refetches of an unindexed plan where the full-source law + // applied, and where the documented short-prefix fallback excused it. + let unindexedRefetchChecks = 0 + let shortUnindexedRefills = 0 let provenReportRequestsBefore: number | undefined // Set by such a report until a step that can change source ordering, and // the first repair request the first query issued meanwhile. @@ -6120,10 +6127,18 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // an unindexed plan whose bounded prefix leaves the window short // (ARCHITECTURE.md, "an underfilled unindexed prefix retains the // established full-source fallback"). + const shortRefill = + !testCase.indexed && prefixLeavesWindowShort(rows, shown) + if (signalsRefetch && unexpressiblePageEnds === unexpressibleBefore) { + if (!testCase.indexed) { + if (shortRefill) shortUnindexedRefills++ + else unindexedRefetchChecks++ + } + } if ( signalsRefetch && unexpressiblePageEnds === unexpressibleBefore && - (testCase.indexed || !prefixLeavesWindowShort(rows, shown)) + !shortRefill ) { expect( fullSourceRequests, @@ -6392,6 +6407,8 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ cooperativeCancels, lateAnswers, ignoredProvenReports, + unindexedRefetchChecks, + shortUnindexedRefills, } } finally { unsubscribe?.() @@ -8343,13 +8360,73 @@ describe(`eager indexed ordered windows`, () => { expect(proven, `reports a first page proved`).toBeGreaterThan(0) }) - fcTest.prop( - [eagerProvenReportCase], - oracleRandomParameters(runs, replay, `ordered-work.eager-proven-report`), - )( - `starts no repair for reports a window's own first page proves for a random or replayed seed`, - checkEagerWindow, - ) + it(`starts no repair for reports a window's own first page proves for a random or replayed seed`, async () => { + const options = oracleRandomParameters( + runs, + replay, + `ordered-work.eager-proven-report`, + ) + const reach = createOracleReach(`ordered-work.eager-proven-report`, [ + `ignoredProvenReports`, + ] as const) + await fc.assert( + fc.asyncProperty(eagerProvenReportCase, async (testCase) => { + reach.beginHistory() + const observed = await runEagerWindow(testCase) + if (observed.ignoredProvenReports > 0) + reach.reach(`ignoredProvenReports`) + }), + options, + ) + reach.assertReached() + }) + + // The refetch law over unindexed joined windows behind a signalling + // provider. A repair must not read the full source, except for the + // documented refill of an unindexed plan whose bounded prefix leaves the + // window short. The campaign must reach both sides of that exception. + // Ineligible rows crowd the bounded prefix and leave the window short, so + // half the cases start with every row eligible, where the law applies. + const eagerUnindexedRefetchCase = fc + .tuple(eagerJoinedCase, fc.boolean()) + .map(([testCase, allEligible]) => ({ + ...testCase, + path: `on-demand` as const, + provider: `refetch-signal` as const, + indexed: false, + peer: false, + initial: allEligible + ? testCase.initial.map((row) => ({ ...row, eligible: true })) + : testCase.initial, + })) + for (const campaign of [`fixed`, `random`] as const) { + it(`reads the full source after a signalling refetch only for a short unindexed refill (${campaign} campaign)`, async () => { + const property = `ordered-work.eager-unindexed-refetch` + const reach = createOracleReach( + property, + [`unindexedRefetchChecks`, `shortUnindexedRefills`] as const, + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(eagerUnindexedRefetchCase, async (testCase) => { + reach.beginHistory() + const observed = await runEagerWindow(testCase) + for (const state of [ + `unindexedRefetchChecks`, + `shortUnindexedRefills`, + ] as const) { + if (observed[state] > 0) reach.reach(state) + } + }), + campaign === `fixed` + ? // This seed kills a loader that repairs every report with a + // full-source read; about half of seeds do. + { numRuns: runs, seed: 2094 } + : oracleRandomParameters(runs, replay, property), + ) + reach.assertReached() + }) + } it(`keeps a window whose loads answer between live changes for a fixed seed`, async () => { // Chosen for reach after nullish ranks joined the row generator. From f9874017e334892510c9df366163053b2f88e56b Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 11:25:51 -0600 Subject: [PATCH 16/40] fix(db): share one replay barrier across a query's subscriptions to a collection A query that joins and includes the same on-demand source has two subscriptions to it, each with its own truncate replay. A truncate's commit can retire one subscription's only demand privately; its emptied replay then closed while the other's still loaded, and demand the graph started for rows the commit exposed fell outside every replay. Its failure ended the query with error 228, the #2224 class through a second subscription. A subscription's replay now neither publishes nor closes as empty while a sibling subscription of the same collection in the graph still has replay work, so demand started meanwhile joins it. - Law: ARCHITECTURE.md "Replay participants and failure", revised here: a live query's subscriptions to one collection share one publication barrier. The replay oracle's participant rule and retention law now apply to them (previously skipped as unattributable). - Reach (createOracleReach, minimum 5): a sibling-demand branch of the failed-replay campaign declares a demand that fails inside the shared barrier of a query that joins and includes markers. - RED: the parent commit's production code fails the named replay "a demand that fails while a sibling subscription's replay still loads" and both failed-replay campaigns at "convergence: terminal error". - GREEN: replay refinement oracle 37 tests; @tanstack/db 285 files / 13,816 tests; test:dist 323; query-db 42 files. - Mutants: removing the hold at replay completion, or the hold for an emptied replay, each fails the named replay and both campaigns. - Size: +175 B gzip in @tanstack/db (+276 to +451). - Limits: histories with a peer query stay unjudged by the retention law. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/mangle-cache.json | 3 +- packages/db/src/collection/subscription.ts | 41 ++++++- packages/db/src/query/live/ARCHITECTURE.md | 7 +- .../query/live/collection-config-builder.ts | 18 +++ .../src/query/live/collection-subscriber.ts | 2 + packages/db/src/types.ts | 5 + ...ion-subscription-reentrancy-oracle.test.ts | 1 + ...ubscription-replay-oracle.property.test.ts | 11 +- .../db/tests/collection-subscription.test.ts | 1 + ...ad-subset-replay-refinement-oracle.test.ts | 109 +++++++++++++++--- 11 files changed, 174 insertions(+), 26 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b1b927ec06..dfcc000310 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. An unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random) over joined windows behind a signalling provider declares reach on both sides of that exception, where the law applies and where the short refill excuses it; the full-source mutant fails its fixed seed and random runs. The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. Limits: the adapter boundary cannot tell which subscription issued a load, and each subscription has its own replay, so the retention law and the participant rule apply only where each recovering source has one subscription in the query (no peer, and not a query that both joins and includes the marker source). The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. A lazy demand that the graph starts after its own subscription's replay finished, while another subscription of the same source still recovers, is outside a replay and may end the query. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. An unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random) over joined windows behind a signalling provider declares reach on both sides of that exception, where the law applies and where the short refill excuses it; the full-source mutant fails its fixed seed and random runs. The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index d53feec048..37b2aa4807 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -416,5 +416,6 @@ "windowRequestGeneration": "av", "latestWindowRequest": "gE", "loadMovedEndTie": "gN", - "ownPrefixEnd": "gT" + "ownPrefixEnd": "gT", + "awaitsSiblingReplays": "gU" } diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 47330f26b6..3d58ba588c 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -20,6 +20,7 @@ import type { ChangeMessage, LoadSubsetOptions, LoadSubsetRequestResult, + SubscribeChangesOptions, Subscription, SubscriptionEvents, SubscriptionLoadSubsetErrorEvent, @@ -89,10 +90,9 @@ type CollectionSubscriptionOptions = { onResumeAcquisition?: () => void } -type TruncateReplayPublicationControl = Readonly<{ - start: () => void - succeed: () => void -}> +type TruncateReplayPublicationControl = NonNullable< + SubscribeChangesOptions[`truncateReplayPublication`] +> type TruncatePublicationState = { loadedInitialState: boolean @@ -616,6 +616,22 @@ export class CollectionSubscription } } + /** + * A graph's subscriptions to one collection share a barrier. While another + * one's replay still loads, this replay stays open, so demand this one + * starts meanwhile joins it; it finishes once those replays settle. + */ + private awaitsSiblingReplays( + replayState: TruncateReplayState, + finish: () => void, + ): boolean { + const siblings = this.options.truncateReplayPublication?.siblingReplays() + void siblings?.then(() => { + if (this.truncateReplayState === replayState) finish() + }) + return siblings !== undefined + } + /** Publish only after every overlapping replay attempt has settled. */ private checkTruncateReplayComplete(replayState: TruncateReplayState): void { if (this.truncateReplayState !== replayState) return @@ -628,6 +644,12 @@ export class CollectionSubscription if (activeFailure) { this.abandonTruncateReplay(replayState, activeFailure[1]) } else { + if ( + this.awaitsSiblingReplays(replayState, () => + this.checkTruncateReplayComplete(replayState), + ) + ) + return this.flushTruncateReplay(replayState) } } finally { @@ -822,6 +844,15 @@ export class CollectionSubscription this.checkTruncateReplayComplete(replayState) } + /** Whether this subscription's replay still waits on its own work. */ + public get hasTruncateReplayWork(): boolean { + const replayState = this.truncateReplayState + return ( + replayState !== undefined && + (replayState.pendingSetups > 0 || replayState.pending.size > 0) + ) + } + public get hasPendingTruncateReplacement(): boolean { return this.truncateReplacementPending } @@ -1384,6 +1415,8 @@ export class CollectionSubscription this.flushTruncateReplay(replayState) return } + if (this.awaitsSiblingReplays(replayState, () => this.retireEmptyReplay())) + return this.discardTruncateReplay() this.stalePublishedRows = new Map(this.publishedRows) this.restorePublishedSnapshotTracking() diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 001cc6809e..410cce83c4 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1460,7 +1460,12 @@ from an on-demand source therefore opens a replay at every truncate, even when it holds no demand. A replay without demand starts no recovery work and reports no loading state. Lazy demand that the graph starts while that replay is open belongs to it, including demand started after the replay failed while it still -holds publication. A rejection of an aborted lazy demand request is obsolete demand, +holds publication. A live query's subscriptions to one collection, such as a +join and an include of the same source, share one publication barrier. None +of their replays publishes, and none whose last demand retired closes, while +another of them still has replay work. So demand that any of them starts +meanwhile joins its own open replay instead of falling outside recovery. +A rejection of an aborted lazy demand request is obsolete demand, not a load failure, as for the subscription. When a truncate supersedes the acquisition, its replay decides the demand, even if the replay settles before the aborted request rejects. A segment that newer coverage replaced settles. A diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 624bfa9fc9..9da1ab53f5 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -644,6 +644,24 @@ export class CollectionConfigBuilder< : undefined } + /** Replays of other subscriptions to a source's collection that still load. */ + siblingReplays(sourceId: string): Promise | undefined { + const collection = this.collectionSources.find( + (source) => source.sourceId === sourceId, + )?.collection + const pending = this.collectionSources.flatMap((source) => { + const subscription = this.subscriptions[source.sourceId] + // A replay whose completion already settled cannot be waited on. + return source.sourceId !== sourceId && + source.collection === collection && + subscription?.hasTruncateReplayWork && + subscription.pendingTruncateReplacement + ? [subscription.pendingTruncateReplacement] + : [] + }) + return pending.length > 0 ? Promise.allSettled(pending) : undefined + } + private hasFailedSourceRecovery(): boolean { return Object.values(this.subscriptions).some( (subscription) => subscription.hasFailedTruncateReplacement, diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index 6c01a41f29..b168343931 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -421,6 +421,8 @@ export class CollectionSubscriber< start: () => { onStart?.() }, + siblingReplays: () => + this.collectionConfigBuilder.siblingReplays(this.sourceId), succeed: () => { if ( syncRunGeneration !== diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 298543dc42..5966f1743b 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -1173,6 +1173,11 @@ export interface SubscribeChangesOptions< truncateReplayPublication?: { readonly start: () => void readonly succeed: () => void + /** + * The replays of the graph's other subscriptions to this collection that + * still load. They share one barrier, so this replay waits for them. + */ + readonly siblingReplays: () => Promise | undefined } } diff --git a/packages/db/tests/collection-subscription-reentrancy-oracle.test.ts b/packages/db/tests/collection-subscription-reentrancy-oracle.test.ts index 1b01c39e1f..e217620b41 100644 --- a/packages/db/tests/collection-subscription-reentrancy-oracle.test.ts +++ b/packages/db/tests/collection-subscription-reentrancy-oracle.test.ts @@ -290,6 +290,7 @@ describe(`Collection subscription callback reentry oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => delegate.push(`start`), succeed: () => delegate.push(`succeed`), }, diff --git a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts index 77a50e7e15..294b98caf2 100644 --- a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts +++ b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts @@ -3599,6 +3599,7 @@ describe(`CollectionSubscription replay oracle`, () => { }) const subscription = collection.subscribeChanges(() => {}, { truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => { if (rejectReplacement) throw listenerFailure @@ -3950,6 +3951,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => {}, }, @@ -4015,6 +4017,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => {}, }, @@ -4203,6 +4206,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => replayStarts++, succeed: () => replaySuccesses++, }, @@ -4326,6 +4330,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => replaySuccesses++, }, @@ -4507,7 +4512,11 @@ describe(`CollectionSubscription replay oracle`, () => { }) const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, - truncateReplayPublication: { start: () => {}, succeed: succeeded }, + truncateReplayPublication: { + start: () => {}, + succeed: succeeded, + siblingReplays: () => undefined, + }, }) try { subscription.requestSnapshot({ where: firstWhere }) diff --git a/packages/db/tests/collection-subscription.test.ts b/packages/db/tests/collection-subscription.test.ts index 2a0958c025..ae3076f91f 100644 --- a/packages/db/tests/collection-subscription.test.ts +++ b/packages/db/tests/collection-subscription.test.ts @@ -1339,6 +1339,7 @@ describe(`CollectionSubscription status tracking`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => { replayStarts += 1 }, diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 8fcce9082d..ba56ac55fa 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1123,8 +1123,8 @@ describe(`loadSubset replay refinement`, () => { * now keeps. * A load that joins an open replay belongs to it as well, which covers a * demand that the graph starts after the commit while the replay still - * loads. The model judges that only where the source has one subscription - * in the query (see Limits). + * loads, from any of the query's subscriptions to that source. The model + * judges that only without a peer (see Limits). * A load that fails before the query first becomes ready is fatal too (an * initial load failure), except on a lazy join source, which owns its * demand. So a query may end in `error` only after a demand load outside a @@ -1161,12 +1161,11 @@ describe(`loadSubset replay refinement`, () => { * * Limits: one ordered source; no full joins (the ordered-work oracle owns * them); includes one level deep; and a provider that reads its rows at the time it - * answers. The adapter boundary cannot tell which subscription issued a load, - * and each subscription has its own replay. So the retention law, and the - * rule that a replay participant counts as recovery, apply only where each - * recovering source has one subscription in the query: the order source - * without a peer, and the marker source unless the query both joins and - * includes it (or a peer reads it too). A release that throws while a replay starts may + * answers. A query's subscriptions to one source share one replay barrier, + * so a load joins it whichever of them issued it. A peer is another query + * with its own barrier, and the adapter boundary cannot tell which query + * issued a load, so the retention law and the rule that a replay + * participant counts as recovery skip histories with a peer. A release that throws while a replay starts may * fail that replay, so the retention law skips a barrier where one did. */ type OrderedRow = { id: number; rank: number } @@ -1308,6 +1307,7 @@ type OrderedReach = { retainedFailedReplays: number participantDemandFailures: number includeDemandFailures: number + siblingDemandFailures: number preloadReplayFailures: number discardedReplayMoves: number } @@ -1502,11 +1502,11 @@ async function checkOrderedReplay( } } } - // Each subscription has its own replay, and the adapter boundary cannot - // tell which subscription issued a load. - const oneSubscription = (source: BarrierSource) => - !scenario.peer && - (source === `rows` || !(scenario.joined && scenario.includes)) + // A query's subscriptions to one source share one replay barrier, so a + // load belongs to it whichever subscription issued it. A peer is another + // query with its own barrier, and the adapter boundary cannot tell which + // query issued a load. + const oneSubscription = (_source: BarrierSource) => !scenario.peer const acquired = new Map() const held: Array<(ok: boolean) => void> = [] let holding = false @@ -1613,6 +1613,7 @@ async function checkOrderedReplay( retainedFailedReplays: 0, participantDemandFailures: 0, includeDemandFailures: 0, + siblingDemandFailures: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -1669,18 +1670,25 @@ async function checkOrderedReplay( load.outcome = `failed` anyLoadFailed = true if (demand) demandFailed = true - // A demand the graph started after the commit, failing - // while its replay still loads. + // A demand the graph started after the commit fails inside + // the replay. if ( participant && - replayLoading && !options.signal?.aborted && oneSubscription(`markers`) && !markerCommitting && isDemand(options) ) { - reach.participantDemandFailures++ - if (scenario.includes) reach.includeDemandFailures++ + // While other participants still load. + if (replayLoading) { + reach.participantDemandFailures++ + if (scenario.includes) reach.includeDemandFailures++ + } + // The query both joins and includes the marker source, so + // two of its subscriptions share the barrier. + if (scenario.includes && scenario.joined) { + reach.siblingDemandFailures++ + } } if (queries.some(({ status }) => status === `loading`)) { failedWhileLoading = true @@ -2535,6 +2543,31 @@ describe(`Ordered windows across truncate replays`, () => { }, reaches: `settledWindowChecks`, }, + { + // A query that joins and includes the marker source has two marker + // subscriptions. The truncate's commit deletes the markers privately, + // so the join filter drops row 1 and the graph demands markers for + // the next rows. One subscription's replay loads settle first, and + // the new demand then fails while the other's replay still loads. + // With one replay per subscription, that demand fell outside every + // replay and ended the query; the source's barrier owns it. + name: `a demand that fails while a sibling subscription's replay still loads`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + reaches: `siblingDemandFailures`, + recovers: true, + }, ] satisfies Array<{ name: string history: Array @@ -2587,6 +2620,7 @@ describe(`Ordered windows across truncate replays`, () => { retainedFailedReplays: 0, participantDemandFailures: 0, includeDemandFailures: 0, + siblingDemandFailures: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -2624,6 +2658,7 @@ describe(`Ordered windows across truncate replays`, () => { `retainedFailedReplays`, `participantDemandFailures`, `includeDemandFailures`, + `siblingDemandFailures`, `preloadReplayFailures`, `discardedReplayMoves`, ]) @@ -2722,12 +2757,46 @@ describe(`Ordered windows across truncate replays`, () => { included, ]) }) + // A query that joins and includes the marker source has two marker + // subscriptions under one barrier. Settling some held loads can finish one + // subscription's own replay loads while the other's still load; the graph + // then demands markers for rows the commit privately exposed, and the + // newest load fails. + const siblingDemandCase = fc + .tuple( + failedReplayScenario, + fc.constantFrom(`some` as const, `none` as const, `inner` as const), + fc.integer({ min: 1, max: 2 }), + fc.boolean(), + fc.array(orderedCommand, { maxLength: 6 }), + ) + .map( + ([scenario, joined, settles, newest, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + ...Array.from({ length: settles }, () => ({ + type: `settle` as const, + ok: true, + keepHolding: true, + newest, + })), + { type: `settle`, ok: false, newest: true, keepHolding: true }, + ...rest, + ], + { ...scenario, joined, includes: true, markerMode: `on-demand` }, + ], + ) const failedReplayCase = fc.oneof( { weight: 3, arbitrary: fc.tuple(failedReplayHistory, failedReplayScenario), }, { weight: 1, arbitrary: includeDemandCase }, + { weight: 1, arbitrary: siblingDemandCase }, ) const failedReplayProperty = `replay.ordered-failed-replay` const failedReplayCampaigns = @@ -2898,6 +2967,7 @@ describe(`Ordered windows across truncate replays`, () => { `heldPublicationChecks`, `participantDemandFailures`, `includeDemandFailures`, + `siblingDemandFailures`, ] as const, { minimum: 5 }, ) @@ -2917,6 +2987,9 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.includeDemandFailures > 0) { reach.reach(`includeDemandFailures`) } + if (observed.siblingDemandFailures > 0) { + reach.reach(`siblingDemandFailures`) + } }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(200) } From aec47ad00816dc2ac892966d4b448f08c6135d6f Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 11:26:18 -0600 Subject: [PATCH 17/40] chore: describe the shared replay barrier in the changeset Co-authored-by: Isaac --- .changeset/keep-live-query-through-failed-replay.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index ef19984f1f..b763f96468 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -4,6 +4,8 @@ Keep a live query running when a lazy join or include load fails as part of a source truncate. The truncate's replay fails instead, and the query keeps its last complete result until a later replay of that source succeeds. Before, the query could end in error 228, for example when a truncate made an ordered window demand a row that it never showed. A load that fails before the query first becomes ready still ends it. +A query that joins and includes the same source has two subscriptions to it. Their truncate replays now share one barrier, so a lazy demand that fails while either replay still loads no longer ends the query. + An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window reads the new end's tie group. It no longer trusts a row that an older tie request installed. From 944a0c0b88c1700032aa45983ca5f2d19aadf35e Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 15:06:42 -0600 Subject: [PATCH 18/40] test(db): drop the one-run replay override from this branch's campaigns A path replay now runs once inside oraclePropertyOptions, so the override is redundant, and oracle/no-disabled rejects it. Co-authored-by: Isaac --- .../query/load-subset-replay-refinement-oracle.test.ts | 10 ++-------- .../query/ordered-source-loader-state-oracle.test.ts | 5 +---- 2 files changed, 3 insertions(+), 12 deletions(-) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 47d3baf6be..6922009d3f 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -2938,10 +2938,7 @@ describe(`Ordered windows across truncate replays`, () => { // dies in about half of 200-run seeds. campaign === `fixed` ? { seed: 20261027, numRuns: oracleRuns(300) } - : { - ...oraclePropertyOptions(300, preloadProperty), - ...(campaign === `replay` ? { numRuns: 1 } : {}), - }, + : oraclePropertyOptions(300, preloadProperty), ) reach.assertReached() }, @@ -2990,10 +2987,7 @@ describe(`Ordered windows across truncate replays`, () => { }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(200) } - : { - ...oraclePropertyOptions(200, failedReplayProperty), - ...(campaign === `replay` ? { numRuns: 1 } : {}), - }, + : oraclePropertyOptions(200, failedReplayProperty), ) reach.assertReached() }, diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 11f11f8c15..3e8a2a8bec 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -3944,10 +3944,7 @@ describe(`Ordered source request authority`, () => { }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(300) } - : { - ...oraclePropertyOptions(300, unprovenReportProperty), - ...(campaign === `replay` ? { numRuns: 1 } : {}), - }, + : oraclePropertyOptions(300, unprovenReportProperty), ) reach.assertReached() }, From fe48cbe546b266081cafb0b83bdeeb69087d2544 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 16:29:16 -0600 Subject: [PATCH 19/40] fix(db): grow a short unindexed prefix instead of reading the full source An ordered, limited window over a source without an order index used to read the whole source when its bounded prefix left the window short. That read is unbounded. Maintainer decision: the prefix grows instead, doubling the last prefix while growth still adds rows, and never reads the full source. Law: after a signalling refetch, a repair starts no full-source request unless a bounded page ended on a null or NaN rank (ARCHITECTURE.md, "an underfilled unindexed prefix grows"). A pending optimistic departure from such a window refills it before settlement with a bounded request. Reach: `ordered-work.eager-unindexed-refetch` (fixed and random) declares windows the prefix fills and windows it leaves short. Sixteen new optimistic-window cases cover an unindexed one-row window behind a joined or anti-joined filter that installs only its grown prefix, so a departure leaves the replacement row missing. RED (unchanged production): the named short-window replay and the fixed unindexed-refetch campaign fail with "full-source requests after a signalling refetch at step N: expected 2 to be +0". GREEN: ordered-work oracle 293/293; @tanstack/db 284 of 285 files pass, and oracle-replay.test.ts fails only on child-run timeouts (spawnSync ETIMEDOUT) under a concurrent mutant sweep. Mutants: no stop rule -> "ordered loading did not reach a fixed point"; stopping whenever the prefix reads short -> the 16 optimistic cases leave the window empty after settlement ("settled rows: [] instead of [2]"). Expectation changes (maintainer approved): the existing unindexed optimistic cases behind a joined filter now allow, without requiring, a request when the replacement row is installed, because growth proves rows only up to its prefix. The optimistic refill bound for unindexed plans is twice the installed prefix plus the overlaid row. Limits: the new optimistic cases are fixed histories with one-row windows; growth's request count is bounded but not minimal (an answer with new rows resets the growth record, so a short source can cost one more read). Size: @tanstack/db +468 B gzip vs main (+17 B for this change). Co-authored-by: Isaac --- .../keep-live-query-through-failed-replay.md | 2 + docs/contributing/oracle-coverage.md | 2 +- packages/db/mangle-cache.json | 3 +- packages/db/src/query/live/ARCHITECTURE.md | 6 +- .../src/query/live/ordered-source-loader.ts | 34 +++++++-- .../ordered-work-oracle.property.test.ts | 74 ++++++++++++++----- 6 files changed, 91 insertions(+), 30 deletions(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index b763f96468..0450635b77 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -11,3 +11,5 @@ An aborted subset request that a source rejects with `AbortError`, as the source Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window reads the new end's tie group. It no longer trusts a row that an older tie request installed. A window move made while a truncate replay holds a live query's results now applies after that replay, even when a newer truncate discards it. Before, the move could reject, and an older move's window stayed. + +An ordered, limited query over a source without an order index can read too few rows to fill its window. A join that drops rows can cause this. It no longer reads the whole source then. It reads a prefix twice as long instead. It grows again while that adds rows, so every request stays bounded. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 393912ceb7..b30776d771 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture lets refill a short window from the full source; random seeds found it on main. The law now allows that read only when the model's bounded prefix leaves the window short, a named replay reaches that premise, and a mutant that repairs every report with a full-source read still fails the fixed campaigns. An unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random) over joined windows behind a signalling provider declares reach on both sides of that exception, where the law applies and where the short refill excuses it; the full-source mutant fails its fixed seed and random runs. The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the source holds fewer rows than the last prefix asked for and the last growth added none. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index 37b2aa4807..e510d37fa0 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -417,5 +417,6 @@ "latestWindowRequest": "gE", "loadMovedEndTie": "gN", "ownPrefixEnd": "gT", - "awaitsSiblingReplays": "gU" + "awaitsSiblingReplays": "gU", + "grownPrefixRows": "gV" } diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 410cce83c4..63051055fd 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1187,7 +1187,11 @@ join-key update, or a root change, and for Collection and Effect consumers. The finite root requests; this contract does not imply that it can evaluate the relation filter remotely. Finite continuation still requires an order index; an underfilled unindexed -prefix retains the established full-source fallback. +prefix grows. Each growth reads a prefix twice as long as the last one, so +every read stays bounded and none reads the full source. The prefix grows +while growing still adds rows to the source. When the source holds fewer rows +than the last prefix asked for and the last growth added none, the source +holds no more rows and the window stays short. A custom string comparator is a local ordering contract. If any resolved source order term uses `stringSort: 'custom'`, bounded and unbounded queries diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 4ad9ecbf17..3753136bd8 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -118,6 +118,8 @@ export class OrderedSourceLoader { private authorityGeneration = 0 private lastPage: { count: number; boundary: unknown } | undefined private lastPrefixCount: number | undefined + // The rows the source's prefix held when it last grew. + private grownPrefixRows: number | undefined // The count of the last request that read from the start of the order, // until a continuation reads past it. private ownPrefixCount: number | undefined @@ -482,14 +484,31 @@ export class OrderedSourceLoader { continuesOrderedPrefixRepair = false, ): void { if (!this.active || this.pending || this.stagedContinuation) return - if (this.lastPrefixCount === count) { - if ((this.info.dataNeeded?.(this.settledSourceBoundary) ?? 0) > 0) { - if (continuesOrderedPrefixRepair) this.abandonOrderedPrefixRepair() - this.loadFullSource(windowOperationGeneration) - } else if (continuesOrderedPrefixRepair) { - this.finishOrderedPrefixRepair(windowOperationGeneration) + const last = this.lastPrefixCount ?? 0 + if (last >= count) { + if ((this.info.dataNeeded?.(this.settledSourceBoundary) ?? 0) === 0) { + if (continuesOrderedPrefixRepair) { + this.finishOrderedPrefixRepair(windowOperationGeneration) + } + return } - return + // A filter after the read dropped rows, so the prefix doubles: each read + // stays bounded by twice the last one, and none reads the full source. + // A short prefix can hide optimistic rows from this count, so it grows + // while growing still adds rows. Once growing adds none, the source + // holds no more rows. + const rows = this.subscription.readOrderedSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: last, + }).length + if (rows < last && rows <= (this.grownPrefixRows ?? -1)) { + if (continuesOrderedPrefixRepair) { + this.finishOrderedPrefixRepair(windowOperationGeneration) + } + return + } + this.grownPrefixRows = rows + count = last * 2 } this.requestAndObserve( (tracking) => @@ -637,6 +656,7 @@ export class OrderedSourceLoader { } this.lastPage = undefined this.lastPrefixCount = undefined + this.grownPrefixRows = undefined } invalidateSourceOrdering(): void { diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index af5345ecd6..1ed1658b69 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -6122,11 +6122,10 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ } // The repair a report starts reads each window's bounded prefix, // tie, and refill; a whole-source read would be the cost the report - // exists to avoid. Two documented fallbacks read the full source - // instead: a page that ends on a null or NaN rank, and the refill of - // an unindexed plan whose bounded prefix leaves the window short - // (ARCHITECTURE.md, "an underfilled unindexed prefix retains the - // established full-source fallback"). + // exists to avoid. One documented fallback reads the full source + // instead: a page that ends on a null or NaN rank. An unindexed plan + // whose bounded prefix leaves the window short grows the prefix + // (ARCHITECTURE.md, "an underfilled unindexed prefix grows"). const shortRefill = !testCase.indexed && prefixLeavesWindowShort(rows, shown) if (signalsRefetch && unexpressiblePageEnds === unexpressibleBefore) { @@ -6135,11 +6134,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ else unindexedRefetchChecks++ } } - if ( - signalsRefetch && - unexpressiblePageEnds === unexpressibleBefore && - !shortRefill - ) { + if (signalsRefetch && unexpressiblePageEnds === unexpressibleBefore) { expect( fullSourceRequests, `full-source requests after a signalling refetch at step ${step}`, @@ -6431,6 +6426,14 @@ type OptimisticWindowCheckpoint = const MAX_OPTIMISTIC_WINDOW_REQUESTS = 8 const OPTIMISTIC_ACQUISITION_OBSERVATION_TURNS = 8 +// The most rows a bounded refill may ask for. An indexed plan pages past +// the window. An unindexed plan grows its prefix to twice the rows it +// installed (ARCHITECTURE.md, "an underfilled unindexed prefix grows"), and +// a request also covers the one row that the pending transaction overlays. +function refillBound({ indexed, width, installedIds }: OptimisticWindowCase) { + return indexed ? width + 2 : 2 * installedIds.length + 1 +} + type OptimisticWindowCase = { shape: OptimisticWindowShape mutation: OptimisticWindowMutation @@ -7185,7 +7188,7 @@ function optimisticWindowViolations( needsAcquisition && (refillAttempt?.direction !== (testCase.direction ?? `desc`) || refillAttempt.limit === undefined || - refillAttempt.limit > width + 2) + refillAttempt.limit > refillBound(testCase)) ) { violations.push(`refill used an unbounded provider request`) } else if ( @@ -7396,10 +7399,9 @@ describe(`eager indexed ordered windows`, () => { // An unindexed joined window whose only domain row is ineligible. After a // signalling refetch the repair's bounded prefix holds just that row, so - // the window is short and the refill takes the documented full-source - // fallback. The refetch law allows that read for an unindexed plan only - // when its prefix leaves the window short. - it(`allows the full-source refill of a short unindexed window after a signalling refetch`, async () => { + // the window is short. The refill grows the prefix instead of reading the + // full source, which the refetch law forbids. + it(`grows the prefix of a short unindexed window after a signalling refetch`, async () => { await checkEagerWindow({ path: `on-demand`, provider: `refetch-signal`, @@ -7636,7 +7638,17 @@ describe(`eager indexed ordered windows`, () => { ? [4, 3] : [4, 3, 2] : [1, 2, 3, 4], - acquisition: extent === `missing` ? `required` : `forbidden`, + // An unindexed plan proves rows only up to its bounded + // prefix, so behind a joined filter it may grow the prefix + // even when the replacement row is installed (ARCHITECTURE.md, + // "an underfilled unindexed prefix grows"). A source filter + // leaves no rows out of the prefix, so it needs no request. + acquisition: + extent === `missing` + ? `required` + : shape === `source` + ? `forbidden` + : `allowed`, } }), ), @@ -7679,6 +7691,27 @@ describe(`eager indexed ordered windows`, () => { })), ), ), + // An unindexed one-row window behind a joined filter installs only the + // prefix it grew to: the rejected leading row 4 and the window row 3. + // The departure of row 3 leaves the replacement row 2 missing, so the + // refill must grow the prefix before persistence settles. + ...([`joined`, `anti-joined`] as const).flatMap((shape) => + ([`update`, `delete`] as const).flatMap((mutation) => + ([`acknowledge`, `rollback`] as const).flatMap((outcome) => + ([`collection`, `effect`] as const).map((consumer) => ({ + shape, + mutation, + consumer, + outcome, + width: 1 as const, + indexed: false, + extent: `missing` as const, + installedIds: [4, 3], + acquisition: `required` as const, + })), + ), + ), + ), { shape: `source`, mutation: `update`, @@ -8382,9 +8415,10 @@ describe(`eager indexed ordered windows`, () => { }) // The refetch law over unindexed joined windows behind a signalling - // provider. A repair must not read the full source, except for the - // documented refill of an unindexed plan whose bounded prefix leaves the - // window short. The campaign must reach both sides of that exception. + // provider. A repair must not read the full source, including the refill + // of an unindexed plan whose bounded prefix leaves the window short, which + // grows the prefix. The campaign must reach windows the prefix fills and + // windows it leaves short. // Ineligible rows crowd the bounded prefix and leave the window short, so // half the cases start with every row eligible, where the law applies. const eagerUnindexedRefetchCase = fc @@ -8400,7 +8434,7 @@ describe(`eager indexed ordered windows`, () => { : testCase.initial, })) for (const campaign of [`fixed`, `random`] as const) { - it(`reads the full source after a signalling refetch only for a short unindexed refill (${campaign} campaign)`, async () => { + it(`grows a short unindexed prefix instead of reading the full source after a signalling refetch (${campaign} campaign)`, async () => { const property = `ordered-work.eager-unindexed-refetch` const reach = createOracleReach( property, From 248f60bd070dc8ab44848394100a92e235ea5f24 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 18:46:25 -0600 Subject: [PATCH 20/40] refactor(db): drop the implied short-read term from the growth stop rule A prefix stops growing once its last growth added no rows. The rule also required the prefix to read fewer rows than it asked for, but that follows: the growth record holds the rows before the last doubling, so a count that did not exceed it is below the doubled prefix. Every path that shrinks the prefix count resets the record too. Evidence: a deletion sweep found the term's mutant surviving the ordered, replay, include, pagination, subset, and join oracles. With the term removed, the ordered-work, pagination, and ordered source loader oracles pass (659 tests), and the full @tanstack/db suite passes (285 files, 13,833 tests). The 16 optimistic-window cases that kill a stop rule without the growth record still pass. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 5 ++--- packages/db/src/query/live/ordered-source-loader.ts | 8 ++++---- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b30776d771..a402f4d76b 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the source holds fewer rows than the last prefix asked for and the last growth added none. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 63051055fd..b4ac6ae725 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1189,9 +1189,8 @@ relation filter remotely. Finite continuation still requires an order index; an underfilled unindexed prefix grows. Each growth reads a prefix twice as long as the last one, so every read stays bounded and none reads the full source. The prefix grows -while growing still adds rows to the source. When the source holds fewer rows -than the last prefix asked for and the last growth added none, the source -holds no more rows and the window stays short. +while growing still adds rows to the source. When the last growth added +none, the source holds no more rows and the window stays short. A custom string comparator is a local ordering contract. If any resolved source order term uses `stringSort: 'custom'`, bounded and unbounded queries diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 3753136bd8..cdc80b5c8b 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -494,14 +494,14 @@ export class OrderedSourceLoader { } // A filter after the read dropped rows, so the prefix doubles: each read // stays bounded by twice the last one, and none reads the full source. - // A short prefix can hide optimistic rows from this count, so it grows - // while growing still adds rows. Once growing adds none, the source - // holds no more rows. + // A pending optimistic departure can hide a row from this count, so the + // prefix grows while growing still adds rows. Once growing adds none, + // the source holds no more rows. const rows = this.subscription.readOrderedSnapshot({ orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), limit: last, }).length - if (rows < last && rows <= (this.grownPrefixRows ?? -1)) { + if (rows <= (this.grownPrefixRows ?? -1)) { if (continuesOrderedPrefixRepair) { this.finishOrderedPrefixRepair(windowOperationGeneration) } From d1bf75c1b77c5fbec676cfb721fdd30814f87f53 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 21:44:17 -0600 Subject: [PATCH 21/40] test(db): judge every loaded-subset report against a faithful fake source A deletion sweep found the report path's guards surviving every oracle: deleting the moved-end early return in revalidateLoadedCoverage, its past-window term, or loadMovedEndTie's owed-debt term changed nothing that an oracle saw. Law (ARCHITECTURE.md, the loaded-subset report paragraph): a loader whose coverage is a held full-source read, or whose window its own first page proves, ignores the report, so it reads nothing for it. A loader that already owes a repair repairs instead, so no tie or page comes before the repair's prefix. Oracle changes: - The fake subscription reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit. Its reads ignored the limit, which made the first-page ignore branch unreachable. - The cost law's per-chain bound of three requests was an artifact of that fake. It is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which now grows its prefix instead of reading the full source. - A head insert moves the page's end before the tie the loader read, and a window that shrinks counts as quiet: it shows only rows already proven. - The ignored-report law forbids any request, not only authoritative ones, and a new owed-repair law judges reports while a repair is owed. Reach: `ordered-loader.unproven-report` (600 runs) starts each history with one of seven prefixes and requires at least 20 histories for each premise: a proven first page, a page past a shrunk window, a held full-source read, a moved end, an owed repair, authoritative work in flight, and an older tie's release that truncates or disposes the loader. RED (hostile mutants; production is unchanged): - the early return removed: "ignored report: request 2 was read for a report over coverage that the loader's own first page proves"; - its past-window term removed: the same failure; - the owed-debt term removed: "owed repair: request 2, an ordinary read, came before the repair a report owed". GREEN: ordered-source-loader-state oracle 77/77. Limits: the unexpressible-end term (the fake's comparator cannot place NaN or nulls), the stale-run check after the older tie's release (the fake's dataNeeded ignores its boundary), and the in-flight-work, staged- continuation, and covered-source checks, each decided by another check in every reached state, still survive. Recorded in the coverage map. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- ...ordered-source-loader-state-oracle.test.ts | 388 ++++++++++++++---- 2 files changed, 303 insertions(+), 87 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index a402f4d76b..b15008368c 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. The fake's installed rows ignore a read's limit, so its first-prefix ignore branch is unreachable there. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 600 runs) now starts each history with one of seven prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, and a release of the older tie that truncates or disposes the loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. Still open there: the unexpressible-end term (the fake's comparator cannot place NaN or nulls), the stale-run check after the older tie's release (the fake's `dataNeeded` ignores its boundary), and the in-flight-work, staged-continuation, and covered-source checks, each of which another check decides in every state the grammar reaches. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 3e8a2a8bec..baf86e8d29 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -1820,26 +1820,27 @@ describe(`Successful finite demand retirement`, () => { * invalidation reads no full source, and it issues at most one chain (in * this fake, a prefix, a tie, and a refill) per ordering epoch move. * - **Ignored report:** a sync adapter's report that served pages may have - * shifted issues no authoritative request when nothing it describes can - * be wrong: before any request settled (a request in flight still owes an - * exact result for its own options), or while the loader's coverage is - * its own first ordered prefix of at least `offset + limit` rows, with - * only that prefix's tie after it, and a row past the page sorts strictly - * after its end with no more rows needed (ARCHITECTURE.md). That row is - * evidence only when the page read it, or when the tie the loader read is - * the tie of the page's end. The law judges only quiet histories: every - * earlier step settled a request successfully, ran a pass, inserted a row, - * reported, or changed joined work, and no release callback has side - * effects, so no earlier event owes a repair. + * shifted issues no authoritative request before any request settled: a + * request in flight still owes an exact result for its own options. The + * loader ignores the report, and so issues no request at all, while its + * coverage is a held full-source read, or while it is the loader's own + * first ordered prefix of at least `offset + limit` rows, with only that + * prefix's tie after it, and a row past the page sorts strictly after its + * end with no more rows needed (ARCHITECTURE.md). That row is evidence + * only when the page read it, or when the tie the loader read is the tie + * of the page's end. The law judges only quiet histories: every earlier + * step settled a request successfully, ran a pass, inserted a row, + * reported, shrank the window, or changed joined work, and no release + * callback has side effects, so no earlier event owes a repair. + * - **Owed repair:** a report while the loader already owes a repair + * repairs instead (ARCHITECTURE.md): the pass that reacts to it reads no + * tie or page before the repair's prefix. * - **Unproven report:** in such a quiet history, a report over the loader's * own first prefix that no row it read proves starts a repair on its pass, * or reads the tie of the page's end, which completes the proof. * An installed row past the page can come from another request, and a * refetch can move the page's end before the tie the loader read, so the - * loader cannot use either as proof. This fake's installed rows ignore a - * read's limit, so its tie is always for the last installed row; the - * ignored-report branch for a first page is therefore unreachable here, - * and only the unproven-report law runs on such a page. + * loader cannot use either as proof. * - **One request at a time:** a request issued while another is in flight * starts authority, or an authority start lies between them. A repair * chain that a newer start replaced issues no more steps. @@ -1901,6 +1902,9 @@ describe(`Successful finite demand retirement`, () => { * - `invalidate` invalidates source ordering, and `report` delivers a * loaded-subset report; either can run an ordinary pass at once or leave * the reaction deferred, as pending joined work or a later graph run can; + * - `headInsert` delivers a new row that sorts before every installed row, + * as a refetch that moved a provider row into the first page does, so the + * page's end moves one row earlier; * - `pass` runs an ordinary `loadMore()`; * - `move` is an explicit window move, which may also widen the window, so * an unindexed loader reads a longer prefix; @@ -1960,8 +1964,9 @@ describe(`Successful finite demand retirement`, () => { * * ## Limits * - * The fake subscription's rows do not move except for the unexpressible - * boundary, so cursor values are fixed. Publication holds are outside this + * The fake subscription's installed rows move only through the unexpressible + * boundary and a head insert. Its ordered reads apply a tie's equality, a + * cursor, and a limit as the subscription's reads do. Publication holds are outside this * property; other tests here own them. The laws do not judge requests issued * before a truncate, which the replay owns. The loader has no request * timeout, so a hung authoritative request may hold its work slot: progress @@ -2003,6 +2008,7 @@ type AuthorityCommand = | { type: `settle`; pick: number; ok: boolean } | { type: `invalidate`; react: boolean } | { type: `report`; react: boolean } + | { type: `headInsert` } | { type: `failOldest` } | { type: `settleNewest` } | { type: `invalidateThenFailOldest` } @@ -2087,6 +2093,7 @@ const authorityCommand: fc.Arbitrary = fc.oneof( react: fc.boolean(), }), }, + { weight: 1, arbitrary: fc.constant({ type: `headInsert` as const }) }, // The oldest in-flight request is the one most often superseded. { weight: 1, arbitrary: fc.constant({ type: `failOldest` as const }) }, // A provider can answer newer requests first, so a repair chain finishes @@ -2194,7 +2201,17 @@ type AuthorityObservation = { joinedDeferrals: number ignoredReportsBeforeSettle: number ignoredReportsInsideFirstPage: number + ignoredReportsOverFullSource: number repairedReportsOnOwnPage: number + // Reports after a refetch moved the page's end, while a repair was owed, + // while authoritative work was in flight, and reports whose release of an + // older tie truncated or disposed the loader. + movedEndReports: number + // Reports whose own first page read past a window that then shrank. + reportsPastWindow: number + reportsOwingRepair: number + reportsDuringWork: number + reportReleasesEndingLoader: number truncates: number replayFailures: number optimisticDepartures: number @@ -2221,6 +2238,11 @@ async function checkAuthorityCase( let joinedPending = false let truncatePending = false let boundaryUnexpressible = false + // Rows a refetch delivered before every installed row. + let headInserts = 0 + // Whether an event since the last authoritative start owes a repair: an + // ordering invalidation, a visible delete, or an unexpressible boundary. + let repairOwed = false const info = createOrderByInfo({ limit: testCase.limit, ...(testCase.indexed ? {} : { index: undefined }), @@ -2287,21 +2309,25 @@ async function checkAuthorityCase( let releaseFaults = 0 let syncRepairs = 0 let invalidatingPages = 0 + // The rank each continuation's cursor starts after, so a read of its + // acquisition's rows applies the cursor as the subscription does. + const cursorStarts = new WeakMap() const record = (method: Observed[`method`], options: RequestOptions) => { // Like the real subscription, a continuation's acquisition carries a // cursor, so the loader does not take it for a prefix read. + const cursor = + options.minValues === undefined + ? undefined + : ({ + whereFrom: new PropRef([`row`, `rank`]), + whereCurrent: new PropRef([`row`, `rank`]), + } as unknown as NonNullable) + if (cursor) cursorStarts.set(cursor, options.minValues![0]) const acquisition: LoadSubsetOptions = { orderBy: options.orderBy, limit: options.limit, where: options.where, - ...(options.minValues === undefined - ? {} - : { - cursor: { - whereFrom: new PropRef([`row`, `rank`]), - whereCurrent: new PropRef([`row`, `rank`]), - } as unknown as NonNullable, - }), + ...(cursor ? { cursor } : {}), } const fullSource = method === `snapshot` && @@ -2350,7 +2376,10 @@ async function checkAuthorityCase( ) { concurrencyViolation ??= `request ${request.index}, a step of a repair chain whose epoch moved, was issued before its replacement` } - if (startsAuthority(request)) epochMovesAtLastStart = epochMoves + if (startsAuthority(request)) { + epochMovesAtLastStart = epochMoves + repairOwed = false + } // An issued request consumes an owed continuation. An ordinary page also // shows the chain finished, since none is issued while a repair runs. continuationOwed = false @@ -2470,25 +2499,44 @@ async function checkAuthorityCase( // The page's own delivery changes source order before it returns. invalidateNextPage = false invalidatingPages++ + repairOwed = true loader.invalidateSourceOrdering() } answering.push(request) options.onLoadSubsetResult?.(request.deferred.promise, acquisition, release) } // Installed rows in ascending rank. An unexpressible boundary makes the - // last one NaN, which no cursor can express. - const snapshot = () => - Array.from({ length: testCase.snapshotRows }, (_, index) => ({ - type: `insert`, - key: index + 1, - value: { - id: index + 1, - rank: - boundaryUnexpressible && index === testCase.snapshotRows - 1 - ? Number.NaN - : index + 1, - }, - })) + // last one NaN, which no cursor can express. A read applies a tie's + // equality, a cursor (rows strictly after its start), and the limit, as + // the subscription's ordered read does. + const snapshot = (options?: LoadSubsetOptions) => { + const tie = options === undefined ? undefined : tieValue(options) + const after = options?.cursor && cursorStarts.get(options.cursor) + return [ + ...Array.from({ length: headInserts }, (_, index) => ({ + type: `insert`, + key: 1000 + headInserts - index, + value: { id: 1000 + headInserts - index, rank: index - headInserts }, + })), + ...Array.from({ length: testCase.snapshotRows }, (_, index) => ({ + type: `insert`, + key: index + 1, + value: { + id: index + 1, + rank: + boundaryUnexpressible && index === testCase.snapshotRows - 1 + ? Number.NaN + : index + 1, + }, + })), + ] + .filter( + ({ value }) => + (tie === undefined || value.rank === tie) && + (after === undefined || value.rank > (after as number)), + ) + .slice(0, options?.limit) + } const subscription = { readOrderedSnapshot: snapshot, setOrderByIndex: () => {}, @@ -2561,7 +2609,13 @@ async function checkAuthorityCase( let joinedDeferrals = 0 let ignoredReportsBeforeSettle = 0 let ignoredReportsInsideFirstPage = 0 + let ignoredReportsOverFullSource = 0 let repairedReportsOnOwnPage = 0 + let movedEndReports = 0 + let reportsPastWindow = 0 + let reportsOwingRepair = 0 + let reportsDuringWork = 0 + let reportReleasesEndingLoader = 0 // Steps that settle a request successfully, run a pass, report, or change // joined work. None of them owes a repair. const quietCommand = (command: AuthorityCommand) => @@ -2569,17 +2623,20 @@ async function checkAuthorityCase( command.type === `settleNewest` || command.type === `pass` || command.type === `report` || + command.type === `headInsert` || + // A window that shrinks shows only rows that were already proven. + (command.type === `move` && command.delta === -1) || command.type === `joinedBusy` || command.type === `joinedIdle` // An unproven report whose pass pending joined work deferred: the ledger // index at the report and the window size it judged. - let owedProof: { from: number; windowSize: number } | undefined - const expectProofRequest = (from: number, windowSize: number) => { + let owedProof: { from: number; endRank: unknown } | undefined + const expectProofRequest = (from: number, endRank: unknown) => { const issued = ledger .slice(from) .some( ({ authoritative, options }) => - authoritative || tieValue(options) === windowSize, + authoritative || tieValue(options) === endRank, ) if (!issued) { throw new Error( @@ -2845,6 +2902,7 @@ async function checkAuthorityCase( // or a later graph run defers it, while the oldest in-flight request // fails before the loader reacts. case `invalidateThenFailOldest`: { + repairOwed = true loader.invalidateSourceOrdering() const request = inflight()[0] if (!request) break @@ -2858,6 +2916,7 @@ async function checkAuthorityCase( break } case `invalidate`: + repairOwed = true loader.invalidateSourceOrdering() if (command.react) graphPass() await flush() @@ -2875,12 +2934,15 @@ async function checkAuthorityCase( const windowSize = testCase.offset + info.limit const [first, ...later] = ledger const beforeSettle = ledger.every(({ outcome }) => outcome !== `ok`) + // The page's end is the installed row at the window's last position. + const ranks = snapshot().map(({ value }) => value.rank) + const endRank = ranks[windowSize - 1] // The loader's coverage is its own first page with only tie // requests after it. const ownFirstPage = !testCase.requiresFullSource && !testCase.needsMore && - testCase.snapshotRows > windowSize && + ranks.length > windowSize && first !== undefined && first.outcome === `ok` && !first.authoritative && @@ -2900,16 +2962,60 @@ async function checkAuthorityCase( // A row past the page proves the source's next row only when the // page read it, or when it ties with the page's end and the loader // read that tie. Any other installed row can come from another - // request, so it proves nothing. The installed ranks are 1..n, so - // the end's rank is the window size. + // request, so it proves nothing, and neither does a tie of an older + // end that a head insert moved. const insideFirstPage = ownFirstPage && ((first.options.limit ?? 0) > windowSize || - later.some(({ options }) => tieValue(options) === windowSize)) + later.some(({ options }) => tieValue(options) === endRank)) + if (ownFirstPage && (first.options.limit ?? 0) > windowSize) { + reportsPastWindow++ + } + if ( + ownFirstPage && + later.some(({ options }) => tieValue(options) !== undefined) && + !insideFirstPage + ) { + movedEndReports++ + } + // Coverage is a held full-source read. + const coversSource = ledger.some( + ({ fullSource, outcome, released }) => + fullSource && outcome === `ok` && !released, + ) + const owesRepair = + repairOwed && !replayPending() && !isDisposed() && command.react + if (owesRepair) reportsOwingRepair++ + if ( + ledger.some( + ({ outcome, authoritative, hung }) => + outcome === `inflight` && authoritative && !hung, + ) + ) { + reportsDuringWork++ + } const before = ledger.length + const releasedBefore = ledger.filter(({ released }) => released) loader.revalidateLoadedCoverage() + if ( + (testCase.releaseTruncates || testCase.releaseDisposes) && + ledger.some( + (request) => + request.released && !releasedBefore.includes(request), + ) + ) { + reportReleasesEndingLoader++ + } if (command.react) graphPass() await flush() + // A report while a repair is owed repairs instead: no tie or page + // comes before the repair's prefix. + const firstIssued = ledger[before] + if (owesRepair && firstIssued && !firstIssued.authoritative) { + throw new Error( + `owed repair: request ${firstIssued.index}, an ordinary read, came before the repair a report owed`, + ) + } // A report the loader cannot prove covers nothing it could change // owes a repair, which its next pass starts, or a tie request for // the page's end, which completes the proof. Pending joined work @@ -2923,13 +3029,13 @@ async function checkAuthorityCase( !insideFirstPage ) { if (joinedPending && testCase.joined) { - owedProof = { from: before, windowSize } + owedProof = { from: before, endRank } } else { - expectProofRequest(before, windowSize) + expectProofRequest(before, endRank) repairedReportsOnOwnPage++ } } - if (quiet && (beforeSettle || insideFirstPage)) { + if (quiet && beforeSettle) { const issued = ledger .slice(before) .find(({ authoritative }) => authoritative) @@ -2938,11 +3044,34 @@ async function checkAuthorityCase( `ignored report: request ${issued.index} re-read the source for a report that covers nothing it could change`, ) } - if (beforeSettle) ignoredReportsBeforeSettle++ + ignoredReportsBeforeSettle++ + } else if (quiet && (insideFirstPage || coversSource)) { + // The loader ignores the report, so it reads nothing for it. + const issued = ledger[before] + if (issued) { + throw new Error( + `ignored report: request ${issued.index} was read for a report over coverage that ${coversSource ? `a full-source read` : `the loader's own first page`} proves`, + ) + } + if (coversSource) ignoredReportsOverFullSource++ else ignoredReportsInsideFirstPage++ } break } + // A refetch delivered a row that sorts before every installed row. + case `headInsert`: { + headInserts++ + const row = { id: 1000 + headInserts, rank: -headInserts } + loader.onSourceChanges( + [{ type: `insert`, key: row.id, value: row }], + sentRows, + () => false, + ) + sentRows.set(row.id, row) + graphPass() + await flush() + break + } case `pass`: graphPass() await flush() @@ -2967,6 +3096,7 @@ async function checkAuthorityCase( case `unexpressibleBoundary`: boundaryUnexpressible = !boundaryUnexpressible if (boundaryUnexpressible) unexpressibleBoundaries++ + repairOwed = true loader.invalidateSourceOrdering() graphPass() await flush() @@ -2981,7 +3111,7 @@ async function checkAuthorityCase( graphPass() await flush() if (owedProof) { - expectProofRequest(owedProof.from, owedProof.windowSize) + expectProofRequest(owedProof.from, owedProof.endRank) owedProof = undefined repairedReportsOnOwnPage++ } @@ -3014,6 +3144,7 @@ async function checkAuthorityCase( await flush() break case `sourceDelete`: + repairOwed = true sourceDeletes++ loader.onSourceChanges( [{ type: `delete`, key: 1, value: { id: 1, rank: 1 } }], @@ -3203,16 +3334,18 @@ async function checkAuthorityCase( // Cost: a full-source read is very expensive, so a repair avoids one // whenever a bounded read is available (ARCHITECTURE.md). That holds // after a settled request in this incarnation, for a plan that does - // not need the full source, with an expressible boundary, unless an - // unindexed window still needs rows past a repeated prefix (the - // documented fallback). One repair is then one chain per ordering - // epoch move: in this fake, a prefix, a tie, and a refill. + // not need the full source, with an expressible boundary. An + // unindexed window that still needs rows grows its prefix instead + // (ARCHITECTURE.md, "an underfilled unindexed prefix grows"). One + // repair is then one chain per ordering epoch move: a prefix, then a + // tie and a refill page for each boundary it reaches (ARCHITECTURE.md, + // ordered repair). Each boundary is a different installed row, so a + // chain issues at most one prefix plus two requests per row. await drain() const repair = ledger.slice(before) const boundedRepair = !testCase.requiresFullSource && !boundaryUnexpressible && - (testCase.indexed || !testCase.needsMore) && repair.every(({ outcome }) => outcome === `ok`) && ledger.some( (request) => @@ -3232,7 +3365,7 @@ async function checkAuthorityCase( (request) => request.authoritative, ).length const chains = epochMoves - movesBefore - if (authoritative > 3 * chains) { + if (authoritative > chains * (1 + 2 * snapshot().length)) { throw new Error( `cost: one invalidation issued ${authoritative} authoritative requests for ${chains} epoch moves`, ) @@ -3271,7 +3404,13 @@ async function checkAuthorityCase( joinedDeferrals, ignoredReportsBeforeSettle, ignoredReportsInsideFirstPage, + ignoredReportsOverFullSource, repairedReportsOnOwnPage, + movedEndReports, + reportsPastWindow, + reportsOwingRepair, + reportsDuringWork, + reportReleasesEndingLoader, truncates, replayFailures, optimisticDepartures, @@ -3893,29 +4032,99 @@ describe(`Ordered source request authority`, () => { 120_000, ) - // The unproven-report law needs a quiet history whose coverage is the - // loader's own first page, longer than the window, with no more rows to - // read, and then a report whose pass reacts. The broad grammar rarely lines - // these up, so each history here settles the first page, settles the read - // after it, and reports, over random case parameters, before a random - // suffix. A release that truncates or disposes would end the quiet history. + // The report laws need premises that the broad grammar rarely lines up. + // Each history here starts with one of these prefixes over random case + // parameters, then a random suffix: + // - the first page and its tie settle, then a report: the window is + // inside the loader's own first page, so the report is ignored; + // - the same, but a head insert first moves the page's end before the tie + // the loader read, so the report is unproven; + // - an invalidation whose reaction waits, then a report that reacts: a + // repair is owed; + // - an invalidation that starts its repair, then a report while that + // repair is in flight; + // - the moved-end prefix again with a release callback that truncates or + // disposes the loader, so releasing the older tie ends the loader; + // - a two-row window whose first page and tie settle, then shrinks to one + // row before a report: the page read past the window, which proves it; + // - a plan that reads only the full source, then a report over it. + // A release that truncates or disposes would end the quiet history, so + // only the fifth prefix allows one. const unprovenReportProperty = `ordered-loader.unproven-report` - const unprovenReportCase: fc.Arbitrary = fc - .tuple(authorityCaseArbitrary, fc.array(authorityCommand, { maxLength: 8 })) - .map(([testCase, rest]) => ({ - ...testCase, - settledFirst: true, - requiresFullSource: false, - snapshotRows: 3, - needsMore: false, - releaseTruncates: false, - releaseDisposes: false, + const reportPrefixes: Array<{ + history: Array + endsLoader?: boolean + fullSource?: boolean + wide?: boolean + }> = [ + { history: [ { type: `settle`, pick: 0, ok: true }, { type: `settle`, pick: 0, ok: true }, { type: `report`, react: true }, - ...rest, ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert` }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `invalidate`, react: false }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `invalidate`, react: true }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert` }, + { type: `report`, react: true }, + ], + endsLoader: true, + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `move`, delta: -1 }, + { type: `report`, react: true }, + ], + wide: true, + }, + { history: [{ type: `report`, react: true }], fullSource: true }, + ] + const unprovenReportCase: fc.Arbitrary = fc + .tuple( + authorityCaseArbitrary, + fc.constantFrom(...reportPrefixes), + fc.boolean(), + fc.array(authorityCommand, { maxLength: 8 }), + ) + .map(([testCase, prefix, truncates, rest]) => ({ + ...testCase, + settledFirst: true, + requiresFullSource: prefix.fullSource === true, + ...(prefix.wide ? { limit: 2 as const, offset: 0 as const } : {}), + snapshotRows: 3, + needsMore: false, + releaseTruncates: prefix.endsLoader === true && truncates, + releaseDisposes: prefix.endsLoader === true && !truncates, + history: [...prefix.history, ...rest], })) const unprovenReportCampaigns = authorityReplay.replayPath === undefined @@ -3925,26 +4134,33 @@ describe(`Ordered source request authority`, () => { : ([] as const) it.each(unprovenReportCampaigns)( - `reads the end's tie or repairs after an unproven report (%s campaign)`, + `answers each loaded-subset report as its coverage requires (%s campaign)`, async (campaign) => { - // Declared reach: histories where the unproven-report law judged a - // report over the loader's own first page. - const reach = createOracleReach( - unprovenReportProperty, - [`repairedReportsOnOwnPage`] as const, - { minimum: 20 }, - ) + // Declared reach: histories where each report law judged a report. + const reportStates = [ + `repairedReportsOnOwnPage`, + `ignoredReportsInsideFirstPage`, + `ignoredReportsOverFullSource`, + `movedEndReports`, + `reportsPastWindow`, + `reportsOwingRepair`, + `reportsDuringWork`, + `reportReleasesEndingLoader`, + ] as const + const reach = createOracleReach(unprovenReportProperty, reportStates, { + minimum: 20, + }) await fc.assert( fc.asyncProperty(unprovenReportCase, async (testCase) => { reach.beginHistory() const observation = await checkAuthorityCase(testCase) - if (observation.repairedReportsOnOwnPage > 0) { - reach.reach(`repairedReportsOnOwnPage`) + for (const state of reportStates) { + if (observation[state] > 0) reach.reach(state) } }), campaign === `fixed` - ? { seed: 20261023, numRuns: oracleRuns(300) } - : oraclePropertyOptions(300, unprovenReportProperty), + ? { seed: 20261023, numRuns: oracleRuns(600) } + : oraclePropertyOptions(600, unprovenReportProperty), ) reach.assertReached() }, From 3ead20ae4464dff230ef20608eb8621849512333 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 22:34:18 -0600 Subject: [PATCH 22/40] fix(db): judge a lazy demand's failure against its source's shared barrier A query that joins and includes one source has two subscriptions to it, which share one replay barrier. When one subscription's replay failed and the other's finished, demand that the finished subscription then started and lost ended the query: the subscriber judged the failure by its own subscription's replay. Law (ARCHITECTURE.md, "Replay participants and failure", now extended): a failed replay of any of a query's subscriptions to one collection fails the shared barrier. Until a later replay of the source succeeds, a lazy demand of any of them that fails keeps the last publication and does not end the query. Reach: the failed-replay campaign's sibling branch may now move a row and fail the oldest held load; `failedSiblingDemandFailures` (minimum 5) is reached 11 to 19 times per 400-run campaign. RED (unchanged production): the random campaign found seed -1842904352, path 22:1:0:0:0:0:2:0:2:5:5:5 (also on the loader before prefix growth); the named replay "a demand that fails after the sibling subscription's replay failed" and both campaigns fail at "convergence: terminal error". GREEN: load-subset replay oracle 38/38; @tanstack/db 285 files, 13,834 tests. Mutant: the per-subscription rule is the RED above. A rule that judged against any failed replay of the query, across collections, survives: the convergence check follows an explicit window move, which retries a failed row demand. ARCHITECTURE.md keeps that failure fatal; recorded as a limit. Also, the report path's own `!active` checks go: dispose() advances the generation that the stale-run check after the older tie's release reads, and a disposed loader issues nothing. The report campaign gains a dispose-before-report prefix (reached 69 to 90 times per campaign), and it runs 1,000 histories, since at 600 a random seed could miss a minimum. GREEN: ordered source loader state oracle 77/77, query-db refetch oracle 40/40, ordered and replay oracles 1093/1093. Size: @tanstack/db +501 B gzip vs main (the barrier check costs about 33 B after a smaller form replaced a shared lookup helper). Co-authored-by: Isaac --- .../keep-live-query-through-failed-replay.md | 2 +- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 4 ++ .../query/live/collection-config-builder.ts | 12 ++++ .../src/query/live/collection-subscriber.ts | 10 ++-- .../src/query/live/ordered-source-loader.ts | 9 ++- ...ad-subset-replay-refinement-oracle.test.ts | 59 ++++++++++++++++++- ...ordered-source-loader-state-oracle.test.ts | 23 +++++++- 8 files changed, 104 insertions(+), 17 deletions(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index 0450635b77..e641a4fc87 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -4,7 +4,7 @@ Keep a live query running when a lazy join or include load fails as part of a source truncate. The truncate's replay fails instead, and the query keeps its last complete result until a later replay of that source succeeds. Before, the query could end in error 228, for example when a truncate made an ordered window demand a row that it never showed. A load that fails before the query first becomes ready still ends it. -A query that joins and includes the same source has two subscriptions to it. Their truncate replays now share one barrier, so a lazy demand that fails while either replay still loads no longer ends the query. +A query that joins and includes the same source has two subscriptions to it. Their truncate replays now share one barrier. A lazy demand that fails while either replay still loads, or after either replay failed, no longer ends the query. An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b15008368c..03f48f61e0 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 600 runs) now starts each history with one of seven prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, and a release of the older tie that truncates or disposes the loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. Still open there: the unexpressible-end term (the fake's comparator cannot place NaN or nulls), the stale-run check after the older tie's release (the fake's `dataNeeded` ignores its boundary), and the in-flight-work, staged-continuation, and covered-source checks, each of which another check decides in every state the grammar reaches. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. Still open there: the unexpressible-end term (the fake's comparator cannot place NaN or nulls), the stale-run check after the older tie's release (the fake's `dataNeeded` ignores its boundary), and the in-flight-work, staged-continuation, and covered-source checks, each of which another check decides in every state the grammar reaches. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index b4ac6ae725..b957772c51 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1468,6 +1468,10 @@ join and an include of the same source, share one publication barrier. None of their replays publishes, and none whose last demand retired closes, while another of them still has replay work. So demand that any of them starts meanwhile joins its own open replay instead of falling outside recovery. +A failed replay of any of them fails the barrier: until a later replay of the +source succeeds, a lazy demand of any of them that fails keeps the last +publication and does not end the query, even when its own subscription's +replay finished. A rejection of an aborted lazy demand request is obsolete demand, not a load failure, as for the subscription. When a truncate supersedes the acquisition, its replay decides the demand, even if the replay settles before diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 9da1ab53f5..e236216479 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -662,6 +662,18 @@ export class CollectionConfigBuilder< return pending.length > 0 ? Promise.allSettled(pending) : undefined } + /** + * Whether a failed replay of a collection still holds publication. The + * query's subscriptions to one collection share that replay's barrier. + */ + hasFailedReplay(collection: Collection): boolean { + return this.collectionSources.some( + (source) => + source.collection === collection && + this.subscriptions[source.sourceId]?.hasFailedTruncateReplacement, + ) + } + private hasFailedSourceRecovery(): boolean { return Object.values(this.subscriptions).some( (subscription) => subscription.hasFailedTruncateReplacement, diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index b168343931..de81ffb847 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -225,9 +225,11 @@ export class CollectionSubscriber< () => this.collectionConfigBuilder.settleDemand(plan.id, generation), (error) => { // A truncate replay aborts the acquisition it supersedes. Its - // replacement decides the demand. A failure that fails the - // replay keeps the last publication until a later replay of this - // source succeeds, so it does not end the query. + // replacement decides the demand. A failure while a replay of + // this source failed, including another subscription's replay + // behind the same barrier, keeps the last publication until a + // later replay of the source succeeds, so it does not end the + // query. const replacement = subscription.pendingTruncateReplacement if (replacement) settleWith(replacement) else @@ -235,7 +237,7 @@ export class CollectionSubscriber< plan.id, generation, error, - !subscription.hasFailedTruncateReplacement, + !this.collectionConfigBuilder.hasFailedReplay(this.collection), ) }, ) diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index cdc80b5c8b..86bd287b01 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -685,8 +685,7 @@ export class OrderedSourceLoader { revalidateLoadedCoverage(): void { // A `requiresFullSource` plan's coverage is its full-source read: nothing // has settled before it, and it covers the source after. - if (!this.active || !this.hasSettledSourceRequest || this.coversSource()) - return + if (!this.hasSettledSourceRequest || this.coversSource()) return const end = this.ownPrefixEnd() if (end === undefined) { this.invalidateSourceOrdering() @@ -739,9 +738,9 @@ export class OrderedSourceLoader { } finally { this.retiringSupersededAcquisitions = false } - // A release that resets the cursor hands the window to the replay. - if (!this.active || orderedLoadGeneration !== this.orderedLoadGeneration) - return + // A release that resets the cursor, or disposes the loader, hands the + // window to the replay or ends it. + if (orderedLoadGeneration !== this.orderedLoadGeneration) return this.settledSourceBoundary = end // The pass that follows the report reads the tie, as a repair's first // request would start then, so a start that throws fails that pass. diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 6922009d3f..6c3f171771 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1308,6 +1308,7 @@ type OrderedReach = { participantDemandFailures: number includeDemandFailures: number siblingDemandFailures: number + failedSiblingDemandFailures: number preloadReplayFailures: number discardedReplayMoves: number } @@ -1614,6 +1615,7 @@ async function checkOrderedReplay( participantDemandFailures: 0, includeDemandFailures: 0, siblingDemandFailures: 0, + failedSiblingDemandFailures: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -1661,6 +1663,10 @@ async function checkOrderedReplay( held.push(resolve), ) if (!ok) { + // The source's replay had already failed and still holds + // publication when this load fails. + const barrierFailed = + (barriers.get(`markers`)?.failures.size ?? 0) > 0 const replayLoading = settleParticipant( `markers`, load, @@ -1688,6 +1694,9 @@ async function checkOrderedReplay( // two of its subscriptions share the barrier. if (scenario.includes && scenario.joined) { reach.siblingDemandFailures++ + // After the shared replay failed: the subscription whose + // demand fails may have finished its own replay. + if (barrierFailed) reach.failedSiblingDemandFailures++ } } if (queries.some(({ status }) => status === `loading`)) { @@ -2568,6 +2577,31 @@ describe(`Ordered windows across truncate replays`, () => { reaches: `siblingDemandFailures`, recovers: true, }, + { + // Found by the failed-replay random campaign. One subscription's + // replay fails, the other's finishes once it settles, and a rank move + // starts demand on that other subscription, which then fails. Its own + // replay had finished, so a per-subscription judgment ended the query, + // while the source's shared barrier still held the failed replay. + name: `a demand that fails after the sibling subscription's replay failed`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + { type: `move`, id: 2, rank: 1 }, + { type: `settle`, ok: false, keepHolding: false, newest: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + reaches: `failedSiblingDemandFailures`, + recovers: true, + }, ] satisfies Array<{ name: string history: Array @@ -2621,6 +2655,7 @@ describe(`Ordered windows across truncate replays`, () => { participantDemandFailures: 0, includeDemandFailures: 0, siblingDemandFailures: 0, + failedSiblingDemandFailures: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -2656,6 +2691,7 @@ describe(`Ordered windows across truncate replays`, () => { `participantDemandFailures`, `includeDemandFailures`, `siblingDemandFailures`, + `failedSiblingDemandFailures`, `preloadReplayFailures`, `discardedReplayMoves`, ]) @@ -2759,16 +2795,26 @@ describe(`Ordered windows across truncate replays`, () => { // subscription's own replay loads while the other's still load; the graph // then demands markers for rows the commit privately exposed, and the // newest load fails. + // After the failing load, half the histories move a row, which starts + // demand once the shared replay has failed, and then fail the oldest held + // load: the other subscription may have finished its own replay by then. const siblingDemandCase = fc .tuple( failedReplayScenario, fc.constantFrom(`some` as const, `none` as const, `inner` as const), fc.integer({ min: 1, max: 2 }), fc.boolean(), + fc.option( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + ), fc.array(orderedCommand, { maxLength: 6 }), ) .map( - ([scenario, joined, settles, newest, rest]): [ + ([scenario, joined, settles, newest, move, rest]): [ Array, OrderedScenario, ] => [ @@ -2782,6 +2828,9 @@ describe(`Ordered windows across truncate replays`, () => { newest, })), { type: `settle`, ok: false, newest: true, keepHolding: true }, + ...(move + ? [move, { type: `settle` as const, ok: false, newest: false }] + : []), ...rest, ], { ...scenario, joined, includes: true, markerMode: `on-demand` }, @@ -2962,6 +3011,7 @@ describe(`Ordered windows across truncate replays`, () => { `participantDemandFailures`, `includeDemandFailures`, `siblingDemandFailures`, + `failedSiblingDemandFailures`, ] as const, { minimum: 5 }, ) @@ -2984,10 +3034,13 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.siblingDemandFailures > 0) { reach.reach(`siblingDemandFailures`) } + if (observed.failedSiblingDemandFailures > 0) { + reach.reach(`failedSiblingDemandFailures`) + } }), campaign === `fixed` - ? { seed: 20261023, numRuns: oracleRuns(200) } - : oraclePropertyOptions(200, failedReplayProperty), + ? { seed: 20261023, numRuns: oracleRuns(400) } + : oraclePropertyOptions(400, failedReplayProperty), ) reach.assertReached() }, diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index baf86e8d29..6e773ad9fd 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -2207,8 +2207,10 @@ type AuthorityObservation = { // while authoritative work was in flight, and reports whose release of an // older tie truncated or disposed the loader. movedEndReports: number - // Reports whose own first page read past a window that then shrank. + // Reports whose own first page read past a window that then shrank, and + // reports to a disposed loader. reportsPastWindow: number + reportsAfterDispose: number reportsOwingRepair: number reportsDuringWork: number reportReleasesEndingLoader: number @@ -2613,6 +2615,7 @@ async function checkAuthorityCase( let repairedReportsOnOwnPage = 0 let movedEndReports = 0 let reportsPastWindow = 0 + let reportsAfterDispose = 0 let reportsOwingRepair = 0 let reportsDuringWork = 0 let reportReleasesEndingLoader = 0 @@ -2983,6 +2986,7 @@ async function checkAuthorityCase( ({ fullSource, outcome, released }) => fullSource && outcome === `ok` && !released, ) + if (isDisposed()) reportsAfterDispose++ const owesRepair = repairOwed && !replayPending() && !isDisposed() && command.react if (owesRepair) reportsOwingRepair++ @@ -3408,6 +3412,7 @@ async function checkAuthorityCase( repairedReportsOnOwnPage, movedEndReports, reportsPastWindow, + reportsAfterDispose, reportsOwingRepair, reportsDuringWork, reportReleasesEndingLoader, @@ -4047,6 +4052,8 @@ describe(`Ordered source request authority`, () => { // disposes the loader, so releasing the older tie ends the loader; // - a two-row window whose first page and tie settle, then shrinks to one // row before a report: the page read past the window, which proves it; + // - the moved-end prefix with the loader disposed before the report, which + // the dispose law then judges; // - a plan that reads only the full source, then a report over it. // A release that truncates or disposes would end the quiet history, so // only the fifth prefix allows one. @@ -4106,6 +4113,15 @@ describe(`Ordered source request authority`, () => { ], wide: true, }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert` }, + { type: `dispose` }, + { type: `report`, react: true }, + ], + }, { history: [{ type: `report`, react: true }], fullSource: true }, ] const unprovenReportCase: fc.Arbitrary = fc @@ -4143,6 +4159,7 @@ describe(`Ordered source request authority`, () => { `ignoredReportsOverFullSource`, `movedEndReports`, `reportsPastWindow`, + `reportsAfterDispose`, `reportsOwingRepair`, `reportsDuringWork`, `reportReleasesEndingLoader`, @@ -4159,8 +4176,8 @@ describe(`Ordered source request authority`, () => { } }), campaign === `fixed` - ? { seed: 20261023, numRuns: oracleRuns(600) } - : oraclePropertyOptions(600, unprovenReportProperty), + ? { seed: 20261023, numRuns: oracleRuns(1000) } + : oraclePropertyOptions(1000, unprovenReportProperty), ) reach.assertReached() }, From e0e4390bd410fe7d2f9c3deeebcf3c6b110f071d Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 08:15:51 -0600 Subject: [PATCH 23/40] refactor(db): close the report path's remaining guards with laws or deletions A deletion sweep left five guards on the loaded-subset report path that no oracle distinguished. Each now has a law that fails without it, or it is deleted because another check decides every state it covered. Laws (ARCHITECTURE.md, the loaded-subset report and truncate rules; `canExpressCursorOrder`): - the loader never reads the tie of a rank no cursor can express; - the first finite request after a truncate starts with no cursor; - a report over coverage that a held full-source read completed issues no request, whatever came before, since such coverage owes nothing. Oracle changes: a head insert can carry a `-Infinity` rank; the fake's dataNeeded counts the window rows not at or before the boundary it gets, where it returned a constant; and the report campaign gains prefixes for an unexpressible end, a release that truncates before the window grows, and a paged loader whose repair fell back to a full-source read. The cost law's premise now asks whether every installed rank is expressible. Reach: each new report state reaches its minimum of 20 with a margin (the lowest state across six random seeds was 47). RED (hostile mutants; production unchanged for these): - the stale-run check after the older tie's release removed: "reset after step 4: request 2, the first finite request after a truncate, continued from a cursor" in the fixed and random report campaigns; - loadBoundary's unexpressible-value check removed: "unexpressible tie after the drain: request 3 read the tie of rank -Infinity"; - the work-slot term in loadMovedEndTie removed (with the deletions below): "ignored report: request 2 was read for a report over coverage that a full-source read proves"; - the unexpressible-end term in loadMovedEndTie removed: the unindexed-refetch random campaign found seed 1620448148, where an unindexed loader reads the full source through the boundary fallback; a named replay of that trace fails with "full-source requests after a signalling refetch at step 2: expected 2 to be +0". Deleted, with the oracles green: the covered-source early return in revalidateLoadedCoverage (the work-slot check decides it) and the staged-continuation term in loadMovedEndTie (a staged finite continuation cannot coexist with a proven end; a staged repair step owes a repair). GREEN: ordered source loader state 77/77; ordered, replay, and pagination oracles 1095/1095; query-db refetch oracle 40/40; @tanstack/db 285 files, 13,835 tests. Limits: the moved-end check's `Object.is` term still survives; it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. Size: @tanstack/db +497 B gzip vs main (was +501). Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 3 +- .../src/query/live/ordered-source-loader.ts | 16 +- ...ordered-source-loader-state-oracle.test.ts | 216 ++++++++++++++++-- .../ordered-work-oracle.property.test.ts | 38 +++ 5 files changed, 243 insertions(+), 32 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 03f48f61e0..81a25c54a6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. Still open there: the unexpressible-end term (the fake's comparator cannot place NaN or nulls), the stale-run check after the older tie's release (the fake's `dataNeeded` ignores its boundary), and the in-flight-work, staged-continuation, and covered-source checks, each of which another check decides in every state the grammar reaches. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index b957772c51..82bb1586ca 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1018,7 +1018,8 @@ tie completes the proof without a repair. It then releases the older tie, whose rows sort after the end and outside the window, and the deletes that release causes are not an order change. A loader that already owes a repair, has other work in flight, or has an end that no cursor can express repairs -instead. A window that needs a row +instead: reading that end's tie would take the boundary's full-source +fallback, while the repair's bounded prefix can prove the window. A window that needs a row past the page, such as one whose page holds rows a joined filter rejects, still repairs. So does any loader while its source holds accepted sync rows back, as behind a persisting optimistic diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 86bd287b01..06b04858d5 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -684,8 +684,10 @@ export class OrderedSourceLoader { */ revalidateLoadedCoverage(): void { // A `requiresFullSource` plan's coverage is its full-source read: nothing - // has settled before it, and it covers the source after. - if (!this.hasSettledSourceRequest || this.coversSource()) return + // has settled before it. Coverage that one full-source read completed + // holds the work slot, so the end's tie below defers to it, and an + // ordering invalidation owes it nothing. + if (!this.hasSettledSourceRequest) return const end = this.ownPrefixEnd() if (end === undefined) { this.invalidateSourceOrdering() @@ -709,9 +711,12 @@ export class OrderedSourceLoader { // tie completes the proof without a repair. The older tie's rows sort // after the end and outside the window, and they prove nothing now. private loadMovedEndTie(end: Record): void { - // Other work owns the boundary, a repair is already owed, a replay owns - // the leases, or no cursor can express the end's tie, so only the - // repair's full-source fallback can read it. + // Other work owns the boundary, a repair is already owed, or a replay + // owns the leases, so the repair reads it. A staged continuation is + // replaced: the window needs no more rows, and a staged repair step owes + // a repair. No cursor can express some ends' ties; reading one would + // take the boundary's full-source fallback, while the repair's bounded + // prefix can prove the window without it. if ( !canExpressCursorOrder( normalizeOrderByPaths(this.info.orderBy, this.alias).slice(0, 1), @@ -720,7 +725,6 @@ export class OrderedSourceLoader { this.debt !== `none` || this.work.kind !== `idle` || this.pending || - this.stagedContinuation || this.subscription.hasPendingTruncateReplacement ) { this.invalidateSourceOrdering() diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 6e773ad9fd..6f44f426d3 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -1823,7 +1823,9 @@ describe(`Successful finite demand retirement`, () => { * shifted issues no authoritative request before any request settled: a * request in flight still owes an exact result for its own options. The * loader ignores the report, and so issues no request at all, while its - * coverage is a held full-source read, or while it is the loader's own + * coverage is a held full-source read since the last truncate, whatever + * came before (an ordering invalidation owes such coverage nothing), or, + * in a quiet history, while its coverage is the loader's own * first ordered prefix of at least `offset + limit` rows, with only that * prefix's tie after it, and a row past the page sorts strictly after its * end with no more rows needed (ARCHITECTURE.md). That row is evidence @@ -1835,6 +1837,11 @@ describe(`Successful finite demand retirement`, () => { * - **Owed repair:** a report while the loader already owes a repair * repairs instead (ARCHITECTURE.md): the pass that reacts to it reads no * tie or page before the repair's prefix. + * - **Unexpressible tie:** the loader never reads the tie of a rank that no + * cursor can express, such as NaN or an infinity; it reads the full source + * or repairs instead (ARCHITECTURE.md; `canExpressCursorOrder`). + * - **Reset:** a truncate resets ordered loading, so the first finite + * request after it starts at the beginning of the order, with no cursor. * - **Unproven report:** in such a quiet history, a report over the loader's * own first prefix that no row it read proves starts a repair on its pass, * or reads the tie of the page's end, which completes the proof. @@ -1904,7 +1911,8 @@ describe(`Successful finite demand retirement`, () => { * the reaction deferred, as pending joined work or a later graph run can; * - `headInsert` delivers a new row that sorts before every installed row, * as a refetch that moved a provider row into the first page does, so the - * page's end moves one row earlier; + * page's end moves one row earlier. The row's rank may be `-Infinity`, + * which no cursor can express; * - `pass` runs an ordinary `loadMore()`; * - `move` is an explicit window move, which may also widen the window, so * an unindexed loader reads a longer prefix; @@ -2008,7 +2016,8 @@ type AuthorityCommand = | { type: `settle`; pick: number; ok: boolean } | { type: `invalidate`; react: boolean } | { type: `report`; react: boolean } - | { type: `headInsert` } + // `unexpressible` gives the row a rank that no cursor can express. + | { type: `headInsert`; unexpressible?: boolean } | { type: `failOldest` } | { type: `settleNewest` } | { type: `invalidateThenFailOldest` } @@ -2093,7 +2102,13 @@ const authorityCommand: fc.Arbitrary = fc.oneof( react: fc.boolean(), }), }, - { weight: 1, arbitrary: fc.constant({ type: `headInsert` as const }) }, + { + weight: 1, + arbitrary: fc.record({ + type: fc.constant(`headInsert` as const), + unexpressible: fc.boolean(), + }), + }, // The oldest in-flight request is the one most often superseded. { weight: 1, arbitrary: fc.constant({ type: `failOldest` as const }) }, // A provider can answer newer requests first, so a repair chain finishes @@ -2202,6 +2217,7 @@ type AuthorityObservation = { ignoredReportsBeforeSettle: number ignoredReportsInsideFirstPage: number ignoredReportsOverFullSource: number + ignoredReportsOverFallback: number repairedReportsOnOwnPage: number // Reports after a refetch moved the page's end, while a repair was owed, // while authoritative work was in flight, and reports whose release of an @@ -2211,6 +2227,11 @@ type AuthorityObservation = { // reports to a disposed loader. reportsPastWindow: number reportsAfterDispose: number + // Reports whose page's end, moved by a head insert, has no cursor. + unexpressibleEndReports: number + // First finite requests after a report's release truncated the loader, + // which the reset law judges. + resetsAfterReportRelease: number reportsOwingRepair: number reportsDuringWork: number reportReleasesEndingLoader: number @@ -2240,8 +2261,8 @@ async function checkAuthorityCase( let joinedPending = false let truncatePending = false let boundaryUnexpressible = false - // Rows a refetch delivered before every installed row. - let headInserts = 0 + // Rows a refetch delivered before every installed row, in rank order. + const headRows: Array<{ id: number; rank: number }> = [] // Whether an event since the last authoritative start owes a repair: an // ordering invalidation, a visible delete, or an unexpressible boundary. let repairOwed = false @@ -2252,7 +2273,22 @@ async function checkAuthorityCase( ...(testCase.syncRepair ? { sourceHoldsAllRows: true } : {}), requiresFullSource: testCase.requiresFullSource, ...(testCase.joined ? { joinedFilterSourceId: `markers` } : {}), - dataNeeded: () => (testCase.needsMore ? 1 : 0), + // A window that needs more always needs one row, as behind a joined + // filter that drops rows. Otherwise it needs the window's rows that do + // not sort at or before the loaded boundary. + dataNeeded: (boundary?: Record) => + testCase.needsMore + ? 1 + : Math.max( + 0, + info.offset + + info.limit - + (boundary === undefined + ? 0 + : snapshot().filter( + ({ value }) => value.rank <= (boundary.rank as number), + ).length), + ), offset: testCase.offset, }) let revision = 0 @@ -2290,6 +2326,10 @@ async function checkAuthorityCase( } } let releaseViolation: string | undefined + let unexpressibleTieViolation: string | undefined + let resetViolation: string | undefined + // The truncate count after a report's release truncated the loader. + let truncatedByReport = -1 // A disposed loader owns nothing: it issues no request and releases no // lease, even as requests it issued earlier settle. let disposed = false @@ -2365,6 +2405,41 @@ async function checkAuthorityCase( failedAt: -1, } ledger.push(request) + // No cursor expresses a NaN or infinite rank, so the loader reads the + // full source or repairs instead of reading that value's tie + // (ARCHITECTURE.md; `canExpressCursorOrder`). + // A truncate resets ordered loading (ARCHITECTURE.md), so the first + // finite request after it starts at the beginning of the order. A cursor + // there would continue from a boundary the truncate discarded. + if ( + truncates > 0 && + !fullSource && + options.minValues !== undefined && + ledger.every( + (other) => + other === request || + other.fullSource || + other.issuedTruncates !== truncates, + ) + ) { + resetViolation ??= `request ${request.index}, the first finite request after a truncate, continued from a cursor` + } + if ( + truncatedByReport === truncates && + !fullSource && + ledger.every( + (other) => + other === request || + other.fullSource || + other.issuedTruncates !== truncates, + ) + ) { + resetsAfterReportRelease++ + } + const tie = tieValue(options) + if (typeof tie === `number` && !Number.isFinite(tie)) { + unexpressibleTieViolation ??= `request ${request.index} read the tie of rank ${tie}, which no cursor can express` + } if (disposed) { disposeViolation ??= `request ${request.index} was issued after dispose` } @@ -2515,11 +2590,7 @@ async function checkAuthorityCase( const tie = options === undefined ? undefined : tieValue(options) const after = options?.cursor && cursorStarts.get(options.cursor) return [ - ...Array.from({ length: headInserts }, (_, index) => ({ - type: `insert`, - key: 1000 + headInserts - index, - value: { id: 1000 + headInserts - index, rank: index - headInserts }, - })), + ...headRows.map((row) => ({ type: `insert`, key: row.id, value: row })), ...Array.from({ length: testCase.snapshotRows }, (_, index) => ({ type: `insert`, key: index + 1, @@ -2612,10 +2683,13 @@ async function checkAuthorityCase( let ignoredReportsBeforeSettle = 0 let ignoredReportsInsideFirstPage = 0 let ignoredReportsOverFullSource = 0 + let ignoredReportsOverFallback = 0 let repairedReportsOnOwnPage = 0 let movedEndReports = 0 let reportsPastWindow = 0 let reportsAfterDispose = 0 + let unexpressibleEndReports = 0 + let resetsAfterReportRelease = 0 let reportsOwingRepair = 0 let reportsDuringWork = 0 let reportReleasesEndingLoader = 0 @@ -2775,6 +2849,14 @@ async function checkAuthorityCase( if (releaseViolation) { throw new Error(`release ${checkpoint}: ${releaseViolation}`) } + if (resetViolation) { + throw new Error(`reset ${checkpoint}: ${resetViolation}`) + } + if (unexpressibleTieViolation) { + throw new Error( + `unexpressible tie ${checkpoint}: ${unexpressibleTieViolation}`, + ) + } if (concurrencyViolation) { throw new Error( `one request at a time ${checkpoint}: ${concurrencyViolation}`, @@ -2981,12 +3063,27 @@ async function checkAuthorityCase( ) { movedEndReports++ } - // Coverage is a held full-source read. - const coversSource = ledger.some( - ({ fullSource, outcome, released }) => - fullSource && outcome === `ok` && !released, - ) + // Coverage is a held full-source read since the last truncate. It + // owes nothing whatever came before, so this premise needs no quiet + // history. + const coversSource = + !replayPending() && + !isDisposed() && + ledger.some( + ({ fullSource, outcome, released, issuedTruncates }) => + fullSource && + outcome === `ok` && + !released && + issuedTruncates === truncates, + ) if (isDisposed()) reportsAfterDispose++ + if ( + ownFirstPage && + typeof endRank === `number` && + !Number.isFinite(endRank) + ) { + unexpressibleEndReports++ + } const owesRepair = repairOwed && !replayPending() && !isDisposed() && command.react if (owesRepair) reportsOwingRepair++ @@ -3000,7 +3097,9 @@ async function checkAuthorityCase( } const before = ledger.length const releasedBefore = ledger.filter(({ released }) => released) + const truncatesBefore = truncates loader.revalidateLoadedCoverage() + if (truncates > truncatesBefore) truncatedByReport = truncates if ( (testCase.releaseTruncates || testCase.releaseDisposes) && ledger.some( @@ -3049,7 +3148,7 @@ async function checkAuthorityCase( ) } ignoredReportsBeforeSettle++ - } else if (quiet && (insideFirstPage || coversSource)) { + } else if ((quiet && insideFirstPage) || coversSource) { // The loader ignores the report, so it reads nothing for it. const issued = ledger[before] if (issued) { @@ -3057,15 +3156,31 @@ async function checkAuthorityCase( `ignored report: request ${issued.index} was read for a report over coverage that ${coversSource ? `a full-source read` : `the loader's own first page`} proves`, ) } - if (coversSource) ignoredReportsOverFullSource++ - else ignoredReportsInsideFirstPage++ + if (!coversSource) ignoredReportsInsideFirstPage++ + else if (testCase.requiresFullSource) ignoredReportsOverFullSource++ + // A paged loader whose boundary fell back to a full-source read. + else ignoredReportsOverFallback++ } break } // A refetch delivered a row that sorts before every installed row. case `headInsert`: { - headInserts++ - const row = { id: 1000 + headInserts, rank: -headInserts } + // One `-Infinity` row at most: a second would tie with it, which + // the fake's subtraction order cannot compare. + const unexpressible = + command.unexpressible === true && + headRows.every(({ rank }) => Number.isFinite(rank)) + const row = { + id: 1001 + headRows.length, + rank: unexpressible + ? Number.NEGATIVE_INFINITY + : -1 - + headRows.filter(({ rank }) => Number.isFinite(rank)).length, + } + headRows.push(row) + headRows.sort((left, right) => + left.rank === right.rank ? 0 : left.rank < right.rank ? -1 : 1, + ) loader.onSourceChanges( [{ type: `insert`, key: row.id, value: row }], sentRows, @@ -3349,7 +3464,9 @@ async function checkAuthorityCase( const repair = ledger.slice(before) const boundedRepair = !testCase.requiresFullSource && - !boundaryUnexpressible && + // Every installed rank is expressible, so no boundary needs the + // full-source fallback. + snapshot().every(({ value }) => Number.isFinite(value.rank)) && repair.every(({ outcome }) => outcome === `ok`) && ledger.some( (request) => @@ -3409,10 +3526,13 @@ async function checkAuthorityCase( ignoredReportsBeforeSettle, ignoredReportsInsideFirstPage, ignoredReportsOverFullSource, + ignoredReportsOverFallback, repairedReportsOnOwnPage, movedEndReports, reportsPastWindow, reportsAfterDispose, + unexpressibleEndReports, + resetsAfterReportRelease, reportsOwingRepair, reportsDuringWork, reportReleasesEndingLoader, @@ -4054,6 +4174,13 @@ describe(`Ordered source request authority`, () => { // row before a report: the page read past the window, which proves it; // - the moved-end prefix with the loader disposed before the report, which // the dispose law then judges; + // - the moved-end prefix with a `-Infinity` head row, so no cursor can + // express the page's new end; + // - the moved-end prefix whose release of the older tie truncates the + // loader, then a window that grows, which needs a page after the reset; + // - a one-row window whose page and tie settle, then a `-Infinity` head + // row and an invalidation: the repair's boundary has no cursor, so it + // falls back to a full-source read, and a report follows over it; // - a plan that reads only the full source, then a report over it. // A release that truncates or disposes would end the quiet history, so // only the fifth prefix allows one. @@ -4061,7 +4188,9 @@ describe(`Ordered source request authority`, () => { const reportPrefixes: Array<{ history: Array endsLoader?: boolean + truncatesLoader?: boolean fullSource?: boolean + narrow?: boolean wide?: boolean }> = [ { @@ -4122,6 +4251,39 @@ describe(`Ordered source request authority`, () => { { type: `report`, react: true }, ], }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert`, unexpressible: true }, + { type: `report`, react: true }, + ], + // The end is the `-Infinity` row only in a one-row window. + narrow: true, + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert` }, + { type: `report`, react: true }, + { type: `move`, delta: 1 }, + ], + truncatesLoader: true, + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert`, unexpressible: true }, + { type: `invalidate`, react: true }, + { type: `settleNewest` }, + { type: `settleNewest` }, + { type: `settleNewest` }, + { type: `report`, react: true }, + ], + narrow: true, + }, { history: [{ type: `report`, react: true }], fullSource: true }, ] const unprovenReportCase: fc.Arbitrary = fc @@ -4136,9 +4298,12 @@ describe(`Ordered source request authority`, () => { settledFirst: true, requiresFullSource: prefix.fullSource === true, ...(prefix.wide ? { limit: 2 as const, offset: 0 as const } : {}), + ...(prefix.narrow ? { limit: 1 as const, offset: 0 as const } : {}), snapshotRows: 3, needsMore: false, - releaseTruncates: prefix.endsLoader === true && truncates, + releaseTruncates: + prefix.truncatesLoader === true || + (prefix.endsLoader === true && truncates), releaseDisposes: prefix.endsLoader === true && !truncates, history: [...prefix.history, ...rest], })) @@ -4157,9 +4322,12 @@ describe(`Ordered source request authority`, () => { `repairedReportsOnOwnPage`, `ignoredReportsInsideFirstPage`, `ignoredReportsOverFullSource`, + `ignoredReportsOverFallback`, `movedEndReports`, `reportsPastWindow`, `reportsAfterDispose`, + `unexpressibleEndReports`, + `resetsAfterReportRelease`, `reportsOwingRepair`, `reportsDuringWork`, `reportReleasesEndingLoader`, diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 1ed1658b69..967c9af1c2 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -7425,6 +7425,44 @@ describe(`eager indexed ordered windows`, () => { }) }) + // Found by the unindexed-refetch random campaign. A refetch report moves + // the page's end onto a null rank, which no cursor can express. Reading + // that end's tie would take the boundary's full-source fallback, so the + // loader repairs with its bounded prefix instead; a loader that read the + // tie broke the refetch law. + it(`repairs instead of reading an unexpressible end's tie after a signalling refetch`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `joined`, + via: `marker`, + indexed: false, + direction: `desc`, + nulls: `first`, + terms: 1, + limit: 2, + offset: 1, + initial: [ + { id: 0, rank: null as unknown as number, eligible: true }, + { id: 2, rank: Number.NaN, eligible: true }, + { id: 1, rank: 0, eligible: true }, + { id: 3, rank: 0, eligible: true }, + ], + history: [ + { + type: `upsert`, + id: 10, + rank: null as unknown as number, + eligible: false, + }, + { type: `dup`, id: 0 }, + ], + }) + }) + // A peer's tie answer changes, so the provider reports, but the first // query's page and tie answers stay the same and the peer installed the // row after its page. That page proves the window, so the report must From cd3d062fb73cb11a12374bdd065bbd9f356dfd0b Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 11:19:30 -0600 Subject: [PATCH 24/40] fix(db): fail a truncate replay when a demand's load throws inside it A source whose loadSubset throws instead of rejecting could end a live query during a truncate replay: the subscription dropped the thrown demand, so its failure did not fail the replay, and the subscriber treated the synchronous throw as fatal. Law (ARCHITECTURE.md, "Replay participants and failure", extended by maintainer decision): a load that throws while it starts inside an open replay fails that replay as a rejected load does. The demand stays with the subscription, so the replay keeps the last publication, a later replay retries it, and its owner releases it; the adapter acquired nothing, so it receives no unloadSubset for that load. Oracle: `markerThrows` arms the next marker load to throw while it starts. A thrown load joins a replay only if that replay still has a participant or a failure, and an armed throw expires before the drain. Reach: `thrownReplayDemands` (a throw into a replay that is only loading) reaches 10 to 24 per 600-run failed-replay campaign. RED (parent code): the named replays "a replay load that throws while it starts" and "a demand whose load throws inside its open replay", both failed-replay campaigns, and the broad fixed campaign fail at "convergence: terminal error". GREEN: load-subset replay oracle 41/41; @tanstack/db 285 files, 13,838 tests; query-db, electric, powersync, trailbase, and react suites pass. Mutant: unloading the never-acquired load fails with "adapter release: marker unloadSubset released a demand it did not hold". The subscriber-side alternative (judge the thrown demand against a failed replay) fails like the parent code and was dropped. Expectation changes (maintainer decision): the subscription replay oracle's `new request throws` case and five live-query cases expected a thrown new demand to roll back; they now expect the rejection outcome. Model corrections: a thrown demand's retirement is invisible at the adapter boundary, so the retention law skips the marker barrier from a throw to the end of its episode, a barrier raised later in an ambiguous episode inherits that (which also corrects the existing release-throw case), and a failure after the throw's command may end the query. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it. Open: a retention counterexample with no throw, found at five times the run count and failing on the parent code too, is pinned with it.fails and recorded in the coverage map. Size: @tanstack/db +498 B gzip vs main. Co-authored-by: Isaac --- .../keep-live-query-through-failed-replay.md | 2 + docs/contributing/oracle-coverage.md | 2 +- packages/db/src/collection/subscription.ts | 30 +- packages/db/src/query/live/ARCHITECTURE.md | 5 + ...ubscription-replay-oracle.property.test.ts | 5 +- .../tests/query/live-query-collection.test.ts | 23 +- ...ad-subset-replay-refinement-oracle.test.ts | 392 ++++++++++++++---- 7 files changed, 339 insertions(+), 120 deletions(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index e641a4fc87..b764df9233 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -6,6 +6,8 @@ Keep a live query running when a lazy join or include load fails as part of a so A query that joins and includes the same source has two subscriptions to it. Their truncate replays now share one barrier. A lazy demand that fails while either replay still loads, or after either replay failed, no longer ends the query. +A source whose `loadSubset` throws instead of returning a rejected promise now fails a truncate replay the same way. Before, a throw during a replay could end the query. + An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window reads the new end's tie group. It no longer trusts a row that an older tie request installed. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 81a25c54a6..125019dc0d 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. Open: the failed-replay random campaign at five times its run count found a retention counterexample with no throw (a join and an include of one marker source, two failed replay loads around a row move, then a marker change publishes an empty window); it fails the same way on the parent code and is pinned by an `it.fails` witness until it is fixed or the model is corrected. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 3d58ba588c..168fd3bf3e 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -1114,21 +1114,25 @@ export class CollectionSubscription replayState.currentAttempt === replayAttempt)), ) } catch (error) { + cancelAcquisition(acquisition) const demandIndex = this.subsetDemands.indexOf(demand) - if (demandIndex !== -1) { - if ( - replayState && - replayAttempt && - this.truncateReplayState === replayState && - replayState.currentAttempt === replayAttempt - ) { - replayState.failures.set(demand, normalizeError(error)) - replayState.failed = true - } - this.subsetDemands.splice(demandIndex, 1) + // A load that throws inside an open replay fails that replay as a + // rejected load does. The demand stays, so the replay keeps + // publication until a later replay retries it, and its owner still + // releases it; the adapter acquired nothing to unload. + if ( + demandIndex !== -1 && + replayState && + replayAttempt && + this.truncateReplayState === replayState && + replayState.currentAttempt === replayAttempt + ) { + acquisition.releaseAttempted = true + result = Promise.reject(normalizeError(error)) + } else { + if (demandIndex !== -1) this.subsetDemands.splice(demandIndex, 1) + throw error } - cancelAcquisition(acquisition) - throw error } if (!this.isSyncRunGenerationCurrent(syncRunGeneration)) { diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 82bb1586ca..df17a52b01 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1473,6 +1473,11 @@ A failed replay of any of them fails the barrier: until a later replay of the source succeeds, a lazy demand of any of them that fails keeps the last publication and does not end the query, even when its own subscription's replay finished. +A load that throws while it starts inside an open replay fails that replay as +a rejected load does. Its demand stays with the subscription, so the replay +keeps the last publication, a later replay retries the demand, and its owner +still releases it; the adapter acquired nothing, so it receives no +`unloadSubset` for that load. A rejection of an aborted lazy demand request is obsolete demand, not a load failure, as for the subscription. When a truncate supersedes the acquisition, its replay decides the demand, even if the replay settles before diff --git a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts index 294b98caf2..25778ddf23 100644 --- a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts +++ b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts @@ -2902,12 +2902,15 @@ describe(`CollectionSubscription replay oracle`, () => { commit() await flushPromises() if (site === `new request throws`) { + // A request that throws inside the open replay fails that replay + // as a rejected load does (maintainer decision), so it does not + // throw to its caller. expect(() => subscription.requestSnapshot({ where: new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]), optimizedOnly: false, }), - ).toThrow(`request failed`) + ).not.toThrow() } subscription.releaseSnapshot(where) await flushPromises() diff --git a/packages/db/tests/query/live-query-collection.test.ts b/packages/db/tests/query/live-query-collection.test.ts index 99f211ceb6..521e15c55d 100644 --- a/packages/db/tests/query/live-query-collection.test.ts +++ b/packages/db/tests/query/live-query-collection.test.ts @@ -2656,8 +2656,10 @@ describe(`createLiveQueryCollection`, () => { return { loadSubset: (options) => { if (!recovering) return true - // Choose by request shape, not callback order: a startup - // throw rolls back new demand but retains a replayed owner. + // Choose by request shape, not callback order. A startup + // throw inside the replay fails it as a rejection does, for + // new demand and a replayed owner alike (maintainer + // decision; ARCHITECTURE.md, replay participants). const target = demand === `retained` ? options.limit !== undefined @@ -2700,17 +2702,12 @@ describe(`createLiveQueryCollection`, () => { () => ({ status: `fulfilled` as const }), (error: unknown) => ({ status: `rejected` as const, error }), ) - if (demand === `new` && delivery === `throw`) { - expect(settlement.status).toBe(`fulfilled`) - expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) - } else { - expect(settlement.status).toBe(`rejected`) - if (settlement.status !== `rejected`) - throw new Error(`Expected replay rejection`) - expect(settlement.error).toBe(reportedError) - expect(settlement.error).toBeInstanceOf(Error) - expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) - } + expect(settlement.status).toBe(`rejected`) + if (settlement.status !== `rejected`) + throw new Error(`Expected replay rejection`) + expect(settlement.error).toBe(reportedError) + expect(settlement.error).toBeInstanceOf(Error) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) expect(live.utils.lastSubsetError).toBe(reportedError) } finally { replayGate.resolve() diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 6c3f171771..d10bde292c 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1124,7 +1124,12 @@ describe(`loadSubset replay refinement`, () => { * A load that joins an open replay belongs to it as well, which covers a * demand that the graph starts after the commit while the replay still * loads, from any of the query's subscriptions to that source. The model - * judges that only without a peer (see Limits). + * judges that only without a peer (see Limits). A load that throws while + * it starts, as a synchronous adapter does, belongs to a recovery on the + * same terms as one that rejects: a throw inside an open replay fails that + * replay. The throw happens while the subscription registers the demand, + * so it joins a replay only if that replay still has a participant or a + * failure. * A load that fails before the query first becomes ready is fatal too (an * initial load failure), except on a lazy join source, which owns its * demand. So a query may end in `error` only after a demand load outside a @@ -1145,7 +1150,9 @@ describe(`loadSubset replay refinement`, () => { * or newest held load with success or failure (optionally keeping later * loads held), * truncates (optionally arming the next unload once the replay has started), - * and arming the next unload to throw. A history may start while the initial + * arming the next unload to throw, and arming the next marker load to throw + * while it starts. An armed throw that no history load consumed expires + * before the drain, whose loads succeed. A history may start while the initial * preload's loads are held. A marker load whose request was aborted either * resolves without writing or rejects with `AbortError`, as the source * contract asks; an `AbortError` is not a failed load. The driver uses a real on-demand Collection with an index and an @@ -1167,6 +1174,14 @@ describe(`loadSubset replay refinement`, () => { * issued a load, so the retention law and the rule that a replay * participant counts as recovery skip histories with a peer. A release that throws while a replay starts may * fail that replay, so the retention law skips a barrier where one did. + * A load that throws acquires nothing, so the adapter never sees its demand + * retire, and any graph run can retire it. The retention law therefore skips + * the marker barrier from a throw to the end of its episode (a barrier + * raised while an earlier one in the episode is ambiguous inherits that), and + * a failure that joins a barrier that already failed or became ambiguous may + * end the query. A throw that joins a replay that is only loading is judged + * strictly. So a design that dropped a thrown demand and let its replay + * succeed would pass wherever only retention could see it. */ type OrderedRow = { id: number; rank: number } type OrderedCommand = @@ -1182,6 +1197,9 @@ type OrderedCommand = | { type: `window`; offset: 0 | 1; limit: 1 | 2 | 3 } | { type: `mark`; id: number; on: boolean } | { type: `truncateMarkers` } + // The next marker load throws while it starts, as an adapter that fails + // synchronously does, instead of returning a promise that rejects. + | { type: `markerThrows` } // The query's order and window, whether ranks may tie, and whether a second // live query reads the same source. @@ -1262,6 +1280,7 @@ const orderedCommand: fc.Arbitrary = fc.oneof( weight: 1, arbitrary: fc.constant({ type: `truncateMarkers` as const }), }, + { weight: 1, arbitrary: fc.constant({ type: `markerThrows` as const }) }, { weight: 2, arbitrary: fc.record({ @@ -1309,6 +1328,8 @@ type OrderedReach = { includeDemandFailures: number siblingDemandFailures: number failedSiblingDemandFailures: number + // Lazy demand whose load threw while it started, inside its replay. + thrownReplayDemands: number preloadReplayFailures: number discardedReplayMoves: number } @@ -1436,7 +1457,9 @@ async function checkOrderedReplay( } barriers.set(source, { failures: new Map(), - ambiguous: false, + // The retained publication dates from the first barrier, so an + // ambiguous barrier earlier in this episode leaves it ambiguous. + ambiguous: [...barriers.values()].some(({ ambiguous }) => ambiguous), pending: new Map(barriers.get(source)?.pending), own: new Set(), }) @@ -1523,12 +1546,21 @@ async function checkOrderedReplay( let markerCommitting = false let markerTruncates = 0 let demandFailed = false + let markerThrowArmed = false + let loadThrew = false + // The adapter sets the flag, so read it through a call. + const threw = () => loadThrew // A failed initial load is fatal too (error 229). The queries are created // below; a load can only fail after they start. const queries: Array<{ status: string }> = [] let failedWhileLoading = false // Adapter callbacks set these flags, so read them through a call. - const fatalLoadFailed = () => demandFailed || failedWhileLoading + // A failure that joins a barrier that was already ambiguous, after a load + // threw, may fall outside the replay the model still holds open, so it may + // end the query (see Limits). + let ambiguousFailure = false + const fatalLoadFailed = () => + demandFailed || failedWhileLoading || ambiguousFailure // Lazy join demand asks for keys; ordered and full-source loads do not. // An ordered tie request also has a `where` and no `orderBy`, but it asks // for rows by rank, so it is not demand. @@ -1616,6 +1648,7 @@ async function checkOrderedReplay( includeDemandFailures: 0, siblingDemandFailures: 0, failedSiblingDemandFailures: 0, + thrownReplayDemands: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -1647,99 +1680,150 @@ async function checkOrderedReplay( operations.markReady() return { // Marker loads share the history's held and failing loads, so - // joined work can stay pending or fail. - loadSubset: async (options) => { - acquired.set(options, (acquired.get(options) ?? 0) + 1) - const load: Load = { epoch: truncates, outcome: `pending` } - loads.push(load) - reach.markerLoads++ - const participant = joinBarrier(`markers`, load, options) - const recovery = - belongsToRecovery(options, markerTruncates, markerCommitting) || - (participant && oneSubscription(`markers`)) - const demand = !recovery && isDemand(options) - if (holding) { - const ok = await new Promise((resolve) => - held.push(resolve), - ) - if (!ok) { - // The source's replay had already failed and still holds - // publication when this load fails. - const barrierFailed = - (barriers.get(`markers`)?.failures.size ?? 0) > 0 - const replayLoading = settleParticipant( - `markers`, - load, - options, - false, + // joined work can stay pending or fail. An armed load throws while + // it starts: it joins its source's replay like any load, fails at + // once, and is judged by the same recovery rule as a rejection. + loadSubset: (options) => { + if (markerThrowArmed) { + markerThrowArmed = false + loadThrew = true + const load: Load = { epoch: truncates, outcome: `failed` } + loads.push(load) + reach.markerLoads++ + // The throw happens while the subscription registers the + // demand, so the barrier here is the replay's state then. One + // whose participants all retired or settled has completed, and + // the demand does not join it. + const open = barriers.get(`markers`) + const participant = + open !== undefined && + (open.pending.size > 0 || open.failures.size > 0) && + joinBarrier(`markers`, load, options) + // After a failure, the model cannot see whether the failed + // demands retired, so a throw that joins a failed or ambiguous + // barrier may fall outside the replay (see Limits). A throw that + // joins a replay that is only loading is judged strictly. + const strict = + participant && !open.ambiguous && open.failures.size === 0 + if (participant && !strict) ambiguousFailure = true + const recovery = + belongsToRecovery(options, markerTruncates, markerCommitting) || + (participant && oneSubscription(`markers`)) + settleParticipant(`markers`, load, options, false) + // A load that threw acquired nothing, so the adapter never sees + // its demand retire, and any graph run can retire it. The + // marker barrier's retention is ambiguous from here to the end + // of its episode (see Limits). + if (open) open.ambiguous = true + anyLoadFailed = true + if (!recovery && isDemand(options)) demandFailed = true + // A throw inside a replay that is only loading, which the + // recovery rule judges strictly. + if (recovery && strict && isDemand(options)) { + reach.thrownReplayDemands++ + } + if (queries.some(({ status }) => status === `loading`)) { + failedWhileLoading = true + } + reach.failedLoads++ + throw inject(new Error(`marker load threw`)) + } + return (async () => { + acquired.set(options, (acquired.get(options) ?? 0) + 1) + const load: Load = { epoch: truncates, outcome: `pending` } + loads.push(load) + reach.markerLoads++ + const participant = joinBarrier(`markers`, load, options) + const recovery = + belongsToRecovery(options, markerTruncates, markerCommitting) || + (participant && oneSubscription(`markers`)) + const demand = !recovery && isDemand(options) + if (holding) { + const ok = await new Promise((resolve) => + held.push(resolve), ) - load.outcome = `failed` - anyLoadFailed = true - if (demand) demandFailed = true - // A demand the graph started after the commit fails inside - // the replay. - if ( - participant && - !options.signal?.aborted && - oneSubscription(`markers`) && - !markerCommitting && - isDemand(options) - ) { - // While other participants still load. - if (replayLoading) { - reach.participantDemandFailures++ - if (scenario.includes) reach.includeDemandFailures++ + if (!ok) { + // The source's replay had already failed and still holds + // publication when this load fails. + const barrierFailed = + (barriers.get(`markers`)?.failures.size ?? 0) > 0 + if (participant && barriers.get(`markers`)?.ambiguous) { + ambiguousFailure = true } - // The query both joins and includes the marker source, so - // two of its subscriptions share the barrier. - if (scenario.includes && scenario.joined) { - reach.siblingDemandFailures++ - // After the shared replay failed: the subscription whose - // demand fails may have finished its own replay. - if (barrierFailed) reach.failedSiblingDemandFailures++ + const replayLoading = settleParticipant( + `markers`, + load, + options, + false, + ) + load.outcome = `failed` + anyLoadFailed = true + if (demand) demandFailed = true + // A demand the graph started after the commit fails inside + // the replay. + if ( + participant && + !options.signal?.aborted && + oneSubscription(`markers`) && + !markerCommitting && + isDemand(options) + ) { + // While other participants still load. + if (replayLoading) { + reach.participantDemandFailures++ + if (scenario.includes) reach.includeDemandFailures++ + } + // The query both joins and includes the marker source, so + // two of its subscriptions share the barrier. + if (scenario.includes && scenario.joined) { + reach.siblingDemandFailures++ + // After the shared replay failed: the subscription whose + // demand fails may have finished its own replay. + if (barrierFailed) reach.failedSiblingDemandFailures++ + } } - } - if (queries.some(({ status }) => status === `loading`)) { - failedWhileLoading = true - // Lazy demand in a replay failed before the first ready. - if (participant && isDemand(options)) { - reach.preloadReplayFailures++ + if (queries.some(({ status }) => status === `loading`)) { + failedWhileLoading = true + // Lazy demand in a replay failed before the first ready. + if (participant && isDemand(options)) { + reach.preloadReplayFailures++ + } } + reach.failedLoads++ + throw inject(new Error(`marker load failed`)) } - reach.failedLoads++ - throw inject(new Error(`marker load failed`)) + } else await Promise.resolve() + if (options.signal?.aborted) { + load.outcome = `ok` + settleParticipant(`markers`, load, options, true) + if (scenario.abortedMarkerLoads === `resolve`) return + reach.abortRejections++ + throw inject( + Object.assign(new Error(`marker load aborted`), { + name: `AbortError`, + }), + ) } - } else await Promise.resolve() - if (options.signal?.aborted) { + operations.begin() + for (const id of marked) { + const value = { id, rowId: id } + if ( + options.where && + evaluateReferenceExpression(options.where, value) !== true + ) { + continue + } + operations.write({ + type: markers.has(id) ? `update` : `insert`, + value, + }) + installedMarkers.add(id) + } + await operations.commit() load.outcome = `ok` + // The load settles once its rows apply. settleParticipant(`markers`, load, options, true) - if (scenario.abortedMarkerLoads === `resolve`) return - reach.abortRejections++ - throw inject( - Object.assign(new Error(`marker load aborted`), { - name: `AbortError`, - }), - ) - } - operations.begin() - for (const id of marked) { - const value = { id, rowId: id } - if ( - options.where && - evaluateReferenceExpression(options.where, value) !== true - ) { - continue - } - operations.write({ - type: markers.has(id) ? `update` : `insert`, - value, - }) - installedMarkers.add(id) - } - await operations.commit() - load.outcome = `ok` - // The load settles once its rows apply. - settleParticipant(`markers`, load, options, true) + })() }, unloadSubset: (options) => { retireParticipants(`markers`, options) @@ -2038,6 +2122,10 @@ async function checkOrderedReplay( try { if (!scenario.historyDuringPreload) await preload for (const command of history) { + // A later command can retire a thrown demand, which the adapter never + // sees, and so retire its failed replay; demand that waited on that + // replay may then fail outside it (see Limits). + if (threw()) ambiguousFailure = true try { switch (command.type) { case `move`: { @@ -2103,6 +2191,9 @@ async function checkOrderedReplay( } // The joined source truncates. An eager source redelivers every // marker; an on-demand one replays its loaded demands. + case `markerThrows`: + markerThrowArmed = true + break case `truncateMarkers`: { if (!readsMarkers) break reach.markerTruncates++ @@ -2160,6 +2251,9 @@ async function checkOrderedReplay( } holding = false throwNextUnload = false + // The drain is the checkpoint, where recovery loads succeed, so an + // armed throw that no history load consumed expires with the history. + markerThrowArmed = false await settleAll() await Promise.all(pendingMoves.splice(0)) await settleAll() @@ -2602,6 +2696,41 @@ describe(`Ordered windows across truncate replays`, () => { reaches: `failedSiblingDemandFailures`, recovers: true, }, + { + // A marker truncate replays the include's demand, and its load throws + // while it starts. The throw fails the replay as a rejection does, so + // it keeps the last publication; it used to end the query. + name: `a replay load that throws while it starts`, + history: [{ type: `markerThrows` }, { type: `truncateMarkers` }], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `none`, + markerMode: `on-demand`, + includes: true, + }, + recovers: true, + }, + { + // A row move starts join demand inside an open marker replay, and its + // load throws while it starts. The subscription dropped that demand, + // so its failure did not fail the replay, and the query ended. + name: `a demand whose load throws inside its open replay`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `markerThrows` }, + { type: `move`, id: 3, rank: 1 }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `none`, + markerMode: `on-demand`, + }, + reaches: `thrownReplayDemands`, + recovers: true, + }, ] satisfies Array<{ name: string history: Array @@ -2624,6 +2753,39 @@ describe(`Ordered windows across truncate replays`, () => { }, ) + // Open: found by the failed-replay random campaign at five times its run + // count, on this branch's code and its parent alike. A join and an include + // share the marker source; the replay's newest load fails, a row move + // starts demand whose load fails too, and a marker change then publishes + // an empty window while the model still retains the last publication. The + // witness pins the gap until it is fixed or the model is corrected. + it.fails( + `keeps the last publication when a marker change follows two failed replay loads`, + async () => { + await checkOrderedReplay( + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + { type: `move`, id: 1, rank: 1 }, + { type: `settle`, ok: false, newest: false }, + { type: `hold` }, + { type: `mark`, id: 4, on: false }, + ], + { + ...ordinaryOrderedScenario, + direction: `desc`, + offset: 1, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + ) + }, + ) + it.each(orderedReplayCampaigns)( `keeps the ordered window under load and release faults (%s campaign)`, async (campaign) => { @@ -2656,6 +2818,7 @@ describe(`Ordered windows across truncate replays`, () => { includeDemandFailures: 0, siblingDemandFailures: 0, failedSiblingDemandFailures: 0, + thrownReplayDemands: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -2692,6 +2855,7 @@ describe(`Ordered windows across truncate replays`, () => { `includeDemandFailures`, `siblingDemandFailures`, `failedSiblingDemandFailures`, + `thrownReplayDemands`, `preloadReplayFailures`, `discardedReplayMoves`, ]) @@ -2836,13 +3000,53 @@ describe(`Ordered windows across truncate replays`, () => { { ...scenario, joined, includes: true, markerMode: `on-demand` }, ], ) + // A marker replay is open when a change starts marker demand whose load + // throws while it starts. The throw fails the replay, like a rejection, + // so it must not end the query. + const thrownDemandCase = fc + .tuple( + failedReplayScenario, + fc.boolean(), + fc.oneof( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + fc.record({ + type: fc.constant(`mark` as const), + id: fc.integer({ min: 1, max: 5 }), + on: fc.boolean(), + }), + ), + fc.array(orderedCommand, { maxLength: 6 }), + ) + .map( + ([scenario, settleFirst, change, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + ...(settleFirst + ? [{ type: `settle` as const, ok: true, keepHolding: true }] + : []), + { type: `markerThrows` }, + change, + ...rest, + ], + { ...scenario, markerMode: `on-demand` }, + ], + ) const failedReplayCase = fc.oneof( { weight: 3, arbitrary: fc.tuple(failedReplayHistory, failedReplayScenario), }, { weight: 1, arbitrary: includeDemandCase }, - { weight: 1, arbitrary: siblingDemandCase }, + { weight: 2, arbitrary: siblingDemandCase }, + { weight: 2, arbitrary: thrownDemandCase }, ) const failedReplayProperty = `replay.ordered-failed-replay` const failedReplayCampaigns = @@ -3012,6 +3216,7 @@ describe(`Ordered windows across truncate replays`, () => { `includeDemandFailures`, `siblingDemandFailures`, `failedSiblingDemandFailures`, + `thrownReplayDemands`, ] as const, { minimum: 5 }, ) @@ -3037,10 +3242,13 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.failedSiblingDemandFailures > 0) { reach.reach(`failedSiblingDemandFailures`) } + if (observed.thrownReplayDemands > 0) { + reach.reach(`thrownReplayDemands`) + } }), campaign === `fixed` - ? { seed: 20261023, numRuns: oracleRuns(400) } - : oraclePropertyOptions(400, failedReplayProperty), + ? { seed: 20261023, numRuns: oracleRuns(600) } + : oraclePropertyOptions(600, failedReplayProperty), ) reach.assertReached() }, From e8ec44ff8574d2c1bc750dbc86fdbc1c8c076221 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 11:45:01 -0600 Subject: [PATCH 25/40] fix(db): hold publication for a demand that fails behind a sibling's failed replay A query that joins and includes one source has two subscriptions to it. When one subscription's replay failed and the other's finished, a demand on the finished subscription that then failed was kept from ending the query, but it held nothing. A later change that retired the first replay's failed demand discarded that replay and published the window without the second demand's rows. Law (ARCHITECTURE.md, "Replay participants and failure"): until a later replay of the source succeeds, a lazy demand of any of the query's subscriptions to it that fails keeps the last publication. Such a failure now fails its own subscription's replacement too; it holds until a later replay succeeds or that subscription's last demand retires. Reach: `partialFailureRetirements` (a failed demand retires while another failure still holds the barrier) reaches 17 to 27 per failed-replay campaign. RED (parent code, also before this branch): the failed-replay random campaign at five times its run count found the trace; its named replay fails with "retained publication after mark: expected [] to deeply equal [ { id: 4, rank: 40, ... } ]". The campaigns do not fail without the hold, so the named replay is the witness. GREEN: load-subset replay oracle 41/41 three times; @tanstack/db 285 files, 13,839 tests. Mutant: the subscriber without the hold is the RED above. Also keeps the trace from #2221 as a named replay in the ordered-work oracle: the loader before prefix growth reads the full source twice after a signalling refetch past a NaN boundary ("expected 2 to be 1"); with growth it reads none. Size: @tanstack/db +570 B gzip vs main (+72 B for this fix). Fixes #2221 Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/collection/subscription.ts | 22 ++++++ packages/db/src/query/live/ARCHITECTURE.md | 5 +- .../src/query/live/collection-subscriber.ts | 38 +++++++--- ...ad-subset-replay-refinement-oracle.test.ts | 76 +++++++++++-------- .../ordered-work-oracle.property.test.ts | 32 ++++++++ 6 files changed, 132 insertions(+), 43 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 125019dc0d..adaad7c49b 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. Open: the failed-replay random campaign at five times its run count found a retention counterexample with no throw (a join and an include of one marker source, two failed replay loads around a row move, then a marker change publishes an empty window); it fails the same way on the parent code and is pinned by an `it.fails` witness until it is fixed or the model is corrected. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 168fd3bf3e..cb061977a4 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -783,6 +783,28 @@ export class CollectionSubscription this.stalePublishedRows.clear() } + /** + * A lazy demand that failed outside this subscription's replay, while a + * failed replay of another of the graph's subscriptions to this collection + * holds their shared barrier, fails this subscription's replacement too. + * It keeps the last publication until a later replay succeeds, or until + * this subscription's last demand retires. + */ + failReplacement(error: unknown): void { + if (!this.options.truncateReplayPublication || this.truncateReplayState) { + return + } + const replayState = this.createTruncateReplayState( + this.collection._sync.getSyncRunGeneration(), + () => new Map(), + ) + replayState.pendingSetups = 0 + replayState.failed = true + replayState.completion.reject(normalizeError(error)) + this.truncateReplayState = replayState + this.truncateReplacementPending = true + } + /** Drop the replay without publishing; an unfinished wait rejects as aborted. */ private discardTruncateReplay(): void { const replayState = this.truncateReplayState diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index df17a52b01..bfdf757180 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1472,7 +1472,10 @@ meanwhile joins its own open replay instead of falling outside recovery. A failed replay of any of them fails the barrier: until a later replay of the source succeeds, a lazy demand of any of them that fails keeps the last publication and does not end the query, even when its own subscription's -replay finished. +replay finished. Such a failure fails its own subscription's replacement too, +so retiring the other replay's failed demand cannot publish without this +demand's rows; it holds until a later replay of the source succeeds or the +subscription's last demand retires. A load that throws while it starts inside an open replay fails that replay as a rejected load does. Its demand stays with the subscription, so the replay keeps the last publication, a later replay retries the demand, and its owner diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index de81ffb847..9255283b22 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -203,8 +203,12 @@ export class CollectionSubscriber< if (!Object.is(subscription.lastError, error)) throw error const isInitialSync = this.collectionConfigBuilder.liveQueryCollection?.status === `loading` - const generation = this.collectionConfigBuilder.beginDemand(plan.id) - this.collectionConfigBuilder.failDemand(plan.id, generation, error) + this.failDemand( + subscription, + plan.id, + this.collectionConfigBuilder.beginDemand(plan.id), + error, + ) if (isInitialSync) throw error return } @@ -232,13 +236,7 @@ export class CollectionSubscriber< // query. const replacement = subscription.pendingTruncateReplacement if (replacement) settleWith(replacement) - else - this.collectionConfigBuilder.failDemand( - plan.id, - generation, - error, - !this.collectionConfigBuilder.hasFailedReplay(this.collection), - ) + else this.failDemand(subscription, plan.id, generation, error) }, ) settleWith(update.ready) @@ -247,6 +245,28 @@ export class CollectionSubscriber< } } + // A failure while a failed replay of this source holds the shared barrier + // belongs to that recovery: it does not end the query, and it holds this + // subscription's publication too, so retiring the other replay's failed + // demand cannot publish without this demand's rows. + private failDemand( + subscription: CollectionSubscription, + planId: string, + generation: number, + error: unknown, + ): void { + const recovering = this.collectionConfigBuilder.hasFailedReplay( + this.collection, + ) + if (recovering) subscription.failReplacement(error) + this.collectionConfigBuilder.failDemand( + planId, + generation, + error, + !recovering, + ) + } + private sendChangesToPipeline( changes: Iterable>, ): void { diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index d10bde292c..54fb7fc0fd 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1330,6 +1330,8 @@ type OrderedReach = { failedSiblingDemandFailures: number // Lazy demand whose load threw while it started, inside its replay. thrownReplayDemands: number + // A failed demand retires while another failure still holds its barrier. + partialFailureRetirements: number preloadReplayFailures: number discardedReplayMoves: number } @@ -1520,11 +1522,16 @@ async function checkOrderedReplay( ) => { const barrier = barriers.get(source) if (!barrier) return + const failures = barrier.failures.size for (const owned of [barrier.pending, barrier.failures]) { for (const [load, held] of owned) { if (held === options) owned.delete(load) } } + // One failed demand retires while another still holds the barrier. + if (barrier.failures.size > 0 && barrier.failures.size < failures) { + reach.partialFailureRetirements++ + } } // A query's subscriptions to one source share one replay barrier, so a // load belongs to it whichever subscription issued it. A peer is another @@ -1649,6 +1656,7 @@ async function checkOrderedReplay( siblingDemandFailures: 0, failedSiblingDemandFailures: 0, thrownReplayDemands: 0, + partialFailureRetirements: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -2753,38 +2761,36 @@ describe(`Ordered windows across truncate replays`, () => { }, ) - // Open: found by the failed-replay random campaign at five times its run - // count, on this branch's code and its parent alike. A join and an include - // share the marker source; the replay's newest load fails, a row move - // starts demand whose load fails too, and a marker change then publishes - // an empty window while the model still retains the last publication. The - // witness pins the gap until it is fixed or the model is corrected. - it.fails( - `keeps the last publication when a marker change follows two failed replay loads`, - async () => { - await checkOrderedReplay( - [ - { type: `hold` }, - { type: `truncateMarkers` }, - { type: `settle`, ok: true, keepHolding: true, newest: false }, - { type: `settle`, ok: false, newest: true, keepHolding: true }, - { type: `move`, id: 1, rank: 1 }, - { type: `settle`, ok: false, newest: false }, - { type: `hold` }, - { type: `mark`, id: 4, on: false }, - ], - { - ...ordinaryOrderedScenario, - direction: `desc`, - offset: 1, - limit: 1, - joined: `some`, - markerMode: `on-demand`, - includes: true, - }, - ) - }, - ) + // Found by the failed-replay random campaign at five times its run count. + // A join and an include share the marker source. One subscription's replay + // fails; the other's finishes, and a row move starts demand on it whose + // load fails too, which the shared barrier keeps from ending the query. A + // marker change then retires the first replay's failed demand, which + // discarded that replay and published an empty window while the second + // failure held nothing; that failure now holds its own subscription. + it(`keeps the last publication when a marker change follows two failed replay loads`, async () => { + await checkOrderedReplay( + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + { type: `move`, id: 1, rank: 1 }, + { type: `settle`, ok: false, newest: false }, + { type: `hold` }, + { type: `mark`, id: 4, on: false }, + ], + { + ...ordinaryOrderedScenario, + direction: `desc`, + offset: 1, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + ) + }) it.each(orderedReplayCampaigns)( `keeps the ordered window under load and release faults (%s campaign)`, @@ -2819,6 +2825,7 @@ describe(`Ordered windows across truncate replays`, () => { siblingDemandFailures: 0, failedSiblingDemandFailures: 0, thrownReplayDemands: 0, + partialFailureRetirements: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, } @@ -2856,6 +2863,7 @@ describe(`Ordered windows across truncate replays`, () => { `siblingDemandFailures`, `failedSiblingDemandFailures`, `thrownReplayDemands`, + `partialFailureRetirements`, `preloadReplayFailures`, `discardedReplayMoves`, ]) @@ -3217,6 +3225,7 @@ describe(`Ordered windows across truncate replays`, () => { `siblingDemandFailures`, `failedSiblingDemandFailures`, `thrownReplayDemands`, + `partialFailureRetirements`, ] as const, { minimum: 5 }, ) @@ -3245,6 +3254,9 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.thrownReplayDemands > 0) { reach.reach(`thrownReplayDemands`) } + if (observed.partialFailureRetirements > 0) { + reach.reach(`partialFailureRetirements`) + } }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(600) } diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 967c9af1c2..58f338a8e8 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -7425,6 +7425,38 @@ describe(`eager indexed ordered windows`, () => { }) }) + // The trace from #2221. A descending, nulls-first, unindexed window behind + // an anti-joined marker has a NaN rank, so its boundary has no cursor. + // After a signalling refetch the loader read the full source twice: the + // short prefix took the full-source fallback. The prefix now grows, so the + // refetch law, which allows no full-source read here, holds. + it(`reads no full source after a signalling refetch past a NaN boundary`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `anti-joined`, + via: `marker`, + indexed: false, + direction: `desc`, + nulls: `first`, + terms: 1, + limit: 2, + offset: 0, + initial: [ + { id: 0, rank: 0, eligible: true }, + { id: 2, rank: Number.NaN, eligible: true }, + { id: 1, rank: 0, eligible: false }, + ], + history: [ + { type: `delete-visible` }, + { type: `move`, offset: 0, limit: 1 }, + ], + }) + }) + // Found by the unindexed-refetch random campaign. A refetch report moves // the page's end onto a null rank, which no cursor can express. Reading // that end's tie would take the boundary's full-source fallback, so the From 5b4c304d443c7af9e8ad2906b8125f548a7ec6db Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 11:55:49 -0600 Subject: [PATCH 26/40] test(db): keep the two traces reported on #2221 as named replays Both traces failed on main and pass on this branch, each fixed by an earlier commit here rather than by a shared marker-filter cause. - Trace 1 (ordered-work oracle): an ascending unindexed window behind an anti-joined marker read the full source twice after a signalling refetch. The loader before prefix growth (944a0c0b8) fails it with "full-source requests after a signalling refetch at step 1: expected 2 to be +0"; prefix growth (fe48cbe54) reads none. - Trace 2 (load-subset replay oracle): a one-row move deferred behind a marker replay that a source truncate discards was dropped, so the window kept three rows. main's production fails it with "settled window: expected [ 10, 20, 30 ] to deeply equal [ 10 ]"; bisection places the fix at ce1299ded, which applies a deferred move after the newer replay. Co-authored-by: Isaac --- ...ad-subset-replay-refinement-oracle.test.ts | 21 +++++++++++++++ .../ordered-work-oracle.property.test.ts | 27 +++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 54fb7fc0fd..9c34d8be52 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -2704,6 +2704,27 @@ describe(`Ordered windows across truncate replays`, () => { reaches: `failedSiblingDemandFailures`, recovers: true, }, + { + // Trace 2 from a comment on #2221. The window grows to three rows while + // loads are held, shrinks to one behind a marker replay, and a source + // truncate then discards that replay. The deferred one-row move was + // dropped with it, so the settled window kept three rows. + name: `a window shrink deferred behind a replay that a truncate discards`, + history: [ + { type: `hold` }, + { type: `window`, offset: 0, limit: 3 }, + { type: `truncateMarkers` }, + { type: `window`, offset: 0, limit: 1 }, + { type: `truncate`, armAfterStart: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: false, + markerMode: `on-demand`, + includes: true, + }, + }, { // A marker truncate replays the include's demand, and its load throws // while it starts. The throw fails the replay as a rejection does, so diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 58f338a8e8..293e8249f9 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -7457,6 +7457,33 @@ describe(`eager indexed ordered windows`, () => { }) }) + // Trace 1 from a comment on #2221: an ascending unindexed window behind an + // anti-joined marker whose only row is ineligible. Before prefix growth the + // short prefix took the full-source fallback, twice, after a signalling + // refetch; the refetch law allows none. + it(`reads no full source after a signalling refetch of a short anti-joined window`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `anti-joined`, + via: `marker`, + indexed: false, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 1, + offset: 0, + initial: [], + history: [ + { type: `upsert`, id: 0, rank: 0, eligible: false }, + { type: `refetch` }, + ], + }) + }) + // Found by the unindexed-refetch random campaign. A refetch report moves // the page's end onto a null rank, which no cursor can express. Reading // that end's tie would take the boundary's full-source fallback, so the From 766d7f6354034b6b274b5082450b2a74786a50bc Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 12:06:48 -0600 Subject: [PATCH 27/40] fix(db): keep a window move from settling on behalf of itself A window move whose load throws while it starts bumps the request generation before it replaces the latest request. An older move that still waited then saw itself as superseded and returned the latest request, which was its own promise: a chaining cycle, which escaped as a TypeError. Law (the oracle's newest-move rule; ARCHITECTURE.md, window moves): a superseded move resolves after the newest move settled. A newer move that threw while it started has settled, failed, so the older move resolves on its own. Oracle: a move whose setWindow throws counts as a settled, failed move in the model, as a rejected move does; it stayed pending before, so the newest-move law misjudged the older move. Reach: `thrownMoves`. RED (parent code): the broad random campaign at five times its run count found the trace; its named replay fails with "escaped error: only injected faults may surface", caused by "TypeError: Chaining cycle detected for promise". It fails the same way before this branch's synchronous-throw and hold changes. GREEN: load-subset replay oracle 43/43; @tanstack/db 285 files, 13,842 tests. Mutant: the settlement without the self guard is the RED above. Limits: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming. The grammar flushes between commands, so it never reaches that microtask gap; a guard there had no witness and was left out. Recorded in the coverage map. Size: @tanstack/db +588 B gzip vs main (+18 B). Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- .../query/live/collection-config-builder.ts | 12 ++++-- ...ad-subset-replay-refinement-oracle.test.ts | 40 ++++++++++++++++++- 3 files changed, 48 insertions(+), 6 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index adaad7c49b..79870ff9b1 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. The settlement now never returns itself; a named replay fails without that. Open: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming; that microtask gap is outside the grammar, which flushes between commands, so the deferred path is unchanged. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index e236216479..ff4e6487ea 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -431,14 +431,17 @@ export class CollectionConfigBuilder< this.latestWindowRequest = undefined return true } - return (this.latestWindowRequest = settlement.then( + const settled: Promise = settlement.then( () => { if (windowOperationGeneration === this.windowOperationGeneration) { this.settledWindow = requestedWindow } - return request === this.windowRequestGeneration + // A newer call wins; one that threw while it started left this one + // latest, and a promise cannot settle on behalf of itself. + const latest = this.latestWindowRequest + return request === this.windowRequestGeneration || latest === settled ? undefined - : this.latestWindowRequest + : latest }, (error) => { if (windowOperationGeneration === this.windowOperationGeneration) { @@ -447,7 +450,8 @@ export class CollectionConfigBuilder< } throw error }, - )) + ) + return (this.latestWindowRequest = settled) } getWindow(): { offset: number; limit: number } | undefined { diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 9c34d8be52..87c3451f35 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1330,6 +1330,8 @@ type OrderedReach = { failedSiblingDemandFailures: number // Lazy demand whose load threw while it started, inside its replay. thrownReplayDemands: number + // Window moves whose load threw while the move started. + thrownMoves: number // A failed demand retires while another failure still holds its barrier. partialFailureRetirements: number preloadReplayFailures: number @@ -1656,6 +1658,7 @@ async function checkOrderedReplay( siblingDemandFailures: 0, failedSiblingDemandFailures: 0, thrownReplayDemands: 0, + thrownMoves: 0, partialFailureRetirements: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, @@ -2217,7 +2220,17 @@ async function checkOrderedReplay( const requested = { offset: command.offset, limit: command.limit } reach.windowMoves++ const move = ++latestMove - const settled = live.utils.setWindow(requested) + let settled: true | Promise + try { + settled = live.utils.setWindow(requested) + } catch (error) { + // A move whose load throws while it starts has settled, failed, + // as a rejected one has; the window stays where it was. + settledMoves.add(move) + failedMoves.add(move) + reach.thrownMoves++ + throw error + } if (settled === true) { settleMove(move) window = requested @@ -2704,6 +2717,29 @@ describe(`Ordered windows across truncate replays`, () => { reaches: `failedSiblingDemandFailures`, recovers: true, }, + { + // Found by the broad random campaign at five times its run count. A + // window move's marker load throws while the move starts, after an + // older move that still waits. The older move's settlement returned + // the latest request, which was still itself: a chaining cycle. + name: `a window move whose load throws while an older move waits`, + history: [ + { type: `window`, offset: 0, limit: 1 }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `markerThrows` }, + { type: `window`, offset: 0, limit: 2 }, + { type: `mark`, id: 1, on: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 3, + joined: false, + markerMode: `on-demand`, + includes: true, + historyDuringPreload: true, + }, + reaches: `thrownMoves`, + }, { // Trace 2 from a comment on #2221. The window grows to three rows while // loads are held, shrinks to one behind a marker replay, and a source @@ -2846,6 +2882,7 @@ describe(`Ordered windows across truncate replays`, () => { siblingDemandFailures: 0, failedSiblingDemandFailures: 0, thrownReplayDemands: 0, + thrownMoves: 0, partialFailureRetirements: 0, preloadReplayFailures: 0, discardedReplayMoves: 0, @@ -2885,6 +2922,7 @@ describe(`Ordered windows across truncate replays`, () => { `failedSiblingDemandFailures`, `thrownReplayDemands`, `partialFailureRetirements`, + `thrownMoves`, `preloadReplayFailures`, `discardedReplayMoves`, ]) From d6453b55c8152ea3fc4cf094dc8cd4ae16215ae5 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 14:47:56 -0600 Subject: [PATCH 28/40] refactor(db): derive the live query's error state and share loading-promise release - collection-subscriber: the loading-promise map stores the bare resolve function, and the resolve-then-delete sequence that the status handler and the unsubscribe cleanup each repeated is one helper. - collection-config-builder: `isInErrorState` is a getter over the two flags it always mirrored (a fatal error, or any unrecovered source error), which removes four assignments. No behavior change: setErrorState is reached only through a fatal transition or a source error, recovery clears the source id, and sync start and clear reset both flags together. Evidence: @tanstack/db 285 files, 13,842 tests; ordered, replay, and subscription oracles pass. Size: -12 B gzip. Co-authored-by: Isaac --- .../query/live/collection-config-builder.ts | 11 ++++---- .../src/query/live/collection-subscriber.ts | 25 ++++++++----------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index ff4e6487ea..678535f0b6 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -119,11 +119,15 @@ export class CollectionConfigBuilder< public currentSyncState: FullSyncState | undefined // Error state tracking - private isInErrorState = false private fatalQueryError = false private readonly erroredSourceIds = new Set() private lastSubsetError: unknown | undefined + // A fatal error or any unrecovered source error puts the live query in error. + private get isInErrorState(): boolean { + return this.fatalQueryError || this.erroredSourceIds.size > 0 + } + // Reference to the live query collection for error state transitions public liveQueryCollection?: Collection @@ -834,7 +838,6 @@ export class CollectionConfigBuilder< // Store reference to the live query collection for error state transitions this.liveQueryCollection = config.collection // Reset error state from any previous sync run so a restarted sync can become ready again. - this.isInErrorState = false this.fatalQueryError = false this.erroredSourceIds.clear() this.lastSubsetError = undefined @@ -917,7 +920,6 @@ export class CollectionConfigBuilder< this.loadMoreFn = undefined this.currentWindow = undefined this.settledWindow = this.initialWindow - this.isInErrorState = false this.fatalQueryError = false this.erroredSourceIds.clear() @@ -1244,7 +1246,6 @@ export class CollectionConfigBuilder< !this.fatalQueryError && this.erroredSourceIds.size === 0 ) { - this.isInErrorState = false this.maybeRunGraphFn?.() } if ( @@ -1310,8 +1311,6 @@ export class CollectionConfigBuilder< } private setErrorState(message: string, error?: unknown) { - this.isInErrorState = true - // Log error to console for debugging console.error( devBuild() && process.env.NODE_ENV !== `production` diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index 9255283b22..28b9a9b0b8 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -33,7 +33,7 @@ export class CollectionSubscriber< // Track deferred promises for subscription loading states private subscriptionLoadingPromises = new Map< CollectionSubscription, - { resolve: () => void } + () => void >() // Exact row last contributed to D2 for each source key. @@ -106,11 +106,7 @@ export class CollectionSubscriber< this.ensureLoadingPromise(subscription) } else { // status is 'ready' - const deferred = this.subscriptionLoadingPromises.get(subscription) - if (deferred) { - this.subscriptionLoadingPromises.delete(subscription) - deferred.resolve() - } + this.resolveLoadingPromise(subscription) } } const onLoadSubsetError = (event: SubscriptionLoadSubsetErrorEvent) => { @@ -168,11 +164,7 @@ export class CollectionSubscriber< ): void { const unsubscribe = () => { // If subscription has a pending promise, resolve it before unsubscribing - const deferred = this.subscriptionLoadingPromises.get(subscription) - if (deferred) { - this.subscriptionLoadingPromises.delete(subscription) - deferred.resolve() - } + this.resolveLoadingPromise(subscription) try { this.demand.clear() @@ -502,9 +494,14 @@ export class CollectionSubscriber< resolve = res }) - this.subscriptionLoadingPromises.set(subscription, { - resolve: resolve!, - }) + this.subscriptionLoadingPromises.set(subscription, resolve!) this.collectionConfigBuilder.trackSubsetLoadPromise(promise) } + + private resolveLoadingPromise(subscription: CollectionSubscription): void { + const resolve = this.subscriptionLoadingPromises.get(subscription) + if (!resolve) return + this.subscriptionLoadingPromises.delete(subscription) + resolve() + } } From 0bdf113ffacf70c79ae1276204bcb52b8843e54f Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 14:48:02 -0600 Subject: [PATCH 29/40] refactor(db): repair a report whose page end moved instead of reading its tie When a refetch report moves the end of a loader's own first page before the tie it read, the window is no longer proven. The loader read only the new end's tie; it now starts the bounded repair it already used for every other unproven report. Maintainer decision: the tie-only read saved one page request per such refetch but cost shipped bytes. The correctness half of that earlier fix stays: a report is ignored only when the page read past the window or the loader read the tie of the current end, so a row that ties with a moved end is still never skipped. Oracle: the report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything; the ignored-report, owed-repair, unexpressible-tie, and reset laws still run, and the unproven-report law accepts the repair. The query-db replay of an insert into a top-N list now expects the repair's bounded prefix and the new end's tie, with the Query count unchanged. Evidence: @tanstack/db 285 files, 13,842 tests; query-db 21 files, 925 tests; the query-db refetch random campaign passes at three times its run count. Size: @tanstack/db +449 B gzip vs main (was +576 after the previous commit; -127 B). Co-authored-by: Isaac --- .../keep-live-query-through-failed-replay.md | 2 +- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 13 ++- .../src/query/live/ordered-source-loader.ts | 81 +++---------------- ...ordered-source-loader-state-oracle.test.ts | 75 ++--------------- .../refetch-ordered-prefix-oracle.test.ts | 25 +++--- 6 files changed, 38 insertions(+), 160 deletions(-) diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md index b764df9233..b37ec27db3 100644 --- a/.changeset/keep-live-query-through-failed-replay.md +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -10,7 +10,7 @@ A source whose `loadSubset` throws instead of returning a rejected promise now f An aborted subset request that a source rejects with `AbortError`, as the source contract asks, no longer ends a live query. A truncate replay or newer demand decides that request instead. -Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window reads the new end's tie group. It no longer trusts a row that an older tie request installed. +Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window repairs its first page. It no longer trusts a row that an older tie request installed. A window move made while a truncate replay holds a live query's results now applies after that replay, even when a newer truncate discards it. Before, the move could reject, and an older move's window stayed. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 79870ff9b1..78092604c2 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. The settlement now never returns itself; a named replay fails without that. Open: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming; that microtask gap is outside the grammar, which flushes between commands, so the deferred path is unchanged. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. The settlement now never returns itself; a named replay fails without that. Open: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming; that microtask gap is outside the grammar, which flushes between commands, so the deferred path is unchanged. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index bfdf757180..be028e7ad6 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1012,14 +1012,11 @@ of the page's current end: the refetch re-read that tie, so every row that ties with the end is installed. A row that another request installed proves nothing. Neither does a tie that the loader read for an older end, because a refetch can move the end earlier, so a row the provider never sent can tie -with the new end. When the end moved, the loader reads only the new end's tie -on the next pass: the refetched page holds every row before the end, so that -tie completes the proof without a repair. It then releases the older tie, -whose rows sort after the end and outside the window, and the deletes that -release causes are not an order change. A loader that already owes a repair, -has other work in flight, or has an end that no cursor can express repairs -instead: reading that end's tie would take the boundary's full-source -fallback, while the repair's bounded prefix can prove the window. A window that needs a row +with the new end. When the end moved, the loader repairs. (Reading only the +new end's tie would also complete the proof, since the refetched page holds +every row before the end, and saves one request; the bounded repair is kept +because that read cost shipped bytes for one saved request per such +refetch.) A window that needs a row past the page, such as one whose page holds rows a joined filter rejects, still repairs. So does any loader while its source holds accepted sync rows back, as behind a persisting optimistic diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 06b04858d5..c98a4d3b3c 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -90,15 +90,11 @@ export class OrderedSourceLoader { private work: AuthoritativeWork = { kind: `idle` } // Keep callbacks, not copied requests or rows. Successful full-source work // subsumes these logical owners; unfinished transports remain observed. - // Each records its prefix count, for a repair's own acquisitions the epoch - // of the repair chain it belongs to, and for a tie request its boundary row. + // Each records its prefix count and, for a repair's own acquisitions, the + // epoch of the repair chain it belongs to. private settledFiniteAcquisitions = new Map< ReleaseLoadSubset, - { - count: number | undefined - chain: number | undefined - tie: Record | undefined - } + { count: number | undefined; chain: number | undefined } >() // The record's presence blocks automatic retry, including initial requests // that have no explicit window-operation generation. @@ -685,77 +681,22 @@ export class OrderedSourceLoader { revalidateLoadedCoverage(): void { // A `requiresFullSource` plan's coverage is its full-source read: nothing // has settled before it. Coverage that one full-source read completed - // holds the work slot, so the end's tie below defers to it, and an - // ordering invalidation owes it nothing. + // owes an ordering invalidation nothing, so it reads nothing below. if (!this.hasSettledSourceRequest) return const end = this.ownPrefixEnd() - if (end === undefined) { - this.invalidateSourceOrdering() - return - } // An installed row past the page proves the source's next row only when // the page read it, or when it ties with an end whose tie the loader - // read. A refetch can move the end before that tie; reading the new - // end's tie then proves the window. An `undefined` end also matches a + // read. A refetch can move the end before that tie, and then the bounded + // repair proves the window again. An `undefined` end also matches a // loader that attempted no tie. No cursor expresses a nullish end, so the // loader has started or will start a full-source read, which covers it. if ( - (this.ownPrefixCount ?? 0) > this.info.offset + this.info.limit || - Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary) + end !== undefined && + ((this.ownPrefixCount ?? 0) > this.info.offset + this.info.limit || + Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary)) ) return - this.loadMovedEndTie(end) - } - - // The refetched page still holds every row before its end, so the end's - // tie completes the proof without a repair. The older tie's rows sort - // after the end and outside the window, and they prove nothing now. - private loadMovedEndTie(end: Record): void { - // Other work owns the boundary, a repair is already owed, or a replay - // owns the leases, so the repair reads it. A staged continuation is - // replaced: the window needs no more rows, and a staged repair step owes - // a repair. No cursor can express some ends' ties; reading one would - // take the boundary's full-source fallback, while the repair's bounded - // prefix can prove the window without it. - if ( - !canExpressCursorOrder( - normalizeOrderByPaths(this.info.orderBy, this.alias).slice(0, 1), - [this.info.valueExtractorForRawRow(end)], - ) || - this.debt !== `none` || - this.work.kind !== `idle` || - this.pending || - this.subscription.hasPendingTruncateReplacement - ) { - this.invalidateSourceOrdering() - return - } - const orderedLoadGeneration = this.orderedLoadGeneration - // The release can delete rows only the older tie held. Those deletes - // leave rows past the end, so they are not an order change. - this.retiringSupersededAcquisitions = true - try { - this.releaseOwned( - this.settledFiniteAcquisitions, - (_, { tie }) => tie !== undefined && this.info.comparator(tie, end) > 0, - ) - } finally { - this.retiringSupersededAcquisitions = false - } - // A release that resets the cursor, or disposes the loader, hands the - // window to the replay or ends it. - if (orderedLoadGeneration !== this.orderedLoadGeneration) return - this.settledSourceBoundary = end - // The pass that follows the report reads the tie, as a repair's first - // request would start then, so a start that throws fails that pass. - this.stagedContinuation = { - orderedLoadGeneration, - authorityGeneration: this.authorityGeneration, - orderingInvalidationGeneration: this.orderingInvalidationGeneration, - role: `finite`, - kind: `boundary`, - continuesOrderedPrefixRepair: false, - } + this.invalidateSourceOrdering() } // The refetched first page holds every row from the start of the order @@ -1102,7 +1043,6 @@ export class OrderedSourceLoader { ): Promise | undefined { const isFullSource = kind === `full-source` const isOrderedRepair = kind === `ordered-repair` - const tie = kind === `boundary` ? this.settledSourceBoundary : undefined const isAuthoritativeRequest = isFullSource || isOrderedRepair || continuesOrderedPrefixRepair let observed: @@ -1219,7 +1159,6 @@ export class OrderedSourceLoader { this.settledFiniteAcquisitions.set(release, { count: prefixCount, chain, - tie, }) // Full-source coverage also retires a repair request that // settles after it. diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 6f44f426d3..20bf075465 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -2220,8 +2220,7 @@ type AuthorityObservation = { ignoredReportsOverFallback: number repairedReportsOnOwnPage: number // Reports after a refetch moved the page's end, while a repair was owed, - // while authoritative work was in flight, and reports whose release of an - // older tie truncated or disposed the loader. + // and while authoritative work was in flight. movedEndReports: number // Reports whose own first page read past a window that then shrank, and // reports to a disposed loader. @@ -2229,12 +2228,8 @@ type AuthorityObservation = { reportsAfterDispose: number // Reports whose page's end, moved by a head insert, has no cursor. unexpressibleEndReports: number - // First finite requests after a report's release truncated the loader, - // which the reset law judges. - resetsAfterReportRelease: number reportsOwingRepair: number reportsDuringWork: number - reportReleasesEndingLoader: number truncates: number replayFailures: number optimisticDepartures: number @@ -2328,8 +2323,6 @@ async function checkAuthorityCase( let releaseViolation: string | undefined let unexpressibleTieViolation: string | undefined let resetViolation: string | undefined - // The truncate count after a report's release truncated the loader. - let truncatedByReport = -1 // A disposed loader owns nothing: it issues no request and releases no // lease, even as requests it issued earlier settle. let disposed = false @@ -2424,18 +2417,6 @@ async function checkAuthorityCase( ) { resetViolation ??= `request ${request.index}, the first finite request after a truncate, continued from a cursor` } - if ( - truncatedByReport === truncates && - !fullSource && - ledger.every( - (other) => - other === request || - other.fullSource || - other.issuedTruncates !== truncates, - ) - ) { - resetsAfterReportRelease++ - } const tie = tieValue(options) if (typeof tie === `number` && !Number.isFinite(tie)) { unexpressibleTieViolation ??= `request ${request.index} read the tie of rank ${tie}, which no cursor can express` @@ -2689,10 +2670,8 @@ async function checkAuthorityCase( let reportsPastWindow = 0 let reportsAfterDispose = 0 let unexpressibleEndReports = 0 - let resetsAfterReportRelease = 0 let reportsOwingRepair = 0 let reportsDuringWork = 0 - let reportReleasesEndingLoader = 0 // Steps that settle a request successfully, run a pass, report, or change // joined work. None of them owes a repair. const quietCommand = (command: AuthorityCommand) => @@ -3096,19 +3075,7 @@ async function checkAuthorityCase( reportsDuringWork++ } const before = ledger.length - const releasedBefore = ledger.filter(({ released }) => released) - const truncatesBefore = truncates loader.revalidateLoadedCoverage() - if (truncates > truncatesBefore) truncatedByReport = truncates - if ( - (testCase.releaseTruncates || testCase.releaseDisposes) && - ledger.some( - (request) => - request.released && !releasedBefore.includes(request), - ) - ) { - reportReleasesEndingLoader++ - } if (command.react) graphPass() await flush() // A report while a repair is owed repairs instead: no tie or page @@ -3532,10 +3499,8 @@ async function checkAuthorityCase( reportsPastWindow, reportsAfterDispose, unexpressibleEndReports, - resetsAfterReportRelease, reportsOwingRepair, reportsDuringWork, - reportReleasesEndingLoader, truncates, replayFailures, optimisticDepartures, @@ -4168,27 +4133,21 @@ describe(`Ordered source request authority`, () => { // repair is owed; // - an invalidation that starts its repair, then a report while that // repair is in flight; - // - the moved-end prefix again with a release callback that truncates or - // disposes the loader, so releasing the older tie ends the loader; // - a two-row window whose first page and tie settle, then shrinks to one // row before a report: the page read past the window, which proves it; // - the moved-end prefix with the loader disposed before the report, which // the dispose law then judges; // - the moved-end prefix with a `-Infinity` head row, so no cursor can // express the page's new end; - // - the moved-end prefix whose release of the older tie truncates the - // loader, then a window that grows, which needs a page after the reset; // - a one-row window whose page and tie settle, then a `-Infinity` head // row and an invalidation: the repair's boundary has no cursor, so it // falls back to a full-source read, and a report follows over it; // - a plan that reads only the full source, then a report over it. // A release that truncates or disposes would end the quiet history, so - // only the fifth prefix allows one. + // no prefix allows one. const unprovenReportProperty = `ordered-loader.unproven-report` const reportPrefixes: Array<{ history: Array - endsLoader?: boolean - truncatesLoader?: boolean fullSource?: boolean narrow?: boolean wide?: boolean @@ -4224,15 +4183,6 @@ describe(`Ordered source request authority`, () => { { type: `report`, react: true }, ], }, - { - history: [ - { type: `settle`, pick: 0, ok: true }, - { type: `settle`, pick: 0, ok: true }, - { type: `headInsert` }, - { type: `report`, react: true }, - ], - endsLoader: true, - }, { history: [ { type: `settle`, pick: 0, ok: true }, @@ -4261,16 +4211,6 @@ describe(`Ordered source request authority`, () => { // The end is the `-Infinity` row only in a one-row window. narrow: true, }, - { - history: [ - { type: `settle`, pick: 0, ok: true }, - { type: `settle`, pick: 0, ok: true }, - { type: `headInsert` }, - { type: `report`, react: true }, - { type: `move`, delta: 1 }, - ], - truncatesLoader: true, - }, { history: [ { type: `settle`, pick: 0, ok: true }, @@ -4290,10 +4230,9 @@ describe(`Ordered source request authority`, () => { .tuple( authorityCaseArbitrary, fc.constantFrom(...reportPrefixes), - fc.boolean(), fc.array(authorityCommand, { maxLength: 8 }), ) - .map(([testCase, prefix, truncates, rest]) => ({ + .map(([testCase, prefix, rest]) => ({ ...testCase, settledFirst: true, requiresFullSource: prefix.fullSource === true, @@ -4301,10 +4240,8 @@ describe(`Ordered source request authority`, () => { ...(prefix.narrow ? { limit: 1 as const, offset: 0 as const } : {}), snapshotRows: 3, needsMore: false, - releaseTruncates: - prefix.truncatesLoader === true || - (prefix.endsLoader === true && truncates), - releaseDisposes: prefix.endsLoader === true && !truncates, + releaseTruncates: false, + releaseDisposes: false, history: [...prefix.history, ...rest], })) const unprovenReportCampaigns = @@ -4327,10 +4264,8 @@ describe(`Ordered source request authority`, () => { `reportsPastWindow`, `reportsAfterDispose`, `unexpressibleEndReports`, - `resetsAfterReportRelease`, `reportsOwingRepair`, `reportsDuringWork`, - `reportReleasesEndingLoader`, ] as const const reach = createOracleReach(unprovenReportProperty, reportStates, { minimum: 20, diff --git a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts index 9c88bbc2d1..45ca95a214 100644 --- a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts +++ b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts @@ -1307,12 +1307,14 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => // The ordinary insert into a top-N list. The page's end moves earlier, so // Query DB reports, but every window row is in the window's own first page. // The tie the loader read belongs to the old end, so its rows prove - // nothing about rows that tie with the new end. The refetch sends the - // Queries it refetches and one tie request for the new end, with no - // repair. The loader releases the old end's tie, so the Query count stays - // the same and later refetches do not re-run a tie that proves nothing. + // nothing about rows that tie with the new end, and the loader repairs: + // the refetch sends the Queries it refetches, then the bounded repair's + // ordered prefix and the new end's tie, and nothing unbounded. The repair + // retires the requests it replaces, so the Query count stays the same. + // (Maintainer decision: the repair replaces a cheaper tie-only read, which + // cost shipped bytes for one saved request per such refetch.) it.each([1, 2] as const)( - `a refetch of a window inside its first page starts no repair (%i order terms)`, + `a refetch that moves a first page's end repairs with a bounded prefix and the new end's tie (%i order terms)`, async (terms) => { const scenario: Scenario = { encoding: `cursor`, @@ -1340,11 +1342,16 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => expectedWindow(run.backend, run.markers, scenario), ) const extra = run.requests.slice(requestsBefore + queries) - expect(extra).toHaveLength(1) - // A tie request reads equal leading values, with no order or cursor. - expect(extra[0]!.orderBy).toBeUndefined() + expect(extra).toHaveLength(2) + // The repair's prefix reads the window's rows from the start of the + // order, bounded by the window, with no cursor. + expect(extra[0]!.orderBy).toBeDefined() + expect(extra[0]!.limit).toBe(scenario.offset + scenario.limit) expect(extra[0]!.cursor).toBeUndefined() - expect(extra[0]!.where).toBeDefined() + // A tie request reads equal leading values, with no order or cursor. + expect(extra[1]!.orderBy).toBeUndefined() + expect(extra[1]!.cursor).toBeUndefined() + expect(extra[1]!.where).toBeDefined() expect(run.activeQueries()).toBe(activeBefore) } catch (error) { primary = error From 3347da60f81e0842468aff4fe506fcb1f1d19e63 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 14:58:10 -0600 Subject: [PATCH 30/40] refactor(db): fold the shared-barrier failure check into the builder - The builder's failDemand takes the failing subscription and collection and decides there whether a failed replay of that collection holds the shared barrier; the subscriber's wrapper and the public hasFailedReplay query go. A superseded demand, whose generation is stale, no longer installs a failed replacement. - siblingReplays receives the subscriber's collection instead of finding it by source id. - failReplacement drops a check its only caller cannot fail: every live-query subscription has the replay-publication control. - failDemand's `fatal` default goes with its last defaulting caller. Measured one at a time (gzip): -21, -6, -5, -18. A getter that returned a replay's completion only while it had work grew gzip by 4 B and was dropped. Evidence: @tanstack/db 285 files, 13,842 tests; the replay, subscription, and ordered oracles pass. Size: @tanstack/db +399 B gzip vs main (was +449). Co-authored-by: Isaac --- packages/db/src/collection/subscription.ts | 4 +- .../query/live/collection-config-builder.ts | 51 ++++++++++--------- .../src/query/live/collection-subscriber.ts | 39 +++++--------- 3 files changed, 42 insertions(+), 52 deletions(-) diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index cb061977a4..bef3a401c4 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -791,9 +791,7 @@ export class CollectionSubscription * this subscription's last demand retires. */ failReplacement(error: unknown): void { - if (!this.options.truncateReplayPublication || this.truncateReplayState) { - return - } + if (this.truncateReplayState) return const replayState = this.createTruncateReplayState( this.collection._sync.getSyncRunGeneration(), () => new Map(), diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 678535f0b6..1b3b93c567 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -515,25 +515,40 @@ export class CollectionConfigBuilder< } /** - * A lazy demand load failure ends the query unless it failed a truncate - * replay after the query first became ready: that replay keeps the last - * publication until a later replay of the source reacquires the demand. - * Before the first ready there is no publication to keep, so the failure - * is an initial load failure. A failed demand is settled either way, so it - * holds no ordered loading. + * A lazy demand load failure ends the query unless a failed replay of its + * source's collection holds publication after the query first became + * ready: that replay keeps the last publication until a later replay of + * the source reacquires the demand. A demand that a pending replay decides + * reaches here only once that replay settled. Before the first ready there + * is no publication to keep, so the failure is an initial load failure. A + * failed demand is settled either way, so it holds no ordered loading. */ failDemand( + subscription: CollectionSubscription, + collection: Collection, planId: string, generation: number, error: unknown, - fatal = true, ): void { const demand = this.activeDemands.get(planId) if (!demand || demand.generation !== generation) return demand.settled = true const normalized = this.recordSubsetError(error) demand.participant?.reject(normalized) - if (fatal || this.liveQueryCollection?.status === `loading`) + // A failure while a failed replay of this source holds the shared + // barrier belongs to that recovery: it does not end the query, and it + // holds this subscription's publication too, so retiring the other + // replay's failed demand cannot publish without this demand's rows. + if ( + this.liveQueryCollection?.status !== `loading` && + this.collectionSources.some( + (source) => + source.collection === collection && + this.subscriptions[source.sourceId]?.hasFailedTruncateReplacement, + ) + ) + subscription.failReplacement(normalized) + else this.transitionToError( devBuild() && process.env.NODE_ENV !== `production` ? `Subset demand '${planId}' failed: ${normalized.message}` @@ -653,10 +668,10 @@ export class CollectionConfigBuilder< } /** Replays of other subscriptions to a source's collection that still load. */ - siblingReplays(sourceId: string): Promise | undefined { - const collection = this.collectionSources.find( - (source) => source.sourceId === sourceId, - )?.collection + siblingReplays( + collection: Collection, + sourceId: string, + ): Promise | undefined { const pending = this.collectionSources.flatMap((source) => { const subscription = this.subscriptions[source.sourceId] // A replay whose completion already settled cannot be waited on. @@ -670,18 +685,6 @@ export class CollectionConfigBuilder< return pending.length > 0 ? Promise.allSettled(pending) : undefined } - /** - * Whether a failed replay of a collection still holds publication. The - * query's subscriptions to one collection share that replay's barrier. - */ - hasFailedReplay(collection: Collection): boolean { - return this.collectionSources.some( - (source) => - source.collection === collection && - this.subscriptions[source.sourceId]?.hasFailedTruncateReplacement, - ) - } - private hasFailedSourceRecovery(): boolean { return Object.values(this.subscriptions).some( (subscription) => subscription.hasFailedTruncateReplacement, diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index 28b9a9b0b8..3e9587d0db 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -195,8 +195,9 @@ export class CollectionSubscriber< if (!Object.is(subscription.lastError, error)) throw error const isInitialSync = this.collectionConfigBuilder.liveQueryCollection?.status === `loading` - this.failDemand( + this.collectionConfigBuilder.failDemand( subscription, + this.collection, plan.id, this.collectionConfigBuilder.beginDemand(plan.id), error, @@ -228,7 +229,14 @@ export class CollectionSubscriber< // query. const replacement = subscription.pendingTruncateReplacement if (replacement) settleWith(replacement) - else this.failDemand(subscription, plan.id, generation, error) + else + this.collectionConfigBuilder.failDemand( + subscription, + this.collection, + plan.id, + generation, + error, + ) }, ) settleWith(update.ready) @@ -237,28 +245,6 @@ export class CollectionSubscriber< } } - // A failure while a failed replay of this source holds the shared barrier - // belongs to that recovery: it does not end the query, and it holds this - // subscription's publication too, so retiring the other replay's failed - // demand cannot publish without this demand's rows. - private failDemand( - subscription: CollectionSubscription, - planId: string, - generation: number, - error: unknown, - ): void { - const recovering = this.collectionConfigBuilder.hasFailedReplay( - this.collection, - ) - if (recovering) subscription.failReplacement(error) - this.collectionConfigBuilder.failDemand( - planId, - generation, - error, - !recovering, - ) - } - private sendChangesToPipeline( changes: Iterable>, ): void { @@ -436,7 +422,10 @@ export class CollectionSubscriber< onStart?.() }, siblingReplays: () => - this.collectionConfigBuilder.siblingReplays(this.sourceId), + this.collectionConfigBuilder.siblingReplays( + this.collection, + this.sourceId, + ), succeed: () => { if ( syncRunGeneration !== From 7924514027d799a9775a452dece7f3897a3eb936 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 15:19:35 -0600 Subject: [PATCH 31/40] test(db): require a failed demand outside every recovery to end the query The convergence check runs after an explicit window move, which retries a failed demand, so it accepted a query that stayed alive after a lazy demand failed outside any recovery. A design that let any failed replay of the query, across collections, cover such a failure passed both failed-replay campaigns. Law (ARCHITECTURE.md, error 228): a lazy demand load that fails outside a replay ends the query; no barrier keeps the incomplete join side or bucket private. The oracle now requires that, skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay, which may belong to either query's recovery. An aborted request's rejection and the failure of a load whose segment newer coverage released are not failures, as ARCHITECTURE.md states. RED (hostile mutant): judging the shared barrier against any failed replay of the query fails both failed-replay campaigns with "terminal error: a failed demand outside recovery". GREEN: load-subset replay oracle 43/43 five times; @tanstack/db 285 files, 13,842 tests. Also corrects ARCHITECTURE.md, which still described the old rule that a synchronous loadSubset throw rolls its owner back and ends the query; a throw now follows the same rules as a rejection. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 4 +- ...ad-subset-replay-refinement-oracle.test.ts | 46 ++++++++++++++++++- 3 files changed, 47 insertions(+), 5 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 78092604c2..31212d1b1d 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, survives: the convergence check runs after an explicit window move, which retries a failed row demand, so it accepts a non-fatal row failure; ARCHITECTURE.md keeps that failure fatal. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. The settlement now never returns itself; a named replay fails without that. Open: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming; that microtask gap is outside the grammar, which flushes between commands, so the deferred path is unchanged. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. The settlement now never returns itself; a named replay fails without that. Open: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming; that microtask gap is outside the grammar, which flushes between commands, so the deferred path is unchanged. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index be028e7ad6..99c6f8c0f5 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1490,8 +1490,8 @@ the first ready there is no complete publication to keep, so the failure is an initial load failure and ends the query. A lazy demand load that fails outside a replay ends the query (error 228): the parent rows it serves may already be public, and no barrier keeps their incomplete join side or bucket private. A -synchronous `loadSubset` throw rolls its logical owner back, so it fails no -replay that could reacquire the demand, and it ends the query too. +synchronous `loadSubset` throw follows the same rules as a rejection: inside an +open replay it fails that replay, and outside one it ends the query. ### Mutation boundaries and initial readiness diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 87c3451f35..4859b6302d 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1568,6 +1568,12 @@ async function checkOrderedReplay( // threw, may fall outside the replay the model still holds open, so it may // end the query (see Limits). let ambiguousFailure = false + // A failed demand whose recovery the model cannot place (see Limits). + let unplacedFailure = false + // A failed lazy demand that belongs to no recovery, which must end the + // query. Adapter callbacks set the flags, so read them through a call. + const failedOutsideRecovery = () => + demandFailed && !ambiguousFailure && !unplacedFailure const fatalLoadFailed = () => demandFailed || failedWhileLoading || ambiguousFailure // Lazy join demand asks for keys; ordered and full-source loads do not. @@ -1769,7 +1775,19 @@ async function checkOrderedReplay( ) load.outcome = `failed` anyLoadFailed = true - if (demand) demandFailed = true + // An aborted request's rejection is obsolete demand, and a + // segment that newer coverage replaced settles, so neither + // is a failed load (ARCHITECTURE.md). + if ( + demand && + !options.signal?.aborted && + (acquired.get(options) ?? 0) > 0 + ) { + demandFailed = true + // With a peer, a load that joined a replay may belong to either + // query's recovery, so the model cannot place it. + if (participant && scenario.peer) unplacedFailure = true + } // A demand the graph started after the commit fails inside // the replay. if ( @@ -1881,7 +1899,19 @@ async function checkOrderedReplay( ) load.outcome = `failed` anyLoadFailed = true - if (demand) demandFailed = true + // An aborted request's rejection is obsolete demand, and a + // segment that newer coverage replaced settles, so neither + // is a failed load (ARCHITECTURE.md). + if ( + demand && + !options.signal?.aborted && + (acquired.get(options) ?? 0) > 0 + ) { + demandFailed = true + // With a peer, a load that joined a replay may belong to either + // query's recovery, so the model cannot place it. + if (participant && scenario.peer) unplacedFailure = true + } // A demand the graph started after the commit, failing // while its replay still loads. if ( @@ -2328,6 +2358,18 @@ async function checkOrderedReplay( const revalidated = live.utils.setWindow(window) if (revalidated !== true) await revalidated.catch(swallow) await settleAll() + // A failed lazy demand outside every recovery is fatal: the join side + // can no longer be complete, so the query must end rather than show a + // window without that demand's rows. Only a failure the model cannot + // place, after a load threw, is excused. + // With a peer, the adapter cannot tell which query issued the load, so + // either query may be the one that ends. + if (failedOutsideRecovery()) { + expect( + [live, peer].some((query) => query?.status === `error`), + `terminal error: a failed demand outside recovery`, + ).toBe(true) + } for (const [shown, label, expected] of [ [live, `convergence`, window], ...(peer ? [[peer, `peer convergence`, peerWindow] as const] : []), From 41e231be8a6492c3b3fb71935aa6b18f501ee97f Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 15:35:21 -0600 Subject: [PATCH 32/40] fix(db): let no window move wait on itself after a newer move throws An independent review found that a move deferred behind a replay could hang forever: a newer move that throws while it starts bumps the request generation without recording a settlement, so the deferred move, resuming after the replay, awaited the latest request, which was still itself. The plain path's version of the same shape was fixed earlier with a self check on its settlement. A call now clears the latest request as soon as it bumps the generation. An older move then waits only on a settlement the newer call records, or on nothing if that call threw, so neither path can wait on itself, and the earlier self check goes. Evidence: the named replay "a window move whose load throws while an older move waits" passes, and without the reset fails with "TypeError: Chaining cycle detected for promise". The deferred path needs a newer move inside the microtask gap between a replay settling and the deferred move resuming, which the grammar does not reach; it shares the mechanism the replay witnesses. @tanstack/db 285 files, 13,842 tests. Also corrects revalidateLoadedCoverage's comment: coverage that a full-source read completed still runs the invalidation, which starts no request for it, so the loader ignores the report. Size: @tanstack/db +386 B gzip vs main (-13 B). Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- .../src/query/live/collection-config-builder.ts | 16 ++++++++-------- .../db/src/query/live/ordered-source-loader.ts | 3 ++- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 31212d1b1d..797543d90c 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. The settlement now never returns itself; a named replay fails without that. Open: a move deferred behind a replay has the same shape when a newer move throws between the replay settling and the deferred move resuming; that microtask gap is outside the grammar, which flushes between commands, so the deferred path is unchanged. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 1b3b93c567..3ece9121fd 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -342,6 +342,10 @@ export class CollectionConfigBuilder< limit: options.limit ?? baseWindow?.limit, } const request = ++this.windowRequestGeneration + // This call supersedes every older one. Until it records its own + // settlement, they have nothing newer to wait on, even if it throws + // before it does; none of them can wait on itself. + this.latestWindowRequest = undefined const sourceRecovery = this.pendingSourceRecovery() if (sourceRecovery) { const syncRunGeneration = this.syncRunGeneration @@ -435,17 +439,14 @@ export class CollectionConfigBuilder< this.latestWindowRequest = undefined return true } - const settled: Promise = settlement.then( + return (this.latestWindowRequest = settlement.then( () => { if (windowOperationGeneration === this.windowOperationGeneration) { this.settledWindow = requestedWindow } - // A newer call wins; one that threw while it started left this one - // latest, and a promise cannot settle on behalf of itself. - const latest = this.latestWindowRequest - return request === this.windowRequestGeneration || latest === settled + return request === this.windowRequestGeneration ? undefined - : latest + : this.latestWindowRequest }, (error) => { if (windowOperationGeneration === this.windowOperationGeneration) { @@ -454,8 +455,7 @@ export class CollectionConfigBuilder< } throw error }, - ) - return (this.latestWindowRequest = settled) + )) } getWindow(): { offset: number; limit: number } | undefined { diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index c98a4d3b3c..95b5936454 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -681,7 +681,8 @@ export class OrderedSourceLoader { revalidateLoadedCoverage(): void { // A `requiresFullSource` plan's coverage is its full-source read: nothing // has settled before it. Coverage that one full-source read completed - // owes an ordering invalidation nothing, so it reads nothing below. + // owes an ordering invalidation nothing, so the invalidation below starts + // no request for it, and the loader ignores the report (ARCHITECTURE.md). if (!this.hasSettledSourceRequest) return const end = this.ownPrefixEnd() // An installed row past the page proves the source's next row only when From a43e73b83c851ff3f4a77fc0ccfd9a78d90d8bcf Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 16:24:30 -0600 Subject: [PATCH 33/40] test(db): tighten the oracles an independent review found too permissive An oracle review of this branch found excusals broader than their reasons, a false coverage claim, and gaps in reach and in ORC-016. Production is unchanged; each change was checked against the current code and, where a mutant applies, against that mutant. load-subset replay oracle: - The aborted-move branch was unreachable (no history reached it, though the coverage map said a named replay did) and quietly accepted a move rejected with AbortError. A truncate never aborts a move now (ARCHITECTURE.md), so such a rejection is an escaped error; the branch and its counter are gone. - The terminal-error excuse after a synchronous throw applied to every later failure in the history. It now applies only to the thrown error itself ending the query after the step that threw. Rolling a thrown demand back, the old behavior, again fails both named replays and the broad and failed-replay campaigns. - A load that fails outside every barrier after a failed demand of its source retired is unplaced: the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. - The window check requires each shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. ordered source loader state oracle: - The unproven-report law no longer accepts a full-source read as proof when every installed rank is expressible. A mutant that forced full debt after a report passed before and now fails ("issued no bounded repair and no tie of its end"). - The cost law rejects a repeated read within one invalidation. ordered-work oracle: an optimistic refill that a case allows but does not require is still bounded. A mutant that made the growth read unbounded fails 24 cases ("an allowed refill used an unbounded provider request"). query-db refetch oracle: the campaigns declare reach with createOracleReach (minimum 3), including a refetch that reported while the window was full (12 to 21 histories per campaign); the named replay asserts bounded reads and an unchanged Query count instead of an exact request shape. ORC-016: the zero-limit truncate example moves from the ordered-lifecycle oracle to live-query-collection.test.ts. Limit recorded: with a peer, the strict terminal-error law accepts either query ending, since the adapter cannot attribute a load to a query. Evidence: @tanstack/db 285 files, 13,842 tests; query-db 21 files, 925 tests; the replay oracle passes six runs in a row. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 4 +- .../tests/query/live-query-collection.test.ts | 82 +++++++++++++ ...ad-subset-replay-refinement-oracle.test.ts | 110 ++++++++++++------ .../ordered-lifecycle-oracle.property.test.ts | 81 ------------- ...ordered-source-loader-state-oracle.test.ts | 50 ++++++-- .../ordered-work-oracle.property.test.ts | 23 ++++ .../refetch-ordered-prefix-oracle.test.ts | 69 ++++++----- 7 files changed, 267 insertions(+), 152 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 797543d90c..efb896bf62 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -63,7 +63,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it reads only the new end's tie on the next pass and releases the older tie, so query-db's top-N insert costs one tie request instead of a repair; query-db's first-page witness requires exactly that request and an unchanged count of active Queries, and mutants that keep the older tie, or that let its release count as an order change, fail it. The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch and the aborted-move branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | @@ -683,7 +683,7 @@ generic visitor behavior outside the named collectors. | Drafts and native values | [proxy](https://github.com/TanStack/db/blob/main/packages/db/tests/proxy-oracle.test.ts), [detachment](https://github.com/TanStack/db/blob/main/packages/db/tests/proxy-detachment-contract-oracle.test.ts), [iteration](https://github.com/TanStack/db/blob/main/packages/db/tests/proxy-iteration-contract-oracle.test.ts), [revert](https://github.com/TanStack/db/blob/main/packages/db/tests/proxy-revert-oracle.property.test.ts), [native methods](https://github.com/TanStack/db/blob/main/packages/db/tests/proxy-native-methods-oracle.property.test.ts) | Native-operation controls, exact patches and actual stored rows; alias/cycle/adversarial-key histories. Set and Map reorder histories check draft patches and stored iteration order after replacement and clear-and-readd; RegExp replacement/reversion histories check `lastIndex`. General native-mutator and symbol-write support is not established by a plain-object oracle. The revert owner generates assignment, revert, delete, nested, symbol-key, and `for...of` histories and checks `getChanges()` and the draft against an independent draft-equality model: ordered Map and Set contents (including Sets inside Map values), RegExp `lastIndex`, array holes, and symbol keys inside nested values. Map and Set mutators and top-level symbol writes are outside its grammar; array mutators appear only in the mixed block below. Its round-trip grammar also replaces a field with a new object and restores the original value through a nested write. The law requires every changed key, and permits a written key whose final value equals its original to be reported with that value: a false positive repeats the stored value, and a false negative loses a write ([review](oracle-reviews/proxy-revert-replaced-object.md)). A further revert block mixes array `push`, `pop`, and `reverse` with assignments, a nested object replacement, and a retained handle, and checks that a mutator through a replaced handle changes nothing; pinned witnesses require a native write through a Map or Set value, a typed-array view, or a frozen draft to be reported, and a nested revert inside a Map or Set to keep the container's other write; Map and Set mutators are not mixed with assignments, and an equal-valued assignment keeps a retained handle attached (declared). The proxy owner's stored-function law checks, against a native row, that a function stored as data reads back as stored by every read path and that a stored method sees the draft as `this`, so its writes are tracked. The detachment owner pins which key classes a draft copies: enumerable string keys and every symbol key. The revert owner also generates typed arrays (including a subclass that does not forward its constructor argument), URLs, and a class with only private state as written values, with a property that writes two of them over one field. The native-methods owner generates rows (holes, `undefined`, duplicates, nested arrays, empty rows, typed `-0` and `NaN`) and calls every method of `Array.prototype` and `TypedArray.prototype`, in fixed and random campaigns with replay; it compares the result, whether the method returned the array itself, the identity of each original object, and the published row with native values, and a new built-in method fails until it has arguments. The proxy owner's `defineProperty`, stored-function, and frozen-draft laws are bounded tables against native rows. Its `defineProperty` law compares result, value, descriptor, and changes, for values and accessors. The iteration owner checks that array iterators see a push, removal, index write, or length cut made while they are open. Open: a draft reads a class instance of the original row as a plain object, so private-field getters read `undefined`, and a built-in method called through `this` on a nested Map or Set draft rejects the Proxy receiver; a mutator that changes nothing publishes an equal value; writing back a keyless class instance of the original row is a change, and `DataView` setters are untracked ([draft proxy review](oracle-reviews/code-weight-draft-proxy.md)). | | Query DB and observer | [ownership](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts), [persisted direct-write window](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/persisted-direct-write-window.oracle.test.ts) ([review](oracle-reviews/issue-2029-direct-write-lock-window.md)), [load lifecycle](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/load-subset-lifecycle-oracle.test.ts), [observer histories](https://github.com/TanStack/db/blob/main/packages/db/tests/live-query-observer-history-oracle.property.test.ts), [retained snapshots](https://github.com/TanStack/db/blob/main/packages/db/tests/live-query-observer-snapshot-oracle.property.test.ts) ([review](oracle-reviews/pr-2043-retained-observer-snapshots.md)) | Real QueryClient boundary, applied-settlement barriers for existing and cached demand, mutation-handler Collection identity, terminal fetch-record lifecycle, and a per-listener eligibility ledger, not a duplicate dispatch queue. The bounded handler grammar crosses insert, update, delete, parameter and mutation-alias access, refetch, and clearError. A held clearError retry retains the prior public error through initial, background, and Collection application failures; success clears it, failure advances the count, including a repeated error at clock time zero. An intermediate Query retry attempt does not recount the previous background error; a held final attempt can succeed or fail. A held application after successful Query fetch keeps the error visible until the applied result. A mocked persisted-baseline scan holds an older success while a newer terminal Query failure arrives; the newer public error and count survive application, including when the Error object and clock tick repeat. Native SQLite persistence, concurrent retries across multiple tracked Queries, mutation-handler fetch-boundary settlement, and deferred result application remain outside this witness. The mutation overlap grammar crosses both start orders. Check reentry, peer survival, FIFO and disposal independently of final rows. A retained observer snapshot shows its capture-time rows on a later first read of `data` or `state`. Direct-write witnesses now distinguish an idle pending eager cache seed (success with no new freshness timestamp) from an in-flight pending request, and require update, delete, and upsert to reach both eager Collections through their shared physical Query, in both acquisition orders. They check both public Collections and the Query cache immediately after each write. A held older shared fetch also crosses B's release and reacquisition, then compares both public rows at refetch and load settlement; mixed select wrappers and three or more co-owners remain outside this witness. The load-lifecycle owner also restores an intermediate checkpoint that keeps an explicit subset refetch pending until a post-write request applies, and rejects a no-cancel mutant at the replacement-request assertion. | | Query DB SSR dehydration | `packages/query-db-collection/tests/ssr-dehydration-oracle.test.ts`, [review record](oracle-reviews/issue-1950-ssr-dehydration.md) | A plain Query cache control, a live on-demand compound predicate, a signal-only request, and an ordered cursor with a custom comparator all retain their row through QueryClient dehydration, reconstruction of Seroval's emitted stream payload, and Query Core hydration from that payload. The on-demand query functions still receive the original IR, comparator, and signal; serialized metadata retains enumerable user metadata but excludes request options. The original implementation fails the compound and signal cases at the stream checkpoint. A serializable stream-only request-options leak passes the former chunk-count check but fails the current payload assertion. This owner covers successful Query cache entries at the installed Query Core 5.90.20 and Seroval 1.5.0 versions. A TanStack Start integration owner still needs the reporter's Router 1.171.33 and Seroval 1.6.8 path, browser Collection resume/refetch, and request cancellation across SSR. Query subset and pagination oracles remain owners of predicate/order/cursor meaning and supported value types. The issue's `shouldDehydrateQuery` workaround replaces Query Core's success-only default and admits pending/error queries; any documented workaround must preserve that default filter. | -| Query DB ordered refetch | `packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts` | Issue #2136: after preload, and after `utils.refetch()` or a Query invalidation settles, an on-demand Query Collection behind a LEFT-joined, ordered, limited live query must show the window over the current backend rows. The design under test: query-db reports `invalidateLoadedSubsets()` when an ordered page changes keys and either has a later page or ends earlier in the order than before, and each ordered loader runs its bounded prefix repair. Its model is a pure top-k of marked backend rows. A fixed and a random campaign generate rounds of marker flips (which serve continuation pages), backend changes while the provider lags, a second live query (`observe`) whose own demand can load a page's new row first, optional Query invalidation with further changes, and a refetch, across one or two order terms, a numeric or locale-collated string order, limits 1–3, offsets 0–1, and a static or function Query key. With an unchanged backend, a repeated refetch must serve no more rows, start at most one first-page request, and report nothing. A refetch or invalidation must never start a new full-source request. A coverage law runs at the same checkpoints: every backend row from the start of the order through the window's last row (every row, when the window is short) must be in the source Collection with its current values, so a latent gap fails before a marker exposes it; a named replay of the issue trace with the moved row unmarked passes the window law and fails only the coverage law when Query DB never reports. The grammar can hold an optimistic update across a refetch, with the checkpoint moved to its release, and a premise-directed campaign under a hold (other rows observed, a row moved ahead of the page's end, a new row further ahead, a write to the last-sorting observed row) requires a report during a hold in every case and fails a skip that reads rows held back behind the transaction; the general campaigns did not. The general grammar holds optimistic deletes as well as updates. A named replay of a refetch during a held delete left the window short after release, also with this PR's report and overlap changes reverted, until the fix that resumes a continuation after a synchronous answer; no window is checked during a hold, where #2148 and #2169 remain open. Reports (`loadedSubsets:invalidated`) are bounded: a step reports at most once, and only when some ordered request was answered again with different rows; named replays require no report when a one-page window, or a window's last page, only loses a row, and a sync-boundary replay requires a report when a page ending on a row without an order value has its end move earlier (on the earlier gate, which read that page as empty, it reported nothing), and another requires a report when the provider returns a one-page window's rows in reverse order (the earlier gate took the last array row as the page's end). When a page's end moves earlier, Query DB reports, and a loader whose window lies inside its own first page must skip the repair: a named replay of an insert into a top-N list bounds the refetch's requests by its Queries, and a filtered one-page window whose page end moves past an unseen row must still repair (found by the random campaign; a loader that always skips fails it, the f9 tie replay, and the ordered-work oracle's signalling replays). The skip also refuses while the source holds accepted sync rows back: a replay refetches while an optimistic update persists, so the report arrives before the refetched rows are visible, and a skip that reads those older rows fails it. The ordered loader state oracle checks that a prefix read issued before an ordering invalidation, which settles after the bounded repair, does not retire the repair's prefix; the repair-first order failed before that fix. A `loadSubset` call that acquires a released, invalidated page can report, because Query then refetches that page on its own. A failed repair shows only the last window or the right one. Named replays cover the issue trace after a refetch and after an invalidation, with a page-capping provider, with another live query loading the page's new row first, with an empty later page, with a page whose end moves earlier while a tie request holds its old last row, a second refetch after an invalidation, a repair failure, and a provider that honors neither cursor nor offset. Never signalling, the old overlap-to-full-source rule, ignoring empty later pages, reporting on any page change, removing the repair loop guard, and reporting once per page each fail it by assertion. A replay makes a page's application fail and checks the window shows its last result or the right one, then the next refetch restores it; recording a page before its rows are accepted cannot be distinguished, because the failed attempt reports on its own. Open: the report laws bound reports but do not require each one; a page whose end moves earlier reports even with unique order values, where no row can be missing; it does not check whether a repair waits for in-flight work; the generated campaigns use an uncapped cursor provider only (an offset-only provider and a page-capping provider each hit a loader defect that reproduces on main: #2182, #2183); the generated campaigns do not check the window after a marker flip (a named replay checks the trace where the loader stopped refilling after a joined-side removal, which the repair-retirement fix below closed); a failed repair still recovers through the loader's full-source retry; interval, focus, and reconnect refetches share the invalidation path but are not driven; NaN order values have no executable witness; overlapping `refetch: true` acquisitions on one Query hash are counted, so the first to finish does not clear the loop guard for the other, but no witness separates that from a plain set; a Query refetch that joins a running `refetch: true` acquisition on the same hash does not report, and no witness reaches it (#2185); follower tabs of a multi-tab persisted collection never receive the report (#2174). The offset-only and page-capping loader defects have `it.fails` witnesses in the oracle. A finished bounded repair used to discard its boundary when its retirement made Query DB remove a row the loader had sent; the loader then restarted paging with a smaller first page that could not retire the repair's prefix, so every later refetch read both. A named replay found by the random campaign (at four times the run count) and an ordered loader state witness fail on that, and the fix also closed the joined-side refill defect. At four times the run count, the random campaign found a short window that stopped paging before the last row of the order: a report's repair released continuation Queries, and Query DB answered their reacquisition synchronously from its cache, so the graph turn inside each request asked for more while the loader was still requesting and nothing resumed the staged continuation. Only the coverage law failed. A named replay and an ordered loader state witness (a synchronous answer whose writes the graph already processed must continue the chain) fail on that, and pass with the fix. The loader side of the report also has a second owner: the ordered-work oracle's `refetch-signal` provider re-runs each served request alone and reports when any answer changed, across that oracle's generated plans, and a signalling refetch must start no full-source request unless a bounded page in that step ended on a null or NaN rank, which takes the documented full-source fallback (a random seed found that case). Never signalling and the old overlap-to-full-source rule each fail it. It excludes a peer live query, which can keep a short window after a refetch (#2155). The persisted-collection oracle checks that a source's report reaches the Collection through the managed cache-generation wrapper and that a retired source session's report is dropped. Remaining work for this class is tracked in #2186. | +| Query DB ordered refetch | `packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts` | Issue #2136: after preload, and after `utils.refetch()` or a Query invalidation settles, an on-demand Query Collection behind a LEFT-joined, ordered, limited live query must show the window over the current backend rows. The design under test: query-db reports `invalidateLoadedSubsets()` when an ordered page changes keys and either has a later page or ends earlier in the order than before, and each ordered loader runs its bounded prefix repair. Its model is a pure top-k of marked backend rows. A fixed and a random campaign generate rounds of marker flips (which serve continuation pages), backend changes while the provider lags, a second live query (`observe`) whose own demand can load a page's new row first, optional Query invalidation with further changes, and a refetch, across one or two order terms, a numeric or locale-collated string order, limits 1–3, offsets 0–1, and a static or function Query key. With an unchanged backend, a repeated refetch must serve no more rows, start at most one first-page request, and report nothing. A refetch or invalidation must never start a new full-source request. A coverage law runs at the same checkpoints: every backend row from the start of the order through the window's last row (every row, when the window is short) must be in the source Collection with its current values, so a latent gap fails before a marker exposes it; a named replay of the issue trace with the moved row unmarked passes the window law and fails only the coverage law when Query DB never reports. The grammar can hold an optimistic update across a refetch, with the checkpoint moved to its release, and a premise-directed campaign under a hold (other rows observed, a row moved ahead of the page's end, a new row further ahead, a write to the last-sorting observed row) requires a report during a hold in every case and fails a skip that reads rows held back behind the transaction; the general campaigns did not. The general grammar holds optimistic deletes as well as updates. The generated campaigns declare reach (minimum 3) for a lagging refetch after a continuation, one during a hold, and a refetch that reported while the window was full, where the loader decides between ignoring the report and repairing. A named replay of a refetch during a held delete left the window short after release, also with this PR's report and overlap changes reverted, until the fix that resumes a continuation after a synchronous answer; no window is checked during a hold, where #2148 and #2169 remain open. Reports (`loadedSubsets:invalidated`) are bounded: a step reports at most once, and only when some ordered request was answered again with different rows; named replays require no report when a one-page window, or a window's last page, only loses a row, and a sync-boundary replay requires a report when a page ending on a row without an order value has its end move earlier (on the earlier gate, which read that page as empty, it reported nothing), and another requires a report when the provider returns a one-page window's rows in reverse order (the earlier gate took the last array row as the page's end). When a page's end moves earlier, Query DB reports. A loader whose window lies inside its own first page skips the repair only while its page read past the window or it read the tie of the current end; after an insert into a top-N list moves the end, a named replay requires the loader to prove the window again with bounded reads, no full-source read, and an unchanged Query count. A filtered one-page window whose page end moves past an unseen row must still repair (found by the random campaign; a loader that always skips fails it, the f9 tie replay, and the ordered-work oracle's signalling replays). The skip also refuses while the source holds accepted sync rows back: a replay refetches while an optimistic update persists, so the report arrives before the refetched rows are visible, and a skip that reads those older rows fails it. The ordered loader state oracle checks that a prefix read issued before an ordering invalidation, which settles after the bounded repair, does not retire the repair's prefix; the repair-first order failed before that fix. A `loadSubset` call that acquires a released, invalidated page can report, because Query then refetches that page on its own. A failed repair shows only the last window or the right one. Named replays cover the issue trace after a refetch and after an invalidation, with a page-capping provider, with another live query loading the page's new row first, with an empty later page, with a page whose end moves earlier while a tie request holds its old last row, a second refetch after an invalidation, a repair failure, and a provider that honors neither cursor nor offset. Never signalling, the old overlap-to-full-source rule, ignoring empty later pages, reporting on any page change, removing the repair loop guard, and reporting once per page each fail it by assertion. A replay makes a page's application fail and checks the window shows its last result or the right one, then the next refetch restores it; recording a page before its rows are accepted cannot be distinguished, because the failed attempt reports on its own. Open: the report laws bound reports but do not require each one; a page whose end moves earlier reports even with unique order values, where no row can be missing; it does not check whether a repair waits for in-flight work; the generated campaigns use an uncapped cursor provider only (an offset-only provider and a page-capping provider each hit a loader defect that reproduces on main: #2182, #2183); the generated campaigns do not check the window after a marker flip (a named replay checks the trace where the loader stopped refilling after a joined-side removal, which the repair-retirement fix below closed); a failed repair still recovers through the loader's full-source retry; interval, focus, and reconnect refetches share the invalidation path but are not driven; NaN order values have no executable witness; overlapping `refetch: true` acquisitions on one Query hash are counted, so the first to finish does not clear the loop guard for the other, but no witness separates that from a plain set; a Query refetch that joins a running `refetch: true` acquisition on the same hash does not report, and no witness reaches it (#2185); follower tabs of a multi-tab persisted collection never receive the report (#2174). The offset-only and page-capping loader defects have `it.fails` witnesses in the oracle. A finished bounded repair used to discard its boundary when its retirement made Query DB remove a row the loader had sent; the loader then restarted paging with a smaller first page that could not retire the repair's prefix, so every later refetch read both. A named replay found by the random campaign (at four times the run count) and an ordered loader state witness fail on that, and the fix also closed the joined-side refill defect. At four times the run count, the random campaign found a short window that stopped paging before the last row of the order: a report's repair released continuation Queries, and Query DB answered their reacquisition synchronously from its cache, so the graph turn inside each request asked for more while the loader was still requesting and nothing resumed the staged continuation. Only the coverage law failed. A named replay and an ordered loader state witness (a synchronous answer whose writes the graph already processed must continue the chain) fail on that, and pass with the fix. The loader side of the report also has a second owner: the ordered-work oracle's `refetch-signal` provider re-runs each served request alone and reports when any answer changed, across that oracle's generated plans, and a signalling refetch must start no full-source request unless a bounded page in that step ended on a null or NaN rank, which takes the documented full-source fallback (a random seed found that case). Never signalling and the old overlap-to-full-source rule each fail it. It excludes a peer live query, which can keep a short window after a refetch (#2155). The persisted-collection oracle checks that a source's report reaches the Collection through the managed cache-generation wrapper and that a retired source session's report is dropped. Remaining work for this class is tracked in #2186. | | Ordered acquisition | [pagination](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pagination-oracle.property.test.ts), [ordered work](https://github.com/TanStack/db/blob/main/packages/db/tests/query/ordered-work-oracle.property.test.ts), [ordered lifecycle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts), [ordered loader state](https://github.com/TanStack/db/blob/main/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts), [graph scheduler](https://github.com/TanStack/db/blob/main/packages/db/tests/query/scheduler.test.ts), [issue #1880 ordered-repair review](oracle-reviews/issue-1880-ordered-repair.md), [issue #1882 custom local collation review](oracle-reviews/issue-1882-custom-local-collation.md), [issue #1898 relation-filter review](oracle-reviews/issue-1898-relation-filter.md), [PR #1909 external review](oracle-reviews/pr-1909-external-review.md), [eager ordered-window repair review](oracle-reviews/perf-ordered-window-repair.md), [reversed index reads](https://github.com/TanStack/db/blob/main/packages/db/tests/reverse-index-oracle.property.test.ts) | Complete finite provider results, inherited locale and bounded/unbounded custom collation with exact request options, query-level inheritance across source aliases, real lexical/numeric disagreement, custom comparator ties, scan/index paths, pending windows, ties/nulls, lease ownership, Effect callback gates, including an indexed source update during Effect startup, and documented repair timing. Direct LEFT-joined filters cover local finite prefixes, pending child demand, child removal, and anti-join publication with custom full-source and unindexed fallback loading; unrelated include demand cannot stall the root continuation, and an empty joined replay wakes held Effect callbacks; remote relation hints remain outside this owner. Graph scheduler checks coalescing, dependency order, synchronous loader input, and repeated-alias failure ownership. The ordered-work oracle checks that a synchronous root refill reaches D2 before joined publication; a mutant that omitted the post-loader graph step failed at the second child demand. Custom collation proves local full-source acquisition only; it does not establish provider cursor capability or backend collation fidelity. Sibling-source input during a synchronous ordered continuation returns to graph work before another ordered acquisition. Request completion is not proof of unrequested source extent. A window move during existing joined demand waits for its root continuation, including an independently held root request; current demand failure rejects the move, while retirement and an obsolete rejection do not. A bounded ordered repair resumes its refill after joined demand settles. The ordered-source-loader state test checks that an explicit failed-request retry blocked by joined demand retains its generation until it can issue a request. Multiple simultaneously pending joined plans during a window move and a public-query failed-retry overlap still need dedicated witnesses. The ordered-work oracle also checks an eager indexed window over generated upserts and deletes: the rows equal the filtered, sorted model, and each change delivers a bounded number of source rows; an unbounded or always-full repair read fails it. Its filter dimension adds a function filter and a LEFT-joined filter, and a campaign fixed to the joined premise checks that a window refilled past a distant row the query already received holds the first eligible rows. Joined filters also run as anti-joins, change eligibility through the marker or the root row's join key, and drive an Effect beside the Collection. A full-join filter keeps every root row and adds a null-ranked rootless row for each marker no root row matches; on main a bounded root prefix showed markers whose root rows were not yet loaded as unmatched rows, and restoring the old inner-or-right full-source rule fails eight tests. Null placement is a generated dimension, with the index built to match, and a query that ignores it fails six tests. A full-join campaign has few or no filler rows, a matched null-ranked root row and at least one orphan marker in every case, so its windows reach the null group where rootless rows and null-ranked root rows tie; a reconstruction check samples exactly the cases its fixed-seed run executes and requires such windows (19 rootless and 7 mixed null-group windows of 40 at one times the run count), and the Effect runs beside the Collection. Full joins read their full source, so only the rows law covers them; paging them under the old join rule fails the window law. Pinned cases cover a null-ranked root row past a prefix that ends in the null group. Paging a nulls-last full join passed this oracle but was not adopted: it cost more per delete when the window reached the null group, which no work law here measures, and #2130 tracks full-join paging. Joined filters can match a row through several markers, and the model keeps one result row per matched pair; a model that collapses them fails. The eager harness also runs an on-demand source whose provider follows the LoadSubsetOptions contract (offset only without a cursor, the applied receipt returned). Its test provider combines a sync adapter and its provider (glossary model alignment rule 2); a held request is a physical acquisition whose provider answer is delayed, a served range is the prefix and boundary one answer establishes (a model abstraction), and the peer is a second live-query Collection. The provider sorts by the requested order and keeps its own ledger of sent rows. A forwarding provider sends live changes only for rows it sent or rows that match a served subset, and its served rows obey the work bound; a loader that asks for 100 extra rows fails it. A refetching provider, like query-db-collection, sends nothing until a refetch. The refetch keeps every served ordered range whole, as the LoadSubset contract requires: it sends every row from the start of the order through the last order value a limited page returned, and rows in no range leave; the rows law over the model applies once no change is pending. After every step on either provider, the Collection must hold exactly the rows in the provider's ledger, and the query and the Effect must show their window over those rows; a mutant that drops marker removals fails this law at steps 0 and 1 of the fixed refetching campaigns, earlier than the model's rows law, and in one more campaign. A holding provider answers each load only at an `answer` step, from the rows at that moment, so live changes arrive while loads are in flight. While a load is in flight, the query and its peer either keep the window they showed at the previous checkpoint or show distinct ledger rows that pass the query, in its order; the Effect is not checked then. That is weaker than the architecture's barrier rule, under which a repair or window move keeps the last public snapshot until it settles: a barrier law fails today on on-demand joined plans, where a visible-delete repair and a held `setWindow` publish before they settle (#2131; the law and traces are kept for that fix). The provider records a request as served only once it answers it, and an abort (or an adapter unload) of a held request follows `LoadSubsetOptions.signal`: a cooperative provider installs none of its rows and its load fulfills at once, while a provider that cannot cancel in-flight work (`late` cancellation, generated) answers it anyway and its load fulfills after those rows become visible. A `release-peer` step cleans up the peer while its loads are held, which aborts them; each request counts once although its acquisition release both aborts and unloads it. The fixed held campaign reaches 5 aborted held requests and 1 late answer, and its reconstruction check requires at least 3 cooperative cancellations and 1 late answer. A held load rejects when its applied receipt rejects, and the step then fails, because no signal or cleanup rejects a receipt in these histories; that branch is unwitnessed. The held campaign pins a forwarding provider, so `release-peer` never reaches a refetching provider; that branch is unwitnessed. A request whose signal is already aborted when the provider receives it is never reached; that branch is unwitnessed. The held campaign uses a forwarding provider and an indexed source filter or a joined filter, deletes three visible rows and then upserts an eligible row with an id no other command uses; its reconstruction check runs in the same test as the fixed-seed campaign and reads its per-case counts: 35 of 40 cases issue a held request, and every one of them changes rows while it is in flight. Loading must reach a fixed point: past a request budget the provider rejects and records it, and the initial load, every step and the end of each history assert the record is clear (a loader that re-requests a prefix it already holds can loop without reaching the provider, which this cannot see). A bounded step makes at most four provider requests, and each consumer's initial proven-prefix load is bounded separately, in total transfers (a repeated request for the same prefix counts) and in distinct rows (at most the source's size); a loader that asks for 100 extra rows on every page fails that bound and the initial-load work law. Loader mutants that never resume after a boundary request, or that ignore a load request while another load is pending, fail it; a mutant that drops a window move blocked behind a pending load still survives, even with the settlement check on held moves; whether it is equivalent is open. Open (#2148): on an on-demand source with a joined or anti-joined filter, an optimistic delete of the visible row leaves the window short until the delete persists, because the next row is never loaded. On an on-demand source with a source filter, and on an eager source with any filter, the same delete refills the window correctly; these are the contrast cases. Owner: this oracle. Needed witness: the optimistic-delete dimension (kept as a patch for #2148) in the grammar, plus a pinned replay of the anti-join trace. A refetch that re-runs each page alone, as query-db-collection does today, leaves a row that moved between two pages unloaded; random seeds found that trace, a named replay fails on it, and a query-db receiving witness reproduces it (#2136 tracks the provider fix). An Electric-style provider, which skips updates to rows it never sent, is not modeled; a mocked-stream witness shows a row moving into a served ordered range never arriving under the default replica mode (#2138). At one times the run count, the campaigns reach 87 refetch steps: 39 change the window, 6 remove a visible row, and 13 lead to a further provider request. A random seed found a refetch that empties a joined window, after which the loader refills it over several page requests and publishes `[]`, one row, then the final rows; a named replay keeps that trace for #2131. No incorrect final window has appeared on that path, and no loader mutant tried so far is caught only by it; the work law does not cover it. For an on-demand joined or anti-joined filter, the initial load must not request the whole source when the first window's worth of source rows in query order all pass the join and no boundary value is null or NaN; 35 executed cases meet that premise, and a compiler that reads the whole source for every joined filter fails eight tests on this law alone. The rows served by that initial load stay within the prefix, its boundary ties and a refill, for each consumer. Later steps of those plans have no work bound. An unindexed join filter over it left the window empty on main, until the plan counted the rows it still needs; restoring the old guard fails the campaigns and a pinned replay. A step that changes the visible rows must publish, and its subscriber starts with every visible key, so a removal-only change reaches it; mutants that drop update-only or removal-only publications fail. A step may publish one intermediate window per provider request it made, and only its last publication must show the final window, because a repair or refill can publish after each round trip (#2131); the content of those intermediate windows is unchecked. The model keeps its own copy of the markers, and full-join histories can insert the root rows of orphan markers. A premise-directed campaign inserts a fresh row and then changes an initial row's eligibility alone, so the row leaves the window only through the join; it and a named replay fail when a window row past the loader's settled boundary counts toward a full window. One case in four adds a second live query over the same sources with a larger window, and its rows law holds at every checkpoint; the work laws do not apply to those cases. #2155: a second live-query Collection over a refetching on-demand provider kept a short joined window after a refetch removed a tied row (the peer showed ranks `[0, 0]` where the model shows `[0, 0, 100]`). The loader fix that resumes a staged continuation after a synchronous answer the graph already processed closed that trace. The pinned replay of that indexed trace now expects the right window. Still open: an unindexed peer over a refetching provider keeps a short joined window (the peer shows `[100]` where the model shows `[100, 101]`), which also fails without that fix; each campaign therefore still excludes the peer and refetching pairing, as its last map. Owner: this oracle. Needed witness: a pinned replay of the unindexed trace flipped to pass, and the pairing restored to the grammar. Limits: Effects are checked for rows after each step, not for publication counts or positions. Histories move the first query's window with `setWindow` (150 moves at one times the run count, 56 on demand), and its rows law and ledger law then follow the moved window; the Effect and the peer keep theirs. On a holding provider every window-move promise must have fulfilled after the final drain. A refetching provider models query-db's unload: it drops the unloaded request and removes rows that no remaining request's last answer holds. The per-step work law applies only to forwarding, immediate, single-consumer cases: it skips refetching, holding and peer cases and move steps; a two-term indexed move scanned the source four times, which no law here bounds. Multi-term joined plans stay outside this harness, on both paths; on the on-demand path, the provider's transfer is bounded only for indexed single-term source-predicate plans. Joined windows compare order values, so a wrong row tied with the expected one passes; a window row tied with the boundary counts as proven before its tie request settles, which no witness exercises. A boundary that cannot advance past a window row would repeat one page per new key; reading the boundary's range locally should always reach that row, but no witness shows it. Joined ties are compared as any valid first rows; whether source-filter ties may be fixed to the source key is open. The same grammar includes NaN and null ranks and a second-order term, and bounds both rows read and rows delivered, including the initial load and index refills; a scan law limits each change to one pass over the source. A pinned case checks that an indexed descending window places nulls first, and a lifecycle witness checks that an eager full-source window is ready with its rows at creation when its source answers synchronously. The initial fallback cells check indexed and unindexed locale strings with distinct and tied values, plus a null boundary after a warm ordered request. The warm null-boundary case crosses preload-started and constructor-started live-query sync runs: the former completes at the synchronous cut, while the latter resumes deferred on-demand acquisition asynchronously and publishes after settlement. They require an unfiltered full-source request, no public row at fallback entry, and the expected row and status at the immediate readiness check. A Promise fallback stays loading at that checkpoint. A React receiving witness checks an eager locale-string `findOne()` at the first layout commit. A Query DB receiving witness checks warm cached subsets: both raw adapter calls return literal `true` with the source rows visible at each return, and the live query is immediately ready without a fetch. The same fallback cells observe subscriber changes from before the first source write: literal-true and Promise fallbacks each publish one nonempty final-window insert after the fallback settles, and the callback sees the complete row. Promise cells show no nonempty event at the immediate cut; a duplicate-event fault fails all eight cells. Other adapters and a real on-demand React first commit remain unwitnessed. Full-source work after initial publication and repair remain outside this synchronous guarantee. A separate post-publication oracle crosses marker deletion and retargeting with literal, Promise, and held-applied-receipt fallback responses. It also settles the child demand before holding the receipt, so the public live-query loading flag isolates the root fallback. The model checks final rows and net events, while event replay checks every callback snapshot; a retained or empty intermediate window is allowed. A loader fault that ignores the pending receipt fails at the held loading-flag check. Two literal-true cleanup-failure histories require that the replacement still publishes and the adapter error reaches the public query status without making the unrelated marker commit throw; the provider's applied receipts are recorded and compared outside provider callbacks. The Promise case checks eventual coherence on an asynchronous callback path, without claiming a separate timing cut. Four held-Promise histories apply fallback writes, change another join marker before provider settlement, and compare each callback and the final winner; an injected stale extra publication fails the event comparison. Other marker actions and overlapping root-source changes remain open; the provider has no later write or rejection, so this schedule does not establish when the loader claims full-source authority. The reversed-index oracle checks every reversed read's order and filter work for both null placements. | | Indexed predicate filtering | `packages/db/tests/query/index-path-collision-oracle.test.ts` | A bounded cross product of numeric values checks exact public keys before and after adding a nested-field BTree index. Direct predicates vary argument order, bound inclusivity, reversed operands, and bound field. Public Collection subscription and live-query callbacks run with and without the index. Selected-field ordering compares a direct JavaScript sort with a public `$selected` callback; restoring the dotted-key proxy cache made that comparison fail. The original compound grouping failed four indexed cases while scan controls passed. The original callback proxy caches also failed both unindexed public routes. A hostile cleanup control preserves the primary mismatch and secondary release error. Arbitrary path segments, nullish values, custom collation, and incremental publication need separate witnesses if claimed. | diff --git a/packages/db/tests/query/live-query-collection.test.ts b/packages/db/tests/query/live-query-collection.test.ts index 521e15c55d..88957bb9fb 100644 --- a/packages/db/tests/query/live-query-collection.test.ts +++ b/packages/db/tests/query/live-query-collection.test.ts @@ -5183,3 +5183,85 @@ describe(`createLiveQueryCollection`, () => { }) }) }) + +type TruncateWorkRow = { id: number; rank: number } +const truncateWorkRows: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, +] + +describe(`truncate replay work for an ordered window`, () => { + // A truncate reacquires the demand its subscriptions hold (ARCHITECTURE.md, + // "Replay participants and failure"). A window that holds no demand, such + // as a zero-limit window, owes no provider request and no loading state at + // a truncate. A query subscription still opens a replay then, so demand + // its graph starts during the commit belongs to that replay; the replay + // itself starts no recovery work. The one-row window is the neighbour: its + // truncate must reacquire, so a law that issued nothing would fail there. + it.each([0, 1])( + `reacquires only held demand when a %s-row window truncates`, + async (limit) => { + let sync!: Parameters[`sync`]>[0] + const requests: Array = [] + const source = createCollection({ + id: `ordered-truncate-work-${limit}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + sync.begin() + for (const value of truncateWorkRows) { + sync.write({ + type: source.has(value.id) ? `update` : `insert`, + value, + }) + } + sync.commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(limit), + ) + let observation: + { replayRequests: number; loadingAtCommit: boolean } | undefined + try { + await live.preload() + const before = requests.length + sync.begin() + sync.truncate() + sync.commit() + const loadingAtCommit = live.isLoadingSubset + await flushPromises() + observation = { + replayRequests: requests.length - before, + loadingAtCommit, + } + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + if (limit === 0) { + expect(observation).toEqual({ + replayRequests: 0, + loadingAtCommit: false, + }) + } else { + expect(observation.replayRequests).toBeGreaterThan(0) + } + }, + ) +}) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 4859b6302d..b5eb591291 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1317,7 +1317,6 @@ type OrderedReach = { innerJoinedRuns: number markerLoads: number markerTruncates: number - abortedMoves: number terminalErrors: number replayLoads: number abortRejections: number @@ -1386,15 +1385,16 @@ async function checkOrderedReplay( ) // With a joined source, only marked rows qualify; row 2 starts unmarked. const marked = new Set([1, 3, 4, 5]) + const eligible = (id: number) => + !scenario.joined || + (scenario.joined === `none` ? !marked.has(id) : marked.has(id)) // Ties leave the order among equal ranks open, so the model predicts the - // window's rank sequence. Each shown row must also carry its current rank. + // window's rank sequence. Each shown row must also carry its current rank, + // be eligible, and appear once, so a tie cannot hide a row the filter + // rejects. const expectedRanks = (shown: { offset: number; limit: number }) => [...truth.values()] - .filter( - ({ id }) => - !scenario.joined || - (scenario.joined === `none` ? !marked.has(id) : marked.has(id)), - ) + .filter(({ id }) => eligible(id)) .map(({ rank }) => rank) .sort((left, right) => sign * (left - right)) .slice(shown.offset, shown.offset + shown.limit) @@ -1411,7 +1411,12 @@ async function checkOrderedReplay( expect(row.rank, `${label}: row ${row.id} rank`).toBe( truth.get(row.id)?.rank, ) + expect(eligible(row.id), `${label}: row ${row.id} is eligible`).toBe(true) } + expect( + new Set(rows.map(({ id }) => id)).size, + `${label}: each row once`, + ).toBe(rows.length) if (new Set(rows.map(({ rank }) => rank)).size < rows.length) { reach.tiedWindows++ } @@ -1443,12 +1448,16 @@ async function checkOrderedReplay( own: Set } const barriers = new Map() + // Sources whose failed replay lost a failure to a retired demand in this + // episode. + const retiredFailures = new Set() // The rows the query showed when the first barrier rose, if it had // published by then. Before the first ready there is no complete // publication to keep, and the initial-readiness law governs. let retainedRows: string | undefined let retainedAfterReady = false const raiseBarrier = (source: BarrierSource) => { + retiredFailures.delete(source) lowerSettledBarriers() // A newer truncate discards the replay that a pending window move waits // behind. @@ -1530,6 +1539,9 @@ async function checkOrderedReplay( if (held === options) owned.delete(load) } } + // The replay may still hold its failure through the subscription's other + // demands, which the adapter cannot attribute (see Limits). + if (barrier.failures.size < failures) retiredFailures.add(source) // One failed demand retires while another still holds the barrier. if (barrier.failures.size > 0 && barrier.failures.size < failures) { reach.partialFailureRetirements++ @@ -1556,9 +1568,13 @@ async function checkOrderedReplay( let markerTruncates = 0 let demandFailed = false let markerThrowArmed = false - let loadThrew = false - // The adapter sets the flag, so read it through a call. - const threw = () => loadThrew + // The history step running now, the first step where a marker load threw, + // and the first step after which the query was in error. + let currentStep = 0 + let thrownAt: number | undefined + let errorSeenAt: number | undefined + // Errors that marker loads threw while they started. + const thrownErrors = new WeakSet() // A failed initial load is fatal too (error 229). The queries are created // below; a load can only fail after they start. const queries: Array<{ status: string }> = [] @@ -1568,7 +1584,9 @@ async function checkOrderedReplay( // threw, may fall outside the replay the model still holds open, so it may // end the query (see Limits). let ambiguousFailure = false - // A failed demand whose recovery the model cannot place (see Limits). + // A failed demand whose recovery the model cannot place: with a peer, a + // load that joined a replay; or a load outside every barrier after a failed + // demand of its source retired (see Limits). let unplacedFailure = false // A failed lazy demand that belongs to no recovery, which must end the // query. Adapter callbacks set the flags, so read them through a call. @@ -1652,7 +1670,6 @@ async function checkOrderedReplay( innerJoinedRuns: scenario.joined === `inner` ? 1 : 0, markerLoads: 0, markerTruncates: 0, - abortedMoves: 0, terminalErrors: 0, replayLoads: 0, abortRejections: 0, @@ -1703,7 +1720,7 @@ async function checkOrderedReplay( loadSubset: (options) => { if (markerThrowArmed) { markerThrowArmed = false - loadThrew = true + thrownAt ??= currentStep const load: Load = { epoch: truncates, outcome: `failed` } loads.push(load) reach.markerLoads++ @@ -1733,7 +1750,12 @@ async function checkOrderedReplay( // of its episode (see Limits). if (open) open.ambiguous = true anyLoadFailed = true - if (!recovery && isDemand(options)) demandFailed = true + if (!recovery && isDemand(options)) { + demandFailed = true + if (!participant && retiredFailures.has(`markers`)) { + unplacedFailure = true + } + } // A throw inside a replay that is only loading, which the // recovery rule judges strictly. if (recovery && strict && isDemand(options)) { @@ -1743,7 +1765,9 @@ async function checkOrderedReplay( failedWhileLoading = true } reach.failedLoads++ - throw inject(new Error(`marker load threw`)) + const thrown = inject(new Error(`marker load threw`)) + thrownErrors.add(thrown) + throw thrown } return (async () => { acquired.set(options, (acquired.get(options) ?? 0) + 1) @@ -1786,7 +1810,11 @@ async function checkOrderedReplay( demandFailed = true // With a peer, a load that joined a replay may belong to either // query's recovery, so the model cannot place it. - if (participant && scenario.peer) unplacedFailure = true + if ( + (participant && scenario.peer) || + (!participant && retiredFailures.has(`markers`)) + ) + unplacedFailure = true } // A demand the graph started after the commit fails inside // the replay. @@ -1910,7 +1938,11 @@ async function checkOrderedReplay( demandFailed = true // With a peer, a load that joined a replay may belong to either // query's recovery, so the model cannot place it. - if (participant && scenario.peer) unplacedFailure = true + if ( + (participant && scenario.peer) || + (!participant && retiredFailures.has(`rows`)) + ) + unplacedFailure = true } // A demand the graph started after the commit, failing // while its replay still loads. @@ -2162,11 +2194,8 @@ async function checkOrderedReplay( ) try { if (!scenario.historyDuringPreload) await preload - for (const command of history) { - // A later command can retire a thrown demand, which the adapter never - // sees, and so retire its failed replay; demand that waited on that - // replay may then fail outside it (see Limits). - if (threw()) ambiguousFailure = true + for (const [step, command] of history.entries()) { + currentStep = step try { switch (command.type) { case `move`: { @@ -2279,12 +2308,11 @@ async function checkOrderedReplay( if (error instanceof Error && cleanupFaults.has(error)) { escaped ??= error } - // A truncate aborts the load a move waits on (ARCHITECTURE.md); - // the move then rejects and the window stays where it was. - if (error instanceof Error && error.name === `AbortError`) { - reach.abortedMoves++ - return - } + // A truncate never aborts a move: the move waits for any + // replay that holds publication and then applies + // (ARCHITECTURE.md). Only cleanup rejects a waiting move with + // `AbortError`, and cleanup runs after the checks, so any + // other rejection here is an injected fault or a broken law. swallow(error) }, ) @@ -2299,6 +2327,9 @@ async function checkOrderedReplay( } await flushPromises() checkRetention(`retained publication after ${command.type}`) + if (errorSeenAt === undefined && live.status === `error`) { + errorSeenAt = step + } } holding = false throwNextUnload = false @@ -2375,7 +2406,21 @@ async function checkOrderedReplay( ...(peer ? [[peer, `peer convergence`, peerWindow] as const] : []), ] as const) { if (shown.status === `error`) { - expect(fatalLoadFailed(), `${label}: terminal error`).toBe(true) + // A later command can retire a thrown demand, which the adapter never + // sees, and so retire its failed replay; a demand that waited on that + // replay may then end the query with the thrown error (see Limits). + // Only that error, ending the query after the step that threw, is + // excused this way; the throw's own step is judged strictly. + const cause = shown._lifecycle.getSyncError() + const endedAfterThrow = + thrownAt !== undefined && (errorSeenAt ?? history.length) > thrownAt + expect( + fatalLoadFailed() || + (endedAfterThrow && + cause instanceof Error && + thrownErrors.has(cause)), + `${label}: terminal error`, + ).toBe(true) reach.terminalErrors++ } else checkWindow(label, shown.toArray, expected) } @@ -2912,7 +2957,6 @@ describe(`Ordered windows across truncate replays`, () => { innerJoinedRuns: 0, markerLoads: 0, markerTruncates: 0, - abortedMoves: 0, terminalErrors: 0, replayLoads: 0, abortRejections: 0, @@ -2948,14 +2992,12 @@ describe(`Ordered windows across truncate replays`, () => { // starts, and while it settles; loads fail and stay held across a // truncate; truncates ask again for held demand; and the settled-window // law is checked. - // An aborted move needs a truncate to land inside a held move, and a - // terminal error needs a held marker demand load to fail; the named - // replays above reach both. A failed replay that the retention law can + // A terminal error needs a held marker demand load to fail; the named + // replays above reach it. A failed replay that the retention law can // judge, a demand that fails while its replay is open, a replay that // fails before the first ready, and a window move behind a discarded // replay are rare here; the campaigns below declare them. const ownedElsewhere = new Set([ - `abortedMoves`, `terminalErrors`, `retainedFailedReplays`, `participantDemandFailures`, diff --git a/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts index b32a5b55ea..281ffd4788 100644 --- a/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts @@ -3353,84 +3353,3 @@ describe(`synchronous initial full-source settlement refinement`, () => { }, ) }) - -describe(`truncate replay work for an ordered window`, () => { - // A truncate reacquires the demand its subscriptions hold (ARCHITECTURE.md, - // "Replay participants and failure"). A window that holds no demand, such - // as a zero-limit window, owes no provider request and no loading state at - // a truncate. A query subscription still opens a replay then, so demand - // its graph starts during the commit belongs to that replay; the replay - // itself starts no recovery work. The one-row window is the neighbour: its - // truncate must reacquire, so a law that issued nothing would fail there. - it.each([0, 1])( - `reacquires only held demand when a %s-row window truncates`, - async (limit) => { - let sync!: Parameters[`sync`]>[0] - const requests: Array = [] - const source = createCollection({ - id: `ordered-truncate-work-${limit}`, - getKey: ({ id }) => id, - syncMode: `on-demand`, - autoIndex: `eager`, - defaultIndexType: BTreeIndex, - sync: { - sync: (operations) => { - sync = operations - operations.markReady() - return { - loadSubset: (options) => { - requests.push(options) - sync.begin() - for (const value of fullSourceRows) { - sync.write({ - type: source.has(value.id) ? `update` : `insert`, - value, - }) - } - sync.commit() - return true - }, - } - }, - }, - }) - const live = createLiveQueryCollection((q) => - q - .from({ row: source }) - .orderBy(({ row }) => row.rank) - .limit(limit), - ) - let primaryFailure: CapturedOracleFailure | undefined - let observation: - { replayRequests: number; loadingAtCommit: boolean } | undefined - try { - await live.preload() - const before = requests.length - sync.begin() - sync.truncate() - sync.commit() - const loadingAtCommit = live.isLoadingSubset - await flushPromises() - observation = { - replayRequests: requests.length - before, - loadingAtCommit, - } - } catch (error) { - primaryFailure = { error } - } - await finishOracleCleanup( - primaryFailure, - [() => live.cleanup(), () => source.cleanup()], - `Ordered truncate work oracle cleanup failed`, - ) - if (limit === 0) { - expect(observation).toEqual({ - replayRequests: 0, - loadingAtCommit: false, - }) - } else { - expect(observation!.replayRequests).toBeGreaterThan(0) - } - }, - ) -}) diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 20bf075465..59ef724246 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -2686,17 +2686,27 @@ async function checkAuthorityCase( command.type === `joinedIdle` // An unproven report whose pass pending joined work deferred: the ledger // index at the report and the window size it judged. - let owedProof: { from: number; endRank: unknown } | undefined - const expectProofRequest = (from: number, endRank: unknown) => { + let owedProof: + { from: number; endRank: unknown; expressible: boolean } | undefined + // The proof is a bounded repair or the tie of the page's end. With every + // installed rank expressible, a full-source read is not a repair: the + // bounded repair is available (ARCHITECTURE.md), and a full-source read is + // the cost the report path avoids. + const expectProofRequest = ( + from: number, + endRank: unknown, + expressible: boolean, + ) => { const issued = ledger .slice(from) .some( - ({ authoritative, options }) => - authoritative || tieValue(options) === endRank, + ({ authoritative, fullSource, options }) => + (authoritative && (!fullSource || !expressible)) || + tieValue(options) === endRank, ) if (!issued) { throw new Error( - `unproven report: a report over a first page that no row it read proves issued no repair and no tie of its end`, + `unproven report: a report over a first page that no row it read proves issued no bounded repair and no tie of its end`, ) } } @@ -3001,6 +3011,7 @@ async function checkAuthorityCase( // The page's end is the installed row at the window's last position. const ranks = snapshot().map(({ value }) => value.rank) const endRank = ranks[windowSize - 1] + const expressible = ranks.every((rank) => Number.isFinite(rank)) // The loader's coverage is its own first page with only tie // requests after it. const ownFirstPage = @@ -3099,9 +3110,9 @@ async function checkAuthorityCase( !insideFirstPage ) { if (joinedPending && testCase.joined) { - owedProof = { from: before, endRank } + owedProof = { from: before, endRank, expressible } } else { - expectProofRequest(before, endRank) + expectProofRequest(before, endRank, expressible) repairedReportsOnOwnPage++ } } @@ -3197,7 +3208,11 @@ async function checkAuthorityCase( graphPass() await flush() if (owedProof) { - expectProofRequest(owedProof.from, owedProof.endRank) + expectProofRequest( + owedProof.from, + owedProof.endRank, + owedProof.expressible, + ) owedProof = undefined repairedReportsOnOwnPage++ } @@ -3458,6 +3473,25 @@ async function checkAuthorityCase( `cost: one invalidation issued ${authoritative} authoritative requests for ${chains} epoch moves`, ) } + // Each boundary is a different read: no chain reads the same + // prefix, tie, or page twice. + const reads = new Map() + for (const { authoritative: owed, options } of repair) { + if (!owed) continue + const read = JSON.stringify([ + options.limit, + tieValue(options), + options.minValues, + ]) + reads.set(read, (reads.get(read) ?? 0) + 1) + } + for (const [read, count] of reads) { + if (count > chains) { + throw new Error( + `cost: one invalidation repeated the read ${read} ${count} times for ${chains} epoch moves`, + ) + } + } } } } diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 293e8249f9..7f6e869d79 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -6480,6 +6480,8 @@ type OptimisticWindowObservation = { submittedRows: Array limit: number | undefined direction: string | undefined + // The request had a limit or a predicate, so it read no full source. + bounded?: boolean appliedReceiptPending: boolean providerPending: boolean }> @@ -6627,6 +6629,7 @@ async function checkOptimisticWindow( submittedRows: Array limit: number | undefined direction: string | undefined + bounded: boolean appliedReceiptSettled: boolean providerSettled: boolean provider: Promise @@ -6676,6 +6679,8 @@ async function checkOptimisticWindow( submittedRows: [] as Array, limit: options.limit, direction: options.orderBy?.[0]?.compareOptions.direction, + bounded: + options.limit !== undefined || options.where !== undefined, appliedReceiptSettled: false, providerSettled: false, provider: Promise.resolve(), @@ -6985,6 +6990,7 @@ async function checkOptimisticWindow( submittedRows: [...attempt.submittedRows], limit: attempt.limit, direction: attempt.direction, + bounded: attempt.bounded, appliedReceiptPending: !attempt.appliedReceiptSettled, providerPending: !attempt.providerSettled, })) @@ -7020,6 +7026,7 @@ async function checkOptimisticWindow( submittedRows: [...attempt.submittedRows], limit: attempt.limit, direction: attempt.direction, + bounded: attempt.bounded, appliedReceiptPending: !attempt.appliedReceiptSettled, providerPending: !attempt.providerSettled, })) @@ -7204,6 +7211,22 @@ function optimisticWindowViolations( if (testCase.acquisition === `forbidden` && observation.attempts.length > 0) { violations.push(`refill acquisition started with complete local extent`) } + // A refill the case allows but does not require is still bounded: an + // ordered read within the refill bound, or a predicate read, never the + // full source. + if (testCase.acquisition === `allowed`) { + for (const attempt of observation.attempts) { + if ( + attempt.bounded === false || + (attempt.direction !== undefined && + (attempt.limit === undefined || + attempt.limit > refillBound(testCase))) + ) { + violations.push(`an allowed refill used an unbounded provider request`) + break + } + } + } const requested = new Set() for (const request of observation.requestHistory ?? []) { if (request.phase === `pending` && testCase.indexed) { diff --git a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts index 45ca95a214..5033f02df9 100644 --- a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts +++ b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts @@ -11,6 +11,7 @@ import { } from '@tanstack/db' import fc from 'fast-check' import { describe, expect, it, vi } from 'vitest' +import { createOracleReach } from '../../db/tests/oracle-reach.js' import { oraclePropertyOptions, oracleReplayRequested, @@ -809,6 +810,10 @@ type Observation = { // Such a refetch that also reported, so a loader may have judged the skip // while the refetched rows were held back. reachedReportDuringHold: boolean + // A refetch that reported while the window was full: the loader then + // judges whether its own first page still proves the window, and repairs + // when a moved end leaves it unproven. + reachedReportOverFullWindow: boolean // Reports over the whole run. reports: number } @@ -837,6 +842,7 @@ async function checkCase(testCase: Case): Promise { let reachedLaggingRefetchAfterContinuation = false let reachedLaggingRefetchDuringHold = false let reachedReportDuringHold = false + let reachedReportOverFullWindow = false let primary: unknown const check = (checkpoint: string) => { const expected = expectedWindow(run.backend, run.markers, testCase.scenario) @@ -899,6 +905,13 @@ async function checkCase(testCase: Case): Promise { ) { reachedReportDuringHold = true } + if ( + command.type === `refetch` && + run.reports() > reportsBefore && + run.windowIds().length === testCase.scenario.limit + ) { + reachedReportOverFullWindow = true + } checkReports( command, run.reports() - reportsBefore, @@ -995,6 +1008,7 @@ async function checkCase(testCase: Case): Promise { reachedLaggingRefetchAfterContinuation, reachedLaggingRefetchDuringHold, reachedReportDuringHold, + reachedReportOverFullWindow, reports: run.reports(), } } @@ -1307,14 +1321,12 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => // The ordinary insert into a top-N list. The page's end moves earlier, so // Query DB reports, but every window row is in the window's own first page. // The tie the loader read belongs to the old end, so its rows prove - // nothing about rows that tie with the new end, and the loader repairs: - // the refetch sends the Queries it refetches, then the bounded repair's - // ordered prefix and the new end's tie, and nothing unbounded. The repair - // retires the requests it replaces, so the Query count stays the same. - // (Maintainer decision: the repair replaces a cheaper tie-only read, which - // cost shipped bytes for one saved request per such refetch.) + // nothing about rows that tie with the new end. The loader proves the + // window again with bounded reads (a repair, or the new end's tie; the + // shape is the loader's choice), reads nothing unbounded, and retires the + // requests it replaces, so the Query count stays the same. it.each([1, 2] as const)( - `a refetch that moves a first page's end repairs with a bounded prefix and the new end's tie (%i order terms)`, + `a refetch that moves a first page's end proves the window again with bounded reads (%i order terms)`, async (terms) => { const scenario: Scenario = { encoding: `cursor`, @@ -1342,16 +1354,14 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => expectedWindow(run.backend, run.markers, scenario), ) const extra = run.requests.slice(requestsBefore + queries) - expect(extra).toHaveLength(2) - // The repair's prefix reads the window's rows from the start of the - // order, bounded by the window, with no cursor. - expect(extra[0]!.orderBy).toBeDefined() - expect(extra[0]!.limit).toBe(scenario.offset + scenario.limit) - expect(extra[0]!.cursor).toBeUndefined() - // A tie request reads equal leading values, with no order or cursor. - expect(extra[1]!.orderBy).toBeUndefined() - expect(extra[1]!.cursor).toBeUndefined() - expect(extra[1]!.where).toBeDefined() + expect(extra.length).toBeGreaterThan(0) + // Every read is bounded: an ordered prefix with a limit, or a tie + // that reads equal leading values. None reads the full source. + for (const request of extra) { + expect( + request.limit !== undefined || request.where !== undefined, + ).toBe(true) + } expect(run.activeQueries()).toBe(activeBefore) } catch (error) { primary = error @@ -2009,24 +2019,29 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => it.each(campaigns)( `generated refetch histories (%s campaign)`, async (campaign) => { - let reached = 0 - let reachedDuringHold = 0 + // Declared reach: generated histories serve continuation pages before + // a refetch of a lagging provider (the premise of the gap), do so while + // a hold runs, and report a refetch while the window is full, where the + // loader decides between ignoring the report and repairing. + const states = [ + `reachedLaggingRefetchAfterContinuation`, + `reachedLaggingRefetchDuringHold`, + `reachedReportOverFullWindow`, + ] as const + const reach = createOracleReach(property, states, { minimum: 3 }) await fc.assert( fc.asyncProperty(caseArbitrary, async (testCase) => { + reach.beginHistory() const observation = await withDeadline(checkCase(testCase)) - if (observation.reachedLaggingRefetchAfterContinuation) reached++ - if (observation.reachedLaggingRefetchDuringHold) reachedDuringHold++ + for (const state of states) { + if (observation[state]) reach.reach(state) + } }), campaign === `fixed` ? { seed: 20261013, numRuns: oracleRuns(80) } : oraclePropertyOptions(80, property), ) - // Positive reach witness: generated histories serve continuation pages - // before a refetch, which is the premise of the gap. - if (campaign !== `replay`) { - expect(reached).toBeGreaterThan(0) - expect(reachedDuringHold).toBeGreaterThan(0) - } + reach.assertReached() }, 120_000, ) From c0ffcb858b82e53ab5cc452a0e79c5f7f825585a Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 08:24:47 -0600 Subject: [PATCH 34/40] test(db): judge request errors and per-query initial failures in the replay oracle A high review of the #2168 merge flagged the replay oracle's law-16 rewrite: it had no positive request-error assertion, no check that request errors clear after recovery, and a global initial-failure excuse that a peer could satisfy for `live`. Law (ARCHITECTURE.md law 16 and law 17; glossary, "request error"): after initial-query readiness, a failed lazy demand load is a request error on the query's own node until its demand retires, a later load of the source covers its keys, or a later replay of the source completes. A failure inside a replay is decided by that replay's verdict, once every participant settled. A terminal error needs a load that failed while that same query was loading. Oracle changes: - A request-error account, independent of production's bookkeeping, owes each such failure. At every history cut, the query's node must list each owed failure by its injected error object; after recovery replays every source, no request error may stay. - Initial failures are recorded per query. - A thrown load joins a replay that no participant joined yet (a replay still starting its loads holds publication); without this the model booked a recovery that had not happened. - The failed-sibling branch generates a held retry, and a new branch starts another demand of the source while a failed demand's retry is held. Reach: `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. RED: the review's mutant classes. Demand outcomes that never report an error fail 9 tests ("after settle: request error for "marker load failed" on keys 1"); a successful replay that keeps its request errors fails 22 ("convergence: stale request error"). GREEN: production unchanged; replay oracle 53/53 at 1x and 3x budget; @tanstack/db 300 files, 15,528 tests. Limits: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. The reviewer's failReplacement finding (its replay can complete when a joined demand settles, then clear the failed demand's request error) stays open in the coverage map; a trace confirms the flush, but no reached history showed a cleared owed error. Which failure a deferred move rejects with, when its replay has several, is not specified and awaits a decision. Also: - ARCHITECTURE.md now states what the deferred move compares: the sync runs, only when a replay it waits on rejects. - setReadyIfIdle uses hasTruncateReplayWork, and siblingReplays is required again (-14 B gzip; @tanstack/db +355 B vs main). Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/collection/subscription.ts | 8 +- packages/db/src/query/live/ARCHITECTURE.md | 7 +- packages/db/src/types.ts | 2 +- ...ad-subset-replay-refinement-oracle.test.ts | 304 ++++++++++++++++-- 5 files changed, 289 insertions(+), 34 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 29cce1f995..d2b931b48b 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -515,7 +515,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a trace confirms the flush, but no reached history showed a cleared error that was still owed (witness needed: a per-subscription verdict in the model). Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; this awaits a maintainer decision. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index f7e86c87d9..093af5553b 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -854,13 +854,9 @@ export class CollectionSubscription } private setReadyIfIdle(): void { - const replayState = this.truncateReplayState - const hasPendingReplayWork = - replayState && - (replayState.pendingSetups > 0 || replayState.pending.size > 0) if ( this.pendingLoadSubsetParticipants.size === 0 && - !hasPendingReplayWork + !this.hasTruncateReplayWork ) { this.setStatus(`ready`) } @@ -905,7 +901,7 @@ export class CollectionSubscription replayState: TruncateReplayState, finish: () => void, ): boolean { - const siblings = this.options.truncateReplayPublication?.siblingReplays?.() + const siblings = this.options.truncateReplayPublication?.siblingReplays() void siblings?.then(() => { if (this.truncateReplayState === replayState) finish() }) diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index cc93afa901..616c86b58c 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1354,8 +1354,11 @@ applies only after its replacement is complete. A failed replay rejects the move without advancing the reported window. A replay that a newer truncate or the retirement of its last demand discards did not fail, even though its completion rejects with `AbortError`: the move waits for any replay that still -holds publication and then applies. A reload of the query or of a source ends -the sync run the move waits in, and the move rejects. Replay completion callbacks carry +holds publication and then applies. When a replay it waits on is discarded and +the query's or a source's sync run changed meanwhile, as a reload changes it, +the move rejects with that replay's `AbortError`. The move compares the sync +runs only when a replay it waits on rejects, so a reload of another source +whose replay it does not wait on leaves the move to apply. Replay completion callbacks carry their sync-run identity and become no-ops after cleanup or restart. Cleanup rejects the replay barrier, and therefore every window move waiting on it, with `AbortError`; no waiter may outlive the discarded subscription. diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 4e18b7f1b5..a8b5c0ba58 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -1312,7 +1312,7 @@ export interface SubscribeChangesOptions< * The replays of the graph's other subscriptions to this collection that * still load. They share one barrier, so this replay waits for them. */ - readonly siblingReplays?: () => Promise | undefined + readonly siblingReplays: () => Promise | undefined } } diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 518c4af20d..7f9763113f 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1141,8 +1141,11 @@ describe(`loadSubset replay refinement`, () => { * it starts, as a synchronous adapter does, is a rejection: a throw inside * an open replay fails that replay. The throw happens while the * subscription registers the demand, so it joins a replay only if that - * replay still has a participant or a failure. So a query may end in - * `error` only after a load failed while the query was still loading. + * replay still has a participant, a failure, or no participant yet. So a + * query may end in `error` only after a load failed while that query was + * still loading. At each history cut, the query lists every request failure + * that the request-error account still owes, by its injected error object. + * After recovery replays every source, no request error stays. * - **Status:** with no failed load, a quiet live query and its peer are * `ready`; a release that throws does not change status. * - **Initial readiness:** when the history starts during the initial @@ -1188,8 +1191,13 @@ describe(`loadSubset replay refinement`, () => { * the marker barrier from a throw to the end of its episode (a barrier * raised while an earlier one in the episode is ambiguous inherits that). So * a design that dropped a thrown demand and let its replay succeed would pass - * wherever only retention could see it. The request-error witness reads the - * query's own health node; it does not check which request each error names. + * wherever only retention could see it. The request-error account shares the + * model's replay across a query's subscriptions to a source, so it owes a + * failure in a replay only after every load of that source settled; it does + * not judge one subscription's request error while another load is held. It + * reads only the query's own node and permits extra listed failures. Which + * failure a deferred window move rejects with, when its replay has several, + * is not judged. */ type OrderedFailureState = `initiating-keyed-request-failed` | `keyed-reacquisition-failed` @@ -1332,6 +1340,8 @@ type OrderedReach = { // A failed load after initial-query readiness left a request error on the // query's own node, and the query stayed ready (ARCHITECTURE.md law 16). requestErrors: number + // Cuts where the request-error account owed a request error. + requestErrorChecks: number replayLoads: number abortRejections: number preloadChecks: number @@ -1471,7 +1481,12 @@ async function checkOrderedReplay( const installed = new Map() // Each load records the truncate count it was issued after, so a fault can // be placed before or after the replay's loads settle. - type Load = { epoch: number; outcome: `pending` | `ok` | `failed` } + type Load = { + epoch: number + outcome: `pending` | `ok` | `failed` + // The error this history injected when the load failed. + error?: Error + } const loads: Array = [] // Retention model. Each source with an open or failed replay has a // barrier. `pending` holds the loads that still hold its publication: the @@ -1482,6 +1497,8 @@ async function checkOrderedReplay( type BarrierSource = `rows` | `markers` type Barrier = { failures: Map + // When the barrier rose, on the request-error account's clock. + raisedAt: number // A release threw while this replay started, which may fail it. ambiguous: boolean pending: Map @@ -1504,20 +1521,23 @@ async function checkOrderedReplay( retainedRows = JSON.stringify(live.toArray) retainedAfterReady = live.status === `ready` } - barriers.set(source, { + const barrier: Barrier = { failures: new Map(), + raisedAt: ++requestClock, // The retained publication dates from the first barrier, so an // ambiguous barrier earlier in this episode leaves it ambiguous. ambiguous: [...barriers.values()].some(({ ambiguous }) => ambiguous), pending: new Map(barriers.get(source)?.pending), own: new Set(), - }) + } + barriers.set(source, barrier) } // A replay whose participants all settled without a failure publishes. const lowerSettledBarriers = () => { for (const [source, barrier] of barriers) { if (barrier.failures.size === 0 && barrier.pending.size === 0) { barriers.delete(source) + recoveries.push({ source, raisedAt: barrier.raisedAt }) } } if (barriers.size === 0) retainedRows = undefined @@ -1603,7 +1623,13 @@ async function checkOrderedReplay( // A failed initial load is fatal too (error 229). The queries are created // below; a load can only fail after they start. const queries: Array<{ status: string }> = [] - let failedWhileLoading = false + // The queries that were still loading when a load failed: each may end in + // an initial load failure. + const failedWhileLoading = new Set<{ status: string }>() + const noteLoadingFailure = () => { + for (const query of queries) + if (query.status === `loading`) failedWhileLoading.add(query) + } // The driver observes each initiating keyed snapshot request's result // receipt and matches only the exact errors this history injected into // loads, so it uses no replay state, callback timing, or predicate @@ -1660,6 +1686,95 @@ async function checkOrderedReplay( options.where !== undefined && options.orderBy === undefined && !readsRank(options.where) + // Request-error account (ARCHITECTURE.md law 16; glossary, "request + // error"). After initial-query readiness, a failed lazy demand load is a + // request error on the query's own node. The error stays until one of these + // meets it: + // - its demand retires, which aborts its request and unloads it; + // - a later load of the same source covers its keys; + // - a replay of its source that rose after the failure completes, since a + // replay asks again for every held demand. + // A failure that joins a replay waits for that replay's verdict, which + // comes once every participant settled, so it owes no error before then. Keys come from evaluating the request's + // `where` over a small key domain, independent of production's bookkeeping. + // The account combines the production concepts of a failed request and the + // replay verdict that decides it; it names no production structure. + let requestClock = 0 + const keyDomain = Array.from({ length: 14 }, (_, index) => index - 2) + const requestedKeys = (source: BarrierSource, options: LoadSubsetOptions) => + new Set( + keyDomain.filter( + (id) => + !options.where || + evaluateReferenceExpression( + options.where, + source === `markers` ? { id, rowId: id } : { id, rank: 0 }, + ) === true, + ), + ) + type RequestFailure = { + // The failed load; its injected error is the failure the node lists. + load: Load + source: BarrierSource + options: LoadSubsetOptions + keys: Set + at: number + // The replay the load joined, whose verdict decides it. + barrier?: Barrier + } + const requestFailures: Array = [] + const loadedKeys: Array<{ + source: BarrierSource + keys: Set + at: number + }> = [] + const recoveries: Array<{ source: BarrierSource; raisedAt: number }> = [] + const unloaded = new WeakSet() + const allReady = () => queries.every(({ status }) => status === `ready`) + const recordLoaded = (source: BarrierSource, keys: Set) => + loadedKeys.push({ source, keys, at: ++requestClock }) + const recordRequestFailure = ( + load: Load, + source: BarrierSource, + options: LoadSubsetOptions, + startedReady: boolean, + participant: boolean, + ) => { + if (!startedReady || !allReady() || !isDemand(options)) return + if (options.signal?.aborted) return + requestFailures.push({ + load, + source, + options, + keys: requestedKeys(source, options), + at: ++requestClock, + barrier: participant ? barriers.get(source) : undefined, + }) + } + const requestErrorOwed = (failure: RequestFailure) => { + if (failure.options.signal?.aborted || unloaded.has(failure.options)) + return false + // A replay gives its verdict once every participant settled, failed or + // not (ARCHITECTURE.md law 17). + const { barrier } = failure + if ( + barrier && + barriers.get(failure.source) === barrier && + barrier.pending.size > 0 + ) + return false + const covered = loadedKeys.some( + (loaded) => + loaded.source === failure.source && + loaded.at > failure.at && + [...failure.keys].every((key) => loaded.keys.has(key)), + ) + const recovered = recoveries.some( + (recovery) => + recovery.source === failure.source && recovery.raisedAt > failure.at, + ) + return !covered && !recovered + } // The truncate count of its source when each request was first issued. A // replay asks again for a held demand with the same request data, because // request data is immutable (ARCHITECTURE.md), and a new demand builds a @@ -1732,6 +1847,7 @@ async function checkOrderedReplay( markerTruncates: 0, terminalErrors: 0, requestErrors: 0, + requestErrorChecks: 0, replayLoads: 0, abortRejections: 0, preloadChecks: 0, @@ -1789,11 +1905,15 @@ async function checkOrderedReplay( // The throw happens while the subscription registers the // demand, so the barrier here is the replay's state then. One // whose participants all retired or settled has completed, and - // the demand does not join it. + // the demand does not join it. A replay that no participant + // joined yet is still starting its loads, which hold + // publication (ARCHITECTURE.md), so a throw then joins it. const open = barriers.get(`markers`) const participant = open !== undefined && - (open.pending.size > 0 || open.failures.size > 0) && + (open.pending.size > 0 || + open.failures.size > 0 || + open.own.size === 0) && joinBarrier(`markers`, load, options) // A throw that joins a replay that is only loading. const strict = @@ -1812,15 +1932,15 @@ async function checkOrderedReplay( if (recovery && strict && isDemand(options)) { reach.thrownReplayDemands++ } - if (queries.some(({ status }) => status === `loading`)) { - failedWhileLoading = true - } + noteLoadingFailure() reach.failedLoads++ const thrown = inject(new Error(`marker load threw`)) + load.error = thrown injectedLoadFaults.add(thrown) throw thrown } return (async () => { + const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { epoch: truncates, outcome: `pending` } loads.push(load) @@ -1882,12 +2002,19 @@ async function checkOrderedReplay( } } if (queries.some(({ status }) => status === `loading`)) { - failedWhileLoading = true // Lazy demand in a replay failed before the first ready. if (participant && isDemand(options)) { reach.preloadReplayFailures++ } } + noteLoadingFailure() + recordRequestFailure( + load, + `markers`, + options, + startedReady, + participant, + ) if (!options.signal?.aborted) { if (newDemandDuringReplay) recordReach(`new-demand-failed-during-marker-replay`) @@ -1895,7 +2022,11 @@ async function checkOrderedReplay( recordReach(`settled-marker-demand-replay-failed`) } reach.failedLoads++ - throw injectLoad(new Error(`marker load failed`), options) + load.error = injectLoad( + new Error(`marker load failed`), + options, + ) + throw load.error } } else await Promise.resolve() if (options.signal?.aborted) { @@ -1926,11 +2057,13 @@ async function checkOrderedReplay( } await operations.commit() load.outcome = `ok` + recordLoaded(`markers`, requestedKeys(`markers`, options)) // The load settles once its rows apply. settleParticipant(`markers`, load, options, true) })() }, unloadSubset: (options) => { + unloaded.add(options) retireParticipants(`markers`, options) const count = acquired.get(options) ?? 0 if (count === 0) { @@ -1954,6 +2087,7 @@ async function checkOrderedReplay( sync.markReady() return { loadSubset: async (options) => { + const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) const load: Load = { epoch: truncates, outcome: `pending` } loads.push(load) @@ -1985,14 +2119,22 @@ async function checkOrderedReplay( reach.participantDemandFailures++ } if (queries.some(({ status }) => status === `loading`)) { - failedWhileLoading = true // Lazy demand in a replay failed before the first ready. if (participant && isDemand(options)) { reach.preloadReplayFailures++ } } + noteLoadingFailure() + recordRequestFailure( + load, + `rows`, + options, + startedReady, + participant, + ) reach.failedLoads++ - throw injectLoad(new Error(`load failed`), options) + load.error = injectLoad(new Error(`load failed`), options) + throw load.error } } else await Promise.resolve() if (options.signal?.aborted) { @@ -2037,10 +2179,19 @@ async function checkOrderedReplay( } await sync.commit() load.outcome = `ok` + // An ordered load covers the rows it selected; a keyed load, the + // keys it asked for. + recordLoaded( + `rows`, + options.orderBy + ? new Set(selected.map(({ id }) => id)) + : requestedKeys(`rows`, options), + ) // The load settles once its rows apply. settleParticipant(`rows`, load, options, true) }, unloadSubset: (options) => { + unloaded.add(options) retireParticipants(`rows`, options) const count = acquired.get(options) ?? 0 if (count === 0) { @@ -2121,8 +2272,28 @@ async function checkOrderedReplay( // After initial-query readiness no load failure ends the query // (ARCHITECTURE.md law 16), so a terminal error needs a load that failed // while the query was still loading: an initial load failure. - const checkTerminal = (label: string) => { - expect(failedWhileLoading, `${label}: terminal error`).toBe(true) + const checkTerminal = (label: string, shown: { status: string }) => { + expect(failedWhileLoading.has(shown), `${label}: terminal error`).toBe(true) + } + // At a cut, a query that has not ended lists each failure the account + // owes among its node's open request failures, by the injected error + // object. With a peer, the adapter cannot tell which query issued a load, + // so either query's node may list it. + const checkRequestErrors = (label: string) => { + if (queries.some(({ status }) => status !== `ready`)) return + const owed = requestFailures.filter(requestErrorOwed) + if (owed.length === 0) return + reach.requestErrorChecks++ + for (const failure of owed) { + expect( + [live, peer].some((query) => + query?.health.error?.errors.some( + ({ cause }) => cause === failure.load.error, + ), + ), + `${label}: request error for "${failure.load.error?.message}" on keys ${[...failure.keys].join()}`, + ).toBe(true) + } } // A subscriber keeps each live query from garbage collection while a // history waits on held loads. @@ -2360,10 +2531,11 @@ async function checkOrderedReplay( } await flushPromises() checkRetention(`retained publication after ${command.type}`) + checkRequestErrors(`after ${command.type}`) // Judge a terminal error at the history cut where it appears, before // final recovery can hide an incorrect transition. for (const shown of [live, peer]) { - if (shown?.status === `error`) checkTerminal(`history`) + if (shown?.status === `error`) checkTerminal(`history`, shown) } // A failed load after initial-query readiness shows as a request error // until a later load of its data succeeds. @@ -2434,9 +2606,14 @@ async function checkOrderedReplay( ...(peer ? [[peer, `peer convergence`, peerWindow] as const] : []), ] as const) { if (shown.status === `error`) { - checkTerminal(label) + checkTerminal(label, shown) reach.terminalErrors++ - } else checkWindow(label, shown.toArray, expected) + } else { + checkWindow(label, shown.toArray, expected) + // Recovery replayed every source and revalidated the window, and each + // success decides the request errors before it, so none stays. + expect(shown.health.error, `${label}: stale request error`).toBeNull() + } } } finally { holding = false @@ -3109,6 +3286,7 @@ describe(`Ordered windows across truncate replays`, () => { markerTruncates: 0, terminalErrors: 0, requestErrors: 0, + requestErrorChecks: 0, replayLoads: 0, abortRejections: 0, preloadChecks: 0, @@ -3145,6 +3323,7 @@ describe(`Ordered windows across truncate replays`, () => { `partialFailureRetirements`, `preloadReplayFailures`, `discardedReplayMoves`, + `requestErrorChecks`, ]) // Counts are histories, rather than repeated events in one trace. const dimensions = ( @@ -3284,9 +3463,12 @@ describe(`Ordered windows across truncate replays`, () => { }), ), fc.array(orderedCommand, { maxLength: 6 }), + // The failed demand's retry, which the query starts at once, may stay + // held, so later commands act while that demand is still unloaded. + fc.boolean(), ) .map( - ([scenario, joined, settles, newest, move, rest]): [ + ([scenario, joined, settles, newest, move, rest, retryHeld]): [ Array, OrderedScenario, ] => [ @@ -3301,7 +3483,15 @@ describe(`Ordered windows across truncate replays`, () => { })), { type: `settle`, ok: false, newest: true, keepHolding: true }, ...(move - ? [move, { type: `settle` as const, ok: false, newest: false }] + ? [ + move, + { + type: `settle` as const, + ok: false, + newest: false, + keepHolding: retryHeld, + }, + ] : []), ...rest, ], @@ -3347,6 +3537,67 @@ describe(`Ordered windows across truncate replays`, () => { { ...scenario, markerMode: `on-demand` }, ], ) + // A demand fails behind a sibling's failed replay, and its retry stays + // held. A later command starts another demand of the same source, whose + // load then succeeds, so a replay's success, publication, and request + // errors are judged while a failed demand is still unloaded. The model's + // replay is shared across the query's subscriptions to the source, so it + // gives its verdict only when every load of the source settled; it cannot + // judge one subscription's request error while another load is held (see + // Limits). + const heldRetryCase = fc + .tuple( + failedReplayScenario, + fc.constantFrom(`some` as const, `none` as const, `inner` as const), + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + fc.oneof( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: -5, max: 99 }), + }), + fc.record({ + type: fc.constant(`mark` as const), + id: fc.integer({ min: 1, max: 5 }), + on: fc.boolean(), + }), + fc.record({ + type: fc.constant(`window` as const), + offset: fc.constantFrom(0 as const, 1 as const), + limit: fc.constantFrom(1 as const, 2 as const, 3 as const), + }), + ) as fc.Arbitrary, + fc.array(orderedCommand, { maxLength: 3 }), + ) + .map( + ([scenario, joined, move, change, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + move, + { type: `settle`, ok: false, newest: false, keepHolding: true }, + change, + { type: `settle`, ok: true, newest: true, keepHolding: true }, + ...rest, + ], + { + ...scenario, + limit: scenario.limit === 1 ? 2 : scenario.limit, + joined, + includes: true, + markerMode: `on-demand`, + }, + ], + ) const failedReplayCase = fc.oneof( { weight: 3, @@ -3355,6 +3606,7 @@ describe(`Ordered windows across truncate replays`, () => { { weight: 1, arbitrary: includeDemandCase }, { weight: 2, arbitrary: siblingDemandCase }, { weight: 2, arbitrary: thrownDemandCase }, + { weight: 2, arbitrary: heldRetryCase }, ) const failedReplayProperty = `replay.ordered-failed-replay` const failedReplayCampaigns = @@ -3526,6 +3778,7 @@ describe(`Ordered windows across truncate replays`, () => { `failedSiblingDemandFailures`, `thrownReplayDemands`, `partialFailureRetirements`, + `requestErrorChecks`, ] as const, { minimum: 5 }, ) @@ -3557,6 +3810,9 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.partialFailureRetirements > 0) { reach.reach(`partialFailureRetirements`) } + if (observed.requestErrorChecks > 0) { + reach.reach(`requestErrorChecks`) + } }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(600) } From 0203c3f68a6bdf6fdfb80b17d4af86a14473a2d8 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 09:01:54 -0600 Subject: [PATCH 35/40] fix(db): reject a deferred window move with its replay's own error A window move that waits behind a truncate replay caught the replay's rejection, re-entered setWindow, and rejected with the query's latest subset error. When the replay had two failures, the move reported the later one, not the replay's own error. Law (ARCHITECTURE.md, window moves; maintainer decision): a deferred move whose replay fails rejects with that replay's own error, its earliest failure that still decides it. Law 17 decides which failures count: one whose demand retired, one that a later failure of the same request replaced, and one whose keys a later load of the replay covers no longer count. A replay that a newer truncate or a retired demand discards still lets the move apply. Production: the deferred move's loop rethrows a replay rejection unless it is a LoadSubsetOperationAbortedError from a discarded replay in an unchanged sync run. Oracle: an identity law judges deferred moves that start before any failure of the replays they wait on, with the named replay "a window move behind a replay with two failed loads". The request-error account now decides failures per key: a later load of the source that asks for some of a failure's keys decides them (law 16: a key's latest load decides it). Reach: `deferredMoveRejections` reaches the declared minimum of 5 in each failed-replay campaign. RED (parent code): both failed-replay campaigns and the named replay fail with "replay error identity: move 1 rejected with "load failed", not its replay's earliest failure that still decides it". GREEN: replay oracle 54/54; @tanstack/db 300 files, 15,529 tests. Limits: the law does not judge a move that starts after a replay it waits on already failed. Open, recorded in the coverage map with its trace: after a marker replay fails, a marker change can publish over the retained window; it fails on the parent code too and is not yet classified. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/query/live/ARCHITECTURE.md | 4 +- .../query/live/collection-config-builder.ts | 6 +- ...ad-subset-replay-refinement-oracle.test.ts | 157 +++++++++++++++++- 4 files changed, 160 insertions(+), 9 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d2b931b48b..6e8182029c 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -515,7 +515,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a trace confirms the flush, but no reached history showed a cleared error that was still owed (witness needed: a per-subscription verdict in the model). Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; this awaits a maintainer decision. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a trace confirms the flush, but no reached history showed a cleared error that was still owed (witness needed: a per-subscription verdict in the model). Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. Open, found by the held-retry branch at three times the run budget and failing on the parent code too: after a marker replay fails, a marker change can publish over the retained window (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers: "retained publication once loading is quiet: expected [ { id: 3 ... } ] to deeply equal []"); not yet classified as a production or model defect. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 616c86b58c..5b9134f37e 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1354,7 +1354,9 @@ applies only after its replacement is complete. A failed replay rejects the move without advancing the reported window. A replay that a newer truncate or the retirement of its last demand discards did not fail, even though its completion rejects with `AbortError`: the move waits for any replay that still -holds publication and then applies. When a replay it waits on is discarded and +holds publication and then applies. If a replay it waits on fails, the move +rejects with that replay's own error, its earliest failure that still decides +it (maintainer decision). When a replay it waits on is discarded and the query's or a source's sync run changed meanwhile, as a reload changes it, the move rejects with that replay's `AbortError`. The move compares the sync runs only when a replay it waits on rejects, so a reload of another source diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index d5f99eaec4..3044a18a70 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -415,7 +415,11 @@ export class CollectionConfigBuilder< try { await recovery } catch (error) { - if (syncRuns !== this.syncRuns()) throw error + if ( + !(error instanceof LoadSubsetOperationAbortedError) || + syncRuns !== this.syncRuns() + ) + throw error } } // A newer call wins, even if this deferred one resumes after it. diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 7f9763113f..ff8c79183d 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -1195,9 +1195,10 @@ describe(`loadSubset replay refinement`, () => { * model's replay across a query's subscriptions to a source, so it owes a * failure in a replay only after every load of that source settled; it does * not judge one subscription's request error while another load is held. It - * reads only the query's own node and permits extra listed failures. Which - * failure a deferred window move rejects with, when its replay has several, - * is not judged. + * reads only the query's own node and permits extra listed failures. The + * replay-error identity law judges only rejections with a failure of a replay + * the move waited on; a move that resumes into another failed replay is not + * judged. */ type OrderedFailureState = `initiating-keyed-request-failed` | `keyed-reacquisition-failed` @@ -1342,6 +1343,9 @@ type OrderedReach = { requestErrors: number // Cuts where the request-error account owed a request error. requestErrorChecks: number + // Deferred window moves that rejected with a failure of a replay they + // waited on. + deferredMoveRejections: number replayLoads: number abortRejections: number preloadChecks: number @@ -1423,6 +1427,7 @@ async function checkOrderedReplay( return false } let moveOrderViolation: string | undefined + let moveErrorViolation: string | undefined const settleMove = (move: number) => { if (move !== latestMove && !settledMoves.has(latestMove)) { moveOrderViolation ??= `window move ${move} settled before the newest move ${latestMove}` @@ -1486,6 +1491,10 @@ async function checkOrderedReplay( outcome: `pending` | `ok` | `failed` // The error this history injected when the load failed. error?: Error + // Its source, the keys it asked for, and whether it is an ordered read. + source?: BarrierSource + keys?: Set + ordered?: boolean } const loads: Array = [] // Retention model. Each source with an open or failed replay has a @@ -1497,6 +1506,9 @@ async function checkOrderedReplay( type BarrierSource = `rows` | `markers` type Barrier = { failures: Map + // Every failure of this replay, in failure order, including failures + // whose demand later retired. + history: Array // When the barrier rose, on the request-error account's clock. raisedAt: number // A release threw while this replay started, which may fail it. @@ -1505,6 +1517,8 @@ async function checkOrderedReplay( own: Set } const barriers = new Map() + // Every barrier this history raised, in raise order. + const raisedBarriers: Array = [] // The rows the query showed when the first barrier rose, if it had // published by then. Before the first ready there is no complete // publication to keep, and the initial-readiness law governs. @@ -1523,6 +1537,7 @@ async function checkOrderedReplay( } const barrier: Barrier = { failures: new Map(), + history: [], raisedAt: ++requestClock, // The retained publication dates from the first barrier, so an // ambiguous barrier earlier in this episode leaves it ambiguous. @@ -1531,6 +1546,7 @@ async function checkOrderedReplay( own: new Set(), } barriers.set(source, barrier) + raisedBarriers.push(barrier) } // A replay whose participants all settled without a failure publishes. const lowerSettledBarriers = () => { @@ -1576,6 +1592,7 @@ async function checkOrderedReplay( !options.signal?.aborted ) { barrier.failures.set(load, options) + barrier.history.push(load) } barrier?.pending.delete(load) // A replay completes when its last participant settles, so demand that @@ -1691,6 +1708,11 @@ async function checkOrderedReplay( // request error on the query's own node. The error stays until one of these // meets it: // - its demand retires, which aborts its request and unloads it; + // - a later load of the source asks for some of its keys: that newer + // request of the demand decides them, and keys it no longer asks for + // left the demand (ARCHITECTURE.md law 16: a lazy demand's error clears + // when that demand settles or is released; a key's latest load decides + // it); // - a later load of the same source covers its keys; // - a replay of its source that rose after the failure completes, since a // replay asks again for every held demand. @@ -1763,6 +1785,15 @@ async function checkOrderedReplay( barrier.pending.size > 0 ) return false + const at = loads.indexOf(failure.load) + const askedLater = loads.some( + (later, index) => + index > at && + later.source === failure.source && + !later.ordered && + [...failure.keys].some((key) => later.keys?.has(key)), + ) + if (askedLater) return false const covered = loadedKeys.some( (loaded) => loaded.source === failure.source && @@ -1848,6 +1879,7 @@ async function checkOrderedReplay( terminalErrors: 0, requestErrors: 0, requestErrorChecks: 0, + deferredMoveRejections: 0, replayLoads: 0, abortRejections: 0, preloadChecks: 0, @@ -1899,7 +1931,12 @@ async function checkOrderedReplay( // A throw is a rejection (ARCHITECTURE.md): the load holds its // owner, whose release unloads it. acquired.set(options, (acquired.get(options) ?? 0) + 1) - const load: Load = { epoch: truncates, outcome: `failed` } + const load: Load = { + epoch: truncates, + outcome: `failed`, + source: `markers`, + keys: requestedKeys(`markers`, options), + } loads.push(load) reach.markerLoads++ // The throw happens while the subscription registers the @@ -1942,7 +1979,12 @@ async function checkOrderedReplay( return (async () => { const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) - const load: Load = { epoch: truncates, outcome: `pending` } + const load: Load = { + epoch: truncates, + outcome: `pending`, + source: `markers`, + keys: requestedKeys(`markers`, options), + } loads.push(load) reach.markerLoads++ const participant = joinBarrier(`markers`, load, options) @@ -2089,7 +2131,13 @@ async function checkOrderedReplay( loadSubset: async (options) => { const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) - const load: Load = { epoch: truncates, outcome: `pending` } + const load: Load = { + epoch: truncates, + outcome: `pending`, + source: `rows`, + keys: requestedKeys(`rows`, options), + ordered: options.orderBy !== undefined, + } loads.push(load) const participant = joinBarrier(`rows`, load, options) belongsToRecovery(options, truncates, truncating) @@ -2295,6 +2343,51 @@ async function checkOrderedReplay( ).toBe(true) } } + // Replay error identity (ARCHITECTURE.md, "Replay participants and + // failure"; maintainer decision): a deferred window move whose replay + // fails rejects with that replay's own error, its earliest failure that + // still decides it. Law 17 decides which failures count: one whose demand + // retired, one that a later failure of the same request replaced, as a + // retry does, and one whose keys a later load of the replay covers without + // failing, as a demand's replacement does, no longer count. A failure of a + // load in no waited replay is the move's own window failure after it + // resumed, which this law does not judge. + const covers = (later: Load, earlier: Load) => + later.ordered === earlier.ordered && + later.keys !== undefined && + earlier.keys !== undefined && + [...earlier.keys].every((key) => later.keys!.has(key)) + const stillDecides = (barrier: Barrier, earlier: Load, at: number) => + barrier.failures.has(earlier) && + barrier.history.slice(at + 1).every((later) => !covers(later, earlier)) && + [...barrier.own].every( + (later) => + loads.indexOf(later) < loads.indexOf(earlier) || + later === earlier || + later.outcome === `failed` || + !covers(later, earlier), + ) + const checkReplayError = ( + move: number, + error: Error, + waitedReplays: Array, + ) => { + const candidates = waitedReplays.filter(({ history }) => + history.some((load) => load.error === error), + ) + if (candidates.length === 0) return + reach.deferredMoveRejections++ + const earliest = candidates.some((barrier) => + barrier.history + .slice( + 0, + barrier.history.findIndex((load) => load.error === error), + ) + .every((earlier, at) => !stillDecides(barrier, earlier, at)), + ) + if (!earliest) + moveErrorViolation ??= `move ${move} rejected with "${error.message}", not its replay's earliest failure that still decides it` + } // A subscriber keeps each live query from garbage collection while a // history waits on held loads. // A history during preload starts with every load held, including the @@ -2484,6 +2577,18 @@ async function checkOrderedReplay( const requested = { offset: command.offset, limit: command.limit } reach.windowMoves++ const move = ++latestMove + // A move that starts before a replay's verdict waits for it, and + // for any replay that rises before it resumes. The identity law + // judges a move that starts before any failure of the replays it + // waits on. A move that starts after one already failed may meet + // a failed replay of the source at once, and that rule + // (ARCHITECTURE.md: a failed replay rejects the move) names no + // particular failure. + const waited = [...barriers.values()].filter( + ({ pending }) => pending.size > 0, + ) + const judged = waited.every(({ failures }) => failures.size === 0) + const raisedBefore = raisedBarriers.length // A load that throws while it starts is a rejection // (ARCHITECTURE.md), so a move never throws: a throw here is an // escaped error. @@ -2512,6 +2617,15 @@ async function checkOrderedReplay( if (error instanceof Error && cleanupFaults.has(error)) { escaped ??= error } + if (error instanceof Error && injectedLoadFaults.has(error)) { + checkReplayError( + move, + error, + waited.length > 0 && judged + ? [...waited, ...raisedBarriers.slice(raisedBefore)] + : [], + ) + } // A truncate never aborts a move: the move waits for any // replay that holds publication and then applies // (ARCHITECTURE.md). Only cleanup rejects a waiting move with @@ -2563,6 +2677,9 @@ async function checkOrderedReplay( if (moveOrderViolation) { throw new Error(`newest move wins: ${moveOrderViolation}`) } + if (moveErrorViolation) { + throw new Error(`replay error identity: ${moveErrorViolation}`) + } if (escapedError() !== undefined) { throw new Error(`escaped error: only injected faults may surface`, { cause: escapedError(), @@ -3165,6 +3282,28 @@ describe(`Ordered windows across truncate replays`, () => { }, reaches: `failedLoads`, }, + { + // A window move waits behind a source replay whose two loads then + // fail. The move rejects with the replay's own error, its earliest + // failure (maintainer decision). It rejected with the later failure, + // which the query had last recorded. + name: `a window move behind a replay with two failed loads`, + history: [ + { type: `hold` }, + { type: `truncate`, armAfterStart: false }, + { type: `window`, offset: 0, limit: 1 }, + { type: `settle`, ok: false, newest: true }, + { type: `settle`, ok: false, keepHolding: false, newest: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + offset: 1, + limit: 1, + joined: `none`, + markerMode: `eager`, + }, + reaches: `deferredMoveRejections`, + }, { name: `an established marker replay after a fresh include succeeds`, history: [ @@ -3287,6 +3426,7 @@ describe(`Ordered windows across truncate replays`, () => { terminalErrors: 0, requestErrors: 0, requestErrorChecks: 0, + deferredMoveRejections: 0, replayLoads: 0, abortRejections: 0, preloadChecks: 0, @@ -3324,6 +3464,7 @@ describe(`Ordered windows across truncate replays`, () => { `preloadReplayFailures`, `discardedReplayMoves`, `requestErrorChecks`, + `deferredMoveRejections`, ]) // Counts are histories, rather than repeated events in one trace. const dimensions = ( @@ -3779,6 +3920,7 @@ describe(`Ordered windows across truncate replays`, () => { `thrownReplayDemands`, `partialFailureRetirements`, `requestErrorChecks`, + `deferredMoveRejections`, ] as const, { minimum: 5 }, ) @@ -3813,6 +3955,9 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.requestErrorChecks > 0) { reach.reach(`requestErrorChecks`) } + if (observed.deferredMoveRejections > 0) { + reach.reach(`deferredMoveRejections`) + } }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(600) } From 834c1ec9da4f2a9d627a0f96426b5717509cd7b6 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 09:31:28 -0600 Subject: [PATCH 36/40] test(db): let a loaded retry decide a failed replay load in the replay oracle The held-retry branch found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay. The model was wrong: the query retries a failed demand at once, and when that retry loads the failed keys, the failure no longer counts. Law (ARCHITECTURE.md law 17): a replay's failure counts for a key that no later held load loaded. The model now clears a replay failure when a later load of the replay loads its keys. Trace: hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off (ascending, offset 1, limit 1, includes over on-demand markers): "retained publication once loading is quiet: expected [ { id: 3 ... } ] to deeply equal []" on the parent model, with the same result on the code before the identity fix. Production is unchanged. GREEN: replay oracle 42/42 at the normal budget and three times at three times it. Mutant: production without failReplacement still fails "keeps the last publication when a marker change follows two failed replay loads" with "retained publication after mark". Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- .../load-subset-replay-refinement-oracle.test.ts | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 1976f52871..a2d350c3bc 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -525,7 +525,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a trace confirms the flush, but no reached history showed a cleared error that was still owed (witness needed: a per-subscription verdict in the model). Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. Open, found by the held-retry branch at three times the run budget and failing on the parent code too: after a marker replay fails, a marker change can publish over the retained window (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers: "retained publication once loading is quiet: expected [ { id: 3 ... } ] to deeply equal []"); not yet classified as a production or model defect. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a trace confirms the flush, but no reached history showed a cleared error that was still owed (witness needed: a per-subscription verdict in the model). Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index cf57de6762..24b263b6e7 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -560,6 +560,21 @@ async function checkOrderedReplay( barrier.failures.set(load, options) barrier.history.push(load) } + // A later load of the replay that loads a failed load's keys, such as + // the retry the query starts at once, decides those keys, so that + // failure no longer fails the replay (ARCHITECTURE.md law 17: a failure + // counts for a key no later held load loaded). + if (ok && barrier?.pending.has(load) && !options.signal?.aborted) { + for (const failed of barrier.failures.keys()) { + if ( + loads.indexOf(failed) < loads.indexOf(load) && + failed.ordered === load.ordered && + failed.keys !== undefined && + [...failed.keys].every((key) => load.keys?.has(key)) + ) + barrier.failures.delete(failed) + } + } barrier?.pending.delete(load) // A replay completes when its last participant settles, so demand that // starts after it does not join it. From 9890e2b8fa8ef017f0cc937274d822032d6023a5 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 09:46:49 -0600 Subject: [PATCH 37/40] test(db): pin a failed replacement's hold ending when a later demand settles A review of the #2168 merge found that the replay `failReplacement` creates records no failed demand. A later demand that joins it and settles completes it as a success: it publishes, and a live query's `succeed` clears the failed demand's request error. Law (ARCHITECTURE.md, "Replay participants and failure"): the hold lasts until a later replay of the source succeeds or the subscription's last demand retires. RED: as a plain test, "expected "spy" to not be called at all, but actually been called 1 times". Maintainer decision: pinned with it.fails and left unfixed for now; the coverage map records it. The replay oracle cannot reach it until its model gives each subscription its own verdict. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- .../collection-subscription-replay.test.ts | 54 +++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index a2d350c3bc..bacc0efddd 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -525,7 +525,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a trace confirms the flush, but no reached history showed a cleared error that was still owed (witness needed: a per-subscription verdict in the model). Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a focused subscription test reproduces it: after `failReplacement`, a demand that joins its replay and settles completes it, so `succeed` runs ("expected "spy" to not be called at all, but actually been called 1 times"). By maintainer decision it stays unfixed for now, pinned with `it.fails` as `keeps a failed replacement's hold when a later demand settles` in `collection-subscription-replay.test.ts`; the replay oracle cannot reach it until its model gives each subscription its own verdict. Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/collection-subscription-replay.test.ts b/packages/db/tests/collection-subscription-replay.test.ts index ad5847453a..9712f4fc33 100644 --- a/packages/db/tests/collection-subscription-replay.test.ts +++ b/packages/db/tests/collection-subscription-replay.test.ts @@ -3037,4 +3037,58 @@ describe(`CollectionSubscription replay oracle`, () => { } }, ) + + // An open review finding, recorded in the coverage map (Ordered + // acquisition). A demand that fails while a failed replay of another of the + // query's subscriptions to the source holds publication fails this + // subscription's replacement too. That hold lasts until a later replay of + // the source succeeds or the subscription's last demand retires + // (ARCHITECTURE.md, "Replay participants and failure"). The replay that + // `failReplacement` creates records no failed demand, so a later demand + // that joins it and settles completes it as a success: it publishes, and a + // live query's `succeed` clears the failed demand's request error. + // Maintainer decision: pinned and left unfixed for now. + it.fails( + `keeps a failed replacement's hold when a later demand settles`, + async () => { + const succeed = vi.fn() + const collection = createCollection<{ id: string }>({ + id: `failed-replacement-hold`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => Promise.resolve(), + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + siblingReplays: () => undefined, + start: () => {}, + succeed, + }, + }) + + try { + subscription.failReplacement(new Error(`sibling replay failed`)) + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]), + optimizedOnly: false, + }) + await flushPromises() + + expect(succeed).not.toHaveBeenCalled() + expect(subscription.hasFailedTruncateReplacement).toBe(true) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) }) From 446ad1ff78fac1a9954238e84bb1c445aba0d3af Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 10:03:09 -0600 Subject: [PATCH 38/40] test(db): judge each subscription's replay verdict in the request-error account The replay oracle's request-error account treated a query's subscriptions to a source as one replay, so it could not judge one subscription's request error while another subscription's load was held, which is where a failed replacement's hold can end early. Law (ARCHITECTURE.md law 17): a subscription's replay fails once its own participants settled; only a success also waits for the query's other subscriptions to the source. Each load names the subscription that issued it (LoadSubsetOptions.subscription), so the account waits for that subscription's loads. Publication retention stays shared. Reach: `successWhileSubscriptionOwes` (a subscription's load succeeds while a failure of that subscription is still owed) is declared in the failed-replay campaigns; 30 fixed and 24 random histories at the normal budget. GREEN: replay oracle 42/42 at the normal budget and at three times it. Mutant: a failed replacement's hold that completes when any participant settles fails both failed-replay campaigns with "retained publication after mark". Also: the replay-error identity law skips peer histories, since the adapter cannot tell which query a failed load served. Limits: no reached live-query history violates the request-error or retention law at that premise, because the failed demand's own retry joins the hold; the pinned subscription test asserts the hold law's stricter wording. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- ...ad-subset-replay-refinement-oracle.test.ts | 85 ++++++++++++++----- 2 files changed, 65 insertions(+), 22 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index bacc0efddd..199fbba88f 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -525,7 +525,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a focused subscription test reproduces it: after `failReplacement`, a demand that joins its replay and settles completes it, so `succeed` runs ("expected "spy" to not be called at all, but actually been called 1 times"). By maintainer decision it stays unfixed for now, pinned with `it.fails` as `keeps a failed replacement's hold when a later demand settles` in `collection-subscription-replay.test.ts`; the replay oracle cannot reach it until its model gives each subscription its own verdict. Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a focused subscription test reproduces it: after `failReplacement`, a demand that joins its replay and settles completes it, so `succeed` runs ("expected "spy" to not be called at all, but actually been called 1 times"). By maintainer decision it stays unfixed for now, pinned with `it.fails` as `keeps a failed replacement's hold when a later demand settles` in `collection-subscription-replay.test.ts`; the replay oracle now judges each subscription's request-error verdict by the subscription that a load names, and its failed-replay campaigns declare the premise (a subscription's load succeeds while that subscription's failure is still owed; 30 fixed and 24 random histories at the normal budget). No live-query history violates the request-error or retention law there, because the failed demand's own retry joins the hold and keeps it open until it settles; a mutant that completes such a hold when any participant settles fails both failed-replay campaigns at the retention law. The pinned subscription test asserts the stricter wording of the hold's law, under which no joined demand may end it. Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 24b263b6e7..8a11da8ce3 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -157,12 +157,13 @@ import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' * the marker barrier from a throw to the end of its episode (a barrier * raised while an earlier one in the episode is ambiguous inherits that). So * a design that dropped a thrown demand and let its replay succeed would pass - * wherever only retention could see it. The request-error account shares the - * model's replay across a query's subscriptions to a source, so it owes a - * failure in a replay only after every load of that source settled; it does - * not judge one subscription's request error while another load is held. It - * reads only the query's own node and permits extra listed failures. The - * replay-error identity law judges only rejections with a failure of a replay + * wherever only retention could see it. The request-error account judges a + * failure in a replay by its own subscription's verdict, using the + * subscription that each load names; publication retention stays shared + * across the query's subscriptions to a source. It reads only the query's + * own node and permits extra listed failures. With a peer, either query's + * node may list a failure, and the replay-error identity law skips peer + * histories. The replay-error identity law judges only rejections with a failure of a replay * the move waited on; a move that resumes into another failed replay is not * judged. */ @@ -309,6 +310,9 @@ type OrderedReach = { requestErrors: number // Cuts where the request-error account owed a request error. requestErrorChecks: number + // A subscription's load succeeds while a failure of that subscription is + // still owed. + successWhileSubscriptionOwes: number // Deferred window moves that rejected with a failure of a replay they // waited on. deferredMoveRejections: number @@ -697,8 +701,10 @@ async function checkOrderedReplay( // - a later load of the same source covers its keys; // - a replay of its source that rose after the failure completes, since a // replay asks again for every held demand. - // A failure that joins a replay waits for that replay's verdict, which - // comes once every participant settled, so it owes no error before then. Keys come from evaluating the request's + // A failure that joins a replay waits for its subscription's verdict, + // which comes once that subscription's participants settled, so it owes + // no error before then. The barrier that holds publication stays shared + // across the query's subscriptions to the source. Keys come from evaluating the request's // `where` over a small key domain, independent of production's bookkeeping. // The account combines the production concepts of a failed request and the // replay verdict that decides it; it names no production structure. @@ -734,8 +740,26 @@ async function checkOrderedReplay( const recoveries: Array<{ source: BarrierSource; raisedAt: number }> = [] const unloaded = new WeakSet() const allReady = () => queries.every(({ status }) => status === `ready`) - const recordLoaded = (source: BarrierSource, keys: Set) => + const recordLoaded = ( + source: BarrierSource, + keys: Set, + options: LoadSubsetOptions, + ) => { + // A load of a subscription succeeds while a failure of that subscription + // that it does not cover is still owed: the state in which a hold that + // records no failed demand could complete early. + if ( + requestFailures.some( + (failure) => + failure.source === source && + failure.options.subscription === options.subscription && + requestErrorOwed(failure) && + [...failure.keys].some((key) => !keys.has(key)), + ) + ) + reach.successWhileSubscriptionOwes++ loadedKeys.push({ source, keys, at: ++requestClock }) + } const recordRequestFailure = ( load: Load, source: BarrierSource, @@ -757,13 +781,17 @@ async function checkOrderedReplay( const requestErrorOwed = (failure: RequestFailure) => { if (failure.options.signal?.aborted || unloaded.has(failure.options)) return false - // A replay gives its verdict once every participant settled, failed or - // not (ARCHITECTURE.md law 17). + // A subscription's replay fails once its own participants settled + // (ARCHITECTURE.md law 17); only a success also waits for the query's + // other subscriptions to the source. Each load names the subscription + // that issued it, so the verdict waits for that subscription's loads. const { barrier } = failure if ( barrier && barriers.get(failure.source) === barrier && - barrier.pending.size > 0 + [...barrier.pending.values()].some( + ({ subscription }) => subscription === failure.options.subscription, + ) ) return false const at = loads.indexOf(failure.load) @@ -860,6 +888,7 @@ async function checkOrderedReplay( terminalErrors: 0, requestErrors: 0, requestErrorChecks: 0, + successWhileSubscriptionOwes: 0, deferredMoveRejections: 0, replayLoads: 0, abortRejections: 0, @@ -1080,9 +1109,13 @@ async function checkOrderedReplay( } await operations.commit() load.outcome = `ok` - recordLoaded(`markers`, requestedKeys(`markers`, options)) // The load settles once its rows apply. settleParticipant(`markers`, load, options, true) + recordLoaded( + `markers`, + requestedKeys(`markers`, options), + options, + ) })() }, unloadSubset: (options) => { @@ -1208,6 +1241,8 @@ async function checkOrderedReplay( } await sync.commit() load.outcome = `ok` + // The load settles once its rows apply. + settleParticipant(`rows`, load, options, true) // An ordered load covers the rows it selected; a keyed load, the // keys it asked for. recordLoaded( @@ -1215,9 +1250,8 @@ async function checkOrderedReplay( options.orderBy ? new Set(selected.map(({ id }) => id)) : requestedKeys(`rows`, options), + options, ) - // The load settles once its rows apply. - settleParticipant(`rows`, load, options, true) }, unloadSubset: (options) => { unloaded.add(options) @@ -1568,7 +1602,12 @@ async function checkOrderedReplay( const waited = [...barriers.values()].filter( ({ pending }) => pending.size > 0, ) - const judged = waited.every(({ failures }) => failures.size === 0) + // A peer is another query over the same sources, and the adapter + // cannot tell which query a failed load served, so peer histories + // are not judged (see Limits). + const judged = + !scenario.peer && + waited.every(({ failures }) => failures.size === 0) const raisedBefore = raisedBarriers.length // A load that throws while it starts is a rejection // (ARCHITECTURE.md), so a move never throws: a throw here is an @@ -2414,6 +2453,7 @@ describe(`Ordered windows across truncate replays`, () => { terminalErrors: 0, requestErrors: 0, requestErrorChecks: 0, + successWhileSubscriptionOwes: 0, deferredMoveRejections: 0, replayLoads: 0, abortRejections: 0, @@ -2453,6 +2493,7 @@ describe(`Ordered windows across truncate replays`, () => { `discardedReplayMoves`, `requestErrorChecks`, `deferredMoveRejections`, + `successWhileSubscriptionOwes`, ]) // Counts are histories, rather than repeated events in one trace. const dimensions = ( @@ -2669,11 +2710,9 @@ describe(`Ordered windows across truncate replays`, () => { // A demand fails behind a sibling's failed replay, and its retry stays // held. A later command starts another demand of the same source, whose // load then succeeds, so a replay's success, publication, and request - // errors are judged while a failed demand is still unloaded. The model's - // replay is shared across the query's subscriptions to the source, so it - // gives its verdict only when every load of the source settled; it cannot - // judge one subscription's request error while another load is held (see - // Limits). + // errors are judged while a failed demand is still unloaded. Declared + // reach: a subscription's load succeeds while a failure of that + // subscription is still owed. const heldRetryCase = fc .tuple( failedReplayScenario, @@ -2909,6 +2948,7 @@ describe(`Ordered windows across truncate replays`, () => { `partialFailureRetirements`, `requestErrorChecks`, `deferredMoveRejections`, + `successWhileSubscriptionOwes`, ] as const, { minimum: 5 }, ) @@ -2946,6 +2986,9 @@ describe(`Ordered windows across truncate replays`, () => { if (observed.deferredMoveRejections > 0) { reach.reach(`deferredMoveRejections`) } + if (observed.successWhileSubscriptionOwes > 0) { + reach.reach(`successWhileSubscriptionOwes`) + } }), campaign === `fixed` ? { seed: 20261023, numRuns: oracleRuns(600) } From e2351d0000ba3d8dc833878eb2a391164be63171 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 10:13:09 -0600 Subject: [PATCH 39/40] docs(db): end a failed replacement's hold when its joined loads load A review found that the replay `failReplacement` creates records no failed demand, so it completes once every demand that joined it settles. A pinned subscription test asserted a stricter law: no joined demand may end the hold. Law (ARCHITECTURE.md, "Replay participants and failure"; maintainer decision): the hold lasts until every demand load that joins it loads, a later replay of the source succeeds, or the subscription's last demand retires. The query retries the failed demand at once and the retry joins the hold, so the hold cannot publish without the failed demand's rows. Production unchanged; the failReplacement doc comment states the revised law. The pinned `it.fails` test, which asserted the earlier wording, is removed. The coverage map records the revision. Reach: `successWhileSubscriptionOwes` (a subscription's load succeeds while that subscription's failure is still owed), 30 fixed and 24 random histories at the normal budget. GREEN: replay oracle and subscription replay tests, 107 passed; tsc and pnpm lint:changed clean. Mutant: completing such a hold when any participant settles fails both failed-replay campaigns at the retention law (from 446ad1ff7). Limits: the oracle drives the hold only through live-query histories, where the retry always joins. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 2 +- packages/db/src/collection/subscription.ts | 6 ++- packages/db/src/query/live/ARCHITECTURE.md | 7 ++- .../collection-subscription-replay.test.ts | 54 ------------------- 4 files changed, 10 insertions(+), 59 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 199fbba88f..185e5b9df3 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -525,7 +525,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. That leaves open the reviewer's finding that `failReplacement`'s replay, which has no recorded failure, can complete when a later demand that joined it settles and then clear the failed demand's request error; a focused subscription test reproduces it: after `failReplacement`, a demand that joins its replay and settles completes it, so `succeed` runs ("expected "spy" to not be called at all, but actually been called 1 times"). By maintainer decision it stays unfixed for now, pinned with `it.fails` as `keeps a failed replacement's hold when a later demand settles` in `collection-subscription-replay.test.ts`; the replay oracle now judges each subscription's request-error verdict by the subscription that a load names, and its failed-replay campaigns declare the premise (a subscription's load succeeds while that subscription's failure is still owed; 30 fixed and 24 random histories at the normal budget). No live-query history violates the request-error or retention law there, because the failed demand's own retry joins the hold and keeps it open until it settles; a mutant that completes such a hold when any participant settles fails both failed-replay campaigns at the retention law. The pinned subscription test asserts the stricter wording of the hold's law, under which no joined demand may end it. Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. The reviewer's finding that `failReplacement`'s replay, which has no recorded failure, completes when every demand that joined it settles is closed as a law revision by maintainer decision: the hold ends when every demand load that joins it loads, a later replay of the source succeeds, or the subscription's last demand retires (ARCHITECTURE.md, "Replay participants and failure"). The query retries the failed demand at once and that retry joins the hold, so the hold cannot publish without the failed demand's rows. The replay oracle judges each subscription's request-error verdict by the subscription that a load names, and its failed-replay campaigns declare the premise (a subscription's load succeeds while that subscription's failure is still owed; 30 fixed and 24 random histories at the normal budget). No live-query history violates the request-error or retention law there; a mutant that completes such a hold when any participant settles fails both failed-replay campaigns at the retention law. A subscription-level `it.fails` test that asserted the earlier, stricter wording (no joined demand may end the hold) was removed with the revision. Limit: the oracle drives the hold only through live-query histories, where the retry always joins; a direct `Collection.subscribeChanges` caller that never retries is outside the live-query contract. Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 093af5553b..de491b3325 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -876,8 +876,10 @@ export class CollectionSubscription * A lazy demand that failed outside this subscription's replay, while a * failed replay of another of the graph's subscriptions to this collection * holds publication, fails this subscription's replacement too. It keeps - * the last publication until a later replay succeeds, or until this - * subscription's last demand retires. + * the last publication until every demand load that joins this replay + * loads, a later replay succeeds, or this subscription's last demand + * retires. The failed demand's retry joins it, so it cannot publish + * without that demand's rows. */ failReplacement(error: Error): void { if (this.truncateReplayState) return diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 5b9134f37e..3a79aab4ac 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -1491,8 +1491,11 @@ still has replay work, so demand that any of them starts meanwhile joins its own open replay. While a failed replay of any of them holds publication, a lazy demand of any of them that fails after initial-query readiness fails its own subscription's replacement too, so retiring the other replay's failed -demand cannot publish the window without this demand's rows. It holds until a -later replay of the source succeeds or the subscription's last demand retires. +demand cannot publish the window without this demand's rows. It holds until +every demand load that joins it loads, a later replay of the source succeeds, +or the subscription's last demand retires. The query retries the failed demand +at once and the retry joins the hold, so the hold ends only after the failed +demand's rows load. ### Mutation boundaries and initial readiness diff --git a/packages/db/tests/collection-subscription-replay.test.ts b/packages/db/tests/collection-subscription-replay.test.ts index 9712f4fc33..ad5847453a 100644 --- a/packages/db/tests/collection-subscription-replay.test.ts +++ b/packages/db/tests/collection-subscription-replay.test.ts @@ -3037,58 +3037,4 @@ describe(`CollectionSubscription replay oracle`, () => { } }, ) - - // An open review finding, recorded in the coverage map (Ordered - // acquisition). A demand that fails while a failed replay of another of the - // query's subscriptions to the source holds publication fails this - // subscription's replacement too. That hold lasts until a later replay of - // the source succeeds or the subscription's last demand retires - // (ARCHITECTURE.md, "Replay participants and failure"). The replay that - // `failReplacement` creates records no failed demand, so a later demand - // that joins it and settles completes it as a success: it publishes, and a - // live query's `succeed` clears the failed demand's request error. - // Maintainer decision: pinned and left unfixed for now. - it.fails( - `keeps a failed replacement's hold when a later demand settles`, - async () => { - const succeed = vi.fn() - const collection = createCollection<{ id: string }>({ - id: `failed-replacement-hold`, - getKey: ({ id }) => id, - syncMode: `on-demand`, - sync: { - sync: ({ markReady }) => { - markReady() - return { - loadSubset: () => Promise.resolve(), - unloadSubset: () => {}, - } - }, - }, - }) - const subscription = collection.subscribeChanges(() => {}, { - includeInitialState: false, - truncateReplayPublication: { - siblingReplays: () => undefined, - start: () => {}, - succeed, - }, - }) - - try { - subscription.failReplacement(new Error(`sibling replay failed`)) - subscription.requestSnapshot({ - where: new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]), - optimizedOnly: false, - }) - await flushPromises() - - expect(succeed).not.toHaveBeenCalled() - expect(subscription.hasFailedTruncateReplacement).toBe(true) - } finally { - subscription.unsubscribe() - await collection.cleanup() - } - }, - ) }) From b02796989528d435dc320015a11bff339eef5f61 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 11:24:13 -0600 Subject: [PATCH 40/40] test(persistence): pin two known gaps that random campaigns found The multi-tab Electric campaign (seed -1660880085) and the persisted-peers-full session campaign (seed 654774679, which failed CI on this branch) each found a history that reproduces on main. Pin both with `it.fails` replays through each oracle's own driver and checks, and record them in the coverage map, tracked with #2281. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 17 ++++++++++- .../tests/multi-tab-electric-oracle.test.ts | 30 +++++++++++++++++++ .../sync-metadata-session-persisted.test.ts | 23 ++++++++++++++ 3 files changed, 69 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d3f96d867f..8e2cc45ac6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3649,6 +3649,13 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. commits an older row and closes for good. Generated histories never close a tab for good, and an `it.fails` replay through the same driver and checks pins that history. + A random CI campaign (seed `-1660880085`, path + `55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`) found a second Law 2 history that + reproduces on `main`: row 1 reaches v2 and both tabs catch up, then + tab B stops, tab A catches up, and tab A stops, and the durable rows + lose row 1. The maintainer chose to leave it open with #2281; an + `it.fails` replay through the same driver and checks pins it, and the + random campaign can still draw it. Owner: this oracle. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in @@ -3848,7 +3855,15 @@ must equal the model's, which a restart would read. `persisted-peers-full` repeats the lane under full row updates, where an update replaces its row in storage too; `persisted-peers-on-demand` repeats it on demand with a demand for every row. A reset over metadata of a missing -row is too rare in generated histories to declare; a replay pins it. A reset +row is too rare in generated histories to declare; a replay pins it. A +random CI campaign (seed `654774679`) found a `persisted-peers-full` history +that reproduces on `main`: an outer transaction inserts over a bare seed, an +inner one re-inserts the row and commits, and a peer's full reload leaves the +row with the cached seed's metadata, where the model expects none (law: +the source reads its row metadata). The maintainer chose to leave it open, +tracked with #2281; an `it.fails` replay through the same model, driver, and +comparison pins its shrunk prefix, and the random campaign can still draw it. +Owner: this oracle. A reset may rewrite its rows, which its reload then applies as source writes. `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 01aa7342c6..4a6d74c630 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -1459,4 +1459,34 @@ describe(`multi-tab Electric over one persisted database`, () => { ), 60_000, ) + + // Known gap, pinned from the random campaign (seed -1660880085, path + // 55:0:1:4:4:4:1:1:1:1:3:4:4:4:4): row 1 reaches v2 on the server and both + // tabs catch up, but after tab B stops, tab A catches up, and tab A stops, + // the durable rows lose row 1. Law 2 expects every open tab and the durable + // rows to show the head. This reproduces on main and is tracked with #2281. + // It replays through the same driver and checks with the known-gap detector + // off. Remove `fails` when the durable row survives the stops. + it.fails( + `keeps a durable row after both tabs catch up to it and then stop`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `server change`, key: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `stop`, tab: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `stop`, tab: 0, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) }) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index bef1b39f32..169016d3c5 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -1209,6 +1209,29 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla it.each(peerFullReplays)(`replays under full rows: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerFullLane), ) + // Known gap, pinned from the persisted-peers-full random campaign (seed + // 654774679, shrunk to this prefix): an outer transaction inserts over a + // bare seed, an inner one re-inserts the row and commits, and a peer's full + // reload then leaves row 1 with the cached seed's metadata, where the model + // expects none. This reproduces on main and is tracked with #2281. It + // replays through the same model, driver, and comparison. Remove `fails` + // when the reload stops restoring that metadata. + it.fails( + `replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, + () => + assertPersistedHistory( + [ + { type: `begin` }, + { type: `seed`, key: 1, narrow: false, bare: true }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + { type: `reinsert`, key: 1, metadata: false }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `reload` }, + ], + persistedPeerFullLane, + ), + ) it.each(peerOnDemandReplays)(`replays on demand: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerOnDemandLane), )