diff --git a/.changeset/keep-live-query-through-failed-replay.md b/.changeset/keep-live-query-through-failed-replay.md new file mode 100644 index 0000000000..12b98fdbe7 --- /dev/null +++ b/.changeset/keep-live-query-through-failed-replay.md @@ -0,0 +1,11 @@ +--- +'@tanstack/db': patch +--- + +A query that joins and includes the same source has two subscriptions to it. Their truncate replays now share one barrier, so neither publishes while the other still loads. When a lazy demand of either fails while a failed replay of the source holds the query's results, the query keeps its last complete result until a later replay of that source succeeds. Before, retiring the other replay's failed demand could publish the window without that demand's rows. + +Some sources refetch and then report that their pages shifted. An ordered window over such a source no longer skips a row tied with the refetched first page's new end. When the report shows that the page's end moved, the window repairs its first page. It no longer trusts a row that an older tie request installed. + +A window move made while a truncate replay holds a live query's results now applies after that replay, even when a newer truncate discards it. Before, the move could reject, and an older move's window stayed. A reload of the query or of a source still ends the move. + +An ordered, limited query over a source without an order index can read too few rows to fill its window. A join that drops rows can cause this. It no longer reads the whole source then. It reads a prefix twice as long instead. It grows again while that adds rows, so every request stays bounded. diff --git a/docs/contributing/glossary.md b/docs/contributing/glossary.md index 824ae70abd..dc52e6dac6 100644 --- a/docs/contributing/glossary.md +++ b/docs/contributing/glossary.md @@ -131,7 +131,7 @@ earlier node alone. | demand | The logical need for source data. Demand may outlive or replace physical work. | Request, transport, or row ownership. | | logical subset owner | One subscription claim that keeps a subset demand active. | Physical acquisition. | | request data | The immutable `LoadSubsetOptions` and attached signal passed to an adapter. | Demand or established coverage. | -| acquisition attempt | One invocation that asks a sync adapter to start physical work. A synchronous throw ends the attempt before acceptance. If cleanup invalidates the captured sync run after the adapter returns but before activation, core cancels the tentative acquisition without retaining its lease; cleanup owns the adapter resources. | Physical acquisition or transport. | +| acquisition attempt | One invocation that asks a sync adapter to start physical work. A synchronous throw is a rejection: the attempt fails, keeps its owner, and the owner's release unloads it. If cleanup invalidates the captured sync run after the adapter returns but before activation, core cancels the tentative acquisition without retaining its lease; cleanup owns the adapter resources. | Physical acquisition or transport. | | physical acquisition | Request-scoped work accepted by a sync adapter for a demand. | Acquisition attempt, logical demand, or transport. | | lease | An ownership token that requires a matching release. TanStack DB leases do not expire on a timer. Always qualify which resource the lease owns. | Demand, request, or data coverage. | | acquisition lease | The release obligation created when an adapter accepts a physical acquisition. | The acquisition attempt itself. | @@ -221,7 +221,7 @@ Use nouns for state and verbs for transitions: - A demand **becomes active** or **retires**. - Core **starts an acquisition attempt**. If the adapter accepts it, the attempt **establishes a physical acquisition** and its **acquisition lease**. A - synchronous throw **fails the attempt** before either is established. Cleanup + synchronous throw **fails the attempt** as a rejection does. Cleanup may instead invalidate a returned attempt before activation; core then **cancels the tentative acquisition** without retaining its lease. - A logical owner **retires**. Core **releases an acquisition lease**. The diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index afab073a85..8e2cc45ac6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -525,7 +525,7 @@ that test identifiers must copy production's private data structures. | Drafts and native values | Complete | Native differential behavior, draft change tracking, detachment and its class-instance exception, hostile keys, aliases, cycles, and Map/Set live iteration are literate. | | Query DB and observer | Complete | Query-scope row ownership, subset identity and cancellation, failure and recovery, and the per-listener eligibility ledger are literate. | | Query DB SSR dehydration | Fixed QueryClient/Seroval boundary | Successful on-demand entries with a compound predicate, a request signal, and an ordered cursor with a custom comparator retain their row and user metadata through Seroval's reconstructed stream payload and Query Core hydration; request options are absent from the stream, while query functions retain the original values. The TanStack Start router and browser Collection resume path still need a host witness. | -| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A move whose load a truncate aborts may reject with an `AbortError` and leaves the window; a named replay reaches that branch. Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a newly initiated keyed snapshot request, on either side, rejects with an injected load error, or an initial load fails; a new include request can begin during a truncate commit, whereas reacquisition of an established request remains an ordinary nonterminal replay; named replays reach that branch and the aborted-move branch. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | +| Ordered acquisition | Complete | Exact demand identity, applied settlement, independent pagination recomputation, request work, lifecycle products, replay authority, source-generation readiness, and transaction-refinement abort boundaries are literate. An initial full-source ordered load shares the synchronous readiness cut ([historical review](oracle-reviews/initial-full-source-sync-readiness.md)). The ordered-lifecycle oracle covers an inexpressible-boundary fallback before first publication when every acquisition returns literal `true`. It also checks a later null-boundary fallback after the first joined window publishes, across literal, Promise, and held-receipt responses, including a held receipt after child demand settles ([review](oracle-reviews/issue-2152-initial-ordered-fallback.md)). The ordered loader state oracle also has a generated request-authority lane (`ordered-loader.request-authority`, fixed and random campaigns). It drives the real loader over a fake subscription through settlements in any order, failures, deferred invalidation and report reactions, ordinary passes and window moves, and began with four laws from a TLA+ model (#2191): a successful repair keeps its acquisitions until a covering read succeeds; quiet loading holds nothing older than its last authoritative read; a superseded request's failure is recovered by a full-source read without an explicit retry; and once quiet, an invalidation and a pass issue a read unless a full-source read covers the source. Scenarios also throw from the first release callback, answer a repair synchronously from installed rows, and let a page's own delivery invalidate ordering; named replays keep the traces they found, each of which failed on main (a throwing release that unwound a resumed repair, a synchronous repair waiting behind a superseded request, and a self-invalidating page whose tie was never retired). On main it found a superseded tie that failed after its repair finished and a newer invalidation's reaction was deferred, which left refills blocked; a named replay keeps that trace. A stale prefix that retires the repair's prefix fails the held law, and a stale request held after the repair finished fails the leak law. The lane also crosses unexpressible boundaries, joined work that defers loading, truncate replay that succeeds or stays pending, optimistic departures, source deletes, the bounded backoff retry, window moves that shrink or widen, plans that read only the full source, and reports inside the loader's own first page. A replay owns the requests issued before its truncate, and when the subscription still holds a full-source lease the settled replay covers the source. These dimensions found three more defects, each kept as a named replay: a repair prefix that settled after a replay started its replacement took the replacement's chain tag and was never retired; a replay counted a full-source read as complete after an explicit retry had released it; and a failure from before a truncate dropped the next repair's staged step. Each of the seven defect replays fails on the loader at the branch's merge base with main. A later law, one request at a time, found that a replaced repair chain's prefix still counted as current and issued a tie beside the replacement's prefix; a chain is now superseded by any newer authority start. The failed-lease law also counts a covering replay, which a named replay reaches. Repeated random campaigns then found a repair request that settled after full-source coverage and stayed held; full coverage now retires it. The held law counts a later prefix that a successful replay reacquired while it was in flight. Four histories from TCP's known error classes followed: a request the provider never answers (a lost segment), a release callback that truncates synchronously (a reentrant reset), a window offset other than zero, and an at-most-once release law with no release while a replay owns the demand unless an explicit retry replaces it or a held full-source lease covers it. They found that the backoff retry's release pass kept releasing failed demands after a release started a truncate; every release pass now stops on a reentrant reset. The oracle now observes epoch moves as calls to `invalidateSourceOrdering()`, so it tells a replacement chain from a chain step exactly. The loader's work slot and debt now move only through a transition table that throws `OrderedLoaderInvariantError` on an illegal move or an incoherent work and debt pair; the oracle records any such error as a broken law, so its generated histories fuzz the table. A covering-lease law (a full-source lease whose read succeeded or whose demand a replay reacquired is never released) found that a full-source transport failing after a covering replay turned complete coverage into an own failure; the table then found that a repair request failing after a replay finished its chain did the same. Mutants that drop each new guard fail. A release callback that runs an ordinary pass synchronously, as a graph run does when the release delivers deletes outside a transaction, found that the pass issued a page while another request was settling, and that request then issued its tie; every release pass now blocks refinement until it unwinds. A dispose command and a release callback that disposes the loader check that a disposed loader issues and releases nothing; mutants that drop the loader's own `active` checks in the release pass and in failure handling survive because dispose resets the cursor, so those checks were removed. TLA+ run 3 (`LoaderTable.tla`) models the table with truncate and replay; its safe rules pass with 9,368,122 distinct states, and each of the four guards from this round, removed alone, produces a counterexample. Reverting any one fix fails its replay and a campaign. The generated histories also stack a second truncate on a pending replay and truncate after a failed replay, each with a reach witness. A lease missing from its map during a release pass is now an invariant violation; no history reaches it. A second high review found that a covering replay did not retire a superseded failed lease from before its truncate, because the failed-lease law exempted every replay-owned request. The law now judges a superseded failure that was live (it failed in the incarnation it was issued in) once a covering replay settles after it, and the replay retires those leases. The subscriber now schedules the replay's graph run in a finally. The replay-refinement oracle is the receiving witness: its generated ordered section (`replay.ordered-window-faults`) drives a real on-demand Collection and an ordered live query through rank moves, held and failed loads, truncates, and unloads that throw without a truncate, while a replay starts, and while it settles. Its settled-window law found that a release throwing after a successful ordered load left the window on old rows, because the builder counted the rejected participant as a failed load (also on main); the loader now marks release errors and the builder publishes. It also fails without the subscriber's finally. The section now also crosses tied ranks (its law compares the window's rank sequence and each shown row's current rank), a peer live query on the same source, offset 1, descending order, and explicit window moves; a move that rejects after a failed source recovery leaves the window, and one rejected only by a release failure still moves. A reported-faults law accepts injected faults as rejected loads, thrown releases, rejected moves, or errors the subscription rethrows from a microtask, and rejects any other escaped error. These found that a window move whose load succeeded was recorded as failed when a release threw, so the window stayed private until another move. A throwing release is an adapter bug, so the loader now reports it uncaught from a microtask and changes nothing else; the reported-faults law rejects a move that a release failure rejects, and the request-authority oracle requires each throwing release to be reported uncaught exactly once (a mutant that swallows the error fails). It also crosses a joined marker source: a LEFT join that keeps marked or unmarked rows, and an inner join. The marker source is eager or on-demand, with marker loads that are held, fail, and truncate. This found that an on-demand joined source whose loads finished without delivering a row left the window on old rows, because nothing ran the graph that the pending joined work held; the subscriber now runs it when a joined source becomes ready. A truncate never aborts a window move: the move waits for any replay that holds publication and then applies (ARCHITECTURE.md), so a move that rejects with `AbortError` during a history is an escaped error. (An earlier version of this entry claimed a named replay reached an aborted-move branch; no history reached it, and the branch, which quietly accepted such a rejection, is gone.) Recovery truncates the marker source too, because a failed marker replay keeps the last complete publication until that source replays. A hostile-mutant pass over the loader's guards found guards that no ordered oracle kills. A differential instrument then ran 3,000 generated request-authority histories through the loader and each mutant and compared every request's shape, outcome, and release. Three guards the code makes redundant are deleted: the backoff timer guard, the explicit-move check in the retry callback, and the `requiresFullSource` check on a report. An ignored-report law now kills two of the mutants that changed public requests: a report issues no authoritative request before any request settled (a request in flight still owes an exact result for its own options) or while the window is inside the loader's own first prefix. Named replays reach both, and the random campaigns judge them in quiet histories. A random ordered-work campaign later found that the first-prefix check was unsound: it took the next installed row as the next source row, but a signalling refetch moved the page's end before the tie the loader had read, so that tie's row stood past the page while a row tied with the new end was never sent, and a later window move skipped it. The loader now accepts the row only when its page read it or when it read the tie of the page's current end. When the end moved, it repairs (see the later maintainer decision below); query-db's first-page witness requires bounded reads only, no full-source read, and an unchanged count of active Queries. The optimistic-window cases that allow but do not require a refill still bound any refill that occurs: an ordered read within the refill bound or a predicate read, never the full source (a mutant that made the growth read unbounded fails them). The ordered-work model now sorts `undefined` ranks with `null` ones, as the comparator does, and its histories generate both; nullish ranks tie in its window and in-flight comparisons. An `undefined` end also matches a loader that attempted no tie; a nullish end is never an expressible cursor, so the loader has started or will start a full-source read, which covers the source either way. A guard against that match survived the generated campaigns as an equivalent mutant and was removed. The ordered-work oracle keeps the trace as a named replay that fails on main, and the request-authority oracle has an unproven-report law (a report over a first page that no row it read proves starts a repair or reads the end's tie), whose replay and fixed campaign fail on main; the broad campaigns reach that law in about one history in 10,000, so an unproven-report campaign (`ordered-loader.unproven-report`, fixed and random) settles the first page, settles the read after it, and reports, over random case parameters and a random suffix, and declares its reach (about a third of 300 histories); `main`'s loader fails both runs. Its fake subscription now reads installed rows as the subscription does, applying a tie's equality, a cursor, and a limit; when its reads ignored the limit, the tie was always the last installed row's, so the first-prefix ignore branch was unreachable. The cost law's per-chain bound, three requests, was an artifact of that fake: it is now a prefix plus a tie and a page per installed row, and it also judges an unindexed window that still needs rows, which grows its prefix instead of reading the full source. The report campaign (`ordered-loader.unproven-report`, 1,000 runs; at 600 a random seed could miss the minimum) now starts each history with one of eight prefixes and declares reach for each report premise: a proven first page, a page that read past a window that shrank, a held full-source read, a head insert that moved the page's end, an owed repair, authoritative work in flight, a release of the older tie that truncates or disposes the loader, and a report to a disposed loader. The unproven-report law requires a bounded repair or the end's tie when every installed rank is expressible; a full-source read no longer satisfies it (a mutant that forced full debt after a report passed before). The cost law also rejects a repeated read: within one invalidation, no prefix, tie, or page repeats more often than the ordering epoch moved. Two laws are stronger: an ignored report, over a proven first page or a held full-source read, issues no request at all, where it used to forbid only authoritative ones; and a report while a repair is owed reads no tie or page before the repair. Deleting the moved-end early return in `revalidateLoadedCoverage`, or its past-window term, fails the first law, and deleting the owed-debt term in `loadMovedEndTie` fails the second; a deletion sweep had found all three surviving every oracle. With that reach, the report path's own `!active` checks were deleted: `dispose()` advances the generation that the stale-run check after the release reads, and a disposed loader issues nothing, which the dispose law judges. A head insert can carry a `-Infinity` rank, and two more laws judge every request: the loader never reads the tie of a rank no cursor can express, and the first finite request after a truncate has no cursor. The fake's `dataNeeded` now counts the window rows not at or before the boundary it is given, where it returned a constant. With those, the stale-run check after the older tie's release fails the reset law when a release truncates the loader and the window then grows (a prefix declares that reach), and removing `loadBoundary`'s own unexpressible-value check fails the unexpressible-tie law. The covered-source early return and the staged-continuation term were deleted: coverage that a full-source read completed holds the work slot, whose check then decides the report, and a staged finite continuation cannot coexist with a proven end, while a staged repair step owes a repair. The ignored-report law's full-source branch no longer needs a quiet history, since such coverage owes nothing, and a prefix reaches a paged loader whose repair fell back to a full-source read; removing the work-slot check then fails it. The unexpressible-end term in `loadMovedEndTie` stays: the unindexed-refetch random campaign found that without it an unindexed loader reads the full source through the boundary fallback where its bounded prefix repair would do, and a named replay of that trace fails without the term. Maintainer decision, later: a report whose page end moved now repairs instead of reading only the new end's tie, to cut shipped bytes; `loadMovedEndTie` and its guards are gone, so their mutants above no longer exist. The laws stay and judge the repair path: the ignored-report, owed-repair, unexpressible-tie, and reset laws all still run, and the unproven-report law accepts a repair. The report campaign drops its two prefixes whose release of the older tie truncated or disposed the loader, since a report no longer releases anything, and the query-db replay of an insert into a top-N list now expects the bounded repair's prefix and the new end's tie. Still open there: the `Object.is` term of the moved-end check. It skips work when the loader already read the end's tie, and it decides differently from its fallback only while other work is in flight, which the ignored-report law's quiet premise excludes. The ordered-work oracle owns that branch instead: peers over a refetching provider stay in its grammar for source filters (#2155 excludes only joined filters), and a proven-report law requires that a report which changes nothing the first query's own page and end tie prove, with the model's next row installed and no installed row deleted or reordered, starts no request for it and leaves it owing no `refetch: true` repair. A campaign fixed to that premise must reach it, and a loader that repairs every report fails both its fixed and random runs. A staged end tie that pending joined work defers is read by the pass after that work finishes; a request-authority named replay reaches it, and a loader that drops the staged tie fails it. The unproven-report law judges that later pass, and a truncate, failure, move, or disposal in between voids what the report owed. A history that starts during the initial preload found that a window move deferred behind a marker replay, which the source truncate then discarded, rejected with that replay's `AbortError`, so the older move's window won (also on main). A deferred move now waits for every replay that holds publication and applies unless one failed or cleanup ended the sync run; a named replay fails without the fix. The ordered-work refetch law also required no full-source read after a signalling refetch for an unindexed plan, which the architecture then let refill a short window from the full source; random seeds found it on main. Maintainer decision on this branch: an underfilled unindexed prefix never reads the full source, because that read is unbounded. It grows instead: each growth asks for twice the last prefix, and it stops once the last growth added no rows. A deletion sweep showed that the older condition's other term, a prefix that reads short, follows from that one. The refetch law now forbids a full-source read after a signalling refetch with no unindexed exception. The named replay of a short unindexed window and the fixed unindexed-refetch campaign (`ordered-work.eager-unindexed-refetch`, fixed and random, with reach on windows the prefix fills and windows it leaves short) failed on the full-source fallback and pass with growth; the full-source mutant still fails them. Growth made a new optimistic history possible: an unindexed one-row window behind a joined or anti-joined filter that installs only the prefix it grew to, whose departure leaves the replacement row missing. Sixteen fixed optimistic-window cases (update and delete, both consumers, acknowledge and rollback) cover it, with a refill bound of twice the installed prefix plus the overlaid row. A loader without the stop rule never reaches a fixed point; a loader that stops whenever the prefix reads short leaves those windows empty after settlement, because the pending departure hides a row from the count. The existing unindexed optimistic cases now allow, but do not require, a request behind a joined filter when the replacement row is installed: growth proves rows only up to its prefix, where the full-source fallback had proven every row (a maintainer-approved expectation change). The proven-report random campaign now declares its reach too. One mutant still changes requests without a law: dropping the optimistic-refill flag turns a refill into a refetching prefix. Six more are equivalent in the fake but stay, because the fake cannot reach their premise or they encode a documented rule: the own-first-page check behind a held sync transaction (the fake never holds one), a truncate moving complete coverage back to held, clearing the pending request when a full-source read starts (the fake never answers a full-source read synchronously), staging a joined `more` continuation, the staged-continuation guard before an unindexed prefix, and the drain re-entry guard (a long synchronous chain would otherwise recurse). An include dimension adds a `toArray` child that reads the marker source for each shown row, and an include-children law requires each shown row's markers to equal the model wherever the window law holds; a mutant that never loads newly demanded child keys fails it in an include-only scenario. A newest-move law requires a superseded window move to resolve only after the newest move settles (a superseded move that fails still rejects with its own error), and the model lets the newest move that resolved set the window; it found that a superseded move resolved as soon as its own load settled, so awaiting it showed neither window. The terminal-error law also allows `error` after a load fails while the query is still loading, the documented initial-load failure; a lazy join source is exempt. Limits: includes are one level deep. It also found that a release failure in lazy joined demand during a window move rejected the move, because the subscription reported it as a load error. A window law now lets the newest move win; it found that two moves deferred by a pending source recovery could resume in the opposite order, so the older request replaced the newer window. `setWindow()` now ignores a deferred call that a newer call superseded. A terminal-error law allows a final `error` only after a lazy join demand load, on either side, failed outside a truncate's recovery; named replays reach that branch. A load belongs to a recovery when it asks again for a demand its source held before the truncate (the same request data), or when the graph starts it while the truncate commits. The first model read a flag set only during the commit, but subscriptions start replay loads in a later microtask, so it counted every replay load as new demand and allowed `error` after any failed replay load; a random campaign found the reverse case (#2224): a marker truncate privately moved the window, the include's new demand started outside any replay because releasing its old demand ended the subscription's replay, and its failure ended the query. The demand controller now starts new coverage before it releases retired coverage, a query subscription opens a replay at every truncate even without demand (a replay that starts no recovery work and reports no loading), and a demand failure that fails its subscription's replay keeps the last publication instead of ending the query once the query has first become ready. The section also generates histories that start during the initial preload, with an initial-readiness law (the preload has settled once loading is quiet), marker loads whose aborted requests reject with `AbortError` as the source contract asks, and settles of the newest held load. These found two older defects on main: an aborted lazy demand request that rejected after its truncate replay succeeded ended the query, and so did a superseded segment that the controller released; the controller now treats an aborted request as obsolete demand, which its replay or newer coverage decides. Named replays keep each trace; reverting each production change fails one of them, and a mutant whose replay rebuilds its request data fails the campaigns' replay-load reach witness. The ordered-lifecycle oracle owns the truncate-work witness: a zero-limit window issues no request and reports no loading at a truncate, while a one-row window reacquires. A retention law then judges law 9 at generated checkpoints: a truncate raises a publication barrier, every load its source starts while the barrier is up joins it, a participant's failure fails the replay while its logical demand is held, and while any barrier is up the query shows exactly the rows it showed when the first one rose. It runs after every command and once loading is quiet. A failed-replay campaign (`replay.ordered-failed-replay`, fixed and random) starts each history by holding loads, truncating a source, optionally starting demand, and failing a held load, with a branch that fails an include demand while its marker replay still loads; it declares four reach states with `createOracleReach` (a judged failed replay, a judged held replay, a demand started after the commit that fails while its replay loads, and the include form of that demand). A replay participant now also counts as recovery for the terminal-error law, so a demand that starts after the commit while its replay is open is judged in generated histories, not only by a named replay. A mutant whose failed replay releases the publication gate fails both failed-replay campaigns and the broad campaigns; a mutant that ends the query when such a demand fails fails both failed-replay campaigns; the branch's production files reverted to `main` fail both. A preload-recovery campaign (`replay.ordered-preload-recovery`, fixed and random) starts every history during the initial preload over on-demand markers and declares two more states: lazy demand in a replay that fails before the first ready, and a window move that waits behind a replay a newer truncate discards. A builder whose deferred move a discarded replay rejects, and one that keeps a failure before the first ready non-fatal, each failed only a named replay before; both now fail the fixed campaign, whose seed kills both (each dies in about half of 200-run seeds, so the random campaign kills each about half the time). Building the law found three model errors, each corrected from ARCHITECTURE.md: a failure is scoped to its logical demand, so a retired failed demand and the demand that replaces it publish when the replacement succeeds; a replay completes when its last participant's rows apply, so demand that a deferred window move starts afterwards does not join it; and an ordered tie request (a `where` on rank) is not lazy demand, which the terminal-error law had treated it as, so a failed tie request had excused a terminal error. A query's subscriptions to one collection now share one replay barrier (ARCHITECTURE.md, "Replay participants and failure"): a probe found that in a query that joins and includes the marker source, a marker truncate retired the include's only demand privately, its emptied replay closed while the join's replay still loaded, and include demand the graph then started failed outside every replay and ended the query. A sibling-demand branch of the failed-replay campaign declares that state; the branch's code before the fix fails the named replay and both campaigns at `convergence: terminal error`, and removing either sibling hold (at replay completion, or for an emptied replay) fails all three. The random failed-replay campaign then found the other half: one subscription's replay failed, the other's finished once it settled, and demand that a rank move then started on the finished subscription failed and ended the query, because the subscriber judged the failure by its own subscription. The subscriber now asks whether a replay of the source's collection failed; a named replay (`a demand that fails after the sibling subscription's replay failed`), a `failedSiblingDemandFailures` reach in the sibling branch (which now may move a row and fail the oldest held load), and both campaigns at 400 runs fail at `convergence: terminal error` on the per-subscription rule. A design that judged a failure against any failed replay of the query, across collections, used to survive, because the convergence check runs after an explicit window move, which retries a failed row demand. A strict terminal-error law now requires a failed lazy demand outside every recovery to end the query (ARCHITECTURE.md, error 228), skipping only failures the model cannot place: after a load threw, and, with a peer, a load that joined a replay. Merge with #2257: its review (CF01, [record](oracle-reviews/2026-10-26-pr-2257.md)) judged a new keyed request that a truncate commit starts as terminal, and called non-fatal recovery of that demand a future contract. This PR adopts that contract by maintainer decision. The merged owner keeps CF01's initiating-request receipt check as a second account of every terminal error, at each history cut and at convergence, plus its named replays and the `replay.subscription-demand-origin` and `replay.keyed-request-failures` campaigns, with their two reach-state pairs measured on this owner's model. Both demand origins, new demand during a marker commit and a settled demand that the replay asks for again, must stay nonterminal after the first ready. An always-fatal `failDemand` mutant fails 15 tests, including both ported campaigns; a never-fatal mutant fails the strict law in 8. CF01's duplicate of the #2224 named replay is folded into this owner's existing one. Merge with #2168 (ARCHITECTURE.md law 16): after initial-query readiness no load failure ends a live query, and a failed lazy demand is a request error on the query's node. That supersedes the strict terminal-error law, the error-228 rule, and the always-fatal and never-fatal mutant results above for a query that has been ready. The replay owner now allows a terminal error only after a load failed while the query was still loading, judged at each history cut and at convergence, and it adopts #2168's request-error witness (`requestErrors`); the excusals for thrown and unplaced failures went with the strict law. #2168's design already starts new demand before it releases retired coverage, and it treats a synchronous `loadSubset` throw as a rejection whose owner's release unloads it (the subscription oracles pin that unload), so the adapter-release law now counts a thrown load as acquired. It lacked the shared barrier, the failure hold behind a sibling's failed replay, and the deferred window move's wait, which the merge ports onto it. Without the deferred-move loop, 8 tests fail, #2221 trace 2 among them; without `failReplacement`, the two-failures named replay fails; without the sibling wait, that replay and both failed-replay campaigns fail. A deferred move rejects when a reload ends the sync run of the query or of a source, which the source-health oracle requires. The window-request reset went: a move can no longer throw, and deleting it failed nothing. The replay that a truncate opened without demand went too: under law 16 its demand's failure no longer ends the query, and its named replay passes on #2168's subscription. Review of the merge (2026-10-28): the replay owner gained a request-error account. After initial-query readiness, each failed lazy demand load that is still owed must appear among the query node's open request failures, by its injected error object, at every history cut, and no request error may stay after recovery replays every source (authority: ARCHITECTURE.md law 16, law 17, and the glossary's request error). A failure is met when its demand retires, a later load of the source covers its keys, or a later replay of the source completes; a failure inside a replay owes nothing until every participant settled. A terminal error now needs a load that failed while that same query was loading, so a peer's initial failure no longer excuses `live`. A thrown load joins a replay that no participant joined yet, and the failed-sibling branch now generates a held retry. `requestErrorChecks` reaches the declared minimum of 5 in each failed-replay campaign. Mutants: demand outcomes that never report an error fail 9 tests at the per-cut check; a successful replay that keeps its request errors fails 22 at the stale-error check. Limits and open items: the account shares one replay across a query's subscriptions to a source, so it cannot judge one subscription's request error while another load of that source is held. The reviewer's finding that `failReplacement`'s replay, which has no recorded failure, completes when every demand that joined it settles is closed as a law revision by maintainer decision: the hold ends when every demand load that joins it loads, a later replay of the source succeeds, or the subscription's last demand retires (ARCHITECTURE.md, "Replay participants and failure"). The query retries the failed demand at once and that retry joins the hold, so the hold cannot publish without the failed demand's rows. The replay oracle judges each subscription's request-error verdict by the subscription that a load names, and its failed-replay campaigns declare the premise (a subscription's load succeeds while that subscription's failure is still owed; 30 fixed and 24 random histories at the normal budget). No live-query history violates the request-error or retention law there; a mutant that completes such a hold when any participant settles fails both failed-replay campaigns at the retention law. A subscription-level `it.fails` test that asserted the earlier, stricter wording (no joined demand may end the hold) was removed with the revision. Limit: the oracle drives the hold only through live-query histories, where the retry always joins; a direct `Collection.subscribeChanges` caller that never retries is outside the live-query contract. Which failure a deferred window move rejects with, when its replay has several, is not specified: `main` rejected with the replay's earliest failure, and this branch rejects with `lastSubsetError`; by maintainer decision the move rejects with its replay's earliest failure that still decides it (law 17 decides which count: a retired demand's, a retried request's, and a replaced demand's failures do not). An identity law now judges deferred moves that start before any failure of the replays they wait on; on the parent code both failed-replay campaigns and the named replay `a window move behind a replay with two failed loads` fail with "replay error identity", and the deferred-move loop that rethrows a non-abort replay error passes. The request-error account decides failures per key: a later load of the source that asks for some of a failure's keys decides them, and keys it no longer asks for left the demand. The held-retry branch then found, at three times the run budget, a marker change publishing over a window that the model retained after a failed marker replay (`hold, truncateMarkers, settle ok, settle ok, settle fail newest, mark 3 off`, ascending, offset 1, limit 1, includes over on-demand markers). It was a model defect: the query retries a failed demand at once, and when that retry loads the failed keys the failure no longer counts (ARCHITECTURE.md law 17), so the replay succeeds and publishes. The model now clears a replay failure when a later load of the replay loads its keys. The replay oracle passes at the normal budget and three times at three times it, and production without `failReplacement` still fails its named replay at the retention law. A move whose `setWindow` throws synchronously after the request generation increments is reachable only through invariant errors, which probes for a throwing subscriber, a crashed source, and a throwing `select` did not reach. An aborted request's rejection and the failure of a segment that newer coverage released are not failures. That design now fails both failed-replay campaigns. An independent oracle review then narrowed several judgments. The terminal-error excuse after a synchronous throw applies only to the thrown error itself, ending the query after the step that threw; the throw's own step, and every other failure, are judged strictly (a per-history flag had excused every later failure). A load that fails outside every barrier after a failed demand of its source retired is unplaced, because the failed replay may still hold through the subscription's other demands, which the adapter cannot attribute. Limit: with a peer, the strict law accepts either query ending, since the adapter boundary cannot tell which query's subscription issued a load; attributing each failure to its query's subscription would close that. The window check now also requires every shown row to be eligible and to appear once, so a tie cannot hide a row the join rejects. A synchronous-throw dimension (`markerThrows` arms the next marker load to throw while it starts) found that a load throwing inside an open replay ended the query: the subscription dropped the demand, so its failure did not fail the replay, and the subscriber treated the throw as fatal. The subscription now hands the replay a rejected participant for such a throw and keeps the demand, which a later replay retries; named replays (`a replay load that throws while it starts`, `a demand whose load throws inside its open replay`), a `thrownReplayDemands` reach in a thrown-demand branch, and both failed-replay campaigns (now 600 runs) fail on the parent code, and a mutant that later unloads the never-acquired load fails the adapter-release check. Model corrections, each from the adapter-boundary rule that a thrown load acquires nothing: a thrown load joins a replay only if it still has a participant or a failure; an armed throw expires before the drain; and a thrown demand's retirement is invisible, so the retention law skips the marker barrier from a throw to the end of its episode (a later barrier in the episode inherits the ambiguity, which also corrects the existing release-throw case), and a failure that joins an already failed or ambiguous barrier may end the query. Maintainer decision: a synchronous throw inside an open replay fails that replay as a rejection does, so the subscription replay oracle's `new request throws` case and five `live-query-collection` cases, which expected a thrown new demand to roll back, now expect the rejection outcome. Limits: a design that dropped a thrown demand and let its replay succeed passes wherever only retention could see it; after a history command follows a throw, a terminal error is excused, because the model cannot see whether that command retired the thrown demand and with it its failed replay (a demand that waited on that replay then fails outside it). The named replays judge the throw's own outcome strictly. The failed-replay random campaign at five times its run count then found a retention counterexample with no throw, on the parent code too: a join and an include of one marker source; one subscription's replay fails, the other's finishes, and a row move starts demand on it whose load fails, which the shared barrier kept from ending the query; a marker change then retired the first replay's failed demand, which discarded that replay and published an empty window, because the second failure held nothing. That failure now fails its own subscription's replacement, which holds until a later replay succeeds or the subscription's last demand retires. A named replay of the trace fails without the hold, and a `partialFailureRetirements` reach (a failed demand retires while another failure still holds the barrier) is declared in the failed-replay campaign; the campaigns themselves do not fail without the hold, so the named replay is the witness. A window move whose load throws while it starts counts as a settled, failed move in the model (it used to stay pending, so the newest-move law misjudged the older move), with a `thrownMoves` reach. The broad random campaign at five times its run count then found an older move whose settlement resolved with itself: the newer move threw after it bumped the request generation and before it replaced the latest request, so the older move's settlement returned its own promise, a chaining cycle that escaped as a TypeError. A call now clears the latest request when it bumps the generation, so no older move can wait on itself; a named replay fails without that. The same assignment covers a move deferred behind a replay, which had the same shape (an independent review found it can hang forever when a newer move throws between the replay settling and the deferred move resuming); that microtask gap is outside the grammar, which flushes between commands, so only the shared mechanism, not that path, has a witness. The ordered-work oracle also keeps the trace from #2221 as a named replay: before prefix growth the loader read the full source twice after a signalling refetch past a NaN boundary; with growth it reads none. Limits: the adapter boundary cannot tell which query issued a load, so the retention law and the participant rule skip histories with a peer. The retention law judges barriers that rise after the query first became ready; before that the initial-readiness law governs. It skips a barrier where a release threw while the replay started, which may fail that replay. Core now reports a throwing `unloadSubset` uncaught and fails nothing with it; the subscription lifecycle, replay, Effect disposal, includes-temporal, and subset-error oracles check the uncaught report, once, with the adapter's error. The request-authority oracle now also generates requests that throw while they start, the last history only TLA+ covered, with an error-identity law: a failed request's participant reports that request's own error. It found that a superseded failure whose recovery read threw at its start reported the start error instead (review finding 6). A cost law then checks, after loading is quiet and a bounded repair is available, that one invalidation reads no full source and issues at most one chain per ordering epoch move (in the fake, a prefix, a tie, and a refill); a mutant that always repairs with a full-source read fails it. The replay-refinement section's status law requires a quiet live query and its peer to stay ready when no load failed. Limits: whether a release runs the graph at once or defers it to a transaction belongs to the scheduler, and the oracle generates both forms without checking the scheduler's choice; the fake's rows do not move except for the unexpressible boundary; publication holds stay with other tests; the laws do not judge requests issued before a truncate; a hung authoritative request suspends the leak law for its unfinished chain; a recovery start that throws during a superseded failure is rare in the campaigns, so a named replay owns it. | | Indexed predicate filtering | Complete for bounded dotted-path collision | An independent row filter checks direct predicates and public callback predicates for a dotted scalar field beside a nested path, with and without an index. Selected-field ordering checks the same distinction through `$selected`. | | WHERE predicate publication | Complete for bounded predicate and sync-transaction grammar | The contract, Kleene reference evaluator, snapshot and change-history grammar, three subscriber consumers plus direct snapshot, and per-commit key-set refinement check are literate. A fixed same-key update checks live-row and direct-subscriber payloads; a focused descriptor boundary checks stored-row prefilter safety. Generated cleanup and restart histories for filtered subscribers remain open. | @@ -1491,7 +1491,7 @@ generic visitor behavior outside the named collectors. | Query DB and observer | [ownership](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts), [persisted direct-write window](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/persisted-direct-write-window.oracle.test.ts) ([review](oracle-reviews/issue-2029-direct-write-lock-window.md)), [load lifecycle](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/load-subset-lifecycle.test.ts) (fixed tests; no oracle owner (tracked in #2264)), [observer histories](https://github.com/TanStack/db/blob/main/packages/db/tests/live-query-observer-history-oracle.property.test.ts), [retained snapshots](https://github.com/TanStack/db/blob/main/packages/db/tests/live-query-observer-snapshot-oracle.property.test.ts) ([review](oracle-reviews/pr-2043-retained-observer-snapshots.md)) | Real QueryClient boundary, applied-settlement barriers for existing and cached demand, mutation-handler Collection identity, terminal fetch-record lifecycle, and a per-listener eligibility ledger, not a duplicate dispatch queue. The bounded handler grammar crosses insert, update, delete, parameter and mutation-alias access, refetch, and clearError. A held clearError retry retains the prior public error through initial, background, and Collection application failures; success clears it, failure advances the count, including a repeated error at clock time zero. An intermediate Query retry attempt does not recount the previous background error; a held final attempt can succeed or fail. A held application after successful Query fetch keeps the error visible until the applied result. A mocked persisted-baseline scan holds an older success while a newer terminal Query failure arrives; the newer public error and count survive application, including when the Error object and clock tick repeat. Native SQLite persistence, concurrent retries across multiple tracked Queries, mutation-handler fetch-boundary settlement, and deferred result application remain outside this witness. The mutation overlap grammar crosses both start orders. Check reentry, peer survival, FIFO and disposal independently of final rows. A retained observer snapshot shows its capture-time rows on a later first read of `data` or `state`. Direct-write witnesses now distinguish an idle pending eager cache seed (success with no new freshness timestamp) from an in-flight pending request, and require update, delete, and upsert to reach both eager Collections through their shared physical Query, in both acquisition orders. They check both public Collections and the Query cache immediately after each write. A held older shared fetch also crosses B's release and reacquisition, then compares both public rows at refetch and load settlement; mixed select wrappers and three or more co-owners remain outside this witness. The load-lifecycle tests also restore an intermediate checkpoint that keeps an explicit subset refetch pending until a post-write request applies, and rejects a no-cancel mutant at the replacement-request assertion. . Two sync-metadata session witnesses fail on `30175be48`: a claim that inserts a row keeps applied non-owner metadata, and a claim during a held retention cleanup publishes the cleaned owner (see Sync metadata session)| | Observer hydration under terminal statuses | `packages/db/tests/live-query-observer-hydration-oracle.property.test.ts` | ARCHITECTURE.md law 16 and the glossary "crash": while a hydration seed is active, a Collection `error` or `crashed` shows in the snapshot, attached or detached, and is announced; an attached observer that sees the failure ends the seed, so cleanup then shows `cleaned-up` and live rows, not the stale failure or the seed. Decision (not a law-16 claim): `cleaned-up` and restart under an active seed keep the seed as `ready`, because cleanup is not a failure and the next sync run's readiness hands off. Generated histories cross both modes, attach order, live writes, `markReady()`, `fail()`, `crash()`, cleanup, and restart, with declared reach; replays pin the crash, fail-then-cleanup, and crash-before-attach traces. Limits: rows under a failure status are not claimed; the source is a plain controlled Collection, not a live-query graph or an on-demand subset; streamed client errors, handoff timing, and framework adapters stay with the focused observer and adapter tests; the property is not in the `oracle-config.ts` path-replay registry. | | Query DB SSR dehydration | `packages/query-db-collection/tests/ssr-dehydration-oracle.test.ts`, [review record](oracle-reviews/issue-1950-ssr-dehydration.md) | A plain Query cache control, a live on-demand compound predicate, a signal-only request, and an ordered cursor with a custom comparator all retain their row through QueryClient dehydration, reconstruction of Seroval's emitted stream payload, and Query Core hydration from that payload. The on-demand query functions still receive the original IR, comparator, and signal; serialized metadata retains enumerable user metadata but excludes request options. The original implementation fails the compound and signal cases at the stream checkpoint. A serializable stream-only request-options leak passes the former chunk-count check but fails the current payload assertion. This owner covers successful Query cache entries at the installed Query Core 5.90.20 and Seroval 1.5.0 versions. A TanStack Start integration owner still needs the reporter's Router 1.171.33 and Seroval 1.6.8 path, browser Collection resume/refetch, and request cancellation across SSR. Query subset and pagination oracles remain owners of predicate/order/cursor meaning and supported value types. The issue's `shouldDehydrateQuery` workaround replaces Query Core's success-only default and admits pending/error queries; any documented workaround must preserve that default filter. | -| Query DB ordered refetch | `packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts` | Issue #2136: after preload, and after `utils.refetch()` or a Query invalidation settles, an on-demand Query Collection behind a LEFT-joined, ordered, limited live query must show the window over the current backend rows. The design under test: query-db reports `invalidateLoadedSubsets()` when an ordered page changes keys and either has a later page or ends earlier in the order than before, and each ordered loader runs its bounded prefix repair. Its model is a pure top-k of marked backend rows. A fixed and a random campaign generate rounds of marker flips (which serve continuation pages), backend changes while the provider lags, a second live query (`observe`) whose own demand can load a page's new row first, optional Query invalidation with further changes, and a refetch, across one or two order terms, a numeric or locale-collated string order, limits 1–3, offsets 0–1, and a static or function Query key. With an unchanged backend, a repeated refetch must serve no more rows, start at most one first-page request, and report nothing. A refetch or invalidation must never start a new full-source request. A coverage law runs at the same checkpoints: every backend row from the start of the order through the window's last row (every row, when the window is short) must be in the source Collection with its current values, so a latent gap fails before a marker exposes it; a named replay of the issue trace with the moved row unmarked passes the window law and fails only the coverage law when Query DB never reports. The grammar can hold an optimistic update across a refetch, with the checkpoint moved to its release, and a premise-directed campaign under a hold (other rows observed, a row moved ahead of the page's end, a new row further ahead, a write to the last-sorting observed row) requires a report during a hold in every case and fails a skip that reads rows held back behind the transaction; the general campaigns did not. The general grammar holds optimistic deletes as well as updates. A named replay of a refetch during a held delete left the window short after release, also with this PR's report and overlap changes reverted, until the fix that resumes a continuation after a synchronous answer; no window is checked during a hold, where #2148 and #2169 remain open. Reports (`loadedSubsets:invalidated`) are bounded: a step reports at most once, and only when some ordered request was answered again with different rows; named replays require no report when a one-page window, or a window's last page, only loses a row, and a sync-boundary replay requires a report when a page ending on a row without an order value has its end move earlier (on the earlier gate, which read that page as empty, it reported nothing), and another requires a report when the provider returns a one-page window's rows in reverse order (the earlier gate took the last array row as the page's end). When a page's end moves earlier, Query DB reports, and a loader whose window lies inside its own first page must skip the repair: a named replay of an insert into a top-N list bounds the refetch's requests by its Queries, and a filtered one-page window whose page end moves past an unseen row must still repair (found by the random campaign; a loader that always skips fails it, the f9 tie replay, and the ordered-work oracle's signalling replays). The skip also refuses while the source holds accepted sync rows back: a replay refetches while an optimistic update persists, so the report arrives before the refetched rows are visible, and a skip that reads those older rows fails it. The ordered loader state oracle checks that a prefix read issued before an ordering invalidation, which settles after the bounded repair, does not retire the repair's prefix; the repair-first order failed before that fix. A `loadSubset` call that acquires a released, invalidated page can report, because Query then refetches that page on its own. A failed repair shows only the last window or the right one. Named replays cover the issue trace after a refetch and after an invalidation, with a page-capping provider, with another live query loading the page's new row first, with an empty later page, with a page whose end moves earlier while a tie request holds its old last row, a second refetch after an invalidation, a repair failure, and a provider that honors neither cursor nor offset. Never signalling, the old overlap-to-full-source rule, ignoring empty later pages, reporting on any page change, removing the repair loop guard, and reporting once per page each fail it by assertion. A replay makes a page's application fail and checks the window shows its last result or the right one, then the next refetch restores it; recording a page before its rows are accepted cannot be distinguished, because the failed attempt reports on its own. Open: the report laws bound reports but do not require each one; a page whose end moves earlier reports even with unique order values, where no row can be missing; it does not check whether a repair waits for in-flight work; the generated campaigns use an uncapped cursor provider only (an offset-only provider and a page-capping provider each hit a loader defect that reproduces on main: #2182, #2183); the generated campaigns do not check the window after a marker flip (a named replay checks the trace where the loader stopped refilling after a joined-side removal, which the repair-retirement fix below closed); a failed repair still recovers through the loader's full-source retry; interval, focus, and reconnect refetches share the invalidation path but are not driven; NaN order values have no executable witness; overlapping `refetch: true` acquisitions on one Query hash are counted, so the first to finish does not clear the loop guard for the other, but no witness separates that from a plain set; a Query refetch that joins a running `refetch: true` acquisition on the same hash does not report, and no witness reaches it (#2185); follower tabs of a multi-tab persisted collection never receive the report (#2174). The offset-only and page-capping loader defects have `it.fails` witnesses in the oracle. A finished bounded repair used to discard its boundary when its retirement made Query DB remove a row the loader had sent; the loader then restarted paging with a smaller first page that could not retire the repair's prefix, so every later refetch read both. A named replay found by the random campaign (at four times the run count) and an ordered loader state witness fail on that, and the fix also closed the joined-side refill defect. At four times the run count, the random campaign found a short window that stopped paging before the last row of the order: a report's repair released continuation Queries, and Query DB answered their reacquisition synchronously from its cache, so the graph turn inside each request asked for more while the loader was still requesting and nothing resumed the staged continuation. Only the coverage law failed. A named replay and an ordered loader state witness (a synchronous answer whose writes the graph already processed must continue the chain) fail on that, and pass with the fix. The loader side of the report also has a second owner: the ordered-work oracle's `refetch-signal` provider re-runs each served request alone and reports when any answer changed, across that oracle's generated plans, and a signalling refetch must start no full-source request unless a bounded page in that step ended on a null or NaN rank, which takes the documented full-source fallback (a random seed found that case). Never signalling and the old overlap-to-full-source rule each fail it. It excludes a peer live query, which can keep a short window after a refetch (#2155). The persisted-collection oracle checks that a source's report reaches the Collection through the managed cache-generation wrapper and that a retired source session's report is dropped. Remaining work for this class is tracked in #2186. | +| Query DB ordered refetch | `packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts` | Issue #2136: after preload, and after `utils.refetch()` or a Query invalidation settles, an on-demand Query Collection behind a LEFT-joined, ordered, limited live query must show the window over the current backend rows. The design under test: query-db reports `invalidateLoadedSubsets()` when an ordered page changes keys and either has a later page or ends earlier in the order than before, and each ordered loader runs its bounded prefix repair. Its model is a pure top-k of marked backend rows. A fixed and a random campaign generate rounds of marker flips (which serve continuation pages), backend changes while the provider lags, a second live query (`observe`) whose own demand can load a page's new row first, optional Query invalidation with further changes, and a refetch, across one or two order terms, a numeric or locale-collated string order, limits 1–3, offsets 0–1, and a static or function Query key. With an unchanged backend, a repeated refetch must serve no more rows, start at most one first-page request, and report nothing. A refetch or invalidation must never start a new full-source request. A coverage law runs at the same checkpoints: every backend row from the start of the order through the window's last row (every row, when the window is short) must be in the source Collection with its current values, so a latent gap fails before a marker exposes it; a named replay of the issue trace with the moved row unmarked passes the window law and fails only the coverage law when Query DB never reports. The grammar can hold an optimistic update across a refetch, with the checkpoint moved to its release, and a premise-directed campaign under a hold (other rows observed, a row moved ahead of the page's end, a new row further ahead, a write to the last-sorting observed row) requires a report during a hold in every case and fails a skip that reads rows held back behind the transaction; the general campaigns did not. The general grammar holds optimistic deletes as well as updates. The generated campaigns declare reach (minimum 3) for a lagging refetch after a continuation, one during a hold, and a refetch that reported while the window was full, where the loader decides between ignoring the report and repairing. A named replay of a refetch during a held delete left the window short after release, also with this PR's report and overlap changes reverted, until the fix that resumes a continuation after a synchronous answer; no window is checked during a hold, where #2148 and #2169 remain open. Reports (`loadedSubsets:invalidated`) are bounded: a step reports at most once, and only when some ordered request was answered again with different rows; named replays require no report when a one-page window, or a window's last page, only loses a row, and a sync-boundary replay requires a report when a page ending on a row without an order value has its end move earlier (on the earlier gate, which read that page as empty, it reported nothing), and another requires a report when the provider returns a one-page window's rows in reverse order (the earlier gate took the last array row as the page's end). When a page's end moves earlier, Query DB reports. A loader whose window lies inside its own first page skips the repair only while its page read past the window or it read the tie of the current end; after an insert into a top-N list moves the end, a named replay requires the loader to prove the window again with bounded reads, no full-source read, and an unchanged Query count. A filtered one-page window whose page end moves past an unseen row must still repair (found by the random campaign; a loader that always skips fails it, the f9 tie replay, and the ordered-work oracle's signalling replays). The skip also refuses while the source holds accepted sync rows back: a replay refetches while an optimistic update persists, so the report arrives before the refetched rows are visible, and a skip that reads those older rows fails it. The ordered loader state oracle checks that a prefix read issued before an ordering invalidation, which settles after the bounded repair, does not retire the repair's prefix; the repair-first order failed before that fix. A `loadSubset` call that acquires a released, invalidated page can report, because Query then refetches that page on its own. A failed repair shows only the last window or the right one. Named replays cover the issue trace after a refetch and after an invalidation, with a page-capping provider, with another live query loading the page's new row first, with an empty later page, with a page whose end moves earlier while a tie request holds its old last row, a second refetch after an invalidation, a repair failure, and a provider that honors neither cursor nor offset. Never signalling, the old overlap-to-full-source rule, ignoring empty later pages, reporting on any page change, removing the repair loop guard, and reporting once per page each fail it by assertion. A replay makes a page's application fail and checks the window shows its last result or the right one, then the next refetch restores it; recording a page before its rows are accepted cannot be distinguished, because the failed attempt reports on its own. Open: the report laws bound reports but do not require each one; a page whose end moves earlier reports even with unique order values, where no row can be missing; it does not check whether a repair waits for in-flight work; the generated campaigns use an uncapped cursor provider only (an offset-only provider and a page-capping provider each hit a loader defect that reproduces on main: #2182, #2183); the generated campaigns do not check the window after a marker flip (a named replay checks the trace where the loader stopped refilling after a joined-side removal, which the repair-retirement fix below closed); a failed repair still recovers through the loader's full-source retry; interval, focus, and reconnect refetches share the invalidation path but are not driven; NaN order values have no executable witness; overlapping `refetch: true` acquisitions on one Query hash are counted, so the first to finish does not clear the loop guard for the other, but no witness separates that from a plain set; a Query refetch that joins a running `refetch: true` acquisition on the same hash does not report, and no witness reaches it (#2185); follower tabs of a multi-tab persisted collection never receive the report (#2174). The offset-only and page-capping loader defects have `it.fails` witnesses in the oracle. A finished bounded repair used to discard its boundary when its retirement made Query DB remove a row the loader had sent; the loader then restarted paging with a smaller first page that could not retire the repair's prefix, so every later refetch read both. A named replay found by the random campaign (at four times the run count) and an ordered loader state witness fail on that, and the fix also closed the joined-side refill defect. At four times the run count, the random campaign found a short window that stopped paging before the last row of the order: a report's repair released continuation Queries, and Query DB answered their reacquisition synchronously from its cache, so the graph turn inside each request asked for more while the loader was still requesting and nothing resumed the staged continuation. Only the coverage law failed. A named replay and an ordered loader state witness (a synchronous answer whose writes the graph already processed must continue the chain) fail on that, and pass with the fix. The loader side of the report also has a second owner: the ordered-work oracle's `refetch-signal` provider re-runs each served request alone and reports when any answer changed, across that oracle's generated plans, and a signalling refetch must start no full-source request unless a bounded page in that step ended on a null or NaN rank, which takes the documented full-source fallback (a random seed found that case). Never signalling and the old overlap-to-full-source rule each fail it. It excludes a peer live query, which can keep a short window after a refetch (#2155). The persisted-collection oracle checks that a source's report reaches the Collection through the managed cache-generation wrapper and that a retired source session's report is dropped. Remaining work for this class is tracked in #2186. | | Ordered acquisition | [pagination](https://github.com/TanStack/db/blob/main/packages/db/tests/query/pagination-oracle.property.test.ts), [ordered work](https://github.com/TanStack/db/blob/main/packages/db/tests/query/ordered-work-oracle.property.test.ts), [ordered lifecycle](https://github.com/TanStack/db/blob/main/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts), [ordered loader state](https://github.com/TanStack/db/blob/main/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts), [graph scheduler](https://github.com/TanStack/db/blob/main/packages/db/tests/query/scheduler.test.ts), [issue #1880 ordered-repair review](oracle-reviews/issue-1880-ordered-repair.md), [issue #1882 custom local collation review](oracle-reviews/issue-1882-custom-local-collation.md), [issue #1898 relation-filter review](oracle-reviews/issue-1898-relation-filter.md), [PR #1909 external review](oracle-reviews/pr-1909-external-review.md), [eager ordered-window repair review](oracle-reviews/perf-ordered-window-repair.md), [reversed index reads](https://github.com/TanStack/db/blob/main/packages/db/tests/reverse-index-oracle.property.test.ts) | Complete finite provider results, inherited locale and bounded/unbounded custom collation with exact request options, query-level inheritance across source aliases, real lexical/numeric disagreement, custom comparator ties, scan/index paths, pending windows, ties/nulls, lease ownership, Effect callback gates, including an indexed source update during Effect startup, and documented repair timing. Direct LEFT-joined filters cover local finite prefixes, pending child demand, child removal, and anti-join publication with custom full-source and unindexed fallback loading; unrelated include demand cannot stall the root continuation, and an empty joined replay wakes held Effect callbacks; remote relation hints remain outside this owner. Graph scheduler checks coalescing, dependency order, synchronous loader input, and repeated-alias failure ownership. The ordered-work oracle checks that a synchronous root refill reaches D2 before joined publication; a mutant that omitted the post-loader graph step failed at the second child demand. Custom collation proves local full-source acquisition only; it does not establish provider cursor capability or backend collation fidelity. Sibling-source input during a synchronous ordered continuation returns to graph work before another ordered acquisition. Request completion is not proof of unrequested source extent. A window move during existing joined demand waits for its root continuation, including an independently held root request; current demand failure rejects the move, while retirement and an obsolete rejection do not. A bounded ordered repair resumes its refill after joined demand settles. The ordered-source-loader state test checks that an explicit failed-request retry blocked by joined demand retains its generation until it can issue a request. Multiple simultaneously pending joined plans during a window move and a public-query failed-retry overlap still need dedicated witnesses. The ordered-work oracle also checks an eager indexed window over generated upserts and deletes: the rows equal the filtered, sorted model, and each change delivers a bounded number of source rows; an unbounded or always-full repair read fails it. Its filter dimension adds a function filter and a LEFT-joined filter, and a campaign fixed to the joined premise checks that a window refilled past a distant row the query already received holds the first eligible rows. Joined filters also run as anti-joins, change eligibility through the marker or the root row's join key, and drive an Effect beside the Collection. A full-join filter keeps every root row and adds a null-ranked rootless row for each marker no root row matches; on main a bounded root prefix showed markers whose root rows were not yet loaded as unmatched rows, and restoring the old inner-or-right full-source rule fails eight tests. Null placement is a generated dimension, with the index built to match, and a query that ignores it fails six tests. A full-join campaign has few or no filler rows, a matched null-ranked root row and at least one orphan marker in every case, so its windows reach the null group where rootless rows and null-ranked root rows tie; a reconstruction check samples exactly the cases its fixed-seed run executes and requires such windows (19 rootless and 7 mixed null-group windows of 40 at one times the run count), and the Effect runs beside the Collection. Full joins read their full source, so only the rows law covers them; paging them under the old join rule fails the window law. Pinned cases cover a null-ranked root row past a prefix that ends in the null group. Paging a nulls-last full join passed this oracle but was not adopted: it cost more per delete when the window reached the null group, which no work law here measures, and #2130 tracks full-join paging. Joined filters can match a row through several markers, and the model keeps one result row per matched pair; a model that collapses them fails. The eager harness also runs an on-demand source whose provider follows the LoadSubsetOptions contract (offset only without a cursor, the applied receipt returned). Its test provider combines a sync adapter and its provider (glossary model alignment rule 2); a held request is a physical acquisition whose provider answer is delayed, a served range is the prefix and boundary one answer establishes (a model abstraction), and the peer is a second live-query Collection. The provider sorts by the requested order and keeps its own ledger of sent rows. A forwarding provider sends live changes only for rows it sent or rows that match a served subset, and its served rows obey the work bound; a loader that asks for 100 extra rows fails it. A refetching provider, like query-db-collection, sends nothing until a refetch. The refetch keeps every served ordered range whole, as the LoadSubset contract requires: it sends every row from the start of the order through the last order value a limited page returned, and rows in no range leave; the rows law over the model applies once no change is pending. After every step on either provider, the Collection must hold exactly the rows in the provider's ledger, and the query and the Effect must show their window over those rows; a mutant that drops marker removals fails this law at steps 0 and 1 of the fixed refetching campaigns, earlier than the model's rows law, and in one more campaign. A holding provider answers each load only at an `answer` step, from the rows at that moment, so live changes arrive while loads are in flight. While a load is in flight, the query and its peer either keep the window they showed at the previous checkpoint or show distinct ledger rows that pass the query, in its order; the Effect is not checked then. That is weaker than the architecture's barrier rule, under which a repair or window move keeps the last public snapshot until it settles: a barrier law fails today on on-demand joined plans, where a visible-delete repair and a held `setWindow` publish before they settle (#2131; the law and traces are kept for that fix). The provider records a request as served only once it answers it, and an abort (or an adapter unload) of a held request follows `LoadSubsetOptions.signal`: a cooperative provider installs none of its rows and its load fulfills at once, while a provider that cannot cancel in-flight work (`late` cancellation, generated) answers it anyway and its load fulfills after those rows become visible. A `release-peer` step cleans up the peer while its loads are held, which aborts them; each request counts once although its acquisition release both aborts and unloads it. The fixed held campaign reaches 5 aborted held requests and 1 late answer, and its reconstruction check requires at least 3 cooperative cancellations and 1 late answer. A held load rejects when its applied receipt rejects, and the step then fails, because no signal or cleanup rejects a receipt in these histories; that branch is unwitnessed. The held campaign pins a forwarding provider, so `release-peer` never reaches a refetching provider; that branch is unwitnessed. A request whose signal is already aborted when the provider receives it is never reached; that branch is unwitnessed. The held campaign uses a forwarding provider and an indexed source filter or a joined filter, deletes three visible rows and then upserts an eligible row with an id no other command uses; its reconstruction check runs in the same test as the fixed-seed campaign and reads its per-case counts: 35 of 40 cases issue a held request, and every one of them changes rows while it is in flight. Loading must reach a fixed point: past a request budget the provider rejects and records it, and the initial load, every step and the end of each history assert the record is clear (a loader that re-requests a prefix it already holds can loop without reaching the provider, which this cannot see). A bounded step makes at most four provider requests, and each consumer's initial proven-prefix load is bounded separately, in total transfers (a repeated request for the same prefix counts) and in distinct rows (at most the source's size); a loader that asks for 100 extra rows on every page fails that bound and the initial-load work law. Loader mutants that never resume after a boundary request, or that ignore a load request while another load is pending, fail it; a mutant that drops a window move blocked behind a pending load still survives, even with the settlement check on held moves; whether it is equivalent is open. Open (#2148): on an on-demand source with a joined or anti-joined filter, an optimistic delete of the visible row leaves the window short until the delete persists, because the next row is never loaded. On an on-demand source with a source filter, and on an eager source with any filter, the same delete refills the window correctly; these are the contrast cases. Owner: this oracle. Needed witness: the optimistic-delete dimension (kept as a patch for #2148) in the grammar, plus a pinned replay of the anti-join trace. A refetch that re-runs each page alone, as query-db-collection does today, leaves a row that moved between two pages unloaded; random seeds found that trace, a named replay fails on it, and a query-db receiving witness reproduces it (#2136 tracks the provider fix). An Electric-style provider, which skips updates to rows it never sent, is not modeled; a mocked-stream witness shows a row moving into a served ordered range never arriving under the default replica mode (#2138). At one times the run count, the campaigns reach 87 refetch steps: 39 change the window, 6 remove a visible row, and 13 lead to a further provider request. A random seed found a refetch that empties a joined window, after which the loader refills it over several page requests and publishes `[]`, one row, then the final rows; a named replay keeps that trace for #2131. No incorrect final window has appeared on that path, and no loader mutant tried so far is caught only by it; the work law does not cover it. For an on-demand joined or anti-joined filter, the initial load must not request the whole source when the first window's worth of source rows in query order all pass the join and no boundary value is null or NaN; 35 executed cases meet that premise, and a compiler that reads the whole source for every joined filter fails eight tests on this law alone. The rows served by that initial load stay within the prefix, its boundary ties and a refill, for each consumer. Later steps of those plans have no work bound. An unindexed join filter over it left the window empty on main, until the plan counted the rows it still needs; restoring the old guard fails the campaigns and a pinned replay. A step that changes the visible rows must publish, and its subscriber starts with every visible key, so a removal-only change reaches it; mutants that drop update-only or removal-only publications fail. A step may publish one intermediate window per provider request it made, and only its last publication must show the final window, because a repair or refill can publish after each round trip (#2131); the content of those intermediate windows is unchecked. The model keeps its own copy of the markers, and full-join histories can insert the root rows of orphan markers. A premise-directed campaign inserts a fresh row and then changes an initial row's eligibility alone, so the row leaves the window only through the join; it and a named replay fail when a window row past the loader's settled boundary counts toward a full window. One case in four adds a second live query over the same sources with a larger window, and its rows law holds at every checkpoint; the work laws do not apply to those cases. #2155: a second live-query Collection over a refetching on-demand provider kept a short joined window after a refetch removed a tied row (the peer showed ranks `[0, 0]` where the model shows `[0, 0, 100]`). The loader fix that resumes a staged continuation after a synchronous answer the graph already processed closed that trace. The pinned replay of that indexed trace now expects the right window. Still open: an unindexed peer over a refetching provider keeps a short joined window (the peer shows `[100]` where the model shows `[100, 101]`), which also fails without that fix; each campaign therefore still excludes the peer and refetching pairing, as its last map. Owner: this oracle. Needed witness: a pinned replay of the unindexed trace flipped to pass, and the pairing restored to the grammar. Limits: Effects are checked for rows after each step, not for publication counts or positions. Histories move the first query's window with `setWindow` (150 moves at one times the run count, 56 on demand), and its rows law and ledger law then follow the moved window; the Effect and the peer keep theirs. On a holding provider every window-move promise must have fulfilled after the final drain. A refetching provider models query-db's unload: it drops the unloaded request and removes rows that no remaining request's last answer holds. The per-step work law applies only to forwarding, immediate, single-consumer cases: it skips refetching, holding and peer cases and move steps; a two-term indexed move scanned the source four times, which no law here bounds. Multi-term joined plans stay outside this harness, on both paths; on the on-demand path, the provider's transfer is bounded only for indexed single-term source-predicate plans. Joined windows compare order values, so a wrong row tied with the expected one passes; a window row tied with the boundary counts as proven before its tie request settles, which no witness exercises. A boundary that cannot advance past a window row would repeat one page per new key; reading the boundary's range locally should always reach that row, but no witness shows it. Joined ties are compared as any valid first rows; whether source-filter ties may be fixed to the source key is open. The same grammar includes NaN and null ranks and a second-order term, and bounds both rows read and rows delivered, including the initial load and index refills; a scan law limits each change to one pass over the source. A pinned case checks that an indexed descending window places nulls first, and a lifecycle witness checks that an eager full-source window is ready with its rows at creation when its source answers synchronously. The initial fallback cells check indexed and unindexed locale strings with distinct and tied values, plus a null boundary after a warm ordered request. The warm null-boundary case crosses preload-started and constructor-started live-query sync runs: the former completes at the synchronous cut, while the latter resumes deferred on-demand acquisition asynchronously and publishes after settlement. They require an unfiltered full-source request, no public row at fallback entry, and the expected row and status at the immediate readiness check. A Promise fallback stays loading at that checkpoint. A React receiving witness checks an eager locale-string `findOne()` at the first layout commit. A Query DB receiving witness checks warm cached subsets: both raw adapter calls return literal `true` with the source rows visible at each return, and the live query is immediately ready without a fetch. The same fallback cells observe subscriber changes from before the first source write: literal-true and Promise fallbacks each publish one nonempty final-window insert after the fallback settles, and the callback sees the complete row. Promise cells show no nonempty event at the immediate cut; a duplicate-event fault fails all eight cells. Other adapters and a real on-demand React first commit remain unwitnessed. Full-source work after initial publication and repair remain outside this synchronous guarantee. A separate post-publication oracle crosses marker deletion and retargeting with literal, Promise, and held-applied-receipt fallback responses. It also settles the child demand before holding the receipt, so the public live-query loading flag isolates the root fallback. The model checks final rows and net events, while event replay checks every callback snapshot; a retained or empty intermediate window is allowed. A loader fault that ignores the pending receipt fails at the held loading-flag check. Two literal-true cleanup-failure histories require that the replacement still publishes and the adapter error reaches the public query status without making the unrelated marker commit throw; the provider's applied receipts are recorded and compared outside provider callbacks. The Promise case checks eventual coherence on an asynchronous callback path, without claiming a separate timing cut. Four held-Promise histories apply fallback writes, change another join marker before provider settlement, and compare each callback and the final winner; an injected stale extra publication fails the event comparison. Other marker actions and overlapping root-source changes remain open; the provider has no later write or rejection, so this schedule does not establish when the loader claims full-source authority. The reversed-index oracle checks every reversed read's order and filter work for both null placements. | | Indexed predicate filtering | `packages/db/tests/query/index-path-collision-oracle.test.ts` | A bounded cross product of numeric values checks exact public keys before and after adding a nested-field BTree index. Direct predicates vary argument order, bound inclusivity, reversed operands, and bound field. Public Collection subscription and live-query callbacks run with and without the index. Selected-field ordering compares a direct JavaScript sort with a public `$selected` callback; restoring the dotted-key proxy cache made that comparison fail. The original compound grouping failed four indexed cases while scan controls passed. The original callback proxy caches also failed both unindexed public routes. A hostile cleanup control preserves the primary mismatch and secondary release error. Arbitrary path segments, nullish values, custom collation, and incremental publication need separate witnesses if claimed. | @@ -3649,6 +3649,13 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. commits an older row and closes for good. Generated histories never close a tab for good, and an `it.fails` replay through the same driver and checks pins that history. + A random CI campaign (seed `-1660880085`, path + `55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`) found a second Law 2 history that + reproduces on `main`: row 1 reaches v2 and both tabs catch up, then + tab B stops, tab A catches up, and tab A stops, and the durable rows + lose row 1. The maintainer chose to leave it open with #2281; an + `it.fails` replay through the same driver and checks pins it, and the + random campaign can still draw it. Owner: this oracle. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in @@ -3848,7 +3855,15 @@ must equal the model's, which a restart would read. `persisted-peers-full` repeats the lane under full row updates, where an update replaces its row in storage too; `persisted-peers-on-demand` repeats it on demand with a demand for every row. A reset over metadata of a missing -row is too rare in generated histories to declare; a replay pins it. A reset +row is too rare in generated histories to declare; a replay pins it. A +random CI campaign (seed `654774679`) found a `persisted-peers-full` history +that reproduces on `main`: an outer transaction inserts over a bare seed, an +inner one re-inserts the row and commits, and a peer's full reload leaves the +row with the cached seed's metadata, where the model expects none (law: +the source reads its row metadata). The maintainer chose to leave it open, +tracked with #2281; an `it.fails` replay through the same model, driver, and +comparison pins its shrunk prefix, and the random campaign can still draw it. +Owner: this oracle. A reset may rewrite its rows, which its reload then applies as source writes. `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 01aa7342c6..4a6d74c630 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -1459,4 +1459,34 @@ describe(`multi-tab Electric over one persisted database`, () => { ), 60_000, ) + + // Known gap, pinned from the random campaign (seed -1660880085, path + // 55:0:1:4:4:4:1:1:1:1:3:4:4:4:4): row 1 reaches v2 on the server and both + // tabs catch up, but after tab B stops, tab A catches up, and tab A stops, + // the durable rows lose row 1. Law 2 expects every open tab and the durable + // rows to show the head. This reproduces on main and is tracked with #2281. + // It replays through the same driver and checks with the known-gap detector + // off. Remove `fails` when the durable row survives the stops. + it.fails( + `keeps a durable row after both tabs catch up to it and then stop`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `server change`, key: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `stop`, tab: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `stop`, tab: 0, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) }) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index bef1b39f32..169016d3c5 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -1209,6 +1209,29 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla it.each(peerFullReplays)(`replays under full rows: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerFullLane), ) + // Known gap, pinned from the persisted-peers-full random campaign (seed + // 654774679, shrunk to this prefix): an outer transaction inserts over a + // bare seed, an inner one re-inserts the row and commits, and a peer's full + // reload then leaves row 1 with the cached seed's metadata, where the model + // expects none. This reproduces on main and is tracked with #2281. It + // replays through the same model, driver, and comparison. Remove `fails` + // when the reload stops restoring that metadata. + it.fails( + `replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, + () => + assertPersistedHistory( + [ + { type: `begin` }, + { type: `seed`, key: 1, narrow: false, bare: true }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + { type: `reinsert`, key: 1, metadata: false }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `reload` }, + ], + persistedPeerFullLane, + ), + ) it.each(peerOnDemandReplays)(`replays on demand: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerOnDemandLane), ) diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index abd9bfcb0c..b69634e937 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -371,7 +371,6 @@ "repairNeedsReplacement": "gK", "coversSource": "gL", "standing": "gM", - "windowIsInsideOwnPrefix": "gN", "debt": "gO", "releaseOwned": "gP", "moveWork": "ae", @@ -437,5 +436,8 @@ "views": "hc", "show": "hd", "getWritableSyncTransaction": "ar", - "mergesOverRow": "b_" + "mergesOverRow": "b_", + "awaitsSiblingReplays": "gN", + "grownPrefixRows": "gZ", + "ownPrefixEnd": "he" } diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index ed783da5e1..de491b3325 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -22,6 +22,7 @@ import type { CollectionDependent, LoadSubsetOptions, LoadSubsetRequestResult, + SubscribeChangesOptions, Subscription, SubscriptionEvents, SubscriptionLoadSubsetErrorEvent, @@ -93,14 +94,9 @@ type CollectionSubscriptionOptions = { owner?: CollectionDependent } -type TruncateReplayPublicationControl = Readonly<{ - // `retired`: a reload retired the sync run and no truncate came. - start: (retired: boolean) => void - succeed: () => void - // The replay could not load every active demand again; a live query - // reports each failure on its node, in failure order. - fail?: (errors: Array) => void -}> +type TruncateReplayPublicationControl = NonNullable< + SubscribeChangesOptions[`truncateReplayPublication`] +> type TruncatePublicationState = { loadedInitialState: boolean @@ -737,6 +733,12 @@ export class CollectionSubscription if (activeFailures.length > 0) { this.abandonTruncateReplay(replayState, activeFailures) } else { + if ( + this.awaitsSiblingReplays(replayState, () => + this.checkTruncateReplayComplete(replayState), + ) + ) + return this.flushTruncateReplay(replayState) } } finally { @@ -852,13 +854,9 @@ export class CollectionSubscription } private setReadyIfIdle(): void { - const replayState = this.truncateReplayState - const hasPendingReplayWork = - replayState && - (replayState.pendingSetups > 0 || replayState.pending.size > 0) if ( this.pendingLoadSubsetParticipants.size === 0 && - !hasPendingReplayWork + !this.hasTruncateReplayWork ) { this.setStatus(`ready`) } @@ -874,6 +872,53 @@ export class CollectionSubscription this.stalePublishedRows.clear() } + /** + * A lazy demand that failed outside this subscription's replay, while a + * failed replay of another of the graph's subscriptions to this collection + * holds publication, fails this subscription's replacement too. It keeps + * the last publication until every demand load that joins this replay + * loads, a later replay succeeds, or this subscription's last demand + * retires. The failed demand's retry joins it, so it cannot publish + * without that demand's rows. + */ + failReplacement(error: Error): void { + if (this.truncateReplayState) return + const replayState = this.createTruncateReplayState( + this.collection._sync.getSyncRunGeneration(), + () => new Map(), + ) + replayState.pendingSetups = 0 + replayState.failed = true + replayState.completion.reject(error) + this.truncateReplayState = replayState + this.truncateReplacementPending = true + } + + /** + * A graph's subscriptions to one collection share a barrier. While another + * one's replay still loads, this replay stays open, so demand this one + * starts meanwhile joins it; it finishes once those replays settle. + */ + private awaitsSiblingReplays( + replayState: TruncateReplayState, + finish: () => void, + ): boolean { + const siblings = this.options.truncateReplayPublication?.siblingReplays() + void siblings?.then(() => { + if (this.truncateReplayState === replayState) finish() + }) + return siblings !== undefined + } + + /** Whether this subscription's replay still waits on its own work. */ + public get hasTruncateReplayWork(): boolean { + const replayState = this.truncateReplayState + return ( + replayState !== undefined && + (replayState.pendingSetups > 0 || replayState.pending.size > 0) + ) + } + /** Drop the replay without publishing; an unfinished wait rejects as aborted. */ private discardTruncateReplay(): void { const replayState = this.truncateReplayState @@ -1508,6 +1553,8 @@ export class CollectionSubscription this.flushTruncateReplay(replayState) return } + if (this.awaitsSiblingReplays(replayState, () => this.retireEmptyReplay())) + return this.discardTruncateReplay() this.stalePublishedRows = new Map(this.publishedRows) this.restorePublishedSnapshotTracking() diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 5deefe163d..3a79aab4ac 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -750,6 +750,9 @@ old constant, even after its first load settles: deduplication and query state may retain its identity. Request data uses stable data properties, not stateful getters. The signal and subscription references do not change, but their lifecycle remains live. Cancellation and release are not data mutations. +A truncate replay asks again for each retained demand with that demand's +request data, so an adapter sees the same `where` expression it received +before the truncate. The immutable-demand boundary matrix checks direct and deferred sync startup, adapter return and asynchronous settlement, cancellation, and release identity. @@ -757,7 +760,8 @@ A Collection subscription installs each logical subset owner before it calls the source adapter. Reentrant release during `loadSubset` therefore retires the logical owner at once, but physical release waits until the adapter returns and proves that it established an acquisition. A synchronous `loadSubset` throw -rolls the tentative owner back without calling `unloadSubset`. Logical demand +is a rejection: the load failed, it keeps its owner, and the owner's release +unloads it as it does any failed load. Logical demand retires even when `unloadSubset` fails. Each physical acquisition gets one release attempt, marked before calling adapter or error-listener code. Reentrant and repeated teardown cannot repeat it. Other acquisitions still receive cleanup, @@ -905,7 +909,10 @@ applied settlement. A failed or obsolete replacement cannot retire established coverage, and a retry requests the same union again. A partial shrink may keep one intersecting acquisition without replacement. This avoids repeatedly loading the growing prefix while allowing a changing bounded window to return -to one acquisition. +to one acquisition. The controller starts new coverage before it releases an +acquisition that no longer intersects current demand. A demand change +therefore never leaves its subscription without logical demand partway +through, which would end that subscription's truncate replay. Every sync `commit()` returns an applied receipt: `true` when that transaction's writes and events are already visible, or a promise when the @@ -1021,7 +1028,18 @@ when its window is inside that page: every window row sorts at or before the `offset + limit`-th source row, and the next source row sorts strictly after it in the leading term. The refetched page holds every row from the start through its end, and any row it left out sorts after that end, so the window -is proven. A window that needs a row past the page, such as one whose page +is proven. The loader checks the next source row through an installed row, +which is evidence only when the page read it, or when the loader read the tie +of the page's current end: the refetch re-read that tie, so every row that ties +with the end is installed. A row that another request installed proves +nothing. Neither does a tie that the loader read for an older end, because a +refetch can move the end earlier, so a row the provider never sent can tie +with the new end. When the end moved, the loader repairs. (Reading only the +new end's tie would also complete the proof, since the refetched page holds +every row before the end, and saves one request; the bounded repair is kept +because that read cost shipped bytes for one saved request per such +refetch.) A window that needs a row +past the page, such as one whose page holds rows a joined filter rejects, still repairs. So does any loader while its source holds accepted sync rows back, as behind a persisting optimistic transaction: the visible rows then predate the refetch. Core reissues the @@ -1189,7 +1207,10 @@ join-key update, or a root change, and for Collection and Effect consumers. The finite root requests; this contract does not imply that it can evaluate the relation filter remotely. Finite continuation still requires an order index; an underfilled unindexed -prefix retains the established full-source fallback. +prefix grows. Each growth reads a prefix twice as long as the last one, so +every read stays bounded and none reads the full source. The prefix grows +while growing still adds rows to the source. When the last growth added +none, the source holds no more rows and the window stays short. A custom string comparator is a local ordering contract. If any resolved source order term uses `stringSort: 'custom'`, bounded and unbounded queries @@ -1330,7 +1351,16 @@ late rejection from an abandoned sync run cannot reset the replacement sync run's requested window. A window move started during an active source replay waits for that replay and applies only after its replacement is complete. A failed replay rejects the -move without advancing the reported window. Replay completion callbacks carry +move without advancing the reported window. A replay that a newer truncate or +the retirement of its last demand discards did not fail, even though its +completion rejects with `AbortError`: the move waits for any replay that still +holds publication and then applies. If a replay it waits on fails, the move +rejects with that replay's own error, its earliest failure that still decides +it (maintainer decision). When a replay it waits on is discarded and +the query's or a source's sync run changed meanwhile, as a reload changes it, +the move rejects with that replay's `AbortError`. The move compares the sync +runs only when a replay it waits on rejects, so a reload of another source +whose replay it does not wait on leaves the move to apply. Replay completion callbacks carry their sync-run identity and become no-ops after cleanup or restart. Cleanup rejects the replay barrier, and therefore every window move waiting on it, with `AbortError`; no waiter may outlive the discarded subscription. @@ -1454,6 +1484,19 @@ normalized once by the subscription. The `loadSubset:error` event, `lastSubsetError`, and any window move waiting on that replay expose the same `Error` object. +A live query's subscriptions to one collection, such as a join and an include +of the same source, share one publication barrier. None of their replays +publishes, and none whose last demand retired closes, while another of them +still has replay work, so demand that any of them starts meanwhile joins its +own open replay. While a failed replay of any of them holds publication, a +lazy demand of any of them that fails after initial-query readiness fails its +own subscription's replacement too, so retiring the other replay's failed +demand cannot publish the window without this demand's rows. It holds until +every demand load that joins it loads, a later replay of the source succeeds, +or the subscription's last demand retires. The query retries the failed demand +at once and the retry joins the hold, so the hold ends only after the failed +demand's rows load. + ### Mutation boundaries and initial readiness A transaction `mutationFn` must not start or await collection or live-query diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 69982d45b5..3044a18a70 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -400,17 +400,40 @@ export class CollectionConfigBuilder< const request = ++this.windowRequestGeneration const sourceRecovery = this.pendingSourceRecovery() if (sourceRecovery) { - const deferred: Promise = sourceRecovery.then(async () => { + const syncRuns = this.syncRuns() + const resume = async (): Promise => { + // Wait for every replay that holds publication, including one a + // newer truncate starts meanwhile. A replay that a newer truncate or + // a retired demand discards rejects its completion with an + // AbortError, but it did not fail. Cleanup and a reload end the sync + // run the move waits in, and the move rejects with that error. + for ( + let recovery: Promise | undefined = sourceRecovery; + recovery; + recovery = this.pendingSourceRecovery() + ) { + try { + await recovery + } catch (error) { + if ( + !(error instanceof LoadSubsetOperationAbortedError) || + syncRuns !== this.syncRuns() + ) + throw error + } + } // A newer call wins, even if this deferred one resumes after it. if (request !== this.windowRequestGeneration) { await this.latestWindowRequest return } + // A failed replay rejects the call. Otherwise it applies the window. const settlement = this.setWindow(requestedWindow) // The resumed call settles on behalf of this one. this.latestWindowRequest = deferred if (settlement !== true) await settlement - }) + } + const deferred = resume() return (this.latestWindowRequest = deferred) } if (this.hasFailedSourceRecovery()) { @@ -559,7 +582,13 @@ export class CollectionConfigBuilder< ) } - failDemand(planId: string, generation: number, error: unknown): void { + failDemand( + subscription: CollectionSubscription, + collection: Collection, + planId: string, + generation: number, + error: unknown, + ): void { const demand = this.activeDemands.get(planId) if (!demand || demand.generation !== generation) return const normalized = this.recordSubsetError(error) @@ -567,8 +596,19 @@ export class CollectionConfigBuilder< if (this.liveQueryCollection?._lifecycle.status === `ready`) { // The failure settles the demand: the request errors of its keys show // it, and ordered work must not wait for a load that already ended. - // Output held for this demand publishes now. + // Output held for this demand publishes now, unless a failed replay of + // this source holds publication: the failure then holds this + // subscription's publication too, so retiring the other replay's + // failed demand cannot publish without this demand's rows. demand.settled = true + if ( + this.collectionSources.some( + (source) => + source.collection === collection && + this.subscriptions[source.sourceId]?.hasFailedTruncateReplacement, + ) + ) + subscription.failReplacement(normalized) this.maybeRunGraphFn?.() return } @@ -713,6 +753,34 @@ export class CollectionConfigBuilder< this.liveQueryCollection?._lifecycle.clearRequestError(key) } + /** This query's sync run and its sources' sync runs. */ + private syncRuns(): string { + return [ + this.syncRunGeneration, + ...this.collectionSources.map(({ collection }) => + collection._sync.getSyncRunGeneration(), + ), + ].join() + } + + /** Replays of other subscriptions to a source's collection that still load. */ + siblingReplays( + collection: Collection, + sourceId: string, + ): Promise | undefined { + const pending = this.collectionSources.flatMap((source) => { + const subscription = this.subscriptions[source.sourceId] + // A replay whose completion already settled cannot be waited on. + return source.sourceId !== sourceId && + source.collection === collection && + subscription?.hasTruncateReplayWork && + subscription.pendingTruncateReplacement + ? [subscription.pendingTruncateReplacement] + : [] + }) + return pending.length > 0 ? Promise.allSettled(pending) : undefined + } + hasPendingSourceRecovery(): boolean { return Object.values(this.subscriptions).some( (subscription) => subscription.hasPendingTruncateReplacement, diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index addcc28e75..5fdb3fbe1f 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -255,7 +255,13 @@ export class CollectionSubscriber< this.collectionConfigBuilder.settleDemand(plan.id, generation) if (update.ready instanceof Promise) { void update.ready.then(settle, (error) => - this.collectionConfigBuilder.failDemand(plan.id, generation, error), + this.collectionConfigBuilder.failDemand( + subscription, + this.collection, + plan.id, + generation, + error, + ), ) } else { settle() @@ -482,6 +488,11 @@ export class CollectionSubscriber< } return { start: (retired) => onStart?.(retired), + siblingReplays: () => + this.collectionConfigBuilder.siblingReplays( + this.collection, + this.sourceId, + ), succeed: () => { if (!isCurrent()) return this.orderedLoader?.settleFullSourceReplay() diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts index 5117e550f8..baa705d94b 100644 --- a/packages/db/src/query/live/ordered-source-loader.ts +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -114,6 +114,8 @@ export class OrderedSourceLoader { private authorityGeneration = 0 private lastPage: { count: number; boundary: unknown } | undefined private lastPrefixCount: number | undefined + // The rows the source's prefix held when it last grew. + private grownPrefixRows: number | undefined // The count of the last request that read from the start of the order, // until a continuation reads past it. private ownPrefixCount: number | undefined @@ -479,14 +481,31 @@ export class OrderedSourceLoader { continuesOrderedPrefixRepair = false, ): void { if (!this.active || this.pending || this.stagedContinuation) return - if (this.lastPrefixCount === count) { - if ((this.info.dataNeeded?.(this.settledSourceBoundary) ?? 0) > 0) { - if (continuesOrderedPrefixRepair) this.abandonOrderedPrefixRepair() - this.loadFullSource(windowOperationGeneration) - } else if (continuesOrderedPrefixRepair) { - this.finishOrderedPrefixRepair(windowOperationGeneration) + const last = this.lastPrefixCount ?? 0 + if (last >= count) { + if ((this.info.dataNeeded?.(this.settledSourceBoundary) ?? 0) === 0) { + if (continuesOrderedPrefixRepair) { + this.finishOrderedPrefixRepair(windowOperationGeneration) + } + return } - return + // A filter after the read dropped rows, so the prefix doubles: each read + // stays bounded by twice the last one, and none reads the full source. + // A pending optimistic departure can hide a row from this count, so the + // prefix grows while growing still adds rows. Once growing adds none, + // the source holds no more rows. + const rows = this.subscription.readOrderedSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: last, + }).length + if (rows <= (this.grownPrefixRows ?? -1)) { + if (continuesOrderedPrefixRepair) { + this.finishOrderedPrefixRepair(windowOperationGeneration) + } + return + } + this.grownPrefixRows = rows + count = last * 2 } this.requestAndObserve( (tracking) => @@ -634,6 +653,7 @@ export class OrderedSourceLoader { } this.lastPage = undefined this.lastPrefixCount = undefined + this.grownPrefixRows = undefined } invalidateSourceOrdering(): void { @@ -661,12 +681,21 @@ export class OrderedSourceLoader { */ revalidateLoadedCoverage(): void { // A `requiresFullSource` plan's coverage is its full-source read: nothing - // has settled before it, and it covers the source after. + // has settled before it. Coverage that one full-source read completed + // owes an ordering invalidation nothing, so the invalidation below starts + // no request for it, and the loader ignores the report (ARCHITECTURE.md). + if (!this.hasSettledSourceRequest) return + const end = this.ownPrefixEnd() + // An installed row past the page proves the source's next row only when + // the page read it, or when it ties with an end whose tie the loader + // read. A refetch can move the end before that tie, and then the bounded + // repair proves the window again. An `undefined` end also matches a + // loader that attempted no tie. No cursor expresses a nullish end, so the + // loader has started or will start a full-source read, which covers it. if ( - !this.active || - !this.hasSettledSourceRequest || - this.coversSource() || - this.windowIsInsideOwnPrefix() + end !== undefined && + ((this.ownPrefixCount ?? 0) > this.info.offset + this.info.limit || + Object.is(this.info.valueExtractorForRawRow(end), this.lastBoundary)) ) return this.invalidateSourceOrdering() @@ -676,8 +705,9 @@ export class OrderedSourceLoader { // through its end, and any row it left out sorts after that end. So a // window whose rows all sort at or before the end is right. A row after // the page that ties with its end could sort before a window row; then - // the window is not proven. - private windowIsInsideOwnPrefix(): boolean { + // the window is not proven. Returns the end when the window is inside + // the page. + private ownPrefixEnd(): Record | undefined { const size = this.info.offset + this.info.limit // A report can arrive while the refetched rows are accepted but held // back, as behind a persisting optimistic transaction. The visible rows @@ -687,19 +717,19 @@ export class OrderedSourceLoader { !this.info.dataNeeded || this.subscription.hasHeldSyncTransaction ) { - return false + return undefined } const rows = this.subscription.readOrderedSnapshot({ orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), limit: size + 1, }) // Optimistic rows are left out, so a short read proves nothing. - if (rows.length <= size) return false + if (rows.length <= size) return undefined const end = rows[size - 1]!.value - return ( - this.info.comparator(rows[size]!.value, end) > 0 && + return this.info.comparator(rows[size]!.value, end) > 0 && this.info.dataNeeded(end) === 0 - ) + ? end + : undefined } private finishOrderedPrefixRepair(windowOperationGeneration?: number): void { diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 2f23228eb5..5e54ba9e7b 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -1321,6 +1321,11 @@ export interface SubscribeChangesOptions< readonly succeed: () => void /** The replay could not load every active demand again: its failures, in failure order. */ readonly fail?: (errors: Array) => void + /** + * The replays of the graph's other subscriptions to this collection that + * still load. They share one barrier, so this replay waits for them. + */ + readonly siblingReplays: () => Promise | undefined } } diff --git a/packages/db/tests/collection-subscription-reentrancy.test.ts b/packages/db/tests/collection-subscription-reentrancy.test.ts index f5c755af50..4f383f86c6 100644 --- a/packages/db/tests/collection-subscription-reentrancy.test.ts +++ b/packages/db/tests/collection-subscription-reentrancy.test.ts @@ -163,6 +163,7 @@ describe(`Collection subscription callback reentry oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => delegate.push(`start`), succeed: () => delegate.push(`succeed`), }, diff --git a/packages/db/tests/collection-subscription-replay.test.ts b/packages/db/tests/collection-subscription-replay.test.ts index c4fe4b8422..ad5847453a 100644 --- a/packages/db/tests/collection-subscription-replay.test.ts +++ b/packages/db/tests/collection-subscription-replay.test.ts @@ -1557,6 +1557,7 @@ describe(`CollectionSubscription replay oracle`, () => { }) const subscription = collection.subscribeChanges(() => {}, { truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => { if (rejectReplacement) throw listenerFailure @@ -1908,6 +1909,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => {}, }, @@ -1973,6 +1975,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => {}, }, @@ -2161,6 +2164,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => replayStarts++, succeed: () => replaySuccesses++, }, @@ -2284,6 +2288,7 @@ describe(`CollectionSubscription replay oracle`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => {}, succeed: () => replaySuccesses++, }, @@ -2465,7 +2470,11 @@ describe(`CollectionSubscription replay oracle`, () => { }) const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, - truncateReplayPublication: { start: () => {}, succeed: succeeded }, + truncateReplayPublication: { + start: () => {}, + succeed: succeeded, + siblingReplays: () => undefined, + }, }) try { subscription.requestSnapshot({ where: firstWhere }) diff --git a/packages/db/tests/collection-subscription.test.ts b/packages/db/tests/collection-subscription.test.ts index e2b80b7a48..63eb155f9a 100644 --- a/packages/db/tests/collection-subscription.test.ts +++ b/packages/db/tests/collection-subscription.test.ts @@ -1344,6 +1344,7 @@ describe(`CollectionSubscription status tracking`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, truncateReplayPublication: { + siblingReplays: () => undefined, start: () => { replayStarts += 1 }, diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 8e6fdac1a7..1d5ed4dd06 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -50,7 +50,10 @@ const staticOracleProperties = [ `cursor-pagination.failure`, `query-db.refetch-ordered-prefix`, `ordered-loader.request-authority`, + `ordered-loader.unproven-report`, `replay.ordered-window-faults`, + `replay.ordered-failed-replay`, + `replay.ordered-preload-recovery`, `replay.keyed-request-failures`, `replay.subscription-demand-origin`, `oracle-replay.calibration`, @@ -288,6 +291,8 @@ const staticOracleProperties = [ `ordered-work.eager-joined-gap`, `ordered-work.eager-full-join`, `ordered-work.eager-held-load`, + `ordered-work.eager-proven-report`, + `ordered-work.eager-unindexed-refetch`, `ordered-work.lifecycle`, `ordered-work.nullable-lifecycle`, `pagination.async-cursor`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index fd46c89917..f9d16fbd97 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -285,12 +285,12 @@ const ownerGroups: ReadonlyArray< [ `db/tests/query/ordered-work-oracle.property.test.ts`, `ordered-work`, - `consumer-parity eager-indexed-window eager-joined-window eager-joined-gap eager-full-join eager-held-load`, + `consumer-parity eager-indexed-window eager-joined-window eager-joined-gap eager-full-join eager-held-load eager-proven-report eager-unindexed-refetch`, ], [ `db/tests/query/load-subset-replay-refinement-oracle.test.ts`, `replay`, - `ordered-window-faults keyed-request-failures subscription-demand-origin`, + `ordered-window-faults ordered-failed-replay ordered-preload-recovery keyed-request-failures subscription-demand-origin`, ], [ `db/tests/query/pagination-oracle.property.test.ts`, @@ -342,7 +342,7 @@ const ownerGroups: ReadonlyArray< [ `db/tests/query/ordered-source-loader-state-oracle.test.ts`, `ordered-loader`, - `request-authority`, + `request-authority unproven-report`, ], [ `db-sqlite-persistence-core/tests/sqlite-core-adapter-cli-runtime.test.ts`, @@ -830,6 +830,10 @@ const seedReplayProperties = new Set([`evaluators.like`]) function directReplayTestName(property: string): string | undefined { if (property === `replay.ordered-window-faults`) return `keeps the ordered window under load and release faults \\(replay campaign\\)` + if (property === `replay.ordered-failed-replay`) + return `keeps the last publication through a failed replay \\(replay campaign\\)` + if (property === `replay.ordered-preload-recovery`) + return `recovers replays that start during the initial preload \\(replay campaign\\)` if (property === `replay.keyed-request-failures`) return `distinguishes keyed request failure from reacquisition \\(replay campaign\\)` if (property === `replay.subscription-demand-origin`) diff --git a/packages/db/tests/query/live-query-collection.test.ts b/packages/db/tests/query/live-query-collection.test.ts index 1c71661f5a..520ef8ee29 100644 --- a/packages/db/tests/query/live-query-collection.test.ts +++ b/packages/db/tests/query/live-query-collection.test.ts @@ -5195,3 +5195,85 @@ describe(`createLiveQueryCollection`, () => { }) }) }) + +type TruncateWorkRow = { id: number; rank: number } +const truncateWorkRows: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, +] + +describe(`truncate replay work for an ordered window`, () => { + // A truncate reacquires the demand its subscriptions hold (ARCHITECTURE.md, + // "Replay participants and failure"). A window that holds no demand, such + // as a zero-limit window, owes no provider request and no loading state at + // a truncate. A query subscription still opens a replay then, so demand + // its graph starts during the commit belongs to that replay; the replay + // itself starts no recovery work. The one-row window is the neighbour: its + // truncate must reacquire, so a law that issued nothing would fail there. + it.each([0, 1])( + `reacquires only held demand when a %s-row window truncates`, + async (limit) => { + let sync!: Parameters[`sync`]>[0] + const requests: Array = [] + const source = createCollection({ + id: `ordered-truncate-work-${limit}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + sync.begin() + for (const value of truncateWorkRows) { + sync.write({ + type: source.has(value.id) ? `update` : `insert`, + value, + }) + } + sync.commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(limit), + ) + let observation: + { replayRequests: number; loadingAtCommit: boolean } | undefined + try { + await live.preload() + const before = requests.length + sync.begin() + sync.truncate() + sync.commit() + const loadingAtCommit = live.isLoadingSubset + await flushPromises() + observation = { + replayRequests: requests.length - before, + loadingAtCommit, + } + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + if (limit === 0) { + expect(observation).toEqual({ + replayRequests: 0, + loadingAtCommit: false, + }) + } else { + expect(observation.replayRequests).toBeGreaterThan(0) + } + }, + ) +}) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts index 03fabfe257..8a11da8ce3 100644 --- a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -55,6 +55,22 @@ import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' * it is a settlement-phase release, which must not hold back publication. * An unload that throws while the replay starts can fail the replay, which * law 9 allows, so the law does not judge that case. + * - **Retention:** a truncate replay is one publication barrier (law 9 and + * "Replay participants and failure"). It rises when a source truncates and + * falls when its replay succeeds. Every load the source starts while its + * replay is open joins the barrier; a load that a newer truncate aborts or + * whose demand retires stops participating, and work an older replay + * started still holds the newer one. A participant that fails fails the + * replay while its logical demand is held. Once that demand retires, its + * failure no longer counts, and demand started meanwhile joins the + * replacement, which publishes when it succeeds; otherwise only a later + * truncate of that source can replace the failed replay. While + * any barrier is up, the query shows exactly the rows it showed when the + * first one rose: partly replayed rows, rank moves, and marker changes stay + * private, and a failed replay keeps that result instead of ending the + * query. The law is checked after every command and once loading is quiet, + * for barriers that rise after the query first became ready; before that + * there is no complete publication to keep, and initial readiness governs. * - **Convergence:** after a final truncate with no faults, and an explicit * window move that revalidates a window whose own move failed, the window * equals the model. @@ -69,23 +85,39 @@ import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' * - **Terminal and request errors:** a load that fails before the query * first becomes ready is fatal (an initial load failure), except on a lazy * join source, which owns its demand; the query then stays in `error` and - * convergence does not apply. After initial-query readiness a failed load - * is a request error on the query's own node and the query stays `ready` - * (ARCHITECTURE.md law 16): a newly issued keyed snapshot request (lazy join - * demand on either side; the planner loads an inner join's root by key when - * the marker source is eager), or a load that reacquires an established - * request after truncate, which keeps the last publication until the next - * replay. A demand load asks for keys (a `where` with no `orderBy`). A - * truncate may change the private window and issue a new include request - * while it commits; that request still starts fresh lazy demand. - * Reacquisition of a previously settled request is an ordinary replay, even - * when it starts later. Replay renews demand held by its subscription - * before the truncate. A new subscription's first demand remains ordinary - * during another's replay; after readiness both origins are request errors. - * The check permits `error` only after a newly issued keyed snapshot - * request failed, or a load failed while the query was still loading. + * convergence does not apply. After initial-query readiness no load failure + * ends the query (ARCHITECTURE.md law 16): a failed lazy demand load is a + * request error on the query's own node, and the query stays `ready`. A + * demand load asks for keys (a `where` with no `orderBy`); either side can + * be the lazy one, since the planner loads an inner join's root by key when + * the marker source is eager. A failed load that belongs to a truncate's + * recovery also fails that replay, so the query keeps its last publication + * until a later replay of that source (law 9). Two kinds of load belong to + * a recovery: + * - a replay load, which asks again for a demand the source held before + * the truncate; and + * - a demand load that the graph starts while the truncate commits. The + * commit deletes the source's rows privately, so the query can demand + * keys that it never published, such as a row that a LEFT join filter + * now keeps. + * A load that joins an open replay belongs to it as well, which covers a + * demand that the graph starts after the commit while the replay still + * loads, from any of the query's subscriptions to that source. The model + * judges that only without a peer (see Limits). A load that throws while + * it starts, as a synchronous adapter does, is a rejection: a throw inside + * an open replay fails that replay. The throw happens while the + * subscription registers the demand, so it joins a replay only if that + * replay still has a participant, a failure, or no participant yet. So a + * query may end in `error` only after a load failed while that query was + * still loading. At each history cut, the query lists every request failure + * that the request-error account still owes, by its injected error object. + * After recovery replays every source, no request error stays. * - **Status:** with no failed load, a quiet live query and its peer are * `ready`; a release that throws does not change status. + * - **Initial readiness:** when the history starts during the initial + * preload, that preload has resolved or rejected once loading is quiet. A + * failure before the first ready ends the query, so the preload cannot wait + * for a later truncate. * - **Reported faults:** an injected fault may surface as a rejected load, a * thrown release, a window move rejected by a load failure, or an error the * subscription rethrows from a microtask. A window move whose load a @@ -93,8 +125,15 @@ import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' * window move. No other error may escape. * * History grammar: rank moves, holding the next loads, settling the oldest - * held load with success or failure, truncates (optionally arming the next - * unload once the replay has started), and arming the next unload to throw. The driver uses a real on-demand Collection with an index and an + * or newest held load with success or failure (optionally keeping later + * loads held), + * truncates (optionally arming the next unload once the replay has started), + * arming the next unload to throw, and arming the next marker load to throw + * while it starts. An armed throw that no history load consumed expires + * before the drain, whose loads succeed. A history may start while the initial + * preload's loads are held. A marker load whose request was aborted either + * resolves without writing or rejects with `AbortError`, as the source + * contract asks; an `AbortError` is not a failed load. The driver uses a real on-demand Collection with an index and an * ordered live query. At the end, every held load succeeds in order. * * The joined scenarios cross a LEFT join that keeps marked or unmarked rows @@ -107,10 +146,26 @@ import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' * * Limits: one ordered source; no full joins (the ordered-work oracle owns * them); includes one level deep; and a provider that reads its rows at the time it - * answers. The status check relates an observed initiating request rejection - * to dependent query status; it does not independently establish correct - * settlement of that request receipt. Peer scenarios share one history-level - * failure allowance, so they do not establish receiver-specific fault ownership. + * answers. A query's subscriptions to one source share one replay barrier, + * so a load joins it whichever of them issued it. A peer is another query + * with its own barrier, and the adapter boundary cannot tell which query + * issued a load, so the retention law skips histories with a peer. A + * release that throws while a replay starts may + * fail that replay, so the retention law skips a barrier where one did. + * A load that throws acquires nothing, so the adapter never sees its demand + * retire, and any graph run can retire it. The retention law therefore skips + * the marker barrier from a throw to the end of its episode (a barrier + * raised while an earlier one in the episode is ambiguous inherits that). So + * a design that dropped a thrown demand and let its replay succeed would pass + * wherever only retention could see it. The request-error account judges a + * failure in a replay by its own subscription's verdict, using the + * subscription that each load names; publication retention stays shared + * across the query's subscriptions to a source. It reads only the query's + * own node and permits extra listed failures. With a peer, either query's + * node may list a failure, and the replay-error identity law skips peer + * histories. The replay-error identity law judges only rejections with a failure of a replay + * the move waited on; a move that resumes into another failed replay is not + * judged. */ type OrderedFailureState = `initiating-keyed-request-failed` | `keyed-reacquisition-failed` @@ -119,14 +174,19 @@ type OrderedRow = { id: number; rank: number } type OrderedCommand = | { type: `move`; id: number; rank: number } | { type: `hold` } - // `keepHolding`: loads that start during the step, such as a retry the - // failure starts, stay held. - | { type: `settle`; ok: boolean; keepHolding?: boolean } + // `keepHolding` leaves later loads held, so a history can settle one load + // while the next stays pending. `newest` settles the most recently held + // load instead of the oldest, so a later load can settle before an earlier + // one, as independent provider requests may. + | { type: `settle`; ok: boolean; keepHolding?: boolean; newest?: boolean } | { type: `truncate`; armAfterStart: boolean } | { type: `unloadThrows` } | { type: `window`; offset: 0 | 1; limit: 1 | 2 | 3 } | { type: `mark`; id: number; on: boolean } | { type: `truncateMarkers` } + // The next marker load throws while it starts, as an adapter that fails + // synchronously does, instead of returning a promise that rejects. + | { type: `markerThrows` } // The query's order and window, whether ranks may tie, and whether a second // live query reads the same source. @@ -145,6 +205,12 @@ type OrderedScenario = { markerMode: `eager` | `on-demand` // Whether each shown row includes its markers as a `toArray` child. includes: boolean + // How a marker load whose request was aborted settles. The source contract + // asks a source to reject it with `AbortError` once its accepted rows apply; + // a source that cannot cancel may also resolve it without writing. + abortedMarkerLoads: `resolve` | `reject` + // Whether the history starts while the initial preload is still held. + historyDuringPreload: boolean } const orderedScenario: fc.Arbitrary = fc.record({ @@ -161,6 +227,8 @@ const orderedScenario: fc.Arbitrary = fc.record({ ), markerMode: fc.constantFrom(`eager` as const, `on-demand` as const), includes: fc.boolean(), + abortedMarkerLoads: fc.constantFrom(`resolve` as const, `reject` as const), + historyDuringPreload: fc.boolean(), }) const orderedCommand: fc.Arbitrary = fc.oneof( @@ -181,6 +249,8 @@ const orderedCommand: fc.Arbitrary = fc.oneof( { weight: 3, arbitrary: fc.constant(true) }, { weight: 1, arbitrary: fc.constant(false) }, ), + keepHolding: fc.boolean(), + newest: fc.boolean(), }), }, { @@ -197,6 +267,7 @@ const orderedCommand: fc.Arbitrary = fc.oneof( weight: 1, arbitrary: fc.constant({ type: `truncateMarkers` as const }), }, + { weight: 1, arbitrary: fc.constant({ type: `markerThrows` as const }) }, { weight: 2, arbitrary: fc.record({ @@ -233,11 +304,33 @@ type OrderedReach = { innerJoinedRuns: number markerLoads: number markerTruncates: number - abortedMoves: number terminalErrors: number // A failed load after initial-query readiness left a request error on the // query's own node, and the query stayed ready (ARCHITECTURE.md law 16). requestErrors: number + // Cuts where the request-error account owed a request error. + requestErrorChecks: number + // A subscription's load succeeds while a failure of that subscription is + // still owed. + successWhileSubscriptionOwes: number + // Deferred window moves that rejected with a failure of a replay they + // waited on. + deferredMoveRejections: number + replayLoads: number + abortRejections: number + preloadChecks: number + heldPublicationChecks: number + retainedFailedReplays: number + participantDemandFailures: number + includeDemandFailures: number + siblingDemandFailures: number + failedSiblingDemandFailures: number + // Lazy demand whose load threw while it started, inside its replay. + thrownReplayDemands: number + // A failed demand retires while another failure still holds its barrier. + partialFailureRetirements: number + preloadReplayFailures: number + discardedReplayMoves: number } const replayDemandStates = [ @@ -246,20 +339,22 @@ const replayDemandStates = [ ] as const type ReplayDemandState = (typeof replayDemandStates)[number] -// The demand origin determines the terminal consequence for named replays and -// generated histories alike. Reach checks separately prove each premise ran. +// Two demand origins fail behind the marker source's replay barrier: new +// demand that the graph starts while the marker truncate commits, and a +// settled demand that the replay asks for again. After the first ready, the +// barrier keeps the last publication for both, so neither ends the query +// (ARCHITECTURE.md, "Replay participants and failure"). Reach checks +// separately prove each premise ran. function checkReplayDemandFailure( reach: OrderedReach, entered: ReadonlySet, ) { - // The query is ready before either failure, so neither may end it, and - // it stays available for later replay (ARCHITECTURE.md law 16). An - // ordinary demand failure is a request error on the query's node; a failed - // replay keeps the last publication until the next replay. - for (const state of replayDemandStates) { - if (!entered.has(state)) continue - expect(reach.terminalErrors, `${state} is nonterminal`).toBe(0) - } + if (entered.size > 0) + expect( + reach.terminalErrors, + `a failure behind the replay barrier is nonterminal`, + ).toBe(0) + // New demand that fails is a request error on the query's node. if (entered.has(replayDemandStates[0])) expect(reach.requestErrors, `ordinary failure is a request error`).toBe(1) } @@ -273,6 +368,8 @@ const ordinaryOrderedScenario: OrderedScenario = { joined: false, markerMode: `eager`, includes: false, + abortedMarkerLoads: `resolve`, + historyDuringPreload: false, } async function checkOrderedReplay( @@ -300,6 +397,7 @@ async function checkOrderedReplay( return false } let moveOrderViolation: string | undefined + let moveErrorViolation: string | undefined const settleMove = (move: number) => { if (move !== latestMove && !settledMoves.has(latestMove)) { moveOrderViolation ??= `window move ${move} settled before the newest move ${latestMove}` @@ -312,15 +410,16 @@ async function checkOrderedReplay( ) // With a joined source, only marked rows qualify; row 2 starts unmarked. const marked = new Set([1, 3, 4, 5]) + const eligible = (id: number) => + !scenario.joined || + (scenario.joined === `none` ? !marked.has(id) : marked.has(id)) // Ties leave the order among equal ranks open, so the model predicts the - // window's rank sequence. Each shown row must also carry its current rank. + // window's rank sequence. Each shown row must also carry its current rank, + // be eligible, and appear once, so a tie cannot hide a row the filter + // rejects. const expectedRanks = (shown: { offset: number; limit: number }) => [...truth.values()] - .filter( - ({ id }) => - !scenario.joined || - (scenario.joined === `none` ? !marked.has(id) : marked.has(id)), - ) + .filter(({ id }) => eligible(id)) .map(({ rank }) => rank) .sort((left, right) => sign * (left - right)) .slice(shown.offset, shown.offset + shown.limit) @@ -337,7 +436,12 @@ async function checkOrderedReplay( expect(row.rank, `${label}: row ${row.id} rank`).toBe( truth.get(row.id)?.rank, ) + expect(eligible(row.id), `${label}: row ${row.id} is eligible`).toBe(true) } + expect( + new Set(rows.map(({ id }) => id)).size, + `${label}: each row once`, + ).toBe(rows.length) if (new Set(rows.map(({ rank }) => rank)).size < rows.length) { reach.tiedWindows++ } @@ -352,75 +456,197 @@ async function checkOrderedReplay( const installed = new Map() // Each load records the truncate count it was issued after, so a fault can // be placed before or after the replay's loads settle. - type Load = { epoch: number; outcome: `pending` | `ok` | `failed` } + type Load = { + epoch: number + outcome: `pending` | `ok` | `failed` + // The error this history injected when the load failed. + error?: Error + // Its source, the keys it asked for, and whether it is an ordered read. + source?: BarrierSource + keys?: Set + ordered?: boolean + } const loads: Array = [] + // Retention model. Each source with an open or failed replay has a + // barrier. `pending` holds the loads that still hold its publication: the + // loads started while it is up, and work an older replay started. `own` + // holds the loads this replay started, whose failure fails it. A failure + // is scoped to its logical demand: once that demand retires, it no longer + // fails the replay, and demand started meanwhile joins the replacement. + type BarrierSource = `rows` | `markers` + type Barrier = { + failures: Map + // Every failure of this replay, in failure order, including failures + // whose demand later retired. + history: Array + // When the barrier rose, on the request-error account's clock. + raisedAt: number + // A release threw while this replay started, which may fail it. + ambiguous: boolean + pending: Map + own: Set + } + const barriers = new Map() + // Every barrier this history raised, in raise order. + const raisedBarriers: Array = [] + // The rows the query showed when the first barrier rose, if it had + // published by then. Before the first ready there is no complete + // publication to keep, and the initial-readiness law governs. + let retainedRows: string | undefined + let retainedAfterReady = false + const raiseBarrier = (source: BarrierSource) => { + lowerSettledBarriers() + // A newer truncate discards the replay that a pending window move waits + // behind. + if (barriers.size > 0 && !settledMoves.has(latestMove)) { + reach.discardedReplayMoves++ + } + if (barriers.size === 0) { + retainedRows = JSON.stringify(live.toArray) + retainedAfterReady = live.status === `ready` + } + const barrier: Barrier = { + failures: new Map(), + history: [], + raisedAt: ++requestClock, + // The retained publication dates from the first barrier, so an + // ambiguous barrier earlier in this episode leaves it ambiguous. + ambiguous: [...barriers.values()].some(({ ambiguous }) => ambiguous), + pending: new Map(barriers.get(source)?.pending), + own: new Set(), + } + barriers.set(source, barrier) + raisedBarriers.push(barrier) + } + // A replay whose participants all settled without a failure publishes. + const lowerSettledBarriers = () => { + for (const [source, barrier] of barriers) { + if (barrier.failures.size === 0 && barrier.pending.size === 0) { + barriers.delete(source) + recoveries.push({ source, raisedAt: barrier.raisedAt }) + } + } + if (barriers.size === 0) retainedRows = undefined + } + // Returns whether the load joined a replay. + const joinBarrier = ( + source: BarrierSource, + load: Load, + options: LoadSubsetOptions, + ) => { + const barrier = barriers.get(source) + if (!barrier) return false + barrier.pending.set(load, options) + barrier.own.add(load) + return true + } + const settleParticipant = ( + source: BarrierSource, + load: Load, + options: LoadSubsetOptions, + ok: boolean, + ) => { + const barrier = barriers.get(source) + // The replay has not failed, and other participants still load. + const replayLoading = + barrier !== undefined && + barrier.failures.size === 0 && + barrier.pending.has(load) && + barrier.pending.size > 1 + // An aborted request is obsolete demand, and a released one no longer + // participates, so neither fails the replay. + if ( + !ok && + barrier?.pending.has(load) && + barrier.own.has(load) && + !options.signal?.aborted + ) { + barrier.failures.set(load, options) + barrier.history.push(load) + } + // A later load of the replay that loads a failed load's keys, such as + // the retry the query starts at once, decides those keys, so that + // failure no longer fails the replay (ARCHITECTURE.md law 17: a failure + // counts for a key no later held load loaded). + if (ok && barrier?.pending.has(load) && !options.signal?.aborted) { + for (const failed of barrier.failures.keys()) { + if ( + loads.indexOf(failed) < loads.indexOf(load) && + failed.ordered === load.ordered && + failed.keys !== undefined && + [...failed.keys].every((key) => load.keys?.has(key)) + ) + barrier.failures.delete(failed) + } + } + barrier?.pending.delete(load) + // A replay completes when its last participant settles, so demand that + // starts after it does not join it. + lowerSettledBarriers() + return replayLoading + } + const retireParticipants = ( + source: BarrierSource, + options: LoadSubsetOptions, + ) => { + const barrier = barriers.get(source) + if (!barrier) return + const failures = barrier.failures.size + for (const owned of [barrier.pending, barrier.failures]) { + for (const [load, held] of owned) { + if (held === options) owned.delete(load) + } + } + // One failed demand retires while another still holds the barrier. + if (barrier.failures.size > 0 && barrier.failures.size < failures) { + reach.partialFailureRetirements++ + } + } + // A query's subscriptions to one source share one replay barrier, so a + // load belongs to it whichever subscription issued it. A peer is another + // query with its own barrier, and the adapter boundary cannot tell which + // query issued a load. + const oneSubscription = (_source: BarrierSource) => !scenario.peer const acquired = new Map() - const markerRequests = new WeakSet() const held: Array<(ok: boolean) => void> = [] let holding = false let truncates = 0 let throwNextUnload = false // Window moves whose settlement waits on held loads. const pendingMoves: Array> = [] - // Release-fault reach still distinguishes a throw while truncate commits. + // The source truncate is committing. let truncating = false let anyLoadFailed = false let earlyThrowSinceTruncate = false - // Replay renews the leases held at its boundary. The adapter's subscription - // token identifies an owner; evaluating its immutable predicate on this - // finite key domain identifies the demand independently of production replay - // state. Consume each captured lease once, so a new owner or different keys - // cannot inherit another demand's replay classification. - const demandKey = (options: LoadSubsetOptions, marker: boolean) => - JSON.stringify( - [1, 2, 3, 4, 5].filter((id) => - options.where - ? evaluateReferenceExpression( - options.where, - marker ? { id, rowId: id } : truth.get(id)!, - ) === true - : true, - ), - ) - const replayDemands = { - source: [] as Array, - marker: [] as Array, - } - let markerTruncating = false - let hadMarkerDemand = false - const isReplayDemand = (options: LoadSubsetOptions, marker: boolean) => { - const previous = marker ? replayDemands.marker : replayDemands.source - const index = previous.findIndex( - (previousOptions) => - previousOptions.subscription === options.subscription && - demandKey(previousOptions, marker) === demandKey(options, marker), - ) - if (index < 0) return false - previous.splice(index, 1) - return true - } - let demandFailed = false - let initiatingRequestFailed = false + // The marker truncate is committing, and how many marker truncates ran. + let markerCommitting = false + let markerTruncates = 0 + let markerThrowArmed = false // A failed initial load is fatal too (error 229). The queries are created // below; a load can only fail after they start. const queries: Array<{ status: string }> = [] - let failedWhileLoading = false - // Adapter callbacks set these flags, so read them through a call. - // A terminal result must satisfy both the independent retained-demand - // account and the initiating-request observation at each checked cut. - const fatalLoadFailed = () => demandFailed || failedWhileLoading - const initiatingFailure = () => initiatingRequestFailed || failedWhileLoading - const isDemand = (options: LoadSubsetOptions) => - options.where !== undefined && options.orderBy === undefined - // Observe the initiating snapshot request's result. A new lazy keyed - // request may start while truncate commits, whereas reacquisition of an - // established request does not issue another snapshot request. The receipt - // connects an injected adapter failure to its initiating request without - // using replay state, callback timing, or predicate identity. + // The queries that were still loading when a load failed: each may end in + // an initial load failure. + const failedWhileLoading = new Set<{ status: string }>() + const noteLoadingFailure = () => { + for (const query of queries) + if (query.status === `loading`) failedWhileLoading.add(query) + } + // The driver observes each initiating keyed snapshot request's result + // receipt and matches only the exact errors this history injected into + // loads, so it uses no replay state, callback timing, or predicate + // identity. A request that asks again for an established demand issues no + // new snapshot request. The two failure premises that the keyed-request + // campaign declares are measured this way. const injectedLoadFaults = new WeakSet() - // Exact acquisition receipts are used only to measure the two failure - // premises. This requires no referential stability of predicate expressions. + // Exact acquisition receipts, used only to measure the two failure + // premises; this needs no referential stability of predicate expressions. const snapshotAcquisitions = new WeakSet() + const initiatingRequestRejected = (error: unknown) => { + if (error instanceof Error && injectedLoadFaults.has(error)) { + recordFailureReach?.(`initiating-keyed-request-failed`) + } + } const requestSnapshot = CollectionSubscription.prototype.requestSnapshot const snapshotRequests = vi .spyOn(CollectionSubscription.prototype, `requestSnapshot`) @@ -428,22 +654,190 @@ async function checkOrderedReplay( if (options?.where === undefined || options.orderBy !== undefined) { return requestSnapshot.call(this, options) } - return requestSnapshot.call(this, { - ...options, - onLoadSubsetResult: (result, acquisition, release) => { - snapshotAcquisitions.add(acquisition) - if (result instanceof Promise) { - void result.catch((error: unknown) => { - if (error instanceof Error && injectedLoadFaults.has(error)) { - initiatingRequestFailed = true - recordFailureReach?.(`initiating-keyed-request-failed`) - } - }) - } - options.onLoadSubsetResult?.(result, acquisition, release) - }, - }) + try { + return requestSnapshot.call(this, { + ...options, + onLoadSubsetResult: (result, acquisition, release) => { + snapshotAcquisitions.add(acquisition) + if (result instanceof Promise) { + void result.catch(initiatingRequestRejected) + } + options.onLoadSubsetResult?.(result, acquisition, release) + }, + }) + } catch (error) { + // A load that throws outside a replay rejects its request here. + initiatingRequestRejected(error) + throw error + } }) + // Lazy join demand asks for keys; ordered and full-source loads do not. + // An ordered tie request also has a `where` and no `orderBy`, but it asks + // for rows by rank, so it is not demand. + const readsRank = (expression: unknown): boolean => + Array.isArray(expression) + ? expression.some(readsRank) + : typeof expression === `object` && expression !== null + ? Object.entries(expression).some(([field, value]) => + field === `path` && Array.isArray(value) + ? value.includes(`rank`) + : readsRank(value), + ) + : false + const isDemand = (options: LoadSubsetOptions) => + options.where !== undefined && + options.orderBy === undefined && + !readsRank(options.where) + // Request-error account (ARCHITECTURE.md law 16; glossary, "request + // error"). After initial-query readiness, a failed lazy demand load is a + // request error on the query's own node. The error stays until one of these + // meets it: + // - its demand retires, which aborts its request and unloads it; + // - a later load of the source asks for some of its keys: that newer + // request of the demand decides them, and keys it no longer asks for + // left the demand (ARCHITECTURE.md law 16: a lazy demand's error clears + // when that demand settles or is released; a key's latest load decides + // it); + // - a later load of the same source covers its keys; + // - a replay of its source that rose after the failure completes, since a + // replay asks again for every held demand. + // A failure that joins a replay waits for its subscription's verdict, + // which comes once that subscription's participants settled, so it owes + // no error before then. The barrier that holds publication stays shared + // across the query's subscriptions to the source. Keys come from evaluating the request's + // `where` over a small key domain, independent of production's bookkeeping. + // The account combines the production concepts of a failed request and the + // replay verdict that decides it; it names no production structure. + let requestClock = 0 + const keyDomain = Array.from({ length: 14 }, (_, index) => index - 2) + const requestedKeys = (source: BarrierSource, options: LoadSubsetOptions) => + new Set( + keyDomain.filter( + (id) => + !options.where || + evaluateReferenceExpression( + options.where, + source === `markers` ? { id, rowId: id } : { id, rank: 0 }, + ) === true, + ), + ) + type RequestFailure = { + // The failed load; its injected error is the failure the node lists. + load: Load + source: BarrierSource + options: LoadSubsetOptions + keys: Set + at: number + // The replay the load joined, whose verdict decides it. + barrier?: Barrier + } + const requestFailures: Array = [] + const loadedKeys: Array<{ + source: BarrierSource + keys: Set + at: number + }> = [] + const recoveries: Array<{ source: BarrierSource; raisedAt: number }> = [] + const unloaded = new WeakSet() + const allReady = () => queries.every(({ status }) => status === `ready`) + const recordLoaded = ( + source: BarrierSource, + keys: Set, + options: LoadSubsetOptions, + ) => { + // A load of a subscription succeeds while a failure of that subscription + // that it does not cover is still owed: the state in which a hold that + // records no failed demand could complete early. + if ( + requestFailures.some( + (failure) => + failure.source === source && + failure.options.subscription === options.subscription && + requestErrorOwed(failure) && + [...failure.keys].some((key) => !keys.has(key)), + ) + ) + reach.successWhileSubscriptionOwes++ + loadedKeys.push({ source, keys, at: ++requestClock }) + } + const recordRequestFailure = ( + load: Load, + source: BarrierSource, + options: LoadSubsetOptions, + startedReady: boolean, + participant: boolean, + ) => { + if (!startedReady || !allReady() || !isDemand(options)) return + if (options.signal?.aborted) return + requestFailures.push({ + load, + source, + options, + keys: requestedKeys(source, options), + at: ++requestClock, + barrier: participant ? barriers.get(source) : undefined, + }) + } + const requestErrorOwed = (failure: RequestFailure) => { + if (failure.options.signal?.aborted || unloaded.has(failure.options)) + return false + // A subscription's replay fails once its own participants settled + // (ARCHITECTURE.md law 17); only a success also waits for the query's + // other subscriptions to the source. Each load names the subscription + // that issued it, so the verdict waits for that subscription's loads. + const { barrier } = failure + if ( + barrier && + barriers.get(failure.source) === barrier && + [...barrier.pending.values()].some( + ({ subscription }) => subscription === failure.options.subscription, + ) + ) + return false + const at = loads.indexOf(failure.load) + const askedLater = loads.some( + (later, index) => + index > at && + later.source === failure.source && + !later.ordered && + [...failure.keys].some((key) => later.keys?.has(key)), + ) + if (askedLater) return false + const covered = loadedKeys.some( + (loaded) => + loaded.source === failure.source && + loaded.at > failure.at && + [...failure.keys].every((key) => loaded.keys.has(key)), + ) + const recovered = recoveries.some( + (recovery) => + recovery.source === failure.source && recovery.raisedAt > failure.at, + ) + return !covered && !recovered + } + // The truncate count of its source when each request was first issued. A + // replay asks again for a held demand with the same request data, because + // request data is immutable (ARCHITECTURE.md), and a new demand builds a + // new `where`. So a load whose `where` was first issued before its source's + // latest truncate is a replay load. + const firstIssued = new WeakMap() + // A load belongs to a recovery when it is a replay load, or when the graph + // starts it while its source's truncate commits. + const belongsToRecovery = ( + options: LoadSubsetOptions, + sourceTruncates: number, + committing: boolean, + ) => { + const where = options.where + if (!where) return committing + if (!firstIssued.has(where)) firstIssued.set(where, sourceTruncates) + const replay = firstIssued.get(where)! < sourceTruncates + // The campaigns require replay loads, so production that rebuilt request + // data for a replay fails this witness instead of making every replay + // load look like new demand. + if (replay && isDemand(options)) reach.replayLoads++ + return committing || replay + } let violation: string | undefined // The faults this history injects. They may surface as rejected loads, as // thrown releases, or as errors the subscription rethrows from a microtask, @@ -491,9 +885,24 @@ async function checkOrderedReplay( innerJoinedRuns: scenario.joined === `inner` ? 1 : 0, markerLoads: 0, markerTruncates: 0, - abortedMoves: 0, terminalErrors: 0, requestErrors: 0, + requestErrorChecks: 0, + successWhileSubscriptionOwes: 0, + deferredMoveRejections: 0, + replayLoads: 0, + abortRejections: 0, + preloadChecks: 0, + heldPublicationChecks: 0, + retainedFailedReplays: 0, + participantDemandFailures: 0, + includeDemandFailures: 0, + siblingDemandFailures: 0, + failedSiblingDemandFailures: 0, + thrownReplayDemands: 0, + partialFailureRetirements: 0, + preloadReplayFailures: 0, + discardedReplayMoves: 0, } type Marker = { id: number; rowId: number } let markerSync!: Parameters[`sync`]>[0] @@ -523,71 +932,195 @@ async function checkOrderedReplay( operations.markReady() return { // Marker loads share the history's held and failing loads, so - // joined work can stay pending or fail. - loadSubset: async (options) => { - const replay = isReplayDemand(options, true) - // These are adapter-entry witnesses, independent of the expected - // terminal status. A new logical demand cannot renew a captured - // demand merely because both use the same source or subscription. - const newDemandDuringReplay = - !replay && - isDemand(options) && - markerTruncating && - hadMarkerDemand - const settledReplay = - replay && - queries.length > 0 && - queries.every(({ status }) => status === `ready`) - markerRequests.add(options) - acquired.set(options, (acquired.get(options) ?? 0) + 1) - const load: Load = { epoch: truncates, outcome: `pending` } - loads.push(load) - reach.markerLoads++ - const demand = !replay && isDemand(options) - if (holding) { - const ok = await new Promise((resolve) => - held.push(resolve), - ) - if (!ok) { - if (!options.signal?.aborted) { - if (newDemandDuringReplay) - recordReach(`new-demand-failed-during-marker-replay`) - if (settledReplay) - recordReach(`settled-marker-demand-replay-failed`) + // joined work can stay pending or fail. An armed load throws while + // it starts: it joins its source's replay like any load, fails at + // once, and is judged by the same recovery rule as a rejection. + loadSubset: (options) => { + if (markerThrowArmed) { + markerThrowArmed = false + // A throw is a rejection (ARCHITECTURE.md): the load holds its + // owner, whose release unloads it. + acquired.set(options, (acquired.get(options) ?? 0) + 1) + const load: Load = { + epoch: truncates, + outcome: `failed`, + source: `markers`, + keys: requestedKeys(`markers`, options), + } + loads.push(load) + reach.markerLoads++ + // The throw happens while the subscription registers the + // demand, so the barrier here is the replay's state then. One + // whose participants all retired or settled has completed, and + // the demand does not join it. A replay that no participant + // joined yet is still starting its loads, which hold + // publication (ARCHITECTURE.md), so a throw then joins it. + const open = barriers.get(`markers`) + const participant = + open !== undefined && + (open.pending.size > 0 || + open.failures.size > 0 || + open.own.size === 0) && + joinBarrier(`markers`, load, options) + // A throw that joins a replay that is only loading. + const strict = + participant && !open.ambiguous && open.failures.size === 0 + const recovery = + belongsToRecovery(options, markerTruncates, markerCommitting) || + (participant && oneSubscription(`markers`)) + settleParticipant(`markers`, load, options, false) + // A load that threw acquired nothing, so the adapter never sees + // its demand retire, and any graph run can retire it. The + // marker barrier's retention is ambiguous from here to the end + // of its episode (see Limits). + if (open) open.ambiguous = true + anyLoadFailed = true + // A throw inside a replay that is only loading. + if (recovery && strict && isDemand(options)) { + reach.thrownReplayDemands++ + } + noteLoadingFailure() + reach.failedLoads++ + const thrown = inject(new Error(`marker load threw`)) + load.error = thrown + injectedLoadFaults.add(thrown) + throw thrown + } + return (async () => { + const startedReady = allReady() + acquired.set(options, (acquired.get(options) ?? 0) + 1) + const load: Load = { + epoch: truncates, + outcome: `pending`, + source: `markers`, + keys: requestedKeys(`markers`, options), + } + loads.push(load) + reach.markerLoads++ + const participant = joinBarrier(`markers`, load, options) + // Records the request's first truncate count and replay reach. + belongsToRecovery(options, markerTruncates, markerCommitting) + // The two demand origins behind the marker barrier: a demand + // asked for again, while every query is ready, and new demand + // that the graph starts while the marker truncate commits. + const reload = + options.where !== undefined && + firstIssued.get(options.where)! < markerTruncates + const settledReplay = + reload && + isDemand(options) && + queries.length > 0 && + queries.every(({ status }) => status === `ready`) + const newDemandDuringReplay = + !reload && markerCommitting && isDemand(options) + if (holding) { + const ok = await new Promise((resolve) => + held.push(resolve), + ) + if (!ok) { + // The source's replay had already failed and still holds + // publication when this load fails. + const barrierFailed = + (barriers.get(`markers`)?.failures.size ?? 0) > 0 + const replayLoading = settleParticipant( + `markers`, + load, + options, + false, + ) + load.outcome = `failed` + anyLoadFailed = true + // A demand the graph started after the commit fails inside + // the replay. + if ( + participant && + !options.signal?.aborted && + oneSubscription(`markers`) && + !markerCommitting && + isDemand(options) + ) { + // While other participants still load. + if (replayLoading) { + reach.participantDemandFailures++ + if (scenario.includes) reach.includeDemandFailures++ + } + // The query both joins and includes the marker source, so + // two of its subscriptions share the barrier. + if (scenario.includes && scenario.joined) { + reach.siblingDemandFailures++ + // After the shared replay failed: the subscription whose + // demand fails may have finished its own replay. + if (barrierFailed) reach.failedSiblingDemandFailures++ + } + } + if (queries.some(({ status }) => status === `loading`)) { + // Lazy demand in a replay failed before the first ready. + if (participant && isDemand(options)) { + reach.preloadReplayFailures++ + } + } + noteLoadingFailure() + recordRequestFailure( + load, + `markers`, + options, + startedReady, + participant, + ) + if (!options.signal?.aborted) { + if (newDemandDuringReplay) + recordReach(`new-demand-failed-during-marker-replay`) + if (settledReplay) + recordReach(`settled-marker-demand-replay-failed`) + } + reach.failedLoads++ + load.error = injectLoad( + new Error(`marker load failed`), + options, + ) + throw load.error } - load.outcome = `failed` - anyLoadFailed = true - if (demand) demandFailed = true - if (queries.some(({ status }) => status === `loading`)) { - failedWhileLoading = true + } else await Promise.resolve() + if (options.signal?.aborted) { + load.outcome = `ok` + settleParticipant(`markers`, load, options, true) + if (scenario.abortedMarkerLoads === `resolve`) return + reach.abortRejections++ + throw inject( + Object.assign(new Error(`marker load aborted`), { + name: `AbortError`, + }), + ) + } + operations.begin() + for (const id of marked) { + const value = { id, rowId: id } + if ( + options.where && + evaluateReferenceExpression(options.where, value) !== true + ) { + continue } - reach.failedLoads++ - throw injectLoad(new Error(`marker load failed`), options) + operations.write({ + type: markers.has(id) ? `update` : `insert`, + value, + }) + installedMarkers.add(id) } - } else await Promise.resolve() - if (options.signal?.aborted) { + await operations.commit() load.outcome = `ok` - return - } - operations.begin() - for (const id of marked) { - const value = { id, rowId: id } - if ( - options.where && - evaluateReferenceExpression(options.where, value) !== true - ) { - continue - } - operations.write({ - type: markers.has(id) ? `update` : `insert`, - value, - }) - installedMarkers.add(id) - } - await operations.commit() - load.outcome = `ok` + // The load settles once its rows apply. + settleParticipant(`markers`, load, options, true) + recordLoaded( + `markers`, + requestedKeys(`markers`, options), + options, + ) + })() }, unloadSubset: (options) => { + unloaded.add(options) + retireParticipants(`markers`, options) const count = acquired.get(options) ?? 0 if (count === 0) { violation ??= `marker unloadSubset released a demand it did not hold` @@ -610,28 +1143,65 @@ async function checkOrderedReplay( sync.markReady() return { loadSubset: async (options) => { - const replay = isReplayDemand(options, false) + const startedReady = allReady() acquired.set(options, (acquired.get(options) ?? 0) + 1) - const load: Load = { epoch: truncates, outcome: `pending` } + const load: Load = { + epoch: truncates, + outcome: `pending`, + source: `rows`, + keys: requestedKeys(`rows`, options), + ordered: options.orderBy !== undefined, + } loads.push(load) - const demand = !replay && isDemand(options) + const participant = joinBarrier(`rows`, load, options) + belongsToRecovery(options, truncates, truncating) if (holding) { const ok = await new Promise((resolve) => held.push(resolve), ) if (!ok) { + const replayLoading = settleParticipant( + `rows`, + load, + options, + false, + ) load.outcome = `failed` anyLoadFailed = true - if (demand) demandFailed = true + // A demand the graph started after the commit, failing + // while its replay still loads. + if ( + participant && + replayLoading && + !options.signal?.aborted && + oneSubscription(`rows`) && + !truncating && + isDemand(options) + ) { + reach.participantDemandFailures++ + } if (queries.some(({ status }) => status === `loading`)) { - failedWhileLoading = true + // Lazy demand in a replay failed before the first ready. + if (participant && isDemand(options)) { + reach.preloadReplayFailures++ + } } + noteLoadingFailure() + recordRequestFailure( + load, + `rows`, + options, + startedReady, + participant, + ) reach.failedLoads++ - throw injectLoad(new Error(`load failed`), options) + load.error = injectLoad(new Error(`load failed`), options) + throw load.error } } else await Promise.resolve() if (options.signal?.aborted) { load.outcome = `ok` + settleParticipant(`rows`, load, options, true) return } const rows = [...truth.values()] @@ -671,8 +1241,21 @@ async function checkOrderedReplay( } await sync.commit() load.outcome = `ok` + // The load settles once its rows apply. + settleParticipant(`rows`, load, options, true) + // An ordered load covers the rows it selected; a keyed load, the + // keys it asked for. + recordLoaded( + `rows`, + options.orderBy + ? new Set(selected.map(({ id }) => id)) + : requestedKeys(`rows`, options), + options, + ) }, unloadSubset: (options) => { + unloaded.add(options) + retireParticipants(`rows`, options) const count = acquired.get(options) ?? 0 if (count === 0) { violation ??= `unloadSubset released a demand it did not hold` @@ -691,6 +1274,8 @@ async function checkOrderedReplay( )) if (replayUnsettled) { earlyThrowSinceTruncate = true + const barrier = barriers.get(`rows`) + if (barrier) barrier.ambiguous = true reach.replayStartThrows++ } else if (truncates > 0) reach.throwsAfterReplay++ else reach.settlementThrows++ @@ -747,8 +1332,82 @@ async function checkOrderedReplay( // query's retirements, so its window obeys the same law. const peer = scenario.peer ? query() : undefined if (peer) queries.push(peer) + // After initial-query readiness no load failure ends the query + // (ARCHITECTURE.md law 16), so a terminal error needs a load that failed + // while the query was still loading: an initial load failure. + const checkTerminal = (label: string, shown: { status: string }) => { + expect(failedWhileLoading.has(shown), `${label}: terminal error`).toBe(true) + } + // At a cut, a query that has not ended lists each failure the account + // owes among its node's open request failures, by the injected error + // object. With a peer, the adapter cannot tell which query issued a load, + // so either query's node may list it. + const checkRequestErrors = (label: string) => { + if (queries.some(({ status }) => status !== `ready`)) return + const owed = requestFailures.filter(requestErrorOwed) + if (owed.length === 0) return + reach.requestErrorChecks++ + for (const failure of owed) { + expect( + [live, peer].some((query) => + query?.health.error?.errors.some( + ({ cause }) => cause === failure.load.error, + ), + ), + `${label}: request error for "${failure.load.error?.message}" on keys ${[...failure.keys].join()}`, + ).toBe(true) + } + } + // Replay error identity (ARCHITECTURE.md, "Replay participants and + // failure"; maintainer decision): a deferred window move whose replay + // fails rejects with that replay's own error, its earliest failure that + // still decides it. Law 17 decides which failures count: one whose demand + // retired, one that a later failure of the same request replaced, as a + // retry does, and one whose keys a later load of the replay covers without + // failing, as a demand's replacement does, no longer count. A failure of a + // load in no waited replay is the move's own window failure after it + // resumed, which this law does not judge. + const covers = (later: Load, earlier: Load) => + later.ordered === earlier.ordered && + later.keys !== undefined && + earlier.keys !== undefined && + [...earlier.keys].every((key) => later.keys!.has(key)) + const stillDecides = (barrier: Barrier, earlier: Load, at: number) => + barrier.failures.has(earlier) && + barrier.history.slice(at + 1).every((later) => !covers(later, earlier)) && + [...barrier.own].every( + (later) => + loads.indexOf(later) < loads.indexOf(earlier) || + later === earlier || + later.outcome === `failed` || + !covers(later, earlier), + ) + const checkReplayError = ( + move: number, + error: Error, + waitedReplays: Array, + ) => { + const candidates = waitedReplays.filter(({ history }) => + history.some((load) => load.error === error), + ) + if (candidates.length === 0) return + reach.deferredMoveRejections++ + const earliest = candidates.some((barrier) => + barrier.history + .slice( + 0, + barrier.history.findIndex((load) => load.error === error), + ) + .every((earlier, at) => !stillDecides(barrier, earlier, at)), + ) + if (!earliest) + moveErrorViolation ??= `move ${move} rejected with "${error.message}", not its replay's earliest failure that still decides it` + } // A subscriber keeps each live query from garbage collection while a // history waits on held loads. + // A history during preload starts with every load held, including the + // loads that the first subscriber starts. + holding = scenario.historyDuringPreload const subscriptions = [live, peer].flatMap((collection) => collection ? [collection.subscribeChanges(() => {})] : [], ) @@ -768,6 +1427,27 @@ async function checkOrderedReplay( } }), ) + // Retention: while a barrier is up, the query shows the rows it showed when + // the first one rose. A query in `error` is judged by the terminal-error + // law instead. + const checkRetention = (label: string) => { + lowerSettledBarriers() + if (retainedRows === undefined || !retainedAfterReady) return + // A source with two subscriptions has two replays, which the adapter + // boundary cannot tell apart. + if ([...barriers.keys()].some((barrier) => !oneSubscription(barrier))) { + return + } + const raised = [...barriers.values()] + if (raised.some(({ ambiguous }) => ambiguous)) return + if (live.status === `error`) return + expect(JSON.parse(JSON.stringify(live.toArray)), label).toEqual( + JSON.parse(retainedRows), + ) + if (raised.some(({ failures }) => failures.size > 0)) { + reach.retainedFailedReplays++ + } else reach.heldPublicationChecks++ + } const settleAll = async () => { // Each truncate replays every retained demand of each query, so a long // held history can hold hundreds of loads. A load that starts another @@ -781,15 +1461,11 @@ async function checkOrderedReplay( throw new Error(`held loads did not drain`) } const truncate = async () => { - replayDemands.source = [...acquired].flatMap(([options, count]) => - count > 0 && !markerRequests.has(options) - ? Array.from({ length: count }, () => options) - : [], - ) truncates++ earlyThrowSinceTruncate = false if (held.length > 0) reach.heldAcrossTruncate++ installed.clear() + raiseBarrier(`rows`) truncating = true try { sync.begin() @@ -804,13 +1480,8 @@ async function checkOrderedReplay( // An eager marker source rewrites its rows; an on-demand one replays the // demand it retains. const truncateMarkers = async () => { - replayDemands.marker = [...acquired].flatMap(([options, count]) => - count > 0 && markerRequests.has(options) - ? Array.from({ length: count }, () => options) - : [], - ) - hadMarkerDemand = replayDemands.marker.length > 0 installedMarkers.clear() + raiseBarrier(`markers`) markerSync.begin() markerSync.truncate() if (!onDemandMarkers) { @@ -819,18 +1490,23 @@ async function checkOrderedReplay( installedMarkers.add(id) } } - markerTruncating = true + markerTruncates++ + markerCommitting = true let receipt: true | Promise try { receipt = markerSync.commit() } finally { - markerTruncating = false + markerCommitting = false } if (receipt !== true) await receipt } + // The history's commands can interleave with the initial loads. + let preloadSettled = false + const preload = Promise.allSettled([live.preload(), peer?.preload()]).then( + () => (preloadSettled = true), + ) try { - await live.preload() - await peer?.preload() + if (!scenario.historyDuringPreload) await preload let sawRequestError = false for (const command of history) { try { @@ -863,8 +1539,8 @@ async function checkOrderedReplay( holding = true break case `settle`: { - holding = command.keepHolding === true - held.shift()?.(command.ok) + holding = command.keepHolding === true && holding + ;(command.newest === true ? held.pop() : held.shift())?.(command.ok) break } case `truncate`: @@ -898,6 +1574,9 @@ async function checkOrderedReplay( } // The joined source truncates. An eager source redelivers every // marker; an on-demand one replays its loaded demands. + case `markerThrows`: + markerThrowArmed = true + break case `truncateMarkers`: { if (!readsMarkers) break reach.markerTruncates++ @@ -913,7 +1592,33 @@ async function checkOrderedReplay( const requested = { offset: command.offset, limit: command.limit } reach.windowMoves++ const move = ++latestMove - const settled = live.utils.setWindow(requested) + // A move that starts before a replay's verdict waits for it, and + // for any replay that rises before it resumes. The identity law + // judges a move that starts before any failure of the replays it + // waits on. A move that starts after one already failed may meet + // a failed replay of the source at once, and that rule + // (ARCHITECTURE.md: a failed replay rejects the move) names no + // particular failure. + const waited = [...barriers.values()].filter( + ({ pending }) => pending.size > 0, + ) + // A peer is another query over the same sources, and the adapter + // cannot tell which query a failed load served, so peer histories + // are not judged (see Limits). + const judged = + !scenario.peer && + waited.every(({ failures }) => failures.size === 0) + const raisedBefore = raisedBarriers.length + // A load that throws while it starts is a rejection + // (ARCHITECTURE.md), so a move never throws: a throw here is an + // escaped error. + let settled: true | Promise + try { + settled = live.utils.setWindow(requested) + } catch (error) { + escaped ??= error + throw error + } if (settled === true) { settleMove(move) window = requested @@ -932,12 +1637,20 @@ async function checkOrderedReplay( if (error instanceof Error && cleanupFaults.has(error)) { escaped ??= error } - // A truncate aborts the load a move waits on (ARCHITECTURE.md); - // the move then rejects and the window stays where it was. - if (error instanceof Error && error.name === `AbortError`) { - reach.abortedMoves++ - return + if (error instanceof Error && injectedLoadFaults.has(error)) { + checkReplayError( + move, + error, + waited.length > 0 && judged + ? [...waited, ...raisedBarriers.slice(raisedBefore)] + : [], + ) } + // A truncate never aborts a move: the move waits for any + // replay that holds publication and then applies + // (ARCHITECTURE.md). Only cleanup rejects a waiting move with + // `AbortError`, and cleanup runs after the checks, so any + // other rejection here is an injected fault or a broken law. swallow(error) }, ) @@ -951,15 +1664,12 @@ async function checkOrderedReplay( swallow(error) } await flushPromises() - replayDemands.source = [] - replayDemands.marker = [] - // Classify terminal errors at the failing history cut, before final - // recovery can hide an incorrect transition. - for (const shown of queries) { - if (shown.status === `error`) { - expect(fatalLoadFailed(), `history: terminal error`).toBe(true) - expect(initiatingFailure(), `history: initiating failure`).toBe(true) - } + checkRetention(`retained publication after ${command.type}`) + checkRequestErrors(`after ${command.type}`) + // Judge a terminal error at the history cut where it appears, before + // final recovery can hide an incorrect transition. + for (const shown of [live, peer]) { + if (shown?.status === `error`) checkTerminal(`history`, shown) } // A failed load after initial-query readiness shows as a request error // until a later load of its data succeeds. @@ -969,19 +1679,33 @@ async function checkOrderedReplay( if (sawRequestError) reach.requestErrors++ holding = false throwNextUnload = false + // The drain is the checkpoint, where recovery loads succeed, so an + // armed throw that no history load consumed expires with the history. + markerThrowArmed = false await settleAll() await Promise.all(pendingMoves.splice(0)) await settleAll() await flushPromises() + // Initial readiness: once loading is quiet, the initial preload has + // settled. It resolves when the first window is covered and rejects when + // an initial load fails; it does not wait for a later truncate. + if (scenario.historyDuringPreload) { + reach.preloadChecks++ + expect(preloadSettled, `initial preload settles`).toBe(true) + } if (violation) throw new Error(`adapter release: ${violation}`) if (moveOrderViolation) { throw new Error(`newest move wins: ${moveOrderViolation}`) } + if (moveErrorViolation) { + throw new Error(`replay error identity: ${moveErrorViolation}`) + } if (escapedError() !== undefined) { throw new Error(`escaped error: only injected faults may surface`, { cause: escapedError(), }) } + checkRetention(`retained publication once loading is quiet`) // Status: a release that throws is reported, but it fails no load, so a // quiet query with no failed load stays ready. if (!anyLoadFailedNow()) { @@ -1019,10 +1743,14 @@ async function checkOrderedReplay( ...(peer ? [[peer, `peer convergence`, peerWindow] as const] : []), ] as const) { if (shown.status === `error`) { - expect(fatalLoadFailed(), `${label}: terminal error`).toBe(true) - expect(initiatingFailure(), `${label}: initiating failure`).toBe(true) + checkTerminal(label, shown) reach.terminalErrors++ - } else checkWindow(label, shown.toArray, expected) + } else { + checkWindow(label, shown.toArray, expected) + // Recovery replayed every source and revalidated the window, and each + // success decides the request errors before it, so none stays. + expect(shown.health.error, `${label}: stale request error`).toBeNull() + } } } finally { holding = false @@ -1078,8 +1806,8 @@ describe(`Ordered windows across truncate replays`, () => { // graph that the pending joined work had held, so the window stayed on // old rows until an unrelated write); // - a window move whose joined load a truncate aborts (the model counted - // the move's AbortError as an escaped fault; it is a legal rejection, and - // this replay keeps the branch reached); + // the move's AbortError as an escaped fault; the move now waits for the + // newer replay and applies, so this replay checks the moved window); // - a window move that retires lazy joined demand whose release throws (the // subscription reported the release failure as a load error, so the move // rejected with it; core now reports it uncaught); @@ -1155,7 +1883,7 @@ describe(`Ordered windows across truncate replays`, () => { joined: `none`, markerMode: `on-demand`, }, - reaches: `abortedMoves`, + reaches: `settledWindowChecks`, }, { name: `a window move that retires lazy joined demand whose release throws`, @@ -1195,6 +1923,9 @@ describe(`Ordered windows across truncate replays`, () => { reaches: `requestErrors`, }, { + // The marker truncate's commit moves the private window, and the + // include starts new demand inside the marker barrier, which then + // fails. The barrier holds it, so the query keeps its last publication. name: `a new include demand during another subscription's replay`, history: [ { type: `hold` }, @@ -1211,6 +1942,7 @@ describe(`Ordered windows across truncate replays`, () => { includes: true, }, failureState: replayDemandStates[0], + recovers: true, }, { name: `a failed replay of settled joined demand`, @@ -1226,12 +1958,67 @@ describe(`Ordered windows across truncate replays`, () => { markerMode: `on-demand`, }, failureState: replayDemandStates[1], + recovers: true, + }, + { + name: `two window moves while a load is held`, + history: [ + { type: `hold` }, + { type: `window`, offset: 1, limit: 2 }, + { type: `window`, offset: 0, limit: 1 }, + ], + scenario: { + ...ordinaryOrderedScenario, + offset: 1, + limit: 1, + joined: `none`, + }, + }, + { + name: `an inner join whose lazy root demand load fails`, + history: [ + { type: `hold` }, + { type: `mark`, id: 2, on: true }, + { type: `settle`, ok: false }, + { type: `move`, id: 1, rank: 1 }, + ], + scenario: { ...ordinaryOrderedScenario, limit: 1, joined: `inner` }, + reaches: `requestErrors`, + }, + { + name: `a window move that a deferred newer move supersedes`, + history: [ + { type: `hold` }, + { type: `move`, id: 1, rank: 1 }, + { type: `window`, offset: 0, limit: 1 }, + { type: `truncate`, armAfterStart: false }, + { type: `window`, offset: 0, limit: 1 }, + ], + scenario: { ...ordinaryOrderedScenario, limit: 1 }, + }, + { + name: `two window moves that a pending source recovery defers`, + history: [ + { type: `hold` }, + { type: `truncate`, armAfterStart: false }, + { type: `window`, offset: 1, limit: 1 }, + { type: `move`, id: 1, rank: 1 }, + { type: `truncateMarkers` }, + { type: `window`, offset: 0, limit: 1 }, + { type: `truncateMarkers` }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `none`, + markerMode: `on-demand`, + }, }, { // The random campaign's trace from #2224 (also reported as #2239). With // includes and no joined source, the failed load is new include demand - // issued during the marker replay, so the terminal error is fatal. - name: `a failed include demand during a marker replay`, + // that the marker truncate's commit starts inside the marker barrier. + name: `a failed include demand that a marker truncate starts`, history: [ { type: `hold` }, { type: `truncateMarkers` }, @@ -1244,54 +2031,239 @@ describe(`Ordered windows across truncate replays`, () => { markerMode: `on-demand`, includes: true, }, + reaches: `failedLoads`, failureState: replayDemandStates[0], + recovers: true, }, { - name: `two window moves while a load is held`, + name: `a failed include demand started from no prior demand`, history: [ + { type: `mark`, id: 2, on: true }, { type: `hold` }, - { type: `window`, offset: 1, limit: 2 }, - { type: `window`, offset: 0, limit: 1 }, + { type: `truncateMarkers` }, + { type: `settle`, ok: false }, ], scenario: { ...ordinaryOrderedScenario, - offset: 1, limit: 1, joined: `none`, + markerMode: `on-demand`, + includes: true, }, + reaches: `failedLoads`, }, { - name: `an inner join whose lazy root demand load fails`, + name: `a failed include replay before the first ready`, history: [ - { type: `hold` }, - { type: `mark`, id: 2, on: true }, + { type: `settle`, ok: true, keepHolding: true }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: true, keepHolding: true }, { type: `settle`, ok: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + markerMode: `on-demand`, + includes: true, + abortedMarkerLoads: `reject`, + historyDuringPreload: true, + }, + reaches: `abortRejections`, + }, + { + name: `an aborted marker load that rejects after its replay succeeds`, + history: [ + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: true, keepHolding: false }, + { type: `truncateMarkers` }, + { type: `unloadThrows` }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + peer: true, + joined: `some`, + markerMode: `on-demand`, + abortedMarkerLoads: `reject`, + historyDuringPreload: true, + }, + reaches: `abortRejections`, + }, + { + name: `a replaced include segment whose aborted load rejects`, + history: [ + { type: `hold` }, + { type: `truncate`, armAfterStart: false }, { type: `move`, id: 1, rank: 1 }, + { type: `move`, id: 2, rank: 31 }, ], - scenario: { ...ordinaryOrderedScenario, limit: 1, joined: `inner` }, - reaches: `requestErrors`, + scenario: { + ...ordinaryOrderedScenario, + markerMode: `on-demand`, + includes: true, + abortedMarkerLoads: `reject`, + }, + reaches: `abortRejections`, }, { - name: `a window move that a deferred newer move supersedes`, + name: `an include demand that fails while its replay still loads`, history: [ { type: `hold` }, + { type: `truncateMarkers` }, + { type: `move`, id: 3, rank: 1 }, + { type: `settle`, ok: false, newest: true }, + ], + scenario: { + ...ordinaryOrderedScenario, + markerMode: `on-demand`, + includes: true, + }, + reaches: `failedLoads`, + recovers: true, + }, + { + // A move during the initial load waits behind a marker replay that the + // source truncate then discards. The newer move must wait for the + // replay that still holds publication and then apply. Before the fix, + // the discarded replay rejected it, so the older move's window won. + name: `a newer window move deferred behind a discarded replay`, + history: [ { type: `move`, id: 1, rank: 1 }, + { type: `window`, offset: 0, limit: 2 }, + { type: `truncateMarkers` }, { type: `window`, offset: 0, limit: 1 }, { type: `truncate`, armAfterStart: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + markerMode: `on-demand`, + includes: true, + historyDuringPreload: true, + }, + reaches: `settledWindowChecks`, + }, + { + // A query that joins and includes the marker source has two marker + // subscriptions. The truncate's commit deletes the markers privately, + // so the join filter drops row 1 and the graph demands markers for + // the next rows. One subscription's replay loads settle first, and + // the new demand then fails while the other's replay still loads. + // With one replay per subscription, that demand fell outside every + // replay and ended the query; the source's barrier owns it. + name: `a demand that fails while a sibling subscription's replay still loads`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + reaches: `siblingDemandFailures`, + recovers: true, + }, + { + // Found by the failed-replay random campaign. One subscription's + // replay fails, the other's finishes once it settles, and a rank move + // starts demand on that other subscription, which then fails. Its own + // replay had finished, so a per-subscription judgment ended the query, + // while the source's shared barrier still held the failed replay. + name: `a demand that fails after the sibling subscription's replay failed`, + history: [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + { type: `move`, id: 2, rank: 1 }, + { type: `settle`, ok: false, keepHolding: false, newest: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + reaches: `failedSiblingDemandFailures`, + recovers: true, + }, + { + // Found by the broad random campaign at five times its run count. A + // window move's marker load throws while the move starts, after an + // older move that still waits. The older move's settlement returned + // the latest request, which was still itself: a chaining cycle. A + // throw is now a rejection, so the newer move rejects instead. + name: `a window move whose load throws while an older move waits`, + history: [ { type: `window`, offset: 0, limit: 1 }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `markerThrows` }, + { type: `window`, offset: 0, limit: 2 }, + { type: `mark`, id: 1, on: false }, ], - scenario: { ...ordinaryOrderedScenario, limit: 1 }, + scenario: { + ...ordinaryOrderedScenario, + limit: 3, + joined: false, + markerMode: `on-demand`, + includes: true, + historyDuringPreload: true, + }, + reaches: `failedLoads`, }, { - name: `two window moves that a pending source recovery defers`, + // Trace 2 from a comment on #2221. The window grows to three rows while + // loads are held, shrinks to one behind a marker replay, and a source + // truncate then discards that replay. The deferred one-row move was + // dropped with it, so the settled window kept three rows. + name: `a window shrink deferred behind a replay that a truncate discards`, history: [ { type: `hold` }, - { type: `truncate`, armAfterStart: false }, - { type: `window`, offset: 1, limit: 1 }, - { type: `move`, id: 1, rank: 1 }, + { type: `window`, offset: 0, limit: 3 }, { type: `truncateMarkers` }, { type: `window`, offset: 0, limit: 1 }, + { type: `truncate`, armAfterStart: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: false, + markerMode: `on-demand`, + includes: true, + }, + }, + { + // A marker truncate replays the include's demand, and its load throws + // while it starts. The throw fails the replay as a rejection does, so + // it keeps the last publication; it used to end the query. + name: `a replay load that throws while it starts`, + history: [{ type: `markerThrows` }, { type: `truncateMarkers` }], + scenario: { + ...ordinaryOrderedScenario, + limit: 1, + joined: `none`, + markerMode: `on-demand`, + includes: true, + }, + recovers: true, + }, + { + // A row move starts join demand inside an open marker replay, and its + // load throws while it starts. The subscription dropped that demand, + // so its failure did not fail the replay, and the query ended. + name: `a demand whose load throws inside its open replay`, + history: [ + { type: `hold` }, { type: `truncateMarkers` }, + { type: `markerThrows` }, + { type: `move`, id: 3, rank: 1 }, ], scenario: { ...ordinaryOrderedScenario, @@ -1299,6 +2271,8 @@ describe(`Ordered windows across truncate replays`, () => { joined: `none`, markerMode: `on-demand`, }, + reaches: `thrownReplayDemands`, + recovers: true, }, { name: `a fresh include request that fails during marker truncate`, @@ -1315,9 +2289,9 @@ describe(`Ordered windows across truncate replays`, () => { markerMode: `on-demand`, includes: true, }, - // After initial-query readiness the fresh request's failure is a - // request error, not a terminal one (ARCHITECTURE.md law 16). - reaches: `requestErrors`, + reaches: `failedLoads`, + failureState: replayDemandStates[0], + recovers: true, }, { name: `an established include replay that fails`, @@ -1335,6 +2309,28 @@ describe(`Ordered windows across truncate replays`, () => { }, reaches: `failedLoads`, }, + { + // A window move waits behind a source replay whose two loads then + // fail. The move rejects with the replay's own error, its earliest + // failure (maintainer decision). It rejected with the later failure, + // which the query had last recorded. + name: `a window move behind a replay with two failed loads`, + history: [ + { type: `hold` }, + { type: `truncate`, armAfterStart: false }, + { type: `window`, offset: 0, limit: 1 }, + { type: `settle`, ok: false, newest: true }, + { type: `settle`, ok: false, keepHolding: false, newest: false }, + ], + scenario: { + ...ordinaryOrderedScenario, + offset: 1, + limit: 1, + joined: `none`, + markerMode: `eager`, + }, + reaches: `deferredMoveRejections`, + }, { name: `an established marker replay after a fresh include succeeds`, history: [ @@ -1378,6 +2374,10 @@ describe(`Ordered windows across truncate replays`, () => { history: Array scenario?: OrderedScenario reaches?: keyof OrderedReach + // A replay whose history the law places in a recovery. The pin also + // holds where the model cannot tell which subscription issued a load + // (see Limits). + recovers?: true failureState?: ReplayDemandState }>)( `keeps the ordered window for $name (generated replay)`, @@ -1392,9 +2392,43 @@ describe(`Ordered windows across truncate replays`, () => { const reaches: keyof OrderedReach | undefined = `reaches` in replay ? replay.reaches : undefined if (reaches) expect(reach[reaches], reaches).toBeGreaterThan(0) + if (`recovers` in replay) { + expect(reach.terminalErrors, `recovery failure ends the query`).toBe(0) + } }, ) + // Found by the failed-replay random campaign at five times its run count. + // A join and an include share the marker source. One subscription's replay + // fails; the other's finishes, and a row move starts demand on it whose + // load fails too, which the shared barrier keeps from ending the query. A + // marker change then retires the first replay's failed demand, which + // discarded that replay and published an empty window while the second + // failure held nothing; that failure now holds its own subscription. + it(`keeps the last publication when a marker change follows two failed replay loads`, async () => { + await checkOrderedReplay( + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + { type: `move`, id: 1, rank: 1 }, + { type: `settle`, ok: false, newest: false }, + { type: `hold` }, + { type: `mark`, id: 4, on: false }, + ], + { + ...ordinaryOrderedScenario, + direction: `desc`, + offset: 1, + limit: 1, + joined: `some`, + markerMode: `on-demand`, + includes: true, + }, + ) + }) + it.each(orderedReplayCampaigns)( `keeps the ordered window under load and release faults (%s campaign)`, async (campaign) => { @@ -1416,20 +2450,55 @@ describe(`Ordered windows across truncate replays`, () => { innerJoinedRuns: 0, markerLoads: 0, markerTruncates: 0, - abortedMoves: 0, terminalErrors: 0, requestErrors: 0, + requestErrorChecks: 0, + successWhileSubscriptionOwes: 0, + deferredMoveRejections: 0, + replayLoads: 0, + abortRejections: 0, + preloadChecks: 0, + heldPublicationChecks: 0, + retainedFailedReplays: 0, + participantDemandFailures: 0, + includeDemandFailures: 0, + siblingDemandFailures: 0, + failedSiblingDemandFailures: 0, + thrownReplayDemands: 0, + partialFailureRetirements: 0, + preloadReplayFailures: 0, + discardedReplayMoves: 0, } - // The broad property claims these dimensions, not the rare terminal - // interaction owned by the subscription-specific property below. + // Reach witnesses: releases throw without a truncate, while a replay + // starts, and while it settles; loads fail and stay held across a + // truncate; truncates ask again for held demand; and the settled-window + // law is checked. + // A terminal error needs a failed first load, and a request error a held + // demand load that fails after readiness; the named replays above reach + // both. A failed replay that the retention law can + // judge, a demand that fails while its replay is open, a replay that + // fails before the first ready, and a window move behind a discarded + // replay are rare here; the campaigns below declare them. + const ownedElsewhere = new Set([ + `terminalErrors`, + `requestErrors`, + `retainedFailedReplays`, + `participantDemandFailures`, + `includeDemandFailures`, + `siblingDemandFailures`, + `failedSiblingDemandFailures`, + `thrownReplayDemands`, + `partialFailureRetirements`, + `preloadReplayFailures`, + `discardedReplayMoves`, + `requestErrorChecks`, + `deferredMoveRejections`, + `successWhileSubscriptionOwes`, + ]) + // Counts are histories, rather than repeated events in one trace. const dimensions = ( Object.keys(total) as Array - ).filter( - (dimension) => - dimension !== `abortedMoves` && - dimension !== `terminalErrors` && - dimension !== `requestErrors`, - ) + ).filter((dimension) => !ownedElsewhere.has(dimension)) const declared = createOracleReach(orderedReplayProperty, dimensions) await fc.assert( fc.asyncProperty( @@ -1453,14 +2522,479 @@ describe(`Ordered windows across truncate replays`, () => { Object.fromEntries(declared.counts()), ) declared.assertReached() - // Reach witnesses: releases throw without a truncate, while a replay - // starts, and while it settles; loads fail and stay held across a - // truncate; and the settled-window law is checked. - // An aborted move needs a truncate to land inside a held move, and a - // request error needs a held demand load to fail after readiness; the - // named replays above reach both. A terminal error needs a failed first - // load. - // Counts above are histories, rather than repeated events in one trace. + }, + 120_000, + ) + + // The retention law needs a replay that fails while the query holds its + // last publication. Each history here starts by holding the next loads and + // truncating a source, so the replay's loads stay held. It may then change + // a rank, a marker, or the window, which starts demand while the replay is + // open. It then fails a held load: the oldest is a replay load, and the + // newest is the demand just started. Random commands follow, so later + // moves, marks, window moves, and truncates act on a failed replay. Peers + // stay out because the retention law does not judge them. + const failedReplayTruncate = fc.constantFrom( + { type: `truncate` as const, armAfterStart: false }, + { type: `truncateMarkers` as const }, + ) + const failedReplayHistory: fc.Arbitrary> = fc + .tuple( + failedReplayTruncate, + fc.option( + fc.oneof( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + fc.record({ + type: fc.constant(`mark` as const), + id: fc.integer({ min: 1, max: 5 }), + on: fc.boolean(), + }), + fc.record({ + type: fc.constant(`window` as const), + offset: fc.constantFrom(0 as const, 1 as const), + limit: fc.constantFrom(1 as const, 2 as const, 3 as const), + }), + ), + { nil: undefined }, + ), + fc.boolean(), + fc.array(orderedCommand, { maxLength: 8 }), + ) + .map(([truncate, change, keepHolding, rest]) => [ + { type: `hold` as const }, + truncate, + ...(change + ? [change, { type: `settle` as const, ok: false, newest: true }] + : [{ type: `settle` as const, ok: false, keepHolding }]), + ...rest, + ]) + const failedReplayScenario = orderedScenario.map((scenario) => ({ + ...scenario, + peer: false, + })) + // An include demand that fails while its replay still loads needs an + // include over on-demand markers and a move that brings a row into the + // window during the marker replay; this branch builds that start. The + // window starts at offset 0 without ties, and the moved row starts past + // it, so moving it before every other row shows it. + const includeDemandCase = failedReplayScenario.chain((scenario) => { + const included = { + ...scenario, + offset: 0 as const, + ties: false, + joined: false as const, + includes: true, + markerMode: `on-demand` as const, + } + const ascending = included.direction === `asc` + return fc + .tuple( + fc.integer({ min: 1, max: 9 }), + fc.array(orderedCommand, { maxLength: 8 }), + ) + .map(([rank, rest]): [Array, OrderedScenario] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { + type: `move`, + id: ascending ? 5 : 1, + rank: ascending ? rank : 61 - rank, + }, + { type: `settle`, ok: false, newest: true }, + ...rest, + ], + included, + ]) + }) + // A query that joins and includes the marker source has two marker + // subscriptions under one barrier. Settling some held loads can finish one + // subscription's own replay loads while the other's still load; the graph + // then demands markers for rows the commit privately exposed, and the + // newest load fails. + // After the failing load, half the histories move a row, which starts + // demand once the shared replay has failed, and then fail the oldest held + // load: the other subscription may have finished its own replay by then. + const siblingDemandCase = fc + .tuple( + failedReplayScenario, + fc.constantFrom(`some` as const, `none` as const, `inner` as const), + fc.integer({ min: 1, max: 2 }), + fc.boolean(), + fc.option( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + ), + fc.array(orderedCommand, { maxLength: 6 }), + // The failed demand's retry, which the query starts at once, may stay + // held, so later commands act while that demand is still unloaded. + fc.boolean(), + ) + .map( + ([scenario, joined, settles, newest, move, rest, retryHeld]): [ + Array, + OrderedScenario, + ] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + ...Array.from({ length: settles }, () => ({ + type: `settle` as const, + ok: true, + keepHolding: true, + newest, + })), + { type: `settle`, ok: false, newest: true, keepHolding: true }, + ...(move + ? [ + move, + { + type: `settle` as const, + ok: false, + newest: false, + keepHolding: retryHeld, + }, + ] + : []), + ...rest, + ], + { ...scenario, joined, includes: true, markerMode: `on-demand` }, + ], + ) + // A marker replay is open when a change starts marker demand whose load + // throws while it starts. The throw fails the replay, like a rejection, + // so it must not end the query. + const thrownDemandCase = fc + .tuple( + failedReplayScenario, + fc.boolean(), + fc.oneof( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + fc.record({ + type: fc.constant(`mark` as const), + id: fc.integer({ min: 1, max: 5 }), + on: fc.boolean(), + }), + ), + fc.array(orderedCommand, { maxLength: 6 }), + ) + .map( + ([scenario, settleFirst, change, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + ...(settleFirst + ? [{ type: `settle` as const, ok: true, keepHolding: true }] + : []), + { type: `markerThrows` }, + change, + ...rest, + ], + { ...scenario, markerMode: `on-demand` }, + ], + ) + // A demand fails behind a sibling's failed replay, and its retry stays + // held. A later command starts another demand of the same source, whose + // load then succeeds, so a replay's success, publication, and request + // errors are judged while a failed demand is still unloaded. Declared + // reach: a subscription's load succeeds while a failure of that + // subscription is still owed. + const heldRetryCase = fc + .tuple( + failedReplayScenario, + fc.constantFrom(`some` as const, `none` as const, `inner` as const), + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + fc.oneof( + fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: -5, max: 99 }), + }), + fc.record({ + type: fc.constant(`mark` as const), + id: fc.integer({ min: 1, max: 5 }), + on: fc.boolean(), + }), + fc.record({ + type: fc.constant(`window` as const), + offset: fc.constantFrom(0 as const, 1 as const), + limit: fc.constantFrom(1 as const, 2 as const, 3 as const), + }), + ) as fc.Arbitrary, + fc.array(orderedCommand, { maxLength: 3 }), + ) + .map( + ([scenario, joined, move, change, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + { type: `hold` }, + { type: `truncateMarkers` }, + { type: `settle`, ok: true, keepHolding: true, newest: false }, + { type: `settle`, ok: false, newest: true, keepHolding: true }, + move, + { type: `settle`, ok: false, newest: false, keepHolding: true }, + change, + { type: `settle`, ok: true, newest: true, keepHolding: true }, + ...rest, + ], + { + ...scenario, + limit: scenario.limit === 1 ? 2 : scenario.limit, + joined, + includes: true, + markerMode: `on-demand`, + }, + ], + ) + const failedReplayCase = fc.oneof( + { + weight: 3, + arbitrary: fc.tuple(failedReplayHistory, failedReplayScenario), + }, + { weight: 1, arbitrary: includeDemandCase }, + { weight: 2, arbitrary: siblingDemandCase }, + { weight: 2, arbitrary: thrownDemandCase }, + { weight: 2, arbitrary: heldRetryCase }, + ) + const failedReplayProperty = `replay.ordered-failed-replay` + const failedReplayCampaigns = + orderedReplayConfig.replayPath === undefined + ? ([`fixed`, `random`] as const) + : orderedReplayConfig.replayProperty === failedReplayProperty + ? ([`replay`] as const) + : ([] as const) + + // Recovery during the initial preload. A history starts while the first + // loads are held, over on-demand markers that the query includes or joins. + // Its grammar settles held loads one at a time, truncates either source, + // moves the window, and fails a held load, so replays start, fail, and are + // discarded before the query first becomes ready, with window moves + // waiting behind them. + const preloadCommand: fc.Arbitrary = fc.oneof( + { + weight: 4, + arbitrary: fc.record({ + type: fc.constant(`settle` as const), + ok: fc.oneof( + { weight: 3, arbitrary: fc.constant(true) }, + { weight: 1, arbitrary: fc.constant(false) }, + ), + keepHolding: fc.oneof( + { weight: 3, arbitrary: fc.constant(true) }, + { weight: 1, arbitrary: fc.constant(false) }, + ), + newest: fc.boolean(), + }), + }, + { weight: 2, arbitrary: fc.constant({ type: `truncateMarkers` as const }) }, + { + weight: 1, + arbitrary: fc.constant({ + type: `truncate` as const, + armAfterStart: false, + }), + }, + { + weight: 2, + arbitrary: fc.record({ + type: fc.constant(`window` as const), + offset: fc.constantFrom(0 as const, 1 as const), + limit: fc.constantFrom(1 as const, 2 as const, 3 as const), + }), + }, + { + weight: 1, + arbitrary: fc.record({ + type: fc.constant(`move` as const), + id: fc.integer({ min: 1, max: 5 }), + rank: fc.integer({ min: 1, max: 60 }), + }), + }, + { weight: 1, arbitrary: fc.constant({ type: `hold` as const }) }, + ) + const preloadScenario = orderedScenario.map((scenario) => ({ + ...scenario, + markerMode: `on-demand` as const, + includes: scenario.includes || !scenario.joined, + historyDuringPreload: true, + })) + // A replay of include demand that fails before the first ready: initial + // loads settle one at a time, so the order source delivers rows and the + // include demands them; the marker source truncates; more held loads + // settle; and the oldest or newest held load fails. The oldest is often + // a load the truncate aborted, and the newest a replay load. + const settleHeld = { type: `settle` as const, ok: true, keepHolding: true } + const preloadDemandFailure = fc + .tuple( + preloadScenario, + fc.integer({ min: 1, max: 2 }), + fc.integer({ min: 0, max: 3 }), + fc.boolean(), + fc.array(preloadCommand, { maxLength: 6 }), + ) + .map( + ([scenario, before, after, newest, rest]): [ + Array, + OrderedScenario, + ] => [ + [ + ...Array.from({ length: before }, () => settleHeld), + { type: `truncateMarkers` }, + ...Array.from({ length: after }, () => settleHeld), + { type: `settle`, ok: false, newest }, + ...rest, + ], + { ...scenario, joined: false, includes: true }, + ], + ) + const preloadCase = fc.oneof( + { + weight: 2, + arbitrary: fc.tuple( + fc.array(preloadCommand, { minLength: 1, maxLength: 12 }), + preloadScenario, + ), + }, + { weight: 1, arbitrary: preloadDemandFailure }, + ) + const preloadProperty = `replay.ordered-preload-recovery` + const preloadCampaigns = + orderedReplayConfig.replayPath === undefined + ? ([`fixed`, `random`] as const) + : orderedReplayConfig.replayProperty === preloadProperty + ? ([`replay`] as const) + : ([] as const) + + it.each(preloadCampaigns)( + `recovers replays that start during the initial preload (%s campaign)`, + async (campaign) => { + // Declared reach: a replay participant that fails before the first + // ready, which must end the query or let the preload settle, and a + // window move that waits behind a replay a newer truncate discards, + // which must still apply. + const reach = createOracleReach( + preloadProperty, + [ + `preloadReplayFailures`, + `discardedReplayMoves`, + `preloadChecks`, + ] as const, + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(preloadCase, async ([history, scenario]) => { + reach.beginHistory() + const observed = await checkOrderedReplay(history, scenario) + for (const state of [ + `preloadReplayFailures`, + `discardedReplayMoves`, + `preloadChecks`, + ] as const) { + if (observed[state] > 0) reach.reach(state) + } + }), + // The fixed seed kills both mutants that the evidence names: a + // builder that lets a discarded replay reject a deferred window move, + // and one that keeps a failure before the first ready non-fatal. Each + // dies in about half of 200-run seeds. + campaign === `fixed` + ? { seed: 20261027, numRuns: oracleRuns(300) } + : oraclePropertyOptions(300, preloadProperty), + ) + reach.assertReached() + }, + 120_000, + ) + + it.each(failedReplayCampaigns)( + `keeps the last publication through a failed replay (%s campaign)`, + async (campaign) => { + // Declared reach: histories where the retention law judged a failed + // replay, judged a replay that still held publication, and saw a + // demand that the graph started after the commit fail while its replay + // still loaded, on either source and for an include. An include + // demand's rejection arrives before its replay records the failure, so + // only there does the demand wait for the replay to decide it. + const reach = createOracleReach( + failedReplayProperty, + [ + `retainedFailedReplays`, + `heldPublicationChecks`, + `participantDemandFailures`, + `includeDemandFailures`, + `siblingDemandFailures`, + `failedSiblingDemandFailures`, + `thrownReplayDemands`, + `partialFailureRetirements`, + `requestErrorChecks`, + `deferredMoveRejections`, + `successWhileSubscriptionOwes`, + ] as const, + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(failedReplayCase, async ([history, scenario]) => { + reach.beginHistory() + const observed = await checkOrderedReplay(history, scenario) + if (observed.retainedFailedReplays > 0) { + reach.reach(`retainedFailedReplays`) + } + if (observed.heldPublicationChecks > 0) { + reach.reach(`heldPublicationChecks`) + } + if (observed.participantDemandFailures > 0) { + reach.reach(`participantDemandFailures`) + } + if (observed.includeDemandFailures > 0) { + reach.reach(`includeDemandFailures`) + } + if (observed.siblingDemandFailures > 0) { + reach.reach(`siblingDemandFailures`) + } + if (observed.failedSiblingDemandFailures > 0) { + reach.reach(`failedSiblingDemandFailures`) + } + if (observed.thrownReplayDemands > 0) { + reach.reach(`thrownReplayDemands`) + } + if (observed.partialFailureRetirements > 0) { + reach.reach(`partialFailureRetirements`) + } + if (observed.requestErrorChecks > 0) { + reach.reach(`requestErrorChecks`) + } + if (observed.deferredMoveRejections > 0) { + reach.reach(`deferredMoveRejections`) + } + if (observed.successWhileSubscriptionOwes > 0) { + reach.reach(`successWhileSubscriptionOwes`) + } + }), + campaign === `fixed` + ? { seed: 20261023, numRuns: oracleRuns(600) } + : oraclePropertyOptions(600, failedReplayProperty), + ) + reach.assertReached() }, 120_000, ) @@ -1473,11 +3007,11 @@ describe(`Ordered windows across truncate replays`, () => { * its rank and the failure origin. It uses the same driver, * independent classification and observation cuts as the broad property. * - * An include may start new ordinary demand during another owner's - * replacement; a settled joined owner renews its own demand. The adapter must - * witness each failed acquisition in normal fixed and random campaigns. Both - * failures come after initial-query readiness, so neither may end the query; - * the ordinary demand failure must leave a request error on its node. + * An include may start new demand while the marker truncate commits, inside + * the marker barrier; a settled joined owner renews its own demand. After the + * first ready, the barrier keeps the last publication for both failures + * (ARCHITECTURE.md, "Replay participants and failure"). The adapter must + * witness each failed acquisition in normal fixed and random campaigns. * Aborted work does not count. Named neighbors above calibrate both witnesses. */ const replayDemandProperty = `replay.subscription-demand-origin` @@ -1537,8 +3071,8 @@ it.each(replayDemandCampaigns)( declared.reach(state) }, ) - // The query is ready before either failure, so neither origin may - // end it (ARCHITECTURE.md law 16). + // Neither failure may end the query: each fails behind the marker + // barrier, which keeps the query available for a later replay. checkReplayDemandFailure(reach, entered) }), campaign === `fixed` @@ -1560,7 +3094,7 @@ it.each(replayDemandCampaigns)( // histories vary the marked head's rank and two other markers before // placing failure on either side of that boundary. Reacquisition failures // additionally explore later rank, marker, window, replay, release, and -// settlement actions before recovery; terminal histories need no suffix. +// settlement actions before recovery; fresh-failure histories need no suffix. // A new include fails in the first prefix. In the second it succeeds, and // failure instead reaches the already established marker reacquisition. it.each(orderedFailureCampaigns)( @@ -1607,8 +3141,9 @@ it.each(orderedFailureCampaigns)( undefined, failureReach.reach, ) - // Both failures follow initial-query readiness, so each is a - // request error on the query's node and neither ends the query. + // A fresh request that fails inside the marker barrier keeps the + // query; a later suffix may legitimately end it, so only the + // fresh history is pinned. if (fresh) { expect(reach.terminalErrors, `fresh request failure`).toBe(0) expect(reach.requestErrors, `fresh request failure`).toBe(1) diff --git a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts index 6f7f1d77da..df1ceebb1e 100644 --- a/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts +++ b/packages/db/tests/query/ordered-source-loader-state-oracle.test.ts @@ -5,6 +5,7 @@ import { OrderedSourceLoader, } from '../../src/query/live/ordered-source-loader.js' import { PropRef } from '../../src/query/ir.js' +import { createOracleReach } from '../oracle-reach.js' import { captureMicrotaskErrors } from '../utils.js' import { oraclePropertyOptions, @@ -23,6 +24,14 @@ import type { RequestOptions, } from './ordered-source-loader-state-support.js' +// The leading order value that a tie request asks for: its `where` is +// `eq(rank, value)`. Other requests have no tie value. +function tieValue(options: RequestOptions): unknown { + const where = options.where as + { name?: string; args?: Array<{ value?: unknown }> } | undefined + return where?.name === `eq` ? where.args?.[1]?.value : undefined +} + /** * # Request authority under generated settlement orders * @@ -60,15 +69,34 @@ import type { * invalidation reads no full source, and it issues at most one chain (in * this fake, a prefix, a tie, and a refill) per ordering epoch move. * - **Ignored report:** a sync adapter's report that served pages may have - * shifted issues no authoritative request when nothing it describes can - * be wrong: before any request settled (a request in flight still owes an - * exact result for its own options), or while the loader's coverage is - * its own first ordered prefix of at least `offset + limit` rows, with - * only that prefix's tie after it, and a row past the page sorts strictly - * after its end with no more rows needed (ARCHITECTURE.md). The law judges - * only quiet histories: every earlier step settled a request successfully, - * ran a pass, inserted a row, reported, or changed joined work, and no - * release callback has side effects, so no earlier event owes a repair. + * shifted issues no authoritative request before any request settled: a + * request in flight still owes an exact result for its own options. The + * loader ignores the report, and so issues no request at all, while its + * coverage is a held full-source read since the last truncate, whatever + * came before (an ordering invalidation owes such coverage nothing), or, + * in a quiet history, while its coverage is the loader's own + * first ordered prefix of at least `offset + limit` rows, with only that + * prefix's tie after it, and a row past the page sorts strictly after its + * end with no more rows needed (ARCHITECTURE.md). That row is evidence + * only when the page read it, or when the tie the loader read is the tie + * of the page's end. The law judges only quiet histories: every earlier + * step settled a request successfully, ran a pass, inserted a row, + * reported, shrank the window, or changed joined work, and no release + * callback has side effects, so no earlier event owes a repair. + * - **Owed repair:** a report while the loader already owes a repair + * repairs instead (ARCHITECTURE.md): the pass that reacts to it reads no + * tie or page before the repair's prefix. + * - **Unexpressible tie:** the loader never reads the tie of a rank that no + * cursor can express, such as NaN or an infinity; it reads the full source + * or repairs instead (ARCHITECTURE.md; `canExpressCursorOrder`). + * - **Reset:** a truncate resets ordered loading, so the first finite + * request after it starts at the beginning of the order, with no cursor. + * - **Unproven report:** in such a quiet history, a report over the loader's + * own first prefix that no row it read proves starts a repair on its pass, + * or reads the tie of the page's end, which completes the proof. + * An installed row past the page can come from another request, and a + * refetch can move the page's end before the tie the loader read, so the + * loader cannot use either as proof. * - **One request at a time:** a request issued while another is in flight * starts authority, or an authority start lies between them. A repair * chain that a newer start replaced issues no more steps. @@ -130,6 +158,10 @@ import type { * - `invalidate` invalidates source ordering, and `report` delivers a * loaded-subset report; either can run an ordinary pass at once or leave * the reaction deferred, as pending joined work or a later graph run can; + * - `headInsert` delivers a new row that sorts before every installed row, + * as a refetch that moved a provider row into the first page does, so the + * page's end moves one row earlier. The row's rank may be `-Infinity`, + * which no cursor can express; * - `pass` runs an ordinary `loadMore()`; * - `move` is an explicit window move, which may also widen the window, so * an unindexed loader reads a longer prefix; @@ -189,8 +221,9 @@ import type { * * ## Limits * - * The fake subscription's rows do not move except for the unexpressible - * boundary, so cursor values are fixed. Publication holds are outside this + * The fake subscription's installed rows move only through the unexpressible + * boundary and a head insert. Its ordered reads apply a tie's equality, a + * cursor, and a limit as the subscription's reads do. Publication holds are outside this * property; other tests here own them. The laws do not judge requests issued * before a truncate, which the replay owns. The loader has no request * timeout, so a hung authoritative request may hold its work slot: progress @@ -232,6 +265,8 @@ type AuthorityCommand = | { type: `settle`; pick: number; ok: boolean } | { type: `invalidate`; react: boolean } | { type: `report`; react: boolean } + // `unexpressible` gives the row a rank that no cursor can express. + | { type: `headInsert`; unexpressible?: boolean } | { type: `failOldest` } | { type: `settleNewest` } | { type: `invalidateThenFailOldest` } @@ -316,6 +351,13 @@ const authorityCommand: fc.Arbitrary = fc.oneof( react: fc.boolean(), }), }, + { + weight: 1, + arbitrary: fc.record({ + type: fc.constant(`headInsert` as const), + unexpressible: fc.boolean(), + }), + }, // The oldest in-flight request is the one most often superseded. { weight: 1, arbitrary: fc.constant({ type: `failOldest` as const }) }, // A provider can answer newer requests first, so a repair chain finishes @@ -423,6 +465,20 @@ type AuthorityObservation = { joinedDeferrals: number ignoredReportsBeforeSettle: number ignoredReportsInsideFirstPage: number + ignoredReportsOverFullSource: number + ignoredReportsOverFallback: number + repairedReportsOnOwnPage: number + // Reports after a refetch moved the page's end, while a repair was owed, + // and while authoritative work was in flight. + movedEndReports: number + // Reports whose own first page read past a window that then shrank, and + // reports to a disposed loader. + reportsPastWindow: number + reportsAfterDispose: number + // Reports whose page's end, moved by a head insert, has no cursor. + unexpressibleEndReports: number + reportsOwingRepair: number + reportsDuringWork: number truncates: number replayFailures: number optimisticDepartures: number @@ -449,6 +505,11 @@ async function checkAuthorityCase( let joinedPending = false let truncatePending = false let boundaryUnexpressible = false + // Rows a refetch delivered before every installed row, in rank order. + const headRows: Array<{ id: number; rank: number }> = [] + // Whether an event since the last authoritative start owes a repair: an + // ordering invalidation, a visible delete, or an unexpressible boundary. + let repairOwed = false const info = createOrderByInfo({ limit: testCase.limit, ...(testCase.indexed ? {} : { index: undefined }), @@ -456,7 +517,22 @@ async function checkAuthorityCase( ...(testCase.syncRepair ? { sourceHoldsAllRows: true } : {}), requiresFullSource: testCase.requiresFullSource, ...(testCase.joined ? { joinedFilterSourceId: `markers` } : {}), - dataNeeded: () => (testCase.needsMore ? 1 : 0), + // A window that needs more always needs one row, as behind a joined + // filter that drops rows. Otherwise it needs the window's rows that do + // not sort at or before the loaded boundary. + dataNeeded: (boundary?: Record) => + testCase.needsMore + ? 1 + : Math.max( + 0, + info.offset + + info.limit - + (boundary === undefined + ? 0 + : snapshot().filter( + ({ value }) => value.rank <= (boundary.rank as number), + ).length), + ), offset: testCase.offset, }) let revision = 0 @@ -494,6 +570,8 @@ async function checkAuthorityCase( } } let releaseViolation: string | undefined + let unexpressibleTieViolation: string | undefined + let resetViolation: string | undefined // A disposed loader owns nothing: it issues no request and releases no // lease, even as requests it issued earlier settle. let disposed = false @@ -515,21 +593,25 @@ async function checkAuthorityCase( let releaseFaults = 0 let syncRepairs = 0 let invalidatingPages = 0 + // The rank each continuation's cursor starts after, so a read of its + // acquisition's rows applies the cursor as the subscription does. + const cursorStarts = new WeakMap() const record = (method: Observed[`method`], options: RequestOptions) => { // Like the real subscription, a continuation's acquisition carries a // cursor, so the loader does not take it for a prefix read. + const cursor = + options.minValues === undefined + ? undefined + : ({ + whereFrom: new PropRef([`row`, `rank`]), + whereCurrent: new PropRef([`row`, `rank`]), + } as unknown as NonNullable) + if (cursor) cursorStarts.set(cursor, options.minValues![0]) const acquisition: LoadSubsetOptions = { orderBy: options.orderBy, limit: options.limit, where: options.where, - ...(options.minValues === undefined - ? {} - : { - cursor: { - whereFrom: new PropRef([`row`, `rank`]), - whereCurrent: new PropRef([`row`, `rank`]), - } as unknown as NonNullable, - }), + ...(cursor ? { cursor } : {}), } const fullSource = method === `snapshot` && @@ -565,6 +647,29 @@ async function checkAuthorityCase( failedAt: -1, } ledger.push(request) + // No cursor expresses a NaN or infinite rank, so the loader reads the + // full source or repairs instead of reading that value's tie + // (ARCHITECTURE.md; `canExpressCursorOrder`). + // A truncate resets ordered loading (ARCHITECTURE.md), so the first + // finite request after it starts at the beginning of the order. A cursor + // there would continue from a boundary the truncate discarded. + if ( + truncates > 0 && + !fullSource && + options.minValues !== undefined && + ledger.every( + (other) => + other === request || + other.fullSource || + other.issuedTruncates !== truncates, + ) + ) { + resetViolation ??= `request ${request.index}, the first finite request after a truncate, continued from a cursor` + } + const tie = tieValue(options) + if (typeof tie === `number` && !Number.isFinite(tie)) { + unexpressibleTieViolation ??= `request ${request.index} read the tie of rank ${tie}, which no cursor can express` + } if (disposed) { disposeViolation ??= `request ${request.index} was issued after dispose` } @@ -578,7 +683,10 @@ async function checkAuthorityCase( ) { concurrencyViolation ??= `request ${request.index}, a step of a repair chain whose epoch moved, was issued before its replacement` } - if (startsAuthority(request)) epochMovesAtLastStart = epochMoves + if (startsAuthority(request)) { + epochMovesAtLastStart = epochMoves + repairOwed = false + } // An issued request consumes an owed continuation. An ordinary page also // shows the chain finished, since none is issued while a repair runs. continuationOwed = false @@ -698,25 +806,40 @@ async function checkAuthorityCase( // The page's own delivery changes source order before it returns. invalidateNextPage = false invalidatingPages++ + repairOwed = true loader.invalidateSourceOrdering() } answering.push(request) options.onLoadSubsetResult?.(request.deferred.promise, acquisition, release) } // Installed rows in ascending rank. An unexpressible boundary makes the - // last one NaN, which no cursor can express. - const snapshot = () => - Array.from({ length: testCase.snapshotRows }, (_, index) => ({ - type: `insert`, - key: index + 1, - value: { - id: index + 1, - rank: - boundaryUnexpressible && index === testCase.snapshotRows - 1 - ? Number.NaN - : index + 1, - }, - })) + // last one NaN, which no cursor can express. A read applies a tie's + // equality, a cursor (rows strictly after its start), and the limit, as + // the subscription's ordered read does. + const snapshot = (options?: LoadSubsetOptions) => { + const tie = options === undefined ? undefined : tieValue(options) + const after = options?.cursor && cursorStarts.get(options.cursor) + return [ + ...headRows.map((row) => ({ type: `insert`, key: row.id, value: row })), + ...Array.from({ length: testCase.snapshotRows }, (_, index) => ({ + type: `insert`, + key: index + 1, + value: { + id: index + 1, + rank: + boundaryUnexpressible && index === testCase.snapshotRows - 1 + ? Number.NaN + : index + 1, + }, + })), + ] + .filter( + ({ value }) => + (tie === undefined || value.rank === tie) && + (after === undefined || value.rank > (after as number)), + ) + .slice(0, options?.limit) + } const subscription = { readOrderedSnapshot: snapshot, setOrderByIndex: () => {}, @@ -789,6 +912,53 @@ async function checkAuthorityCase( let joinedDeferrals = 0 let ignoredReportsBeforeSettle = 0 let ignoredReportsInsideFirstPage = 0 + let ignoredReportsOverFullSource = 0 + let ignoredReportsOverFallback = 0 + let repairedReportsOnOwnPage = 0 + let movedEndReports = 0 + let reportsPastWindow = 0 + let reportsAfterDispose = 0 + let unexpressibleEndReports = 0 + let reportsOwingRepair = 0 + let reportsDuringWork = 0 + // Steps that settle a request successfully, run a pass, report, or change + // joined work. None of them owes a repair. + const quietCommand = (command: AuthorityCommand) => + (command.type === `settle` && command.ok) || + command.type === `settleNewest` || + command.type === `pass` || + command.type === `report` || + command.type === `headInsert` || + // A window that shrinks shows only rows that were already proven. + (command.type === `move` && command.delta === -1) || + command.type === `joinedBusy` || + command.type === `joinedIdle` + // An unproven report whose pass pending joined work deferred: the ledger + // index at the report and the window size it judged. + let owedProof: + { from: number; endRank: unknown; expressible: boolean } | undefined + // The proof is a bounded repair or the tie of the page's end. With every + // installed rank expressible, a full-source read is not a repair: the + // bounded repair is available (ARCHITECTURE.md), and a full-source read is + // the cost the report path avoids. + const expectProofRequest = ( + from: number, + endRank: unknown, + expressible: boolean, + ) => { + const issued = ledger + .slice(from) + .some( + ({ authoritative, fullSource, options }) => + (authoritative && (!fullSource || !expressible)) || + tieValue(options) === endRank, + ) + if (!issued) { + throw new Error( + `unproven report: a report over a first page that no row it read proves issued no bounded repair and no tie of its end`, + ) + } + } let truncates = 0 let replayFailures = 0 let optimisticDepartures = 0 @@ -917,6 +1087,14 @@ async function checkAuthorityCase( if (releaseViolation) { throw new Error(`release ${checkpoint}: ${releaseViolation}`) } + if (resetViolation) { + throw new Error(`reset ${checkpoint}: ${resetViolation}`) + } + if (unexpressibleTieViolation) { + throw new Error( + `unexpressible tie ${checkpoint}: ${unexpressibleTieViolation}`, + ) + } if (concurrencyViolation) { throw new Error( `one request at a time ${checkpoint}: ${concurrencyViolation}`, @@ -1008,6 +1186,9 @@ async function checkAuthorityCase( await flush() if (testCase.settledFirst && ledger[0]) await settle(ledger[0], true) for (const [step, command] of testCase.history.entries()) { + // An event outside the quiet set, such as a truncate, a failure, a + // move, or a disposal, can change or end what a deferred report owes. + if (owedProof && !quietCommand(command)) owedProof = undefined switch (command.type) { case `settle`: { const open = inflight() @@ -1044,6 +1225,7 @@ async function checkAuthorityCase( // or a later graph run defers it, while the oldest in-flight request // fails before the loader reacts. case `invalidateThenFailOldest`: { + repairOwed = true loader.invalidateSourceOrdering() const request = inflight()[0] if (!request) break @@ -1057,6 +1239,7 @@ async function checkAuthorityCase( break } case `invalidate`: + repairOwed = true loader.invalidateSourceOrdering() if (command.react) graphPass() await flush() @@ -1069,25 +1252,21 @@ async function checkAuthorityCase( !testCase.releaseTruncates && !testCase.releaseDisposes && !boundaryUnexpressible && - testCase.history - .slice(0, step) - .every( - (earlier) => - (earlier.type === `settle` && earlier.ok) || - earlier.type === `settleNewest` || - earlier.type === `pass` || - earlier.type === `report` || - earlier.type === `joinedBusy` || - earlier.type === `joinedIdle`, - ) && + testCase.history.slice(0, step).every(quietCommand) && ledger.every(({ outcome }) => outcome !== `failed`) const windowSize = testCase.offset + info.limit const [first, ...later] = ledger const beforeSettle = ledger.every(({ outcome }) => outcome !== `ok`) - const insideFirstPage = + // The page's end is the installed row at the window's last position. + const ranks = snapshot().map(({ value }) => value.rank) + const endRank = ranks[windowSize - 1] + const expressible = ranks.every((rank) => Number.isFinite(rank)) + // The loader's coverage is its own first page with only tie + // requests after it. + const ownFirstPage = !testCase.requiresFullSource && !testCase.needsMore && - testCase.snapshotRows > windowSize && + ranks.length > windowSize && first !== undefined && first.outcome === `ok` && !first.authoritative && @@ -1104,11 +1283,89 @@ async function checkAuthorityCase( options.cursor === undefined && options.orderBy === undefined, ) + // A row past the page proves the source's next row only when the + // page read it, or when it ties with the page's end and the loader + // read that tie. Any other installed row can come from another + // request, so it proves nothing, and neither does a tie of an older + // end that a head insert moved. + const insideFirstPage = + ownFirstPage && + ((first.options.limit ?? 0) > windowSize || + later.some(({ options }) => tieValue(options) === endRank)) + if (ownFirstPage && (first.options.limit ?? 0) > windowSize) { + reportsPastWindow++ + } + if ( + ownFirstPage && + later.some(({ options }) => tieValue(options) !== undefined) && + !insideFirstPage + ) { + movedEndReports++ + } + // Coverage is a held full-source read since the last truncate. It + // owes nothing whatever came before, so this premise needs no quiet + // history. + const coversSource = + !replayPending() && + !isDisposed() && + ledger.some( + ({ fullSource, outcome, released, issuedTruncates }) => + fullSource && + outcome === `ok` && + !released && + issuedTruncates === truncates, + ) + if (isDisposed()) reportsAfterDispose++ + if ( + ownFirstPage && + typeof endRank === `number` && + !Number.isFinite(endRank) + ) { + unexpressibleEndReports++ + } + const owesRepair = + repairOwed && !replayPending() && !isDisposed() && command.react + if (owesRepair) reportsOwingRepair++ + if ( + ledger.some( + ({ outcome, authoritative, hung }) => + outcome === `inflight` && authoritative && !hung, + ) + ) { + reportsDuringWork++ + } const before = ledger.length loader.revalidateLoadedCoverage() if (command.react) graphPass() await flush() - if (quiet && (beforeSettle || insideFirstPage)) { + // A report while a repair is owed repairs instead: no tie or page + // comes before the repair's prefix. + const firstIssued = ledger[before] + if (owesRepair && firstIssued && !firstIssued.authoritative) { + throw new Error( + `owed repair: request ${firstIssued.index}, an ordinary read, came before the repair a report owed`, + ) + } + // A report the loader cannot prove covers nothing it could change + // owes a repair, which its next pass starts, or a tie request for + // the page's end, which completes the proof. Pending joined work + // defers ordinary loading, so the pass that runs when that work + // finishes owes the request instead. + if ( + command.react && + quiet && + !beforeSettle && + ownFirstPage && + !insideFirstPage + ) { + if (joinedPending && testCase.joined) { + owedProof = { from: before, endRank, expressible } + } else { + expectProofRequest(before, endRank, expressible) + repairedReportsOnOwnPage++ + } + } + if (quiet && beforeSettle) { const issued = ledger .slice(before) .find(({ authoritative }) => authoritative) @@ -1117,11 +1374,50 @@ async function checkAuthorityCase( `ignored report: request ${issued.index} re-read the source for a report that covers nothing it could change`, ) } - if (beforeSettle) ignoredReportsBeforeSettle++ - else ignoredReportsInsideFirstPage++ + ignoredReportsBeforeSettle++ + } else if ((quiet && insideFirstPage) || coversSource) { + // The loader ignores the report, so it reads nothing for it. + const issued = ledger[before] + if (issued) { + throw new Error( + `ignored report: request ${issued.index} was read for a report over coverage that ${coversSource ? `a full-source read` : `the loader's own first page`} proves`, + ) + } + if (!coversSource) ignoredReportsInsideFirstPage++ + else if (testCase.requiresFullSource) ignoredReportsOverFullSource++ + // A paged loader whose boundary fell back to a full-source read. + else ignoredReportsOverFallback++ } break } + // A refetch delivered a row that sorts before every installed row. + case `headInsert`: { + // One `-Infinity` row at most: a second would tie with it, which + // the fake's subtraction order cannot compare. + const unexpressible = + command.unexpressible === true && + headRows.every(({ rank }) => Number.isFinite(rank)) + const row = { + id: 1001 + headRows.length, + rank: unexpressible + ? Number.NEGATIVE_INFINITY + : -1 - + headRows.filter(({ rank }) => Number.isFinite(rank)).length, + } + headRows.push(row) + headRows.sort((left, right) => + left.rank === right.rank ? 0 : left.rank < right.rank ? -1 : 1, + ) + loader.onSourceChanges( + [{ type: `insert`, key: row.id, value: row }], + sentRows, + () => false, + ) + sentRows.set(row.id, row) + graphPass() + await flush() + break + } case `pass`: graphPass() await flush() @@ -1146,6 +1442,7 @@ async function checkAuthorityCase( case `unexpressibleBoundary`: boundaryUnexpressible = !boundaryUnexpressible if (boundaryUnexpressible) unexpressibleBoundaries++ + repairOwed = true loader.invalidateSourceOrdering() graphPass() await flush() @@ -1159,6 +1456,15 @@ async function checkAuthorityCase( joinedPending = false graphPass() await flush() + if (owedProof) { + expectProofRequest( + owedProof.from, + owedProof.endRank, + owedProof.expressible, + ) + owedProof = undefined + repairedReportsOnOwnPage++ + } break case `truncate`: truncate() @@ -1188,6 +1494,7 @@ async function checkAuthorityCase( await flush() break case `sourceDelete`: + repairOwed = true sourceDeletes++ loader.onSourceChanges( [{ type: `delete`, key: 1, value: { id: 1, rank: 1 } }], @@ -1377,16 +1684,20 @@ async function checkAuthorityCase( // Cost: a full-source read is very expensive, so a repair avoids one // whenever a bounded read is available (ARCHITECTURE.md). That holds // after a settled request in this incarnation, for a plan that does - // not need the full source, with an expressible boundary, unless an - // unindexed window still needs rows past a repeated prefix (the - // documented fallback). One repair is then one chain per ordering - // epoch move: in this fake, a prefix, a tie, and a refill. + // not need the full source, with an expressible boundary. An + // unindexed window that still needs rows grows its prefix instead + // (ARCHITECTURE.md, "an underfilled unindexed prefix grows"). One + // repair is then one chain per ordering epoch move: a prefix, then a + // tie and a refill page for each boundary it reaches (ARCHITECTURE.md, + // ordered repair). Each boundary is a different installed row, so a + // chain issues at most one prefix plus two requests per row. await drain() const repair = ledger.slice(before) const boundedRepair = !testCase.requiresFullSource && - !boundaryUnexpressible && - (testCase.indexed || !testCase.needsMore) && + // Every installed rank is expressible, so no boundary needs the + // full-source fallback. + snapshot().every(({ value }) => Number.isFinite(value.rank)) && repair.every(({ outcome }) => outcome === `ok`) && ledger.some( (request) => @@ -1406,11 +1717,30 @@ async function checkAuthorityCase( (request) => request.authoritative, ).length const chains = epochMoves - movesBefore - if (authoritative > 3 * chains) { + if (authoritative > chains * (1 + 2 * snapshot().length)) { throw new Error( `cost: one invalidation issued ${authoritative} authoritative requests for ${chains} epoch moves`, ) } + // Each boundary is a different read: no chain reads the same + // prefix, tie, or page twice. + const reads = new Map() + for (const { authoritative: owed, options } of repair) { + if (!owed) continue + const read = JSON.stringify([ + options.limit, + tieValue(options), + options.minValues, + ]) + reads.set(read, (reads.get(read) ?? 0) + 1) + } + for (const [read, count] of reads) { + if (count > chains) { + throw new Error( + `cost: one invalidation repeated the read ${read} ${count} times for ${chains} epoch moves`, + ) + } + } } } } @@ -1445,6 +1775,15 @@ async function checkAuthorityCase( joinedDeferrals, ignoredReportsBeforeSettle, ignoredReportsInsideFirstPage, + ignoredReportsOverFullSource, + ignoredReportsOverFallback, + repairedReportsOnOwnPage, + movedEndReports, + reportsPastWindow, + reportsAfterDispose, + unexpressibleEndReports, + reportsOwingRepair, + reportsDuringWork, truncates, replayFailures, optimisticDepartures, @@ -1870,9 +2209,85 @@ describe(`Ordered source request authority`, () => { }, }, { - // Without the first-page check, the report repaired a window that the - // loader's own first prefix already proves. - name: `a report while the window is inside the loader's first page`, + // The first page and its tie settle, then joined work starts. A report + // over the moved end stages only that end's tie, and pending joined + // work defers it. The pass after the joined work finishes must read it; + // a loader that dropped the staged tie read nothing. + name: `a staged end tie that waits for joined work`, + testCase: { + indexed: false, + limit: 1, + settledFirst: true, + releaseFault: false, + syncRepair: false, + ...ordinaryScenario, + joined: true, + snapshotRows: 3, + needsMore: false, + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settleNewest` }, + { type: `joinedBusy` }, + { type: `report`, react: true }, + { type: `joinedIdle` }, + ], + }, + }, + { + // A truncate after a deferred unproven report hands the window to its + // replay, so the pass after the joined work owes no repair or tie. + name: `an unproven report whose deferred pass a truncate replaces`, + testCase: { + indexed: false, + limit: 1, + settledFirst: true, + releaseFault: false, + syncRepair: false, + ...ordinaryScenario, + joined: true, + snapshotRows: 3, + needsMore: false, + history: [ + { type: `joinedBusy` }, + { type: `settle`, pick: 0, ok: true }, + { type: `report`, react: true }, + { type: `truncate` }, + { type: `joinedIdle` }, + ], + }, + }, + { + // Pending joined work defers the pass after an unproven report, so the + // repair or tie that the report owes waits for that work. The pass that + // runs when it finishes must start it. A random campaign found the law + // judging the deferred pass instead. + name: `an unproven report while joined work defers loading`, + testCase: { + indexed: false, + limit: 1, + settledFirst: true, + releaseFault: false, + syncRepair: false, + ...ordinaryScenario, + joined: true, + snapshotRows: 3, + needsMore: false, + history: [ + { type: `joinedBusy` }, + { type: `settle`, pick: 0, ok: true }, + { type: `settleNewest` }, + { type: `report`, react: true }, + { type: `joinedIdle` }, + ], + }, + }, + { + // The first page read only the window, and the loader's tie request + // read a value past it, not the page's end. An installed row past the + // page can come from another request, so it proves nothing, and the + // report must repair. Ignoring it kept a window that missed a row tied + // with a refetched page's end. + name: `a report over a first page that read only the window`, testCase: { indexed: false, limit: 1, @@ -1883,6 +2298,7 @@ describe(`Ordered source request authority`, () => { snapshotRows: 3, needsMore: false, history: [ + { type: `settle`, pick: 0, ok: true }, { type: `settle`, pick: 0, ok: true }, { type: `report`, react: true }, ], @@ -1925,6 +2341,7 @@ describe(`Ordered source request authority`, () => { joinedDeferrals: 0, ignoredReportsBeforeSettle: 0, ignoredReportsInsideFirstPage: 0, + repairedReportsOnOwnPage: 0, truncates: 0, replayFailures: 0, optimisticDepartures: 0, @@ -1986,7 +2403,8 @@ describe(`Ordered source request authority`, () => { for (const [dimension, count] of Object.entries(later)) { if ( dimension === `startThrowsDuringFailure` || - dimension === `ignoredReportsInsideFirstPage` + dimension === `ignoredReportsInsideFirstPage` || + dimension === `repairedReportsOnOwnPage` ) continue expect(count, dimension).toBeGreaterThan(0) @@ -1995,4 +2413,169 @@ describe(`Ordered source request authority`, () => { }, 120_000, ) + + // The report laws need premises that the broad grammar rarely lines up. + // Each history here starts with one of these prefixes over random case + // parameters, then a random suffix: + // - the first page and its tie settle, then a report: the window is + // inside the loader's own first page, so the report is ignored; + // - the same, but a head insert first moves the page's end before the tie + // the loader read, so the report is unproven; + // - an invalidation whose reaction waits, then a report that reacts: a + // repair is owed; + // - an invalidation that starts its repair, then a report while that + // repair is in flight; + // - a two-row window whose first page and tie settle, then shrinks to one + // row before a report: the page read past the window, which proves it; + // - the moved-end prefix with the loader disposed before the report, which + // the dispose law then judges; + // - the moved-end prefix with a `-Infinity` head row, so no cursor can + // express the page's new end; + // - a one-row window whose page and tie settle, then a `-Infinity` head + // row and an invalidation: the repair's boundary has no cursor, so it + // falls back to a full-source read, and a report follows over it; + // - a plan that reads only the full source, then a report over it. + // A release that truncates or disposes would end the quiet history, so + // no prefix allows one. + const unprovenReportProperty = `ordered-loader.unproven-report` + const reportPrefixes: Array<{ + history: Array + fullSource?: boolean + narrow?: boolean + wide?: boolean + }> = [ + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert` }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `invalidate`, react: false }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `invalidate`, react: true }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `move`, delta: -1 }, + { type: `report`, react: true }, + ], + wide: true, + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert` }, + { type: `dispose` }, + { type: `report`, react: true }, + ], + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert`, unexpressible: true }, + { type: `report`, react: true }, + ], + // The end is the `-Infinity` row only in a one-row window. + narrow: true, + }, + { + history: [ + { type: `settle`, pick: 0, ok: true }, + { type: `settle`, pick: 0, ok: true }, + { type: `headInsert`, unexpressible: true }, + { type: `invalidate`, react: true }, + { type: `settleNewest` }, + { type: `settleNewest` }, + { type: `settleNewest` }, + { type: `report`, react: true }, + ], + narrow: true, + }, + { history: [{ type: `report`, react: true }], fullSource: true }, + ] + const unprovenReportCase: fc.Arbitrary = fc + .tuple( + authorityCaseArbitrary, + fc.constantFrom(...reportPrefixes), + fc.array(authorityCommand, { maxLength: 8 }), + ) + .map(([testCase, prefix, rest]) => ({ + ...testCase, + settledFirst: true, + requiresFullSource: prefix.fullSource === true, + ...(prefix.wide ? { limit: 2 as const, offset: 0 as const } : {}), + ...(prefix.narrow ? { limit: 1 as const, offset: 0 as const } : {}), + snapshotRows: 3, + needsMore: false, + releaseTruncates: false, + releaseDisposes: false, + history: [...prefix.history, ...rest], + })) + const unprovenReportCampaigns = + authorityReplay.replayPath === undefined + ? ([`fixed`, `random`] as const) + : authorityReplay.replayProperty === unprovenReportProperty + ? ([`replay`] as const) + : ([] as const) + + it.each(unprovenReportCampaigns)( + `answers each loaded-subset report as its coverage requires (%s campaign)`, + async (campaign) => { + // Declared reach: histories where each report law judged a report. + const reportStates = [ + `repairedReportsOnOwnPage`, + `ignoredReportsInsideFirstPage`, + `ignoredReportsOverFullSource`, + `ignoredReportsOverFallback`, + `movedEndReports`, + `reportsPastWindow`, + `reportsAfterDispose`, + `unexpressibleEndReports`, + `reportsOwingRepair`, + `reportsDuringWork`, + ] as const + const reach = createOracleReach(unprovenReportProperty, reportStates, { + minimum: 20, + }) + await fc.assert( + fc.asyncProperty(unprovenReportCase, async (testCase) => { + reach.beginHistory() + const observation = await checkAuthorityCase(testCase) + for (const state of reportStates) { + if (observation[state] > 0) reach.reach(state) + } + }), + campaign === `fixed` + ? { seed: 20261023, numRuns: oracleRuns(1000) } + : oraclePropertyOptions(1000, unprovenReportProperty), + ) + reach.assertReached() + }, + 120_000, + ) }) diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts index 3c4816f46a..26b27425b5 100644 --- a/packages/db/tests/query/ordered-work-oracle.property.test.ts +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -17,6 +17,7 @@ import { } from '../oracle-config.js' import { declareOracle } from '../oracle-declaration.js' import { evaluateReferenceExpression } from '../reference-expression-oracle.js' +import { createOracleReach } from '../oracle-reach.js' import { flushPromises } from '../utils.js' import { withHistoryCleanup } from '../optimistic-history-oracle.js' import { @@ -3512,7 +3513,7 @@ describe(`ordered source work oracle`, () => { * source's limit), the index (present or absent), the direction, one or two * order terms (rank, then ascending id), limit 1..3, offset 0..1, and a history * of upserts, deletes, deletes of the first visible row, and label-only - * touches over ids 0..29 and ranks 0..15, NaN, or null, so ties, deletes + * touches over ids 0..29 and ranks 0..15, NaN, null, or undefined, so ties, deletes * inside and at the edge of the window, rank changes across its boundary, and * distant rows the query has already received all occur. A second campaign * fixes the premise of the joined law: a joined filter, an index and one order @@ -3611,9 +3612,25 @@ describe(`ordered source work oracle`, () => { * One case in four adds a peer with its own window, the first * `offset + limit + 1` rows. Its rows law holds beside the first query's at * every checkpoint; the work laws count one consumer, so they do not apply - * to those cases. A peer over a refetching provider is excluded from the - * grammar until #2155 is fixed: an indexed trace is fixed and pinned, but an - * unindexed peer still keeps a short joined window after a refetch. + * to those cases. A peer over a refetching provider with a joined filter is + * excluded from the grammar until #2155 is fixed: an indexed trace is fixed + * and pinned, but an unindexed peer still keeps a short joined window after + * a refetch. A peer over a source filter stays in the grammar. + * + * `null` and `undefined` ranks are both nullish, so they tie: either may + * come first, and neither is an expressible cursor value. + * + * - **Proven report:** a signalling refetch can report a shift that changes + * nothing the first query's own first page proves. Its page and the tie + * of the page's end answer the same rows, the page ends on a plain number, + * and the model's next row sorts strictly after that end and is + * installed, as a peer's longer page installs it. The refetch also deletes + * and reorders no installed row the first query received, because such a + * change starts the repair path by itself. Then the report starts no + * request for the first query, and until a step that can change source + * ordering, the first query issues no repair (`refetch: true`) request. + * A campaign fixed to that premise must reach it, and a loader that + * repairs every report fails it. * Joined filters can match a row through several markers, by `dup`, so a * LEFT-joined filter and a full join keep one result row per matched * (row, marker) pair, and an anti-join keeps a row once. @@ -3726,6 +3743,7 @@ function eagerRowOf(maxId: number) { { weight: 6, arbitrary: fc.integer({ min: 0, max: 15 }) }, { weight: 1, arbitrary: fc.constant(Number.NaN) }, { weight: 1, arbitrary: fc.constant(null as unknown as number) }, + { weight: 1, arbitrary: fc.constant(undefined as unknown as number) }, ), eligible: fc.boolean(), }) @@ -3817,13 +3835,15 @@ const eagerCaseBase: fc.Arbitrary = fc.record({ // A peer over a refetching on-demand provider can keep a short joined window // after a refetch (#2155). An indexed plan's trace is fixed and pinned below, // but an unindexed one still fails (a peer shows [100] where the model shows -// [100, 101]). Each campaign excludes only that pairing, as its last map, -// until the fix. +// [100, 101]). Each campaign excludes only a joined, anti-joined, or full-joined +// filter in that pairing, as its last map, until the fix; a source filter +// keeps its peer. function withoutPeerRefetch(testCase: EagerCase): EagerCase { return testCase.path === `on-demand` && (testCase.provider === `refetch` || testCase.provider === `refetch-signal`) && - testCase.peer + testCase.peer && + testCase.filter !== `source` ? { ...testCase, peer: false } : testCase } @@ -3915,6 +3935,7 @@ const gapRank: fc.Arbitrary = fc.oneof( { weight: 6, arbitrary: fc.integer({ min: 0, max: 15 }) }, { weight: 1, arbitrary: fc.constant(Number.NaN) }, { weight: 1, arbitrary: fc.constant(null as unknown as number) }, + { weight: 1, arbitrary: fc.constant(undefined as unknown as number) }, ) const eagerJoinedGapCase: fc.Arbitrary = fc @@ -3985,6 +4006,54 @@ const eagerFullJoinCase: fc.Arbitrary = fc // first deletes a visible row, so the refill request is in flight while the // later commands change rows. Releases answer it from the rows at that // moment, between those changes. +// The proven-report premise: a peer over a signalling refetch provider and a +// source filter, so the peer installs the row after the first query's page. +// Histories upsert rows that tie with the peer's end or sort past both +// windows, then refetch, so a report fires while the first query's own page +// and tie stay the same; later moves check that it owes no repair. +const eagerProvenReportCase: fc.Arbitrary = fc + .record({ + base: eagerCaseBase, + ranks: fc.uniqueArray(fc.integer({ min: 0, max: 15 }), { + minLength: 3, + maxLength: 6, + }), + upserts: fc.array( + fc.record({ id: fc.integer({ min: 10, max: 19 }), offset: fc.nat(3) }), + { minLength: 1, maxLength: 3 }, + ), + rest: eagerJoinedHistory, + }) + .map(({ base, ranks, upserts, rest }) => { + const sorted = [...ranks].sort((left, right) => left - right) + const size = base.offset + base.limit + // Ranks at or after the peer's end, so the first query's page keeps its + // rows while the peer's answers change. + const peerEnd = sorted[Math.min(size, sorted.length - 1)]! + return { + ...base, + path: `on-demand` as const, + provider: `refetch-signal` as const, + latency: `immediate` as const, + peer: true, + filter: `source` as const, + indexed: true, + direction: `asc` as const, + terms: 1 as const, + initial: sorted.map((rank, id) => ({ id, rank, eligible: true })), + history: [ + ...upserts.map(({ id, offset }) => ({ + type: `upsert` as const, + id, + rank: peerEnd + offset, + eligible: true, + })), + { type: `refetch` as const }, + ...rest, + ], + } + }) + const eagerHeldCase: fc.Arbitrary = fc .tuple( eagerCaseBase, @@ -4056,9 +4125,10 @@ function eagerOrder( ): Array { // Documented order (live-queries guide): nulls first in either direction // unless the query asks for them last; NaN is greater than every other - // non-null value; ties by ascending id. - const position = (rank: number | null): [number, number] => { - if (rank === null) return [nulls === `last` ? 3 : 0, 0] + // non-null value; ties by ascending id. `null` and `undefined` are both + // nullish, so they tie with each other. + const position = (rank: number | null | undefined): [number, number] => { + if (rank == null) return [nulls === `last` ? 3 : 0, 0] if (Number.isNaN(rank)) return [direction === `asc` ? 2 : 1, 0] return [direction === `asc` ? 1 : 2, direction === `asc` ? rank : -rank] } @@ -4101,14 +4171,39 @@ function expectEagerWindow( expect(actual, label).toEqual(expected) return } + // `null` and `undefined` tie, so either may come first among nullish rows. const rank = (id: number) => { const row = rows.get(id) - return row?.eligible ? row.rank : `ineligible ${id}` + return row?.eligible + ? ((row.rank as number | null | undefined) ?? null) + : `ineligible ${id}` } expect(actual.map(rank), label).toEqual(expected.map(rank)) expect(new Set(actual).size, `${label}: distinct rows`).toBe(actual.length) } +/** + * Whether a bounded prefix of the source leaves the window short. The prefix + * is the first `offset + limit` source rows in the query's order. A source + * filter runs inside the request, so its prefix holds only eligible rows. A + * joined or function filter drops rows only after the source's limit, so its + * prefix can hold fewer eligible rows than the window needs. + */ +function prefixLeavesWindowShort( + rows: ReadonlyMap, + testCase: EagerCase, +): boolean { + const size = testCase.offset + testCase.limit + const candidates = + testCase.filter === `source` + ? rows + : new Map( + [...rows].map(([rowId, row]) => [rowId, { ...row, eligible: true }]), + ) + const prefix = eagerOrder(candidates, testCase).slice(0, size) + return prefix.filter(({ id }) => rows.get(id)!.eligible).length < size +} + /** Ranks of the last window row and the first row after the window. */ function boundaryRanks( rows: ReadonlyMap, @@ -4133,6 +4228,9 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ heldRequests: number cooperativeCancels: number lateAnswers: number + ignoredProvenReports: number + unindexedRefetchChecks: number + shortUnindexedRefills: number }> { const joined = isJoinedFilter(testCase) const fullJoin = testCase.filter === `full-joined` @@ -4227,6 +4325,10 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // The requests the provider has served. A request without a predicate // covers every row. const servedRequests: Array = [] + // Loader requests per consumer subscription. The first query starts its + // sync run before any peer, so the first request is its own. + const requestsBy = new Map() + let firstSubscription: unknown // A refetching provider has skipped a change since its last refetch, so // the Collection may lag the rows and the window laws wait for a refetch. let stale = false @@ -4330,11 +4432,38 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // The rows each served request last returned, at its answer or at the // latest refetch. A refetching provider keeps exactly these rows. const lastAnswered = new Map>() + let refetchedAnswers = new Map>() // Bounded pages that ended on a null or NaN rank. Neither can be // expressed as a cursor (canExpressCursorOrder), so the loader then takes // the documented full-source fallback. let unexpressiblePageEnds = 0 + // Reports that the first query's own page proved, so it owed no request, + // and the first query's request count before the latest such report. + let ignoredProvenReports = 0 + // Signalling refetches of an unindexed plan where the full-source law + // applied, and where the documented short-prefix fallback excused it. + let unindexedRefetchChecks = 0 + let shortUnindexedRefills = 0 + let provenReportRequestsBefore: number | undefined + // Set by such a report until a step that can change source ordering, and + // the first repair request the first query issued meanwhile. + let provenSinceOrderingChange = false + let repairAfterProvenReport: string | undefined function servePage(options: LoadSubsetOptions): true | Promise { + firstSubscription ??= options.subscription + // A repair marks its requests `refetch: true`. After a report the first + // query's own page proved, it owes none until ordering changes again. + if ( + provenSinceOrderingChange && + options.subscription === firstSubscription && + options.refetch === true + ) { + repairAfterProvenReport ??= `request ${providerRequests + 1}` + } + requestsBy.set( + options.subscription, + (requestsBy.get(options.subscription) ?? 0) + 1, + ) providerRequests++ if (providerRequests > 4 * rows.size + 100) { // A rejected load is a provider failure the loader must handle, so it @@ -4385,7 +4514,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ rangeEnds.set(options, { ...page.at(-1)! }) // Generated rows may carry a null rank behind the `number` row type. const end = page.at(-1)!.rank as number | null - if (end === null || Number.isNaN(end)) unexpressiblePageEnds++ + if (end == null || Number.isNaN(end)) unexpressiblePageEnds++ } served += page.length servedBy.set( @@ -4449,7 +4578,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // the rows apply, and only when some answer changed; the contract lets a // sync adapter report whenever its pages may have shifted, so this // provider judges the loader's response, not query-db's gate. - async function refetch(): Promise { + async function refetch(): Promise { const current = new Map() let changed = false for (const request of servedRequests) { @@ -4466,6 +4595,11 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ lastAnswered.set(request, ids) for (const row of covered) current.set(row.id, row) } + // The answers as this refetch produced them, before a report's repair + // can retire the requests that gave them. + refetchedAnswers = new Map( + [...lastAnswered].map(([request, ids]) => [request, new Set(ids)]), + ) const applied = send( [...current.values()], [...sent.keys()].filter((rowId) => !current.has(rowId)), @@ -4477,6 +4611,74 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // Core reports in a microtask; the repair then runs its requests. await flushPromises() } + return signalsRefetch && changed + } + // The first query's own served requests, when they are only its first + // page and one tie request: the answers a report could change for it. + function firstPageAnswers(): + | { + page: LoadSubsetOptions + tie: LoadSubsetOptions + pageIds: Set + tieIds: Set + } + | undefined { + if (testCase.filter !== `source` || inFlight()) return + const own = servedRequests.filter( + (options) => options.subscription === firstSubscription, + ) + const size = shown.offset + shown.limit + const page = own.find( + (options) => + options.orderBy !== undefined && + options.cursor === undefined && + options.limit === size, + ) + const tie = own.find( + (options) => options.where !== undefined && options.orderBy === undefined, + ) + const pageIds = page && lastAnswered.get(page) + const tieIds = tie && lastAnswered.get(tie) + if (own.length !== 2 || !page || !tie || !pageIds || !tieIds) return + return { page, tie, pageIds: new Set(pageIds), tieIds: new Set(tieIds) } + } + // After a refetch, the first query's window is proven by its own page + // alone when the refetch left its page and tie answers unchanged, the page + // holds the model's first rows through a plain numeric end, the tie reads + // that end, and the model's next row sorts strictly after the end and is + // installed. Then a report has nothing to repair for it. + function provenFirstPage( + before: NonNullable>, + ): boolean { + // Read the answers of the requests judged before the refetch. Requests + // the step added, such as a repair, must not hide the premise. + const pageIds = refetchedAnswers.get(before.page) + const tieIds = refetchedAnswers.get(before.tie) + if (!pageIds || !tieIds) return false + const after = { ...before, pageIds, tieIds } + const sameIds = (left: Set, right: Set) => + left.size === right.size && [...left].every((key) => right.has(key)) + const size = shown.offset + shown.limit + const ordered = eagerOrder(rows, shown) + const end = ordered[size - 1] + const next = ordered[size] + const endRank = end?.rank as number | null | undefined + return ( + sameIds(before.pageIds, after.pageIds) && + sameIds(before.tieIds, after.tieIds) && + end !== undefined && + next !== undefined && + endRank != null && + !Number.isNaN(endRank) && + !Object.is(next.rank, endRank) && + sent.has(next.id) && + after.pageIds.size === size && + ordered.slice(0, size).every((row) => after.pageIds.has(row.id)) && + // The tie admits the end's rank and nothing on either side of it. + matches(after.tie.where, end) && + !matches(after.tie.where, { ...end, rank: endRank + 0.5 }) && + !matches(after.tie.where, { ...end, rank: endRank - 0.5 }) + ) } function pageFor(options: LoadSubsetOptions): Array { let page = [...rows.values()].filter( @@ -4838,7 +5040,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // A full join reads its full source too: a rootless row cannot be shown // unmatched until every root row has loaded. if (testCase.filter !== `source`) return undefined - if (boundary.some((rank) => rank === null || Number.isNaN(rank))) return + if (boundary.some((rank) => rank == null || Number.isNaN(rank))) return return prefixBound(boundary, shown.offset + shown.limit) } // Publication: a change publishes its window once. Each batch a subscriber @@ -4899,7 +5101,7 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ // Generated rows may carry a null rank behind the `number` row type. prefix.every( (row) => - (row.rank as number | null) !== null && !Number.isNaN(row.rank), + (row.rank as number | null) != null && !Number.isNaN(row.rank), ) if (proven) { expect( @@ -4970,9 +5172,10 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ ) for (const [index, rowId] of shownIds.slice(1).entries()) { const predecessor = shownIds[index]! + // `null` and `undefined` ranks tie, so either may come first. const sameRank = Object.is( - ledger.get(predecessor)?.rank, - ledger.get(rowId)?.rank, + ledger.get(predecessor)?.rank ?? null, + ledger.get(rowId)?.rank ?? null, ) expect( sameRank || position.get(predecessor)! < position.get(rowId)!, @@ -5023,23 +5226,54 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ const requestsBefore = providerRequests const visibleBefore = JSON.stringify(visibleIds()) stepChangedInFlight = false + // Only a window move keeps a proven report's no-repair premise; any + // other step can change source ordering. + if (command.type !== `move`) provenSinceOrderingChange = false if (command.type === `refetch`) { const fullSourceBefore = fullSourceRequests const unexpressibleBefore = unexpressiblePageEnds - await refetch() - // The repair a report starts reads each window's bounded prefix, - // tie, and refill; a whole-source read would be the cost the report - // exists to avoid. A page that ends on a null or NaN rank takes the - // documented full-source fallback instead. - // A joined predicate can leave the root prefix underfilled. Without - // an order index, finite continuation is unavailable, so the - // architecture permits full-source fallback. Row checks still apply. - const boundedContinuation = testCase.indexed || !joined + const firstAnswers = firstPageAnswers() + const firstRequestsBefore = requestsBy.get(firstSubscription) ?? 0 + const installedBefore = new Map(sent) + const reported = await refetch() + // A delete or rank change of an installed row that the first query's + // subscription received starts the repair path by itself + // (ARCHITECTURE.md), so only a refetch that changes neither is judged. + // A source-filter subscription receives every eligible row. + const keptInstalledRows = [...installedBefore.values()] + .filter((row) => row.eligible) + .every((row) => { + const now = sent.get(row.id) + return now !== undefined && Object.is(now.rank, row.rank) + }) + // A report that changes nothing the first query's own page proves + // starts no request for it (ARCHITECTURE.md: a window inside the + // loader's own first page needs no repair). + // The report's repair would start on a later pass, so the step's + // checkpoint judges it. if ( - signalsRefetch && - boundedContinuation && - unexpressiblePageEnds === unexpressibleBefore + reported && + keptInstalledRows && + firstAnswers && + provenFirstPage(firstAnswers) ) { + provenReportRequestsBefore = firstRequestsBefore + } + // The repair a report starts reads each window's bounded prefix, + // tie, and refill; a whole-source read would be the cost the report + // exists to avoid. One documented fallback reads the full source + // instead: a page that ends on a null or NaN rank. An unindexed plan + // whose bounded prefix leaves the window short grows the prefix + // (ARCHITECTURE.md, "an underfilled unindexed prefix grows"). + const shortRefill = + !testCase.indexed && prefixLeavesWindowShort(rows, shown) + if (signalsRefetch && unexpressiblePageEnds === unexpressibleBefore) { + if (!testCase.indexed) { + if (shortRefill) shortUnindexedRefills++ + else unindexedRefetchChecks++ + } + } + if (signalsRefetch && unexpressiblePageEnds === unexpressibleBefore) { expect( fullSourceRequests, `full-source requests after a signalling refetch at step ${step}`, @@ -5165,6 +5399,19 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ ) if (changedInFlight()) caseHeldChanges++ expectProviderLedger(`after step ${step}`) + if (provenReportRequestsBefore !== undefined) { + ignoredProvenReports++ + expect( + (requestsBy.get(firstSubscription) ?? 0) - provenReportRequestsBefore, + `requests the first query issued for a report its own page proves, at step ${step}`, + ).toBe(0) + provenReportRequestsBefore = undefined + provenSinceOrderingChange = true + } + expect( + repairAfterProvenReport, + `a repair the first query issued after a report its own page proved, by step ${step}`, + ).toBeUndefined() if (inFlight()) { expectInFlightWindow( visibleIds(), @@ -5293,6 +5540,9 @@ async function runEagerWindow(testCase: EagerCase): Promise<{ heldRequests: caseHeldRequests, cooperativeCancels, lateAnswers, + ignoredProvenReports, + unindexedRefetchChecks, + shortUnindexedRefills, } } finally { unsubscribe?.() @@ -5315,6 +5565,14 @@ type OptimisticWindowCheckpoint = const MAX_OPTIMISTIC_WINDOW_REQUESTS = 8 const OPTIMISTIC_ACQUISITION_OBSERVATION_TURNS = 8 +// The most rows a bounded refill may ask for. An indexed plan pages past +// the window. An unindexed plan grows its prefix to twice the rows it +// installed (ARCHITECTURE.md, "an underfilled unindexed prefix grows"), and +// a request also covers the one row that the pending transaction overlays. +function refillBound({ indexed, width, installedIds }: OptimisticWindowCase) { + return indexed ? width + 2 : 2 * installedIds.length + 1 +} + type OptimisticWindowCase = { shape: OptimisticWindowShape mutation: OptimisticWindowMutation @@ -5361,6 +5619,8 @@ type OptimisticWindowObservation = { submittedRows: Array limit: number | undefined direction: string | undefined + // The request had a limit or a predicate, so it read no full source. + bounded?: boolean appliedReceiptPending: boolean providerPending: boolean }> @@ -5508,6 +5768,7 @@ async function checkOptimisticWindow( submittedRows: Array limit: number | undefined direction: string | undefined + bounded: boolean appliedReceiptSettled: boolean providerSettled: boolean provider: Promise @@ -5557,6 +5818,8 @@ async function checkOptimisticWindow( submittedRows: [] as Array, limit: options.limit, direction: options.orderBy?.[0]?.compareOptions.direction, + bounded: + options.limit !== undefined || options.where !== undefined, appliedReceiptSettled: false, providerSettled: false, provider: Promise.resolve(), @@ -5866,6 +6129,7 @@ async function checkOptimisticWindow( submittedRows: [...attempt.submittedRows], limit: attempt.limit, direction: attempt.direction, + bounded: attempt.bounded, appliedReceiptPending: !attempt.appliedReceiptSettled, providerPending: !attempt.providerSettled, })) @@ -5901,6 +6165,7 @@ async function checkOptimisticWindow( submittedRows: [...attempt.submittedRows], limit: attempt.limit, direction: attempt.direction, + bounded: attempt.bounded, appliedReceiptPending: !attempt.appliedReceiptSettled, providerPending: !attempt.providerSettled, })) @@ -6069,7 +6334,7 @@ function optimisticWindowViolations( needsAcquisition && (refillAttempt?.direction !== (testCase.direction ?? `desc`) || refillAttempt.limit === undefined || - refillAttempt.limit > width + 2) + refillAttempt.limit > refillBound(testCase)) ) { violations.push(`refill used an unbounded provider request`) } else if ( @@ -6085,6 +6350,22 @@ function optimisticWindowViolations( if (testCase.acquisition === `forbidden` && observation.attempts.length > 0) { violations.push(`refill acquisition started with complete local extent`) } + // A refill the case allows but does not require is still bounded: an + // ordered read within the refill bound, or a predicate read, never the + // full source. + if (testCase.acquisition === `allowed`) { + for (const attempt of observation.attempts) { + if ( + attempt.bounded === false || + (attempt.direction !== undefined && + (attempt.limit === undefined || + attempt.limit > refillBound(testCase))) + ) { + violations.push(`an allowed refill used an unbounded provider request`) + break + } + } + } const requested = new Set() for (const request of observation.requestHistory ?? []) { if (request.phase === `pending` && testCase.indexed) { @@ -6278,6 +6559,202 @@ describe(`eager indexed ordered windows`, () => { }) }) + // An unindexed joined window whose only domain row is ineligible. After a + // signalling refetch the repair's bounded prefix holds just that row, so + // the window is short. The refill grows the prefix instead of reading the + // full source, which the refetch law forbids. + it(`grows the prefix of a short unindexed window after a signalling refetch`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `joined`, + via: `marker`, + indexed: false, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 1, + offset: 0, + initial: [], + history: [ + { type: `upsert`, id: 0, rank: 0, eligible: false }, + { type: `dup`, id: 0 }, + { type: `refetch` }, + ], + }) + }) + + // The trace from #2221. A descending, nulls-first, unindexed window behind + // an anti-joined marker has a NaN rank, so its boundary has no cursor. + // After a signalling refetch the loader read the full source twice: the + // short prefix took the full-source fallback. The prefix now grows, so the + // refetch law, which allows no full-source read here, holds. + it(`reads no full source after a signalling refetch past a NaN boundary`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `anti-joined`, + via: `marker`, + indexed: false, + direction: `desc`, + nulls: `first`, + terms: 1, + limit: 2, + offset: 0, + initial: [ + { id: 0, rank: 0, eligible: true }, + { id: 2, rank: Number.NaN, eligible: true }, + { id: 1, rank: 0, eligible: false }, + ], + history: [ + { type: `delete-visible` }, + { type: `move`, offset: 0, limit: 1 }, + ], + }) + }) + + // Trace 1 from a comment on #2221: an ascending unindexed window behind an + // anti-joined marker whose only row is ineligible. Before prefix growth the + // short prefix took the full-source fallback, twice, after a signalling + // refetch; the refetch law allows none. + it(`reads no full source after a signalling refetch of a short anti-joined window`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `anti-joined`, + via: `marker`, + indexed: false, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 1, + offset: 0, + initial: [], + history: [ + { type: `upsert`, id: 0, rank: 0, eligible: false }, + { type: `refetch` }, + ], + }) + }) + + // Found by the unindexed-refetch random campaign. A refetch report moves + // the page's end onto a null rank, which no cursor can express. Reading + // that end's tie would take the boundary's full-source fallback, so the + // loader repairs with its bounded prefix instead; a loader that read the + // tie broke the refetch law. + it(`repairs instead of reading an unexpressible end's tie after a signalling refetch`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `joined`, + via: `marker`, + indexed: false, + direction: `desc`, + nulls: `first`, + terms: 1, + limit: 2, + offset: 1, + initial: [ + { id: 0, rank: null as unknown as number, eligible: true }, + { id: 2, rank: Number.NaN, eligible: true }, + { id: 1, rank: 0, eligible: true }, + { id: 3, rank: 0, eligible: true }, + ], + history: [ + { + type: `upsert`, + id: 10, + rank: null as unknown as number, + eligible: false, + }, + { type: `dup`, id: 0 }, + ], + }) + }) + + // A peer's tie answer changes, so the provider reports, but the first + // query's page and tie answers stay the same and the peer installed the + // row after its page. That page proves the window, so the report must + // start no request for the first query and leave it owing no repair: the + // window move that follows continues from its boundary instead. + it(`starts no request for a window its own first page still proves after a report`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: true, + filter: `source`, + via: `marker`, + indexed: true, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 1, + offset: 0, + initial: [ + { id: 0, rank: 1, eligible: true }, + { id: 1, rank: 2, eligible: true }, + { id: 2, rank: 3, eligible: true }, + ], + history: [ + { type: `upsert`, id: 10, rank: 2, eligible: true }, + { type: `refetch` }, + { type: `move`, offset: 0, limit: 3 }, + ], + }) + }) + + // A signalling refetch moves the first page's end earlier, to a rank that + // a row the provider never sent ties with. The window is still inside the + // page, but the rank-2 row past it is local only because the old tie + // request read it, so it proves nothing about the rows that now tie with + // the end. The loader must read the new end's tie; ignoring the report + // made a later window move read the local rows and skip row 11. + it(`repairs a signalling refetch before a window move reads past its page`, async () => { + await checkEagerWindow({ + path: `on-demand`, + provider: `refetch-signal`, + latency: `immediate`, + cancellation: `cooperative`, + peer: false, + filter: `joined`, + via: `marker`, + indexed: true, + direction: `asc`, + nulls: `first`, + terms: 1, + limit: 2, + offset: 0, + initial: [ + { id: 0, rank: null as unknown as number, eligible: true }, + { id: 1, rank: 2, eligible: true }, + { id: 2, rank: 3, eligible: true }, + { id: 3, rank: Number.NaN, eligible: true }, + ], + history: [ + { type: `upsert`, id: 10, rank: 0, eligible: true }, + { type: `flip`, id: 2 }, + { type: `upsert`, id: 11, rank: 0, eligible: true }, + { type: `refetch` }, + { type: `move`, offset: 0, limit: 3 }, + { type: `delete`, id: 0 }, + ], + }) + }) + // A page with no later page whose end moves earlier. Two rank-0 rows // arrive while the provider lags; the refetched page ends at rank 0, but // the tie request still holds the rank-1 row, so the loader sees no @@ -6420,7 +6897,17 @@ describe(`eager indexed ordered windows`, () => { ? [4, 3] : [4, 3, 2] : [1, 2, 3, 4], - acquisition: extent === `missing` ? `required` : `forbidden`, + // An unindexed plan proves rows only up to its bounded + // prefix, so behind a joined filter it may grow the prefix + // even when the replacement row is installed (ARCHITECTURE.md, + // "an underfilled unindexed prefix grows"). A source filter + // leaves no rows out of the prefix, so it needs no request. + acquisition: + extent === `missing` + ? `required` + : shape === `source` + ? `forbidden` + : `allowed`, } }), ), @@ -6463,6 +6950,27 @@ describe(`eager indexed ordered windows`, () => { })), ), ), + // An unindexed one-row window behind a joined filter installs only the + // prefix it grew to: the rejected leading row 4 and the window row 3. + // The departure of row 3 leaves the replacement row 2 missing, so the + // refill must grow the prefix before persistence settles. + ...([`joined`, `anti-joined`] as const).flatMap((shape) => + ([`update`, `delete`] as const).flatMap((mutation) => + ([`acknowledge`, `rollback`] as const).flatMap((outcome) => + ([`collection`, `effect`] as const).map((consumer) => ({ + shape, + mutation, + consumer, + outcome, + width: 1 as const, + indexed: false, + extent: `missing` as const, + installedIds: [4, 3], + acquisition: `required` as const, + })), + ), + ), + ), { shape: `source`, mutation: `update`, @@ -7120,7 +7628,7 @@ describe(`eager indexed ordered windows`, () => { if ( hasRootless && window.some( - (id) => id >= 0 && (model.get(id)!.rank as number | null) === null, + (id) => id >= 0 && (model.get(id)!.rank as number | null) == null, ) ) { mixedNullGroup++ @@ -7148,8 +7656,92 @@ describe(`eager indexed ordered windows`, () => { // the reach check always reads the cases that test just executed. Reach is // keyed by case, so rerunning an identical case overwrites its entry; // shrinking runs only after a failure, which ends the test first. + // Reconstruction: the fixed campaign must reach the proven-report law, so + // a loader that repairs every report fails it. + it(`starts no repair for reports a window's own first page proves for a fixed seed`, async () => { + const PROVEN_SEED = 2091 + let proven = 0 + await fc.assert( + fc.asyncProperty(eagerProvenReportCase, async (testCase) => { + proven += (await runEagerWindow(testCase)).ignoredProvenReports + }), + { numRuns: runs, seed: PROVEN_SEED }, + ) + expect(proven, `reports a first page proved`).toBeGreaterThan(0) + }) + + it(`starts no repair for reports a window's own first page proves for a random or replayed seed`, async () => { + const options = oracleRandomParameters( + runs, + replay, + `ordered-work.eager-proven-report`, + ) + const reach = createOracleReach(`ordered-work.eager-proven-report`, [ + `ignoredProvenReports`, + ] as const) + await fc.assert( + fc.asyncProperty(eagerProvenReportCase, async (testCase) => { + reach.beginHistory() + const observed = await runEagerWindow(testCase) + if (observed.ignoredProvenReports > 0) + reach.reach(`ignoredProvenReports`) + }), + options, + ) + reach.assertReached() + }) + + // The refetch law over unindexed joined windows behind a signalling + // provider. A repair must not read the full source, including the refill + // of an unindexed plan whose bounded prefix leaves the window short, which + // grows the prefix. The campaign must reach windows the prefix fills and + // windows it leaves short. + // Ineligible rows crowd the bounded prefix and leave the window short, so + // half the cases start with every row eligible, where the law applies. + const eagerUnindexedRefetchCase = fc + .tuple(eagerJoinedCase, fc.boolean()) + .map(([testCase, allEligible]) => ({ + ...testCase, + path: `on-demand` as const, + provider: `refetch-signal` as const, + indexed: false, + peer: false, + initial: allEligible + ? testCase.initial.map((row) => ({ ...row, eligible: true })) + : testCase.initial, + })) + for (const campaign of [`fixed`, `random`] as const) { + it(`grows a short unindexed prefix instead of reading the full source after a signalling refetch (${campaign} campaign)`, async () => { + const property = `ordered-work.eager-unindexed-refetch` + const reach = createOracleReach( + property, + [`unindexedRefetchChecks`, `shortUnindexedRefills`] as const, + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(eagerUnindexedRefetchCase, async (testCase) => { + reach.beginHistory() + const observed = await runEagerWindow(testCase) + for (const state of [ + `unindexedRefetchChecks`, + `shortUnindexedRefills`, + ] as const) { + if (observed[state] > 0) reach.reach(state) + } + }), + campaign === `fixed` + ? // This seed kills a loader that repairs every report with a + // full-source read; about half of seeds do. + { numRuns: runs, seed: 2094 } + : oracleRandomParameters(runs, replay, property), + ) + reach.assertReached() + }) + } + it(`keeps a window whose loads answer between live changes for a fixed seed`, async () => { - const HELD_SEED = 2081 + // Chosen for reach after nullish ranks joined the row generator. + const HELD_SEED = 2083 const heldReach = new Map< string, Awaited> @@ -7173,7 +7765,7 @@ describe(`eager indexed ordered windows`, () => { ).toEqual([]) const reach = [...executed].map((key) => heldReach.get(key)!) // A case whose deletes refill from rows the query already holds issues - // no held request; at one times the run count, 35 of 40 cases issue one. + // no held request; at one times the run count, 33 of 40 cases issue one. expect( reach.filter((count) => count.heldRequests > 0).length, `held cases that issue a held request`, @@ -7185,7 +7777,7 @@ describe(`eager indexed ordered windows`, () => { `held cases with a held request but no in-flight change`, ).toEqual([]) // `release-peer` aborts held requests: at one times the run count, a - // cooperative provider cancels 4 and a late provider answers 1 anyway. + // cooperative provider cancels 6 and a late provider answers 4 anyway. expect( reach.reduce((sum, count) => sum + count.cooperativeCancels, 0), `held requests a cooperative provider cancelled`, diff --git a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts index fe2646d59f..1796539141 100644 --- a/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts +++ b/packages/query-db-collection/tests/refetch-ordered-prefix-oracle.test.ts @@ -10,6 +10,7 @@ import { import { QueryClient } from '@tanstack/query-core' import fc from 'fast-check' import { describe, expect, it, onTestFailed, vi } from 'vitest' +import { createOracleReach } from '../../db/tests/oracle-reach.js' import { oraclePropertyOptions, oracleReplayRequested, @@ -454,6 +455,9 @@ type Run = { failAfter: (count: number | undefined) => void // The Query cache entries this Collection owns: one per tracked request. cachedQueries: () => number + // Queries an observer still holds. A refetch re-runs only these; a released + // Query stays cached until its gcTime. + activeQueries: () => number // Reports that served pages may have shifted (`loadedSubsets:invalidated` // events on the source Collection) so far. reports: () => number @@ -737,6 +741,9 @@ async function startRun(testCase: Case): Promise { }, cachedQueries: () => queryClient.getQueryCache().findAll({ queryKey: [id] }).length, + activeQueries: () => + queryClient.getQueryCache().findAll({ queryKey: [id], type: `active` }) + .length, reports: () => reports, changedReanswers: () => changedReanswers, failNextApplication: () => { @@ -798,6 +805,10 @@ type Observation = { // Such a refetch that also reported, so a loader may have judged the skip // while the refetched rows were held back. reachedReportDuringHold: boolean + // A refetch that reported while the window was full: the loader then + // judges whether its own first page still proves the window, and repairs + // when a moved end leaves it unproven. + reachedReportOverFullWindow: boolean // Reports over the whole run. reports: number } @@ -826,6 +837,7 @@ async function checkCase(testCase: Case): Promise { let reachedLaggingRefetchAfterContinuation = false let reachedLaggingRefetchDuringHold = false let reachedReportDuringHold = false + let reachedReportOverFullWindow = false let primary: unknown const check = (checkpoint: string) => { const expected = expectedWindow(run.backend, run.markers, testCase.scenario) @@ -888,6 +900,13 @@ async function checkCase(testCase: Case): Promise { ) { reachedReportDuringHold = true } + if ( + command.type === `refetch` && + run.reports() > reportsBefore && + run.windowIds().length === testCase.scenario.limit + ) { + reachedReportOverFullWindow = true + } checkReports( command, run.reports() - reportsBefore, @@ -984,6 +1003,7 @@ async function checkCase(testCase: Case): Promise { reachedLaggingRefetchAfterContinuation, reachedLaggingRefetchDuringHold, reachedReportDuringHold, + reachedReportOverFullWindow, reports: run.reports(), } } @@ -1283,10 +1303,14 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => }) // The ordinary insert into a top-N list. The page's end moves earlier, so - // Query DB reports, but every window row is in the window's own first page: - // the refetch must send only the Queries it refetches, with no repair. + // Query DB reports, but every window row is in the window's own first page. + // The tie the loader read belongs to the old end, so its rows prove + // nothing about rows that tie with the new end. The loader proves the + // window again with bounded reads (a repair, or the new end's tie; the + // shape is the loader's choice), reads nothing unbounded, and retires the + // requests it replaces, so the Query count stays the same. it.each([1, 2] as const)( - `a refetch of a window inside its first page starts no repair (%i order terms)`, + `a refetch that moves a first page's end proves the window again with bounded reads (%i order terms)`, async (terms) => { const scenario: Scenario = { encoding: `cursor`, @@ -1307,14 +1331,22 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => await run.apply({ type: `upsert`, id: 5, rank: 5 }) await run.apply({ type: `flip`, id: 5 }) const queries = run.cachedQueries() + const activeBefore = run.activeQueries() const requestsBefore = run.requests.length await run.apply({ type: `refetch` }) expect(run.windowIds()).toEqual( expectedWindow(run.backend, run.markers, scenario), ) - expect(run.requests.length - requestsBefore).toBeLessThanOrEqual( - queries, - ) + const extra = run.requests.slice(requestsBefore + queries) + expect(extra.length).toBeGreaterThan(0) + // Every read is bounded: an ordered prefix with a limit, or a tie + // that reads equal leading values. None reads the full source. + for (const request of extra) { + expect( + request.limit !== undefined || request.where !== undefined, + ).toBe(true) + } + expect(run.activeQueries()).toBe(activeBefore) } catch (error) { primary = error } @@ -1888,8 +1920,16 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => it.each(campaigns)( `generated refetch histories (%s campaign)`, async (campaign) => { - let reached = 0 - let reachedDuringHold = 0 + // Declared reach: generated histories serve continuation pages before + // a refetch of a lagging provider (the premise of the gap), do so while + // a hold runs, and report a refetch while the window is full, where the + // loader decides between ignoring the report and repairing. + const states = [ + `reachedLaggingRefetchAfterContinuation`, + `reachedLaggingRefetchDuringHold`, + `reachedReportOverFullWindow`, + ] as const + const reach = createOracleReach(property, states, { minimum: 3 }) const options = campaign === `fixed` ? { seed: 20261013, numRuns: oracleRuns(80) } @@ -1904,18 +1944,15 @@ describe(`query-db on-demand refetch keeps served ordered prefixes whole`, () => }) await fc.assert( fc.asyncProperty(caseArbitrary, async (testCase) => { + reach.beginHistory() const observation = await withDeadline(checkCase(testCase)) - if (observation.reachedLaggingRefetchAfterContinuation) reached++ - if (observation.reachedLaggingRefetchDuringHold) reachedDuringHold++ + for (const state of states) { + if (observation[state]) reach.reach(state) + } }), { ...options, seed }, ) - // Positive reach witness: generated histories serve continuation pages - // before a refetch, which is the premise of the gap. - if (campaign !== `replay`) { - expect(reached).toBeGreaterThan(0) - expect(reachedDuringHold).toBeGreaterThan(0) - } + reach.assertReached() }, 120_000, )