diff --git a/.changeset/on-demand-network-first.md b/.changeset/on-demand-network-first.md new file mode 100644 index 0000000000..443926353a --- /dev/null +++ b/.changeset/on-demand-network-first.md @@ -0,0 +1,11 @@ +--- +'@tanstack/db-sqlite-persistence-core': minor +--- + +`initialRender: { strategy: 'network-first' }` now works for on-demand persisted Collections, including Electric's `progressive` mode. Previously the configuration threw `InvalidPersistedCollectionConfigError`. + +A subset load that settles online records persisted demand coverage for its exact demand. In a later session, a load of the same demand settles on its rows from SQLite once `networkTimeoutMs` (default 3 seconds) has passed, or at once if the network load fails. The network load keeps running, and its rows publish when they arrive. If it fails after that, the query stays ready. A demand that never settled online still waits for the network. Coverage proves that the rows were complete when they were loaded, not that they are still current. A source truncate, a snapshot transaction, scoped recovery, or a cache-generation rotation clears it. A load that a reset made obsolete, a load that was released before it settled, and a load during the rest of a scoped-recovery sync run record none. Coverage also ends when SQLite holds fewer rows for the demand than when it settled, for example after a source delete or Query DB retention expiry. A `refetch: true` load always waits for the network. An ordered window with a `limit` stays covered only while SQLite holds the same rows for it. + +A query that joins eager and on-demand sources becomes ready when every eager source is ready, which network-first allows from its restored rows after the deadline, and every demand the query reaches on an on-demand source has settled. A network failure alone does not settle an on-demand demand that is not covered. + +This applies only with the default single-process coordinator. With a multi-tab coordinator, such as `BrowserCollectionCoordinator`, on-demand loads still wait for the network. diff --git a/.github/workflows/oracle-nightly.yml b/.github/workflows/oracle-nightly.yml new file mode 100644 index 0000000000..b93bf13492 --- /dev/null +++ b/.github/workflows/oracle-nightly.yml @@ -0,0 +1,35 @@ +name: Nightly oracles + +# Longer oracle campaigns than PR CI can afford. The generated demand +# coverage campaign runs with ten times its PR budget, and every hostile +# mutant in scripts/demand-coverage-mutants.mjs must fail a named test. +on: + schedule: + - cron: '17 6 * * *' + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + demand-coverage: + name: Demand coverage campaigns and mutants + runs-on: ubuntu-latest + timeout-minutes: 240 + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + - name: Setup Tools + uses: tanstack/config/.github/setup@e4b48f16568324f76f467aa4c2aac2f05db632c3 + - name: Run the generated campaign with a larger budget + env: + TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: '10' + run: pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest run --coverage.enabled=false tests/persisted-demand-coverage-campaign-oracle.test.ts + - name: Run the demand coverage mutants + run: node scripts/demand-coverage-mutants.mjs diff --git a/docs/collections/electric-collection.md b/docs/collections/electric-collection.md index e075073f38..912c806058 100644 --- a/docs/collections/electric-collection.md +++ b/docs/collections/electric-collection.md @@ -406,6 +406,14 @@ isolated but needs a separate cross-version cleanup policy while older tabs may still use it. Scoped recovery does not certify the new generation for offline reuse. +With `initialRender: { strategy: 'network-first' }`, an on-demand or +progressive Collection can render a demand offline from SQLite after that exact +demand settled online in an earlier session. Each demand keeps its own +coverage in the current persisted cache generation. Scoped recovery, cache-claim +expiry, and a must-refetch end it, so after any of them a demand waits for the +network again. A demand that loads during the rest of a scoped-recovery sync +run records no coverage. See [On-demand Collections offline](../guides/sqlite-persistence.md#on-demand-collections-offline). + During scoped recovery, coordinator notifications about SQLite changes do not hydrate, truncate, or replace public source rows. The Electric change stream and demanded source snapshots own those rows until the sync run ends. A change diff --git a/docs/contributing/glossary.md b/docs/contributing/glossary.md index 824ae70abd..6de5659ee2 100644 --- a/docs/contributing/glossary.md +++ b/docs/contributing/glossary.md @@ -61,7 +61,9 @@ production queues, caches, or semantic helpers merely to share their names. | persisted readiness | The persisted wrapper's internal record that the current sync run's persisted restore completed or failed. Network-first initial render reads it; Collection status is the only public signal. | Collection readiness, network availability, or row nonemptiness. | | persisted cache generation | One version of an on-demand Collection's durable rows and resume metadata, identified by a monotonic generation number and a distinct storage ID. One current generation accepts new claims; a retired generation remains available to sync runs that already claim it. A stale run may create a new retired generation for its own recovery without replacing the current one. Rotating a generation does not change the public Collection identity. | Sync run, provider session, or source snapshot. | | persisted cache claim | A sync run's expiring authority to read and write one persisted cache generation. It is checked before durable access and is separate from a TanStack DB acquisition lease. | Acquisition lease, subset demand, or proof that cached rows are authoritative. | -| network-first initial render | An opt-in persisted Collection setting, `initialRender: { strategy: 'network-first', networkTimeoutMs }`. The Collection waits for its sync adapter until the deadline passes or the remote fails, then a completed persisted restore that installed rows makes it `ready`. An empty restore keeps it `loading`. A remote failure before the restore completes waits for the restore. Network sync continues. | Collection readiness from the sync adapter, an SSR seed, or a separate status. | +| persisted subset restore | The persisted wrapper's read of one demand's request data from the current persisted cache generation, and publication of those rows to the Collection, in the current sync run. The result may be empty. | Persisted restore of an eager Collection, or proof that the rows are complete. | +| persisted demand coverage | A record, in one persisted cache generation, that a load of one exact demand reached applied settlement with its rows durably committed, and how many rows the demand read then. It counts only while the persisted subset restore returns at least that many rows; for an ordered window, only while it returns the same rows. It is the durable, cross-session form of the coverage that a settled acquisition establishes in one sync run: when a covered demand settles from SQLite, it is covered in that sense too. It proves completeness when that load settled, not freshness or broader extent. | Rows present in SQLite, a broader or overlapping demand, or persisted readiness. | +| network-first initial render | An opt-in persisted Collection setting, `initialRender: { strategy: 'network-first', networkTimeoutMs }`. The Collection waits for its sync adapter until the deadline passes or the remote fails, then a completed persisted restore that installed rows makes it `ready`. An empty restore keeps it `loading`. A remote failure before the restore completes waits for the restore. Network sync continues. An on-demand Collection instead settles each covered demand's load from SQLite once that load's wait passes or its remote load fails (persisted demand coverage); query readiness follows its loads. | Collection readiness from the sync adapter, an SSR seed, or a separate status. | `source` names a role in a data flow; it does not own a lifecycle. Do not coin a `source session`. Use `sync run` for the local Collection sync invocation diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index afab073a85..278e912c5f 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3778,6 +3778,204 @@ an answer, and a Collection that synced to zero rows before (it waits like an empty restore) are outside it. Hooks read Collection status only, so the framework conformance suite's status scenarios cover the binding side. +The persisted demand coverage oracle in +`packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` +owns the on-demand law: an opted-in wrapper may settle a demand's +load from SQLite only when that exact demand has persisted demand coverage, +after the configured wait or an upstream failure. Its bounded enumeration +crosses twelve session-1 histories (not requested, failed, overlapping +filter, same filter with another window, same filter with another order, +rejected durable row write, settled empty, settled, settled then truncated, +three in which a reset lands after the load's rows but before it settles (a +source truncate, a completed scoped recovery that rotates the cache +generation, and a scoped recovery still pending), and one in which the source +deletes a row of D after D settles), two waits, and six +session-2 upstream schedules (pending, early and late failure, early and late +answer, late empty answer), over the real wrapper and +`SQLiteCorePersistenceAdapter` on `node:sqlite`. Session 1 creates coverage +only through real loads, and a positive witness checks whether it sent the +session-2 demand key. A pure trace model predicts status, `preload()` state, +and exact rows at five checkpoints. A fixed lifecycle history checks that each +load measures its own wait and settles on its own restored rows, that +release aborts the upstream load, and that coverage survives release. A joined history checks that a +query over an eager and an on-demand source is ready only when the on-demand +demand is covered, read directly and through an outer live query, with the +joined row visible when ready. A third shape reaches the on-demand source +through two live queries; covering only the inner query's demand must keep +the query loading. +Two fixed same-session histories check that a load of D after a cache +rotation, and a `refetch: true` load of D, both wait for the network. Model-level calibration shows that eighteen wrong designs (fifteen +single-demand, two lifecycle, one joined) each differ from the law at a named +case. Before the implementation, every +production case failed at setup because `initialRender` rejected +`syncMode: 'on-demand'`; without the opt-in, that code matched every +uncovered cell and diverged at the first checkpoint where the law applies for +every covered cell except an early network answer. With the implementation, +every case passes; the [implementation review](oracle-reviews/2026-10-21-persisted-demand-coverage-review.md) records RED, GREEN, and each mutant's outcome. Production mutants that ignore coverage, ignore or halve +the wait, error a covered demand on upstream failure, write coverage before +the load settles, measure the wait from the sync run, keep coverage across a +truncate, let an on-demand opt-in permit fallback alone, or read the root +query's demands instead of the owning inner query's each fail named cases. +A review then found three more defects, and the in-flight histories and the +two-reader shape reproduced each one on unchanged production: an obsolete load +recorded coverage after a truncate or rotation (22 cells), a coverage record +during pending recovery failed the session-1 load (12 cells), and the first +live query reached on a source stood for every reader (one cell). After the +repair, mutants that restore each defect, or that check only for a truncate, +fail those cases. A third review round added the row-deletion history, the +source-failure history, and the rotation and refetch histories; on the parent +production code they failed 14 cases by assertion, and the Query DB witness +failed its retention history. Mutants that drop the row count, require more +rows than recorded, ignore the cache generation, let a refetch settle on +coverage, or let a network failure bypass on-demand settlement each fail +those cases; a gate that ignores the +eager deadline fails the persisted-readiness oracle instead. The Electric +progressive witness in +`packages/electric-db-collection/tests/electric-progressive-offline-readiness-witness.test.ts` +receives the law as a production driver for the shared model: it records +Electric's own log at its `loadSubset` boundary, and session 2 reads either +the subset session 1 loaded or one no query read. A subset-only session 1 +settles the loaded subset offline and leaves the other waiting; after a +completed progressive full sync, the never-loaded subset is ready too, +because Electric settles every later load at once from its complete rows +(model rule `full sync completes`). A mutant that drops the complete resume's +full-snapshot resolution fails only that never-loaded history. The Query DB witness +in `packages/query-db-collection/tests/persisted-on-demand-offline-witness.test.ts` +receives it with a fresh QueryClient per session: a covered on-demand query +stays loading until the wait and then shows its restored rows, a rejected +`queryFn` settles it at once, and a never-loaded query keeps loading; a mutant +that ignores coverage fails both covered histories. A fourth history lets +Query DB retention delete the released rows after a 20 ms `gcTime`; session 2 +must then keep loading. React and Solid Suspense once received a joined +initial-render gate through capability fixtures. The source-health change +(#2168) removed that gate: Suspense now releases at query readiness, and the +React and Solid receiving witnesses below run the shared contract. A later round closed most open limits with enforced cases. The driver checks +that the wrapper forwards a load only after its persisted subset restore is +visible, so no upstream event can precede it. Session 2 reads answer on a +later macrotask, so a load that fails at once reaches the coverage read, and +removing the wrapper's early rejection handler raises unhandled rejections. +Part 2b adds a schema reset over an empty covered demand, an optimistic write +that rolls back on a demand settled from SQLite, and two cache-claim expiry +histories: one before the coverage record and one before the coverage check. +Part 4 covers ordered windows: a row deleted, moved out, or moved in after the +window settled keeps it loading, a swap keeps it covered, and with an index a +page move settles offline only when session 1 loaded that page with a cursor. +The Electric witness adds on-demand must-refetch and checks that Electric +writes nothing from a snapshot that answers after release. React +(`useLiveSuspenseQuery-persisted-on-demand.test.tsx`) and Solid +(`useLiveQuery-persisted-on-demand.test.tsx`) Suspense render a covered demand +from a real persisted on-demand Collection on `node:sqlite`. Vue +(`useLiveQuery-persisted-on-demand.test.ts`), Svelte +(`useLiveQuery-persisted-on-demand.svelte.test.ts`), and Angular +(`injectLiveQuery-persisted-on-demand.test.ts`) witnesses observe `isReady`, +`isLoading`, and rows. All five run the shared receiving contract described below. Mutants that drop +the early rejection handler, keep collection metadata across a schema reset, +key coverage without cursor and offset, ignore coverage, or let a coverage +read under an expired claim fail the load each fail these cases. +A fifth round added Part 2c and five more histories. Part 2c checks that a +failed coverage read, after the load or during the check, keeps the load's +own outcome; that a synchronous answer's coverage counts its rows, so a +later deletion ends it; and that a release during the coverage check stops +the source Collection loading. Part 3 then added a source failure before a +covered demand's wait ends; after the source-health change (#2168) replaced +the initial-render gate with Collection status, those histories belong to +`packages/db/tests/source-health-oracle.test.ts`. Part 4 adds a row replaced below the window boundary and a row gained +by an under-full window, which keep the tie-boundary key. Each failed on the +parent production code. A window check that accepts a superset of the +recorded keys, and a builder that drops its error-state check without +notifying settlement, each fail named cases. +A sixth round (the 2026-10-28 review) closed three more holes, each RED on +the parent production code. The campaign adds a `snapshot` command (a +snapshot transaction ends coverage, as a truncate does) and a source that +stops and resolves an aborted load (a load that core aborted records +nothing); Part 2b adds a load after scoped recovery reloaded the demand (no +load settles or records during the rest of that sync run). The shared model +now judges demand identity from the request data itself instead of core's +demand key. The storage sweep also checks that a failed coverage write +crashes the Collection. + +Remaining limits: coverage relies on the source committing a load's rows +before it settles that load (law 7); an unordered demand that loses one row +and gains another keeps its row count and stays covered, so a row that +retention removed can be missing from a covered restore (maintainer +decision, 2026-10-26; a fixed history pins the rule); and multi-tab +coordinators, outside the law by maintainer decision (2026-10-26). Part 4 +now covers a tie at the window boundary, modelled as a limited demand plus +an unordered tie-boundary demand, and page moves with and without an index. + +The generated campaign in +`packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-campaign-oracle.test.ts` +shares the fixed oracle's driver (`tests/helpers/demand-coverage-driver.ts`). +Its model replays the source's own log of row writes, deletes, truncates, +and settled loads, so it does not predict when the wrapper reacquires a +demand. A fixed and a random campaign of 40 histories each interleave two +unordered demands with load starts, answers, failures, releases, live +writes and deletes, truncates, snapshot transactions, and a source that +rejects, ignores, or stops and resolves an aborted load; declared reach +covers an obsolete load that settles, an aborted load that settles, a demand +settled only by an aborted load, coverage ended by a snapshot, coverage +recorded twice, a row moved between demands, and a covered demand short of +rows. Part 2 fails each +adapter call of a covered history once, before or after its effect, in +either session: safety holds for every call, a failed coverage read keeps +the load ready, and a failed coverage check waits. Part 3 releases the +demand after each step of a load: the source Collection stops loading, and +every load left behind was aborted. Mutants that ignore the row count, let +an obsolete load record coverage, keep coverage across a truncate, let a +failed coverage read fail the load, or keep a released load's wait each fail +it. The campaign also reads the window `group = 'a' orderBy rank limit 2` +over generated ranks with ties: a query is ready offline exactly when every +load it sent in session 2 is covered, which judges the tie-boundary load +without predicting when core sends it. Mutants that ignore window keys, +compare them in order, or record keys for every demand each fail it. Its +limits: real timers in session 1, one process, `node:sqlite`, and no cursor +or offset loads. +A scripted mutation sweep (63 mutants, recorded in the review record) then +added five fixed histories: an empty covered demand after a rotation, a +truncate during an empty load's coverage read, a synchronous answer whose +durable write lands late, claim recovery after an expired claim during the +check, and a coverage read that outlasts the wait. A sixth history holds the +load's cache-claim renewal past the wait and kills the last survivor, a wait +measured from after that renewal. No survivor of the sweep remains open in +this owner. + +Every receiving witness is a production driver for the same judgment +(ORC-014). The shared contract, +`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`, +holds six histories as data (session-1 steps and session-2 premises only): +covered, never loaded, covered then a source delete, a covered window, a +window with a row replaced below the boundary, and a covered demand whose +upstream fails at once. Each host records the source log, the session-2 loads, +and two checkpoint observations, and the shared model in +`tests/persisted-demand-coverage-model-oracle.ts` judges them; no witness +states an expected result. The histories module imports nothing, so the +Playwright spec can load it. + +The hosts run all six through their own entry point: Node (better-sqlite3 +through `createNodeSQLitePersistence`), Chromium OPFS (wa-sqlite in two page +loads, in `playwright.opfs.config.ts`, which the browser E2E job runs), +Electron (a renderer over IPC to a `node:sqlite` main process, two app +launches, in the desktop E2E job), Cloudflare Durable Objects (`wrangler dev` +restarted over one persist directory, in the cloud-mobile job), React Native +(an op-sqlite-shaped driver, in the cloud-mobile job), Expo (an expo-sqlite +shim), and Tauri and Capacitor (their unit-suite SQLite stand-ins). The React, +Solid, Vue, Svelte, and Angular witnesses run the covered, never-loaded, and +failure histories through their own hook or primitive. With the contract's +session-2 source forced to a 60 s wait and an upstream that never answers, +every host and framework reports violations on its covered histories. The +OPFS witness also pins the guide's statement that a multi-tab coordinator +waits for the network; that run is recorded, not judged, because it pins the +guide rather than the law. + +Limits: no CI job runs the Expo, Capacitor, or Tauri e2e suites, and no host +runs on its native runtime here. The Capacitor device e2e needs an iOS +simulator or Android emulator, and `adb` is not installed here; its Node e2e +fails at file load (`SqliteError: cannot start a transaction within a +transaction` at `tests/helpers/capacitor-sqlite-test-db.ts:22`). The Tauri +native e2e needs `cargo`. React Suspense reveals a covered failure about 300 ms +late (its fallback throttle), so the contract's before-wait checkpoint sits at +450 ms. + ## Sync metadata session [Issue #2220](https://github.com/TanStack/db/issues/2220) reports that a diff --git a/docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md b/docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md new file mode 100644 index 0000000000..0218c7d260 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md @@ -0,0 +1,370 @@ +# Draft law: persisted demand coverage + +Status: implemented in PR #2263, which also contains the oracle, the matching +`ARCHITECTURE.md` and glossary edits, and the review record +(`2026-10-21-persisted-demand-coverage-review.md`). The law revised an +accepted contract; it was not a defect report under the old one. It belongs to +issue #2102, item 1. Later sections record maintainer decisions made during +review, with their dates. + +## Question + +A user turned on SQLite persistence for an app. `initialRender: { strategy: +'network-first' }` worked for eager Collections but not for progressive +Electric ones. They expected progressive Collections to render from SQLite +without waiting for the network. + +Electric reports `progressive` to core as `on-demand` +(`packages/electric-db-collection/src/electric.ts:1216`). The persisted wrapper +rejects network-first for `on-demand` +(`packages/db-sqlite-persistence-core/src/persisted.ts:6833`). Every on-demand +source has this limit, not only Electric. + +The product need is offline use. Offline, an on-demand source can serve a query +only from an exact query that was persisted before, unless everything was +synced. Serving everything is one reason to use progressive mode. + +## Evidence gathered + +Two witness files record current behavior on main at 31dad2dd3. They state no +contract. + +`packages/db-sqlite-persistence-core/tests/on-demand-offline-readiness-witness.test.ts` +uses a generic on-demand source: + +| Upstream | Local subset | Query rows | Query ready | +| --- | --- | --- | --- | +| never settles | complete | `a1, a2` | no | +| never settles | partial | `a1` | no | +| never settles | empty | none | no | +| settles at once, no write | complete | `a1, a2` | yes | +| settles at once, no write | empty | none | yes | + +`packages/electric-db-collection/tests/electric-progressive-offline-readiness-witness.test.ts` +runs two Electric `progressive` sessions over one real `node:sqlite` database. +Session 2 is offline: + +| Session 1 | Session 2 resumes stream | Query rows | Query ready | +| --- | --- | --- | --- | +| completed full sync | yes (`10_0`, `shape-1`) | `1, 2` | yes | +| loaded the query subset only | no | `1, 2` | no | +| nothing | no | none | no | + +What the evidence establishes, for these histories: + +1. Restored rows reach the query before readiness in every on-demand case. Only + the readiness signal waits. +2. After a completed progressive full sync, Electric resumes and settles loads + at once. The query is ready on SQLite rows with no network. +3. If the full sync never completed, the persisted subset rows are present, but + the query waits on the network with no deadline. +4. The wrapper cannot tell a complete subset from a partial or never-loaded + one. All three read back as rows, or as no rows. + +The witnesses do not cover React or Solid Suspense timing, native SQLite hosts, +cache-claim rotation, or a must-refetch during the offline session. + +## The law + +**Persisted demand coverage.** An opted-in persisted wrapper over an on-demand +source may settle a demand's load from SQLite. It may do this when all of these +hold: + +1. The persisted subset restore for the demand has published its rows in the + current sync run. The rows may be empty. +2. The exact demand has persisted demand coverage in the current persisted + cache generation. +3. The configured wait has passed since the load started, or the demand's + upstream load has failed. + +All three must hold, in any order of arrival. An upstream failure that +arrives before the persisted subset restore publishes settles the demand +once the restore publishes. Until then, the load waits for the upstream load +as it does today. A demand without coverage never settles from SQLite. + +After the load settles from SQLite, its upstream load continues as work that +the wrapper owns. Rows from that load publish as ordinary source writes. A +later upstream failure does not undo the settlement. Releasing the demand, or +cleanup, aborts the upstream load. Coverage survives release: a later load of +the same demand starts its own wait and may settle from SQLite again. +Settling from SQLite does not change Collection readiness. + +An upstream load that settles successfully records coverage for its exact +demand. Coverage is written durably with the load's rows or after them, and +records how many rows the demand reads from SQLite at that moment. Three more +rules came from review: + +- A load that a reset made obsolete records no coverage (law 6 in + `ARCHITECTURE.md`). +- A load with `refetch: true` never settles from SQLite. It starts a new + acquisition that waits for its own applied rows. +- Coverage counts only while the persisted subset restore returns at least as + many rows as coverage recorded. A row deleted afterwards, by the source or by + Query DB retention, ends coverage until the demand loads online again + (maintainer decision, 2026-10-26). An unordered demand counts only rows: one + that loses a row and gains another keeps its count and stays covered, even + when retention removed the lost row (maintainer decision, 2026-10-26, + chosen over recording every row key, whose storage grows with the demand). +- Coverage is bookkeeping beside a load. A coverage read that fails proves + nothing: the load records no coverage, or waits for its upstream load, and + otherwise keeps its own outcome. A load released during its coverage check + owns no wait. + +The first draft also let a certified complete resume baseline cover every +demand. No path implemented that, and a source that resumes from a complete +baseline already settles its own loads at once, so the clause was dropped +(maintainer decision, 2026-10-26). + +### Terms + +These require glossary entries; the draft entry is in +`docs/contributing/glossary.md`. + +- **persisted subset restore**: the wrapper's read of one demand's request data + from the current persisted cache generation, and publication of those rows to + the Collection, in the current sync run. The result may be empty. +- **persisted demand coverage**: a record, in one persisted cache generation, + that a load of one exact demand reached applied settlement with its rows + durably committed. + +### Why coverage matches by exact demand + +Coverage matches by the canonical demand key (`getLoadSubsetDemandKey`). The +law never infers that one demand covers a different +one. #1797 made the same choice for request deduplication. It removed subset +algebra after it treated "requests as established coverage." Law 7 in +`ARCHITECTURE.md` already says settlement proves no broader extent than the +exact request. + +Without coverage, an empty or partial local result is not evidence. The +witnesses show the wrapper cannot tell "no matching rows" from "never loaded." +A fallback on such a demand would render a false empty or partial result. + +Coverage records completeness at the time of the covered load. It says +nothing about freshness, just as an eager restore can be stale. + +### What follows without new rules + +The live-query architecture routes every readiness consumer through demand +settlement. Settling a covered load therefore reaches all of them: + +- initial-query readiness and `preload()`; +- React and Solid Suspense, which follow initial-query readiness; +- `queryOnce`; +- `skipInitial` Effects; +- atomic ordered-window publication, so a covered next page publishes offline; +- demand added after the first render, such as includes children and new + filters. + +`ARCHITECTURE.md` already states that a source resuming from complete local +rows settles its loads at once. This law applies the same idea to one demand +instead of the whole Collection. The live query adds one query-level signal, +described next. + +### Queries over several sources + +The sources combine. A query may render from local data when every source +permits it: + +- an eager source, after its persisted restore completes in the current sync + run; +- an on-demand source, after every demand the query reaches on it has settled, + from SQLite or from the network. + +A source failure does not change these conditions. It never settles a demand, +and it does not keep a settled demand from permitting a render: a covered +demand that settles from SQLite after a source reports an error still permits +the fallback render, as an eager source's restore does after a source failure. + +This is initial-query readiness with one substitution: an eager source's +persisted restore stands in for its Collection readiness. A query over only +on-demand sources needs no substitution, because its readiness already follows +its loads. The eager network-first law in `persisted-readiness-oracle.test.ts` +is unchanged for a query over only eager sources. + +Today the observer's network-first gate runs only when every source registers a +`PersistedReadinessSource` (`collectPersistedReadinessSources` in +`packages/db/src/live-query-observer.ts`). That interface reports one state per +Collection, but demand settlement differs by query: two queries over the same +on-demand source can be in different states. An opted-in on-demand source +therefore registers only that it opted in, with no settlement state. The gate +reads settlement from the live query itself: the query's own initial +readiness, with each eager source's persisted restore standing in for that +source's Collection readiness. An opt-in alone never permits fallback; that +would release a join before the on-demand side's covered rows are visible. + +The live query does not expose that signal today. `preloadForInitialRender` +combines the network `preload()` with per-Collection persisted snapshots; it +cannot ask whether the query's own demands have settled independently of +eager Collection readiness. The implementation must add that query-level +signal: initial readiness with each eager source's persisted restore in place +of its Collection readiness. + +Update, 2026-10-27: the source-health change (#2168) removed the observer's +network-first gate and `PersistedReadinessSource`. A network-first eager +Collection now becomes `ready` from its restored rows after the deadline or a +remote failure, so the substitution above happens in Collection status +itself. Query readiness is the signal, and this law adds nothing to core: an +on-demand source permits initial readiness, and the query is ready when its +demands on it have settled. How a source error reaches the query's status is +the source-health law (`packages/db/tests/source-health-oracle.test.ts`); the +two failure paragraphs above are superseded by it. + +### The wait + +The wait reuses the network-first option and its default. #1955 set that +default to 3 seconds (`DEFAULT_NETWORK_TIMEOUT_MS`), and 0 is valid. The wait +is a preference for fresher data while online, not a correctness condition: +removing it breaks no property above. At 0, a covered demand behaves like a +demand on a completed progressive resume today. + +Each load measures its wait from its own start. Only that load's own upstream +failure ends the wait early. + +### Invalidation + +Coverage belongs to one persisted cache generation. It ends with that +generation and with any reset that replaces the durable rows: truncate, +must-refetch, schema reset, or cache-claim rotation. A record that memory still +holds after a rotation in the same sync run does not count. A crash between +writing the rows and writing the coverage must not leave coverage without rows. +Coverage also ends when the restore returns fewer rows than it recorded. + +Update, 2026-10-28 (high review of the head after #2168): three consequences +of existing terms are now enforced. + +- A snapshot transaction replaces every synced row, as a truncate at its + start does (glossary, "snapshot transaction"), so it ends coverage too. +- A load that core aborted decides nothing, even if its source resolves it + (glossary, "abort"), so it records no coverage. +- Scoped recovery ends coverage and does not certify the new generation for + offline reuse (`docs/collections/electric-collection.md`). Its persisted + subset restore publishes no rows, so coverage cannot count during the + rest of that sync run: no load settles from SQLite and none records. + +### Limits + +- **Coverage lifetime after release.** After a demand is released, its source + stops sending changes for it. Coverage stays valid until its generation ends + or a later load of the same demand settles. An expiry option, like Query DB's + `persistedGcTime`, is tracked as a separate issue and is not implemented. +- **Keys that depend on other rows.** Cursor pages and includes children build + demand keys from other rows' values. If restored rows differ from the + previous session, the key differs too. That demand is then uncovered and + waits for the network. +- **Uncovered demand while offline.** It waits indefinitely, also after a + source reports an error: the network-first wait neither renders nor + rejects. The in-progress error-reporting work owns how that condition is + reported. +- **A failure after a settlement from SQLite.** The query stays ready and the + failure does not reach it, so the application cannot tell that a refresh + failed. The in-progress error-reporting work owns whether such a failure is + reported elsewhere; the guides must state the behavior. +- **Optimistic writes while offline.** An optimistic mutation that touches a + covered demand's rows is outside this law. Optimistic overlay rules apply as + usual; the oracle checks one write and its rollback on a demand settled + from SQLite. +- **Ordered windows.** A window settles from SQLite only while its restore + holds the rows it held when it settled (maintainer decision, 2026-10-26). + Coverage of a limited load records its row keys, and the restore must + return the same keys in any order. The tie-boundary load's key comes from + the restored boundary row, so most changed windows also ask for a key that + no load covered; the recorded keys catch a row replaced or gained without a + boundary change. +- **Failed coverage reads.** A coverage read that fails, including one under + an expired cache claim, proves nothing. The wrapper records no coverage, and + a load that would settle from SQLite waits for the network. Neither path + fails a load whose rows applied. An expired claim also starts claim + recovery. A failed coverage write is a durable failure, which fails the + runtime as any failed source transaction does. +- **Source settlement.** The coverage read waits behind every source + transaction already committed, so it counts their rows durably, also for a + source that answers synchronously. It relies on the source committing a + load's rows before it settles that load (law 7 in `ARCHITECTURE.md`). +- **Multi-tab coordinators.** By maintainer decision (2026-10-26), the law + covers only the default `SingleProcessCoordinator`. Under a Browser or + Electron multi-tab coordinator, demand routes through the coordinator, so a + covered demand neither settles from SQLite nor records coverage; it waits + for the network as before. +- **Electric's own load.** Electric's upstream load may stay pending after the + wrapper settles from SQLite. Its release and abort ownership need a receiving + test. + +## Oracle design + +**Owner.** `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` +owns this law. It sits beside the eager network-first owner, +`persisted-readiness-oracle.test.ts`, under the same coverage-map section. +The live-query consequences listed above stay with their existing owners. A +receiving witness is needed for the window-publication case. + +**Independent model.** For each demand, the model records the local restore +(`pending`, `done`, or `error`) and the coverage (`covered`, `uncovered`, or +`baseline`). It also records the upstream state (`pending`, `settled`, or +`failed`) and whether the wait has passed. From these it computes whether the +load may settle and which rows are visible at settlement. It shares no code +with the wrapper. + +**History grammar.** Session 1 produces coverage through real loads, not seeded +metadata. Seeded metadata would make the model's coverage and the production +coverage the same assumption. Session 2 steps then arrive in any legal order: + +- the local restore settles or fails; +- the upstream load settles or fails; +- the wait passes; +- a reset or rotation happens; +- the demand is released; +- a second demand starts. + +**Production driver.** The real persisted wrapper over +`SQLiteCorePersistenceAdapter` and `node:sqlite`, with a generic on-demand +source. Receiving witnesses cover Electric `progressive` (full-sync resume and +subset-only) and Query DB. + +**Observations.** After each step: load settlement, query rows, `isReady()`, +whether `preload()` has settled, whether the network-first wait has released, +and, for an ordered query, the published window. A joined history pairs an +eager source with an on-demand source. + +The glossary's persisted readiness and network-first initial render entries +follow this combining rule (edited in this worktree). + +**Wrong designs the oracle must reject.** Each needs a demonstrated failure: + +1. Removing the on-demand guard and using startup readiness. This settles + before the subset restore, with no rows. +2. Treating any completed local restore as evidence. This fails "offline, + empty, uncovered" and "offline, partial, uncovered." +3. Treating nonempty rows as coverage. This fails "covered and genuinely + empty." +4. Writing coverage before the rows commit durably. This fails a crash between + the two writes. +5. Keeping coverage across reset or rotation. +6. Matching a different demand key, such as `limit 10` coverage for `limit 20`. +7. Aborting the upstream load when the wrapper settles from SQLite. Fresh rows + then never arrive. +8. Leaving a covered next page unpublished after the wait. +9. An on-demand opt-in that permits fallback on its own. The gate then releases + a join before the on-demand side's covered rows are visible. +10. Coverage keyed by filter only, or ignoring order while honoring limit. +11. A wait measured from the sync run instead of the load, or coverage cleared + on release. +12. A late upstream failure that undoes a settlement from SQLite. + +## Documents that change with the law + +- Persisted-readiness oracle: prose, model, grammar, driver, and checks. +- `packages/db/src/query/live/ARCHITECTURE.md`: source settlement and the + initial-readiness limits (edited in this worktree). +- `docs/contributing/glossary.md`: new persisted subset restore and persisted + demand coverage entries; revised persisted readiness and network-first + initial render entries (edited in this worktree). +- `docs/guides/sqlite-persistence.md` ("Render after SQLite restore") and + `docs/guides/live-queries.md`. They must say that a covered result, including + an empty one, is complete as of the load that covered it, not fresh, and that + the wait changes when a query is ready, not how fresh its rows are. Rows from + the network can still arrive after a query is ready. +- `docs/contributing/oracle-coverage.md`, "Persisted readiness and network-first + initial rendering." +- `docs/contributing/oracle-reviews/issue-2056-cache-eviction-design.md`, per + issue #2102's completion criteria. diff --git a/docs/contributing/oracle-reviews/2026-10-21-offline-on-demand-explainer.html b/docs/contributing/oracle-reviews/2026-10-21-offline-on-demand-explainer.html new file mode 100644 index 0000000000..fafe962260 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-21-offline-on-demand-explainer.html @@ -0,0 +1,658 @@ + + + + + +Offline On-Demand Readiness + + + + + + + +
+ +
+
TanStack DB · persistence design notes
+

Offline On-Demand Readiness

+

Why a persisted on-demand query can't render from SQLite while offline today, what we measured, and the one-rule model we're exploring to fix it.

+
+ Worktree: .worktrees/progressive-network-first + Base: main @ 31dad2dd3 + Status: proposal, nothing committed +
+
+ +
+

Where you left off

+
+
The report

A Discord user turned on persistence. initialRender: network-first worked for eager collections but not for progressive Electric ones. They expected progressive to render from SQLite without waiting for the network.

+
Why it fails

Electric reports progressive to core as on-demand, and the persisted wrapper rejects network-first for any on-demand source. Not Electric-specific.

+
What we measured

Cached rows already show while offline. Only the ready signal waits. A completed progressive sync already works offline; anything short of that waits forever.

+
Where we landed

One rule: the wrapper may settle a load from SQLite when that exact query finished loading before. Everything downstream follows from existing laws.

+
+
+ +
+
+

How readiness works today

+

A live query is ready when every source gives it evidence that its rows are present. Since #2115, that evidence depends on the source's sync mode.

+
+
+
+ EAGER SOURCE +

Collection readiness is the evidence

+

It loads everything without per-query requests, so the query waits for the Collection to be ready. With persistence, a finished SQLite restore of the whole Collection is a separate signal. Network-first (#1955) lets Suspense render that restore after a deadline.

+
+
+ ON-DEMAND SOURCE +

A settled load is the evidence

+

Each query asks for its subset. The persisted wrapper reads that subset from SQLite, then waits for the upstream load. The query is ready when both settle. There is no per-query "restored" signal, so network-first is rejected.

+
+
+

Progressive Electric is on-demand to core. Its Collection stays loading until the full sync, but its queries go ready as their own loads settle.

+
+ +
+
+

Try it

+

Pick a source, what happened last session, and what the app asks for now. Then compare today's behavior, a naive fix, and the proposed rule. Times are illustrative: SQLite answers at 120 ms, a live network at 900 ms.

+
+ +
+
+
+ Source +
+ + + +
+
+
+ Last session +
+ + + +
+
+
+ The app now asks for +
+ + + +
+
+
+ Network +
+ + +
+
+
+ Model +
+ + + +
+
+
+ +
+ + 0 ms +
+ How long to wait for the network before trusting SQLite. +
+
+ +
+
+
SQLite
+
Network
+
Rows on screen
+
Query ready
+
+
+
+ +
+
Rows visible–
+
isReady / Suspense–
+
Is the answer correct?–
+
+

+
+
+ +
+
+

What we measured

+

Two witness tests record current behavior on main. They pin observations, not a contract. Each checkpoint is 50 ms after the query starts.

+
+

Generic on-demand source

+
+ + + + + + + + +
UpstreamLocal subsetRows visibleQuery ready
Never answersCompletea1, a2no
Never answersPartiala1no
Never answersEmptynoneno
Settles at once, writes nothingCompletea1, a2yes
Settles at once, writes nothingEmptynoneyes
+

Progressive Electric, two sessions on real SQLite, second session offline

+
+ + + + + + +
Session 1Session 2 resumes streamRows visibleQuery ready
Completed full syncyes1, 2yes
Loaded the query subset onlyno1, 2no, forever
Nothingnononeno
+

The wrapper can't tell a complete subset from a partial or never-loaded one; they all just read back as rows. That's why "show whatever SQLite has" is wrong.

+
+ +
+

The model we're exploring

+
+ The one rule +
The persisted wrapper may settle a query's load from SQLite once the local rows are visible, if that exact query is covered in the current cache: a previous load of it finished, or the source holds a complete baseline. The network load keeps running and replaces the rows when it answers.
+
+
Exact query keyMatches by getLoadSubsetDemandKey. No "this query is broader than that one."exists today
+
Covered setPer cache generation: keys whose load finished, including empty results, plus "complete baseline" which matches everything.new, in the wrapper
+
Settle ruleSettle when local rows are visible and the query is covered and the wait has passed or its load failed.new
+
Network continuesUpstream keeps loading; its rows publish normally when they arrive.exists today
+
+

Why it's small: initial readiness, Suspense, queryOnce, skipInitial effects, and ordered-window publication (pagination) all already wait on "this query's load settled." Settle a covered load and every one of them works offline with no live-query changes. ARCHITECTURE.md already says a source that resumes from complete local rows "settles its loads at once." The rule moves that idea from the whole Collection down to one query.

+

The network-first wait turns into an optional knob. No correctness property needs it. At 0 it matches how a completed progressive resume behaves today.

+
+
+ +
+
+

What the rule settles

+

The first draft had nine open decisions. Most went away once the design reduced to the single rule.

+
+
+ + + + + + + + + + + + +
Earlier open decisionUnder the one rule
Where coverage livesThe wrapper, because it has to work the same for every source
loading vs unavailable statusGoes away; on-demand needs no new persisted status
Progressive subsets before the full syncCovered if their load settled (inferred, needs a witness)
Deadline meaning per queryOne wait, measured from each load's start; that load's failure ends it
"Not available offline" statusExposed to the in-progress precise-errors PR
Page keys built from other rowsA known limit: if restored page 1 differs, page 2 is a miss and waits for the network
Publishing then replacing rowsAlready normal for ready queries
API shapeOne wrapper option: on/off plus wait
Oracle ownerThe wrapper's settle rule; existing live-query oracles own the consequences
+

Decisions since the first pass

+
+
1

Coverage lifetime. Valid until revalidated or until the cache generation ends. A persistedGcTime-style expiry gets its own issue and waits until someone asks for it.

+
2

The wait. Reuses the network-first option and its 3 s default from #1955; 0 is valid.

+
3

Mixed joins. On-demand sources register no persisted readiness, so a query that joins an eager source keeps the ordinary readiness path and waits for the eager network offline. That's safe, so it's recorded as a limit.

+
4

Law and ARCHITECTURE.md. Both are updated in the worktree, along with two glossary terms.

+
+

Not done yet: no formal model check, no independent range case, and Electric's own pending load after the wrapper settles is unexamined.

+
+ +
+
+

How we got here

+

Earlier work that shaped this problem. Filled dots are the ones that matter most.

+
+
+
2025-11-05 · #618Electric maps progressive to on-demandCore only ever sees eager or on-demand.
+
2026-03-20 · #1380 (samwillis)Persisted sync metadataQuery DB stores per-row query owners and persistedGcTime retention, "for offline-first flows."
+
2026-09-10 · #1797Subset algebra removedDeleted isPredicateSubset and friends. Dedupe matches exact keys "without inferring broader coverage."
+
2026-09-11 · #1785 / 2026-09-24 · #1846Complete resume baselinesA progressive resume restores and certifies the full baseline.
+
2026-09-30 · #1955Network-first initial renderEager only by design: "An on-demand Collection has no complete startup snapshot."
+
2026-10-11 · #2069Managed cache generations"A partial Electric cache does not certify an offline baseline."
+
2026-10-10 · issue #2102 (open)Design safe offline subset reuseItem 1 asks for durable evidence of what a subset covers. This proposal belongs there.
+
2026-10-13 · #2115On-demand queries ready on their own loadThe per-query readiness rule this design builds on.
+
+
+ +
+

Files in the worktree

+
+ + + + + + + +
FileWhat it is
docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.mdThe law draft, rewritten around the one rule
packages/db-sqlite-persistence-core/tests/on-demand-offline-readiness-witness.test.tsGeneric on-demand witness, 5 cases
packages/electric-db-collection/tests/electric-progressive-offline-readiness-witness.test.tsElectric two-session witness on node:sqlite, 4 cases
docs/contributing/oracle-reviews/2026-10-21-offline-on-demand-explainer.htmlThis page
+
+ +
+ + + + diff --git a/docs/contributing/oracle-reviews/2026-10-21-persisted-demand-coverage-review.md b/docs/contributing/oracle-reviews/2026-10-21-persisted-demand-coverage-review.md new file mode 100644 index 0000000000..897dcbd415 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-21-persisted-demand-coverage-review.md @@ -0,0 +1,676 @@ +# Persisted demand coverage: implementation review + +Reviewed head: `cd2aceaec858e0d090fa330d67a9d511c51ba5da`. +Implementation commits: `4fecdc75835ea7b706772b88e9c02caaa60f5caf` (production +change), `42e15a33fc0a7cff5e569bb00f60d3758035b5a2` (truncate history). +RED head: `85372868c05ed933a2234fcf7c5365aa281e48d9` (oracle before the +production change). Merge base with `main`: +`9aecbab4cacb3f1123b1716e6f063abb5affc74f`. + +The law, the oracle design, and the wrong designs are in +[the law record](2026-10-20-on-demand-network-first-law.md). This record +contains the evidence obtained against the implementation. + +## Law and result + +An opted-in persisted wrapper over an on-demand source may settle a demand's +load from SQLite when the persisted subset restore of that demand is visible, +the exact demand key has persisted demand coverage in the current cache +generation, and the wait since the load started has passed or the upstream +load has failed. The upstream load keeps running. Coverage survives release +and ends at a source truncate. For network-first initial rendering, an eager +source permits local rendering after its restore and deadline; an on-demand +source permits it when the query's demands on it have settled. + +Authority: the maintainer decisions recorded in the law record (exact keys, the +same wait option and default as eager, joined sources combine, a late failure +leaves the query ready, coverage written through the source-transaction +pipeline). `packages/db/src/query/live/ARCHITECTURE.md` (the persisted-wrapper +paragraph, the initial-readiness limit, and law 7) states the same contract. + +Production path: `persistedCollectionOptions` over a generic on-demand source +and `SQLiteCorePersistenceAdapter` on `node:sqlite`. The wrapper's +`PersistedCollectionRuntime.loadSubset` and `loadHydratedSubset` settle a +covered demand. The wrapped `loadSubset` records coverage after the source +load in a metadata-only source transaction. The wrapped `truncate` deletes +coverage keys in its own transaction. In core, `CollectionConfigBuilder` +exposes `initialDemandsSettled()` and notifies subscribers from +`updateLiveQueryStatus`, and `LiveQueryObserver.preloadForInitialRender` +combines the sources. + +## RED and GREEN + +RED, at `85372868c` with the 96-cell grammar: every production case failed at +setup with `InvalidPersistedCollectionConfigError`. A probe without the opt-in +(recipe in the law record) passed all 72 uncovered cells. It failed 22 of 24 +covered cells at the first checkpoint where the law applies; the two +early-network-answer cells pass under both designs. The lifecycle case and the +covered joined case failed, and the uncovered joined case passed. + +GREEN, at the reviewed head: the coverage oracle passes all 139 tests (108 +cells, the lifecycle history, two joined histories, and calibration). Also at +this head, `packages/db` passes 13,422 tests, `db-sqlite-persistence-core` +passes 1,327, and `electric-db-collection` passes 719. The type errors at +`persisted-oracle.test.ts:4730` and `:4871` also occur with this change's +sources reverted, so this change does not cause them. + +Two driver repairs reached production paths for the first time. Neither +changed an expected result: + +- Session 1 attached its rejection handler to `preload()` only after the + answer loop, so the `failed` history produced unhandled rejections. +- The joined comparison included virtual row properties (`$key`, `$synced`, + ...), which the law does not govern. + +## Production mutants (ORC-006) + +Each mutant edited one production line. The script lives outside the +repository. Every kill is an assertion failure at a refinement comparison. No +mutant failed at setup, at a `checkpoint not reached` witness, or by timeout. + +| Mutant | Outcome | +| --- | --- | +| Coverage ignored (always wait for upstream) | Killed: 24 cells, first `settled-empty, wait 0, pending` at `restored` | +| Wait ignored (settle at restore) | Killed: 13 cells, `wait 1000` covered cells at `restored` | +| Half deadline | Killed: 9 cells at `beforeDeadline` | +| Failure errors a covered demand | Killed: `settled` and `settled-empty`, `wait 1000, fails-early` | +| Coverage recorded before the upstream load settles | Killed: 33 cells, first the `failed` history | +| Deadline from the sync run, not the load | Killed: lifecycle history at checkpoint 1000 | +| Truncate keeps coverage | Killed after `42e15a33f`: 11 `truncated` cells; it survived at `4fecdc758`, before that history existed | +| On-demand opt-in permits fallback alone | Killed: joined `never requested` | +| Gate ignores the eager deadline | Survives this oracle (Part 3 uses a 0 ms eager wait); killed by `persisted-readiness-oracle.test.ts` ("does not hide a query failure behind an available persisted source") | + +The "coverage recorded before the upstream load settles" mutant is coarser +than the wrong design it stands for: it records coverage for a load that later +fails, not only for one whose rows fail to persist. The model-level +`durable-write-failed` calibration and the rejecting adapter keep the narrower +design distinguished. No production mutant that writes coverage through a +separate durable path, outside the source-transaction pipeline, was run. + +## Requirement outcomes + +- **ORC-001:** The opening prose names the law, its sources, and its limits. + Truncate moved from the limits into the grammar. +- **ORC-002:** The model reads only the history label. It never reads SQLite, + coverage keys, or the wrapper. The positive witness compares session-1 and + session-2 keys through `getLoadSubsetDemandKey`. That is a trusted base for + the witness only; it does not compute any expected result. +- **ORC-003:** The contract, grammar, model, driver, and refinement check are + separate, and each has prose beside it. The `truncated` history has its own + prose and ablation note. +- **ORC-004:** Not triggered; the grammar is a bounded enumeration, not + generated. The file still records ablation for each history, wait, and + schedule value, and the excluded pre-restore event. +- **ORC-005:** A positive witness checks session-1 demand keys, and a + checkpoint witness checks the restored rows. For `truncated`, another witness + checks that the truncate replayed D and that the load stays pending. No + missing-information hypothesis was identified; this is not a claim of an + exhaustive search. +- **ORC-006:** Sixteen model-level wrong designs (thirteen in Part 1, two + lifecycle, one joined), each distinguished at a named case, and the + production mutants above. +- **ORC-007:** Not triggered; there is no generated property. +- **ORC-008:** The model keeps four facts: covered, visible rows, settled, and + failed. Each one changes a later observation, as the model prose explains. The + `truncated` history adds no state; it maps to "not covered" with no rows. +- **ORC-009:** The glossary terms are persisted subset restore, persisted demand + coverage, and load settlement, and the model prose maps them. +- **ORC-010:** `withCleanup` keeps a primary failure and cleanup failures apart + in one `AggregateError`, and a calibration test checks this. +- **ORC-011:** No shared-fault hypothesis was named. Demand-key identity is the + candidate: the oracle trusts `getLoadSubsetDemandKey` to tell the session-1 + demands apart. A second formulation that derives keys independently was not + added. +- **ORC-013:** The wait threshold has checkpoints at `wait - 1` and `wait`, and + the lifecycle history has checkpoints ±1 ms around each load's own deadline. + The half-deadline and sync-run-deadline designs are rejected there. +- **ORC-014:** The controlled premise is a source whose load settles after its + applied receipt. Two receiving witnesses supply it with a real source. In + the Electric progressive witness, subset-only coverage settles offline and a + never-loaded subset waits. In the Query DB witness + (`persisted-on-demand-offline-witness.test.ts`, added after the reviewed + head), a covered query settles at the wait or at once on a rejected + `queryFn`, and a never-loaded query waits. The coverage-ignored mutant fails + both covered Query DB histories. + +## Open in-scope cells + +The [coverage map](../oracle-coverage.md#persisted-readiness-and-network-first-initial-rendering) +owns these. Each needs a witness: + +- ordered windows and cursor pages in session 2, including a key that ignores + offset or cursor; +- must-refetch, schema reset, and cache-claim rotation in the coverage oracle; + Electric must-refetch goes through `truncate`, but no Electric history + exercises it; +- a failure that arrives before the persisted subset restore publishes; +- optimistic writes on a covered demand; +- nested live queries beyond one outer query over the joined query. After + the reviewed head, Part 3 adds that nested shape and reads rows at release; + a gate that reads the root query's demands fails only the nested cells. +- Suspense over a real persisted on-demand Collection. After the reviewed + head, React and Solid tests receive the joined gate with capability fixtures, + and the gate mutant that lets the on-demand opt-in permit fallback alone + fails both. + +This record claims a new law, not the closure of a bug class. + +## 2026-10-26: code-review findings after merging `main` + +Merge head: `5c0e04dce` (merges `main` at `5649a07d4`; the network-first wait +now lives in `waitForPersistedInitialRender`, and the on-demand gate moved +into it). Oracle extension: `e4add5b18`. Repair: the next commit. + +A code review reported ten findings. Four were defects under the accepted +contract, one needed a product decision, and the rest were refuted or a cost +only. The oracle was extended first and run against unchanged production: + +- three session-1 histories put a reset after a load's rows and before it + settles: a source truncate, a completed scoped recovery that rotates the + cache generation, and a scoped recovery still pending; +- Part 3 adds a shape that reaches the on-demand source through two live + queries, with a history that covers only the inner query's demand. + +RED on `5c0e04dce`: 35 failed and 137 passed of 172, all by assertion at a +named checkpoint. In 22 cells an obsolete load recorded coverage. In 12 cells +a coverage record during pending recovery failed the session-1 load. The +two-reader inner-only history released the wait. + +Repair: a load records coverage only if no lifecycle change, reset (rotation +or scoped recovery), or source truncate happened since it was forwarded +upstream. The observer gives an on-demand source one gate entry for each live +query that reads it. Because scoped recovery marks a reset before it can abort +the coverage commit, the epoch check also closes the failed-record case; a +`.catch` on the commit had no reachable history and was removed. + +GREEN: 172 of 172. Also `db` 13,462, `db-sqlite-persistence-core` 1,535, +`react-db` 601, and the Electric and Query DB witnesses pass. Solid did not +run: its test environment needs a package that the registry proxy blocks. + +Mutants, all killed by assertion with no timeout or unreached checkpoint: + +| Mutant | Outcome | +| --- | --- | +| Restore the original: any settled load records coverage | 34 cells fail across all three in-flight histories | +| Narrow fix: check only for a source truncate | 23 cells fail in the two scoped-recovery histories | +| Restore the original gate: the first live query reached on a source stands for all | The two-reader inner-only history fails | + +By maintainer decision (2026-10-26), multi-tab coordinators are outside the +law: under a coordinator other than `SingleProcessCoordinator`, demand routes +through the coordinator, so a covered demand neither settles from SQLite nor +records coverage. A probe showed a covered demand staying loading offline even +in the leader tab. The coverage map, law record, architecture paragraph, and +guide state the limit. + +Refuted: scoped recovery and reload truncates keep coverage (they refresh +memory only; rotation starts a fresh storage target and schema reset deletes +collection metadata); a raw on-demand root releases early (granular mode waits +for its load; wholesale mode requests no demand online or offline). Cost only, +no law: a source's literal `true` becomes a promise, and each load writes one +metadata record. Accepted: coverage keys grow until a truncate or expiry +(#2211). + +## 2026-10-26: CodeRabbit and xhigh reviews of PR #2263 + +Reviewed head: `3de5a922f` (CI green). CodeRabbit review `5655576117` gave five +findings (CR1-CR5) and an xhigh code review gave fifteen (X1-X15). The oracle +was extended first and run against the parent's production code. Maintainer +decisions: Query DB retention expiry ends coverage, through a row-count check +that also covers deletes in a limited window; the certified-baseline clause +is dropped; multi-tab coordinators stay out of scope. + +Oracle and witness changes: + +- Part 1 adds `settled-then-row-deleted`, and its wrong design "coverage + ignores rows deleted after it settled". +- Part 2b adds two same-session histories: a load of D after a cache rotation, + and a `refetch: true` load of D. Both must wait for the network. +- Part 3 adds a source failure on a never-covered demand, which must not + release the wait. +- The Query DB witness adds a retention-expiry history. +- The Electric witness driver now serializes work, as the driver contract + requires. + +RED on the parent production code (`3de5a922f`): the oracle failed 14 of 188 +(11 row-deletion cells, and the refetch, rotation, and source-failure +histories), and the Query DB witness failed its retention history. All +failures were assertions at a named checkpoint. + +Repair: a coverage record stores the cache generation's storage ID and the +number of rows the demand reads from SQLite. It counts only for the current +generation and while the restore returns at least that many rows. A load +rewrites the record only when that count changes. A `refetch` load never +settles from coverage. An aborted load clears its wait timer. The gate checks +on-demand settlement on the network-failure path too, and the observer keeps +one entry for each source and owning live query. + +GREEN: the oracle passes 188 of 188 and the Query DB witness 4 of 4. Also: +`db` 13,462, persistence 1,415, `react-db` 601, `solid-db` 99, +`electric-db-collection` 719, `query-db-collection` 929. Size against `main`: +`app` +78 B gzip, `db-sqlite-persistence-core` +475 B gzip. + +Mutants, all killed by assertion: + +| Mutant | Outcome | +| --- | --- | +| Drop the row count | 11 row-deletion cells and the Query DB retention history fail | +| Require more rows than recorded | 28 covered cases fail | +| Ignore the cache generation | The rotation history fails | +| Let a refetch settle on coverage | The refetch history fails | +| Let a network failure bypass on-demand settlement | The source-failure history fails | + +Counters, not oracle cases: five loads of one covered demand now cause one +coverage commit, not five. Releasing a covered load leaves no settlement timer +behind, while the unfixed code keeps one. + +| ID | Verdict | Action | +| --- | --- | --- | +| CR1, X11 | Confirmed: the glossary entry claimed demand settlement | Fixed: the entry now names the source status only, and the network-first entry carries the demand rule | +| CR2 | Confirmed | Fixed: the law record states its implementation and the query-level signal | +| CR3 | Confirmed | Fixed in the guide | +| CR4, X4 | Confirmed (RED) | Fixed | +| CR5 | Confirmed by reading the driver | Fixed | +| X1 | Confirmed (RED) | Fixed by the row count | +| X2 | Confirmed (RED) | Fixed by the generation check | +| X3 | Confirmed (RED) | Fixed | +| X5 | Refuted: a truncate clears hydrated demand, so the later load must hydrate, and the wrapper rejects a source transaction that crosses a hydration cycle | Recorded as an excluded, illegal history in the oracle | +| X6 | Confirmed by counter | Fixed for writes; each load still reads its row count, and a synchronous `true` result settles asynchronously | +| X7 | Not reproduced; read from the code | Open in the coverage map | +| X8 | Confirmed by reading; deletes are now reached through `settled-then-row-deleted` | Fixed by the row count for deletes; an order change without a delete stays open | +| X9 | Confirmed by counter | Fixed | +| X10 | Confirmed by reading | Fixed: one entry for each source and owner | +| X12 | Accepted as the design direction | The generation and the row count replace per-path invalidation, except the truncate deletion that empty demands still need | +| X13 | Confirmed | Fixed: the observer reuses the exported type | +| X14 | Confirmed | Partly fixed: the eager deadline is read once per source; the demand key is still computed more than once per load (cost only) | +| X15 | Confirmed | The clause is dropped from the law record and the glossary | + +## 2026-10-26: second CodeRabbit review + +Reviewed head: `f62f9dc19`. CodeRabbit review `5656501808` gave two findings. + +| ID | Verdict | Evidence | Action | +| --- | --- | --- | --- | +| CR6 | Confirmed | A real-timer probe with a driver that answers reads on a later macrotask: a load that rejects as soon as it is forwarded raised one `unhandledRejection` during the coverage read, and none after the fix | Fixed: `settleCoveredDemand` observes the upstream rejection before the read | +| CR7 | Confirmed (RED) | The new Part 2b history "keeps a synchronous answer for an already hydrated demand synchronous" failed by assertion: the second request reported a promise | Fixed: a literal `true` returns unchanged, and its coverage record follows without holding the load | + +The oracle adds the session-2 schedule `fails-at-once`. Its driver answers +SQLite reads within the same microtask turn, so it reaches the failure path but +cannot hold the rejection across the coverage read. Under the fake clock, a +driver that delays reads with a timer stopped session 2 before its first +checkpoint, so CR6 keeps the probe as its witness. GREEN: the oracle passes +215 of 215, persistence 1,442, Electric 719, and Query DB 929. + +The CI failure on `f62f9dc19` in `ordered-work-oracle.property.test.ts` is +issue #2221. Its seed `1535602160` fails the same way on `main` at +`5649a07d4`, in the core ordered loader. The fixed-seed case of that run timed +out under CI load and passes locally in about one second. + +## 2026-10-26: closing open limits and the expired-claim finding + +Starting head: `5ba90763d` (CI green). The maintainer asked to close the open +limits and then evaluate CodeRabbit review `5657028264` (CR8: a coverage read +under an expired cache claim fails a load whose rows applied). + +Test-only commit `0949bc478` turned open limits into enforced cases, with no +production change. Each passed on the existing code except where noted: + +- the restore-before-forward invariant; +- macrotask reads in session 2, which put CR6 inside the oracle (removing the + early rejection handler raises six unhandled rejections); +- a schema reset over an empty covered demand (killed by a mutant that keeps + collection metadata); +- an optimistic write that rolls back on a demand settled from SQLite; +- ordered windows (Part 4): row deleted, moved out, moved in, swapped, and + page 2 with an index (killed by a coverage key that ignores cursor and + offset); +- Electric on-demand must-refetch, and Electric's own load after release; +- React and Solid Suspense over a real persisted on-demand Collection (each + killed by a mutant that ignores coverage). + +An on-demand Electric mock that ended a subset with `snapshot-end` instead of +`subset-end` let Electric settle its load before its rows committed. Coverage +then recorded zero rows, and the offline session went ready on `[]`. The SDK +ends injected subsets with `subset-end`, so the mock now does. The law record +states the dependency on source settlement. + +CR8, confirmed on two paths. RED on the parent production code: the session-1 +query went to `error` when the claim expired before the coverage record, and +D went to `error` when it expired before the coverage check. With coverage +recording disabled, the first history stays `ready`, so the coverage read +caused it. Repair: a coverage read that fails under an expired claim returns +no count and schedules claim recovery; other failures propagate. GREEN: the +oracle passes 227 of 227. + +X7 stays open: a probe that queued an empty load's coverage record during +another demand's hydration, truncated, and started scoped recovery left the +Collection `ready`, but it did not prove that the record was queued when the +truncate listed metadata. + + +## 2026-10-26: high code review after the shared test driver + +Starting head: `5d85ce247` (CI green). A high code review reported nine +findings, H1 to H9. The oracle was extended for every behavioral finding before +any production change, and each extension was run on the starting head. + +| Finding | Law and authority | Oracle change | RED on `5d85ce247` | Repair | +| --- | --- | --- | --- | --- | +| H1: a source error leaves the network-first wait pending forever when the query reaches an on-demand source | The glossary's network-first gate permits a render once every eager source passed its deadline or saw the network fail and every on-demand demand settled. A source failure does not settle a demand and does not block a settled one. | Part 3: P or C reports an error before C's 1000 ms wait ends, observed before the wait and after a late checkpoint. The observation adds rejection and the observer's synchronous gate. | Both covered histories: not released, not rejected; the never-requested histories pass | `initialDemandsSettled` no longer reads the error state, which still blocks `markReady`. `settleDemand` notifies settlement listeners, because a query in error runs no graph and so no status check. | +| H2: a failed coverage read fails the load | Prior behavior: a load settles as its upstream load did. The law record makes coverage bookkeeping. | Part 2c: the read after the load fails; the read during the coverage check fails. | Session-1 query `error`; D `error` before the network answers | Any failed coverage read returns no rows. An expired claim also starts claim recovery. | +| H3: a synchronous answer's coverage counts rows before they are durable | Law: coverage records the rows SQLite holds for the demand when the load settles, so a later deletion ends it. | Part 2c: a synchronous answer, with and without a later deletion of one row. | With the deletion, D settled offline | The coverage read waits on the apply mutex behind every committed source transaction. | +| H4 (= X7): the truncate's metadata list makes a pending coverage transaction fail-stop on abort | — | None: not reachable | — | A coverage transaction has no abort signal. Its only abort is `SyncTransactionAbortedError` after its lifecycle was replaced, and the fail-stop then marks a terminal failure on the replaced lifecycle generation, which `getCurrentTerminalFailure` ignores. X7 is closed. | +| H5: the row count accepts a window that lost one row and gained another | The approved window law (maintainer decision, 2026-10-26): a window settles only while its restore holds the same rows. | Part 4: `row-replaced` (`a1` replaced below the boundary) and `row-gained-under-limit` (an under-full window gains a row). Neither changes the tie-boundary key. | Both settled from SQLite | Coverage of a limited load records its sorted row keys, and the restore must return the same keys. | +| H6: a release during the coverage check leaves the wait's timer running | Law record: a released load owns no wait. | Part 2c: release while the coverage read is held. | The source Collection still loaded after release | `settleCoveredDemand` returns at once when the signal is already aborted. | +| H7: a second SQLite read per load | Cost, not behavior | None | — | Kept. The read now also waits on the apply mutex. Reusing the restore read for the check is a possible optimization. | +| H8: duplicated builder-or-Collection lookup | Maintainability | None | — | Declined: a shared helper does not reduce shipped bytes. | +| H9: settlement listeners fire only from the status check | — | None: not reachable | — | `retireDemand` runs only for an empty demand update inside a graph run, which ends with a status check. Setup sends only nonempty initial keys. `settleDemand` now notifies directly (H1). | + +The unordered part of H5 was a law question, not an enforcement gap: the +approved row-count rule accepts an unordered demand that lost one row and +gained another. The maintainer kept the row count for unordered demands +(2026-10-26) over recording every row key. A fixed history now pins that +choice: an unordered demand that loses `a1` and gains `a3` settles offline +on `a2` and `a3`. A mutant that records keys for every demand fails it. + +Hostile mutants, beyond the starting head: + +- a window check that accepts any superset of the recorded keys survives + `row-replaced` and is killed by `row-gained-under-limit`; +- dropping the error-state check without notifying from `settleDemand` is + killed by both covered source-failure histories. + +GREEN: the oracle passes 262 of 262 with the unordered history (vitest counts its type checks too). + +## 2026-10-26: strengthening the oracle beyond the review findings + +Starting head: `4293b0021`. The maintainer asked for seven strengthening steps. +None changed production; each was checked with mutants against the current +code. + +- **Window limits** (`34642864a`): Part 4 adds four histories with a tie at + the window boundary and runs both page-move histories without an index. + The model treats the window as a limited demand (same rows) plus an + unordered tie-boundary demand (row count). A mutant that ignores the + tie-boundary row count fails `tie-row-outside-deleted`; one that records + keys for every demand fails `tie-row-added`. +- **Generated campaign** (`b9a5e481d`): a fixed and a random campaign of 40 + histories each. The first model predicted when the wrapper reacquires a + demand after a truncate, and three counterexamples showed that this + scheduling differs from the guess (the wrapper reacquires a demand only + after its obsolete load settles, and then reacquires every active + demand). The law leaves that timing open, so the model now replays the + source's own event log and judges each settled load as current or + obsolete by whether a truncate came between its arrival and its + settlement. No counterexample indicted production. Mutants that ignore + the row count, let an obsolete load record coverage, or keep coverage + across a truncate each fail both campaigns. +- **Storage failures** (Part 2 of the campaign): every adapter call of a + covered history fails once, before or after its effect, in either session. + Safety held at every call. A mutant that lets a failed coverage read fail + the load fails the sweep. Losing the acknowledgement of the coverage write + fails session 1 terminally, while session 2 still settles on the durable + rows, which is consistent with the law. +- **Releases** (Part 3): the demand is released after each step of a load, + including while a coverage read is in flight. A mutant that keeps a + released load's wait fails the sweep. +- **Framework witnesses** (`eff968951`): Vue, Svelte, and Angular. These + adapters have no network-first render gate, so the witnesses check + readiness and rows. +- **Native drivers**: a better-sqlite3 witness in `node-db-sqlite-persistence` + and a Chromium OPFS witness in `browser-db-sqlite-persistence`, in the + OPFS Playwright matrix that CI runs. The OPFS witness also pins the + guide's statement that a multi-tab coordinator waits for the network. +- **Mutation sweep**: reported separately below. + +### Mutation sweep + +A scripted sweep applied 63 semantic mutants to the coverage code in +`persisted.ts`, the gate in `persisted-readiness.ts`, the builder's +settlement signal, and the observer's source collection. Each ran against the +fixed demand coverage oracle and the persisted-readiness oracle, and each +survivor also ran against the receiving witnesses. Rerun against the +generated campaign, 24 survivors left 23. + +The oracle then gained five histories, and each kills a survivor: + +- an empty covered demand after a rotation (the record ignores its cache + generation); +- a truncate during the coverage read of an empty load (no check after the + read); +- a synchronous answer whose durable write lands late (the coverage read + does not wait on the apply mutex); +- claim recovery after an expired claim during the check (recovery not + scheduled); +- a coverage read that outlasts the wait (the deadline runs from the check). + +The release sweep also checks that no upstream load starts after a release. + +Classified survivors: + +- Equivalent: an early `isCurrent` check, rewriting an unchanged record, the + on-demand check in `settleCoveredDemand`, the timer clamp, a no-op + unsubscribe, re-settling a settled demand, the hydrated path's refetch + guard (repeated in `settleCoveredDemand`), the hydrated path's + expired-claim guard (a read under an expired claim already proves + nothing), and ignoring the lifecycle generation in the reset epoch (a + replaced lifecycle's sync callbacks are inert). +- Not distinguishable here: the forward guard after a release, because + hydration checks the abort signal at each step before it forwards a load. +- Open, outside this owner or needing a new history: the clock read after + the claim wait; the hydrated mark after a truncate, which the hydration + path owns; and six observer and gate mutants for the persisted-readiness + owner. Those six skip the restore check, the preload registration and its + check, or the client-stream failure in `isInitialRenderReady`; take the + source as the demand owner; or stop rendering at once on network + readiness. + +## 2026-10-27: closing the remaining gaps + +Starting head: `0db1fa68b` (CI green). No production change. + +- **Readiness owner.** Six histories in `persisted-readiness-oracle.test.ts` + kill the six observer and gate survivors: + - a restore held past the deadline, and a gate read before the preload + starts; + - a retried observer after a failed client stream, over an empty and a + nonempty restore; + - a retried observer after the Collection's preload has started; + - two readers of one on-demand source, only one of which has settled; + - a network answer before a 60 s deadline. + + The owner passes 77 of 77. One finding is left for design review: the + on-demand initial-render wait rarely differs from network readiness, + because an on-demand query becomes ready when its demands settle. Only the + synchronous gate distinguishes the two-reader history. +- **Claim renewal.** A history holds the load's own cache-claim renewal past + the wait; a wait measured from after the renewal fails it. The first + attempt held the startup renewal instead, before the load started, and its + failure on unchanged code came from the history, not production. Every + survivor of the sweep in this owner is now killed or classified. +- **Windows in the campaign.** The campaign adds the window `w` over + generated ranks with ties. A query is ready offline exactly when every + load it sent in session 2 is covered, so the model judges the + tie-boundary load without predicting when core sends it. A driver check + that every active demand loads again right after a truncate did not hold + for the window, and the model no longer depends on it, so it was removed. + Mutants that ignore window keys, compare them in order, or record keys for + every demand each fail the campaign. So do the five earlier campaign + mutants. +- **Remaining hosts.** Receiving witnesses run the five native histories on + Electron (two app launches over one file, in the desktop E2E job), + Cloudflare Durable Objects (`wrangler dev` restarted over one persist + directory, in the cloud-mobile job), React Native (an op-sqlite-shaped + driver, cloud-mobile job), Expo (an expo-sqlite shim, not in CI), and Tauri + and Capacitor (their unit-suite SQLite stand-ins). In each, a 60 s + session-2 wait fails the two covered histories. The native Tauri and + Capacitor paths need `cargo` or a device here. The Expo witness failed once + on its first local run, under concurrent load, and passed the next ten + runs; the cause is unknown. + +## 2026-10-27: one judgment for every fixed history and receiving witness + +Starting head: `4855c67e9`. No production change. The maintainer asked for a +shared receiving contract, a committed mutation harness with a nightly run, +and more campaign grammar; reading the current `oracle-authoring` skill and +oracle guide then showed that the fixed oracle and the witnesses broke +ORC-016 and ORC-014 by stating their own expected answers. + +- **Shared judgment** (`696d319a8`, `86efc7f7f`): + `tests/persisted-demand-coverage-model-oracle.ts` holds the model and + comparison. It replays the source's own log and judges each load of a + query against its exact demand's record. It covers: + - per-load waits, generation ends, and refetch loads; + - failed coverage reads and checks; + - settlement by the upstream load, cursors, and offsets; + - an eager-and-on-demand join. + + Parts 2 to 4 of the fixed oracle now record and pass their histories to + it. Their hand-written expectation tables are gone, and two pinned checks + owned elsewhere moved to `persisted-demand-coverage-supplemental.test.ts`. + Part 1 keeps its own bounded trace model. +- **A model error found on the way:** a window's tie-boundary load starts only + after its limited load settles and measures its own wait. A single wait + premise per query predicted the window ready at 1500 ms, while production + correctly still waited. The premise is now per load. Production was right. +- **Campaign grammar** (`696d319a8`): session 2 generates a 0 or 1000 ms + wait and a source that never answers or fails at once, with checkpoints + before and after each load's deadline. A mutant that settles a covered + load without waiting fails both campaigns. +- **Receiving contract** (`544c64340`): `tests/contracts/` holds six + histories as data. Every host and framework records and passes its + observations to the shared judgment. All pass. With the contract's session-2 + source forced to a 60 s wait and a silent upstream, every host and + framework reports violations. +- **Mutation harness and nightly job** (`cd9544ce5`): + `scripts/demand-coverage-mutants.mjs` applies 66 mutants in memory through + `vitest.mutants.config.ts`. The full run on `cd9544ce5` matched every + expected outcome. The 22 mutants tied to the converted histories are each + killed, nearly all by the converted fixed histories themselves. + `.github/workflows/oracle-nightly.yml` runs the campaign with ten times its + PR budget and the full mutant list. + +Limits: the campaign generates no joins and no page moves, which the fixed +oracle and the shared model cover; native-runtime witnesses remain blocked +here as the coverage map records. + +## 2026-10-27: integrating the source-health change (#2168) + +Merge commit: `9262ec82d`. #2168 removed the observer's network-first gate, +`PersistedReadinessSource`, and `packages/db/src/persisted-readiness.ts`. A +network-first eager Collection now becomes `ready` from its restored rows, +and Suspense releases at query readiness. The core half of this PR (the +builder's `initialDemandsSettled()`, its listeners, and the observer's +on-demand sources) therefore has no place left and is gone; this PR changes +no `packages/db/src` code now. The law is unchanged. An on-demand source +already permits initial readiness, so a joined query is ready when its eager +source has opened and its demands on the on-demand source have settled. + +- **Production:** `openFromDisk` skips an on-demand Collection. Without the + guard, an on-demand Collection whose covered rows were restored before the + deadline would become `ready` at the deadline, though its source never + called `markReady()`. For Electric `progressive`, that status claims a + completed full sync. +- **Law added to the shared model:** settling loads from SQLite never changes + the on-demand Collection's own status. The guide already stated it ("only + the load settles from SQLite"), but no oracle observed it: mutant P39, which + drops the guard, survived the first sweep after the merge. The model now + judges an optional Collection observation. The campaign, the fixed + lifecycle history, and the joined histories supply it. With it, P39 is + killed by the fixed campaign. These histories never call the source's own + `fail()`; that path is the source-health law's. +- **Part 3:** the joined histories observe the query's status directly + instead of the removed observer gate. The source-failure histories that + closed H1 moved to the source-health law, which #2168 owns in + `packages/db/tests/source-health-oracle.test.ts`. +- **Mutation harness:** the 29 mutants whose targets #2168 deleted (D01–D11, + B01–B10, O01–O08) are gone; P39 is new. 38 mutants remain. +- **Receiving witnesses:** all 13 hosts and frameworks pass on the merge. The + Vue witness cleans up with `{ dependents: true }`, as #2168's cleanup + admission requires for a query with no component scope. +- **Readiness oracle:** main's case that expected on-demand network-first to + throw contradicts this law and is removed. This owner covers that + configuration in every history. + +Limits: the Collection observation is not yet part of the receiving +contract, so hosts and frameworks do not check it; the wrapper is the only +production path that decides it. + +## 2026-10-28: high code review of the head after #2168 + +Reviewed head: `83b10fd97` (clean worktree, files matched `gh pr diff`). +Repairs: `de9fec4c3`; merge of #2278: `32404fa2a`. Each oracle change ran on +the parent production code before the repair. + +| ID | Claim | Verdict and evidence | Action | +| --- | --- | --- | --- | +| R1 | An unordered demand that loses one row and gains another stays covered | True. This is the approved row-count law (maintainer decision, 2026-10-26); a fixed history pins it. The guide overclaimed that a removed row "therefore ends coverage". | Accepted design. The guide now says a row that arrived meanwhile keeps the count. | +| R2 | A failed coverage commit fails a load whose rows applied; the synchronous path ignores it | The asymmetry is real, the consequence is not. A probe failed the coverage write before and after its effect: the Collection crashed both times, as for any durable write (glossary, "crash"), so the load's outcome decides nothing. The one non-durable rejection, an abort while scoped recovery is pending, also aborts the load's own row transactions. | Refuted as stated. The storage sweep now asserts the crash (`coverage write failed`). No mutant separates this path from the general crash law. | +| R3 | A released load whose source resolves anyway records coverage | Confirmed. The glossary's "abort" says an aborted load's later outcome decides nothing, and the signal contract lets a source stop and resolve. RED: the campaign's new `stops and resolves` source; counterexample `start w, release w` settled `w` offline. A first run reached the state only at teardown, where production cannot record either; teardown now drops waiting loads first, and the declared reach `demand settled only by an aborted load` forced the distinguishing case. | Fixed: `recordCoverage` treats an aborted request as obsolete. Mutant P41 killed. | +| R4 | `begin({ snapshot: true })` does not delete coverage records | Confirmed. A snapshot transaction replaces every row as a truncate does (glossary). RED: new `snapshot` command; counterexample `cover a, snapshot` settled `a` offline. | Fixed: truncate and snapshot share `endDemandCoverage`. Mutant P40 killed. | +| R5 | `readDemandRows` lacks the scoped-recovery guard | Confirmed. Coverage counts only while the persisted subset restore returns the rows (glossary), the restore publishes none during scoped recovery, and scoped recovery does not certify the generation for offline reuse (Electric guide). RED: Part 2b `a load after scoped recovery reloaded the demand` showed `ready` on `a1`, `a2`. The first RED was false: the history started recovery before any sync run, and the driver's hook silently did nothing; the driver now throws on that premise. | Fixed: the coverage read proves nothing during scoped recovery, so no load settles from SQLite or records. The reviewer's proposed fix (reuse the restore read) would have recorded a zero-row coverage that every later restore satisfies. Mutant P42 killed. | +| R6 | Up to three SQLite reads per load | True, and the hydrated fast path now reads when a record exists. | No change: H7 kept the read; no work promise exists. Reusing the restore's rows for the check remains an optimization. | +| R7 | Coverage metadata grows without bound | True: one record per demand key until a truncate or rotation. | Accepted design: expiry is deferred to #2211 by maintainer decision. | +| R8 | The model imports core's demand key | True, against AGENTS.md ("Keep Oracles Independent"). | Fixed: the model compares request data itself (offset 0 as none). Every existing history agreed. Mutant K01 (coverage keyed without cursor and offset) is killed. Mutating core's own key is not distinguishable: core's deduplication uses the same key, so the merged key changes which loads core sends, and the model reads those from the record. | +| R9 | On-demand Collections install a no-op deadline timer; `onDemand` duplicates `syncMode` | True. `failUpstream` also calls `openFromDisk`, so its guard stays either way; skipping the timer adds a branch. | No change. | +| R10 | The abort listener stays after the timer or upstream settles | True but bounded: core creates one signal per subset request (`subscription.ts`, `subset-demand-controller.ts`), so the listener lives only as long as its request. The "repeated loads on one signal" premise does not occur. | Refuted as a leak. | + +Totals: 10 items; 4 fixed (R3, R4, R5, R8), 2 accepted design (R1, R7), +2 refuted (R2, R10), 2 no-change confirmations (R6, R9). R1 also fixed a doc +overclaim, and R2 added a sweep assertion. + +Why the suite missed R3 to R5: the grammar had no snapshot transaction, no +source that resolves an aborted load, and no online load after scoped +recovery in the same sync run. The model shared core's demand key (R8). + +### Never-run queries after a completed progressive full sync + +The guide claims that a query that never ran renders offline once a +progressive full sync completed. No witness checked it: the Electric witness's +session 2 read only the subset session 1 had loaded. The shared model gains a +`full sync completes` source event (authority: `ARCHITECTURE.md`, "A source +that resumes from complete local rows settles its loads at once"), and the +Electric witness's network-first block becomes a production driver for it, +over session 1 ∈ {full sync, subset only, nothing} × session 2 reads +{the loaded subset, a never-loaded subset}. All six histories pass; the +full-sync, never-loaded history shows `ready` on its restored row and reaches +`settled from a completed full sync`. Mutant E01 (drop the complete resume's +`resolveFullSnapshot(true)` in `electric.ts`) fails exactly that history. This +covers #2202 without a new adapter API, unlike #2203's +`isPersistedSnapshotComplete`. + + +## 2026-10-28: final high review of `d9bc29d74` + +Reviewed in a clean worktree whose files matched `gh pr diff`; the reviewer +could not run tests there. No production change followed. + +| ID | Claim | Verdict and evidence | Action | +| --- | --- | --- | --- | +| S1 | A covered demand settles while a source truncate is staged but uncommitted | Refuted on the Electric path: a same-path probe (cover group a, then an open `must-refetch` in session 2, then a group-a query offline) kept the query `loading` with no rows until `up-to-date`. On the generic path the precondition needs a source transaction held open across a new demand's hydration, which the wrapper rejects at commit (`cannot cross a hydration cycle`); the oracle already excludes that history (X5). A probe history reproduced exactly that rejection and was removed. | None | +| S2 | A rejected coverage commit fails a load whose rows applied | Duplicate of R2. The new path, a coverage transaction queued behind hydration and rejected by a later reset, belongs to a load that core already aborted at that reset (see S5), so its outcome decides nothing. | None | +| S3 | A load without a signal, released by `unloadSubset`, records coverage | Refuted: every on-demand subset load carries its own `AbortController` signal (`subscription.ts`), and an acquisition release aborts it (glossary, "acquisition release"). | None | +| S4 | A `BrowserCollectionCoordinator` leader without a subset owner records and settles coverage | Refuted for conforming coordinators: `registerRemoteSubsetOwner` is required, and the wrapper registers the owner synchronously when the source session starts and again inside scoped recovery's mutex turn. The OPFS witness with a real coordinator waits for the network. | None | +| S5 | The reset epoch is captured after the source's `loadSubset` returns | Same-path history added (the source truncates synchronously inside its own load call): it passes on production, because core's truncate replay aborts the in-flight load and an aborted load records nothing. | Regression history kept | +| S6, S7 | Extra SQLite reads and a metadata commit on the online path | Duplicates of H7 and R6. The synchronous path also reads on every deduplicated re-subscription. | No change; no work promise | +| S8 | Unbounded coverage metadata | Duplicate of R7 (#2211). | Accepted design | +| S9 | Abort listener and timer outlive their load | Duplicate of R10; the timer resolves an already-settled load. | None | +| S10 | The wait starts after startup, so offline it is startup plus `networkTimeoutMs` | True, by design: the wait is a preference for the network and runs from when the wrapper asks it, as the eager deadline runs from the sync start when its source starts. | Accepted design | + +The full mutation sweep after the earlier repairs reported three survivors, +because the abort check (R3) and the scoped-recovery guard (R5) now subsume +older guards on the tested paths: + +- P33 (no reset-epoch check) is killed by a new history: an empty synchronous + answer, then a truncate during its coverage read, with the replay left + waiting. A synchronous answer's load has already settled, so core cannot + abort it, and only the epoch check stops the stale record. +- P05 (the epoch ignores the reset sequence) and P10 (the record ignores its + generation) are equivalent in the single-process scope: every in-run + rotation or reset-sequence change that can meet a record or a load goes + through scoped recovery, where the coverage read proves nothing, and + claim recovery after startup also calls `startScopedRecovery`. A new + history with an empty covered demand and an expired claim confirms P10 + survives with its premise reached. + +Totals: 10 items; 0 fixed, 6 refuted or duplicate-refuted (S1 to S5, S9), +2 duplicates of no-change cost items (S6, S7), 2 accepted design (S8, S10). diff --git a/docs/contributing/oracle-reviews/issue-2056-cache-eviction-design.md b/docs/contributing/oracle-reviews/issue-2056-cache-eviction-design.md index 608facb12f..2c66b7aa05 100644 --- a/docs/contributing/oracle-reviews/issue-2056-cache-eviction-design.md +++ b/docs/contributing/oracle-reviews/issue-2056-cache-eviction-design.md @@ -323,3 +323,34 @@ demand beside an active warm tab and a peer notification delivered to an expired receiver. The Electric descriptor owner checks restart, and the installed-SDK owner checks subset-only network work. The remaining limits above are recorded in the coverage map rather than inferred from green tests. + +## Follow-up: persisted demand coverage + +PR #2263 implements item 1 of #2102 in a narrower form than the issue +proposed. The issue asked for certified predicate and window evidence that +overlapping subsets could combine. The PR instead records persisted demand +coverage: evidence, in one persisted cache generation, that a load of one +exact demand reached applied settlement with its rows durably committed. Its +law is in `2026-10-20-on-demand-network-first-law.md`, and its primary owner is +`packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts`. + +Coverage does not change this design's authority rules: + +- It is not a key-set certificate. An uncertified generation stays + uncertified. Coverage only lets a later load of the same demand settle from + rows that the same generation already holds. +- Coverage belongs to its generation. Scoped recovery and claim-expiry + rotation give the next load an empty generation with no coverage, and a + record that memory still holds from the retired generation does not count. +- An ordinary persisted truncate deletes every coverage record in its own + transaction, and a schema reset deletes collection metadata with the rows. +- A row deleted after the covered load settled ends coverage for that demand. + An ordered window whose rows change asks for a tie-boundary key that no load + covered. +- Overlapping demands never combine. A demand covers only its own exact key, + as #1797 requires for request deduplication. + +Shape-identity churn across new runs, described above, can alternate the +current generation. Coverage written under one shape's generation is not +reused under the other's. + diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index cf35e92980..a63b20c40f 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -262,7 +262,7 @@ For more details on framework integration, see the [React](../framework/react/ov For React applications, you can use the `useLiveSuspenseQuery` hook to integrate with React Suspense boundaries. This hook suspends rendering while data loads initially, then streams updates without re-suspending. -For eager SQLite persisted Collections, you can opt into a network-first initial render on every query source. A completed restore can release React or Solid Suspense after a network deadline or source failure. React also handles a failed client query stream this way. See [Render after SQLite restore](./sqlite-persistence.md#render-after-sqlite-restore) for setup and limits. +For SQLite persisted Collections, you can opt into a network-first initial render on every query source. A completed restore can release React or Solid Suspense after a network deadline or source failure. On-demand sources apply this to each query demand that settled online before. React also handles a failed client query stream this way. See [Render after SQLite restore](./sqlite-persistence.md#render-after-sqlite-restore) for setup and limits. ```tsx import { useLiveSuspenseQuery } from '@tanstack/react-db' diff --git a/docs/guides/sqlite-persistence.md b/docs/guides/sqlite-persistence.md index e3d4b2fae2..0ad929dbc4 100644 --- a/docs/guides/sqlite-persistence.md +++ b/docs/guides/sqlite-persistence.md @@ -244,7 +244,7 @@ An on-demand cache is not yet a certified offline snapshot. The first managed ge ## Render after SQLite restore -Without `initialRender`, a persisted Collection becomes `ready` when its sync adapter calls `markReady()`, like any other Collection. The restored rows are in the Collection while it loads, but `status` stays `loading`. +Without `initialRender`, a persisted Collection becomes `ready` when its sync adapter calls `markReady()`, like any other Collection. The restored rows are in the Collection while it loads, but `status` stays `loading`. Eager and on-demand Collections both support `initialRender`; [on-demand Collections](#on-demand-collections-offline) apply it to each query demand. With `initialRender: { strategy: 'network-first', networkTimeoutMs }`, the Collection prefers the network for `networkTimeoutMs` (default 3 seconds). It becomes `ready` from the restored rows when both of these are true: @@ -304,6 +304,45 @@ Place `TodosScreen` inside an [Error Boundary](./error-handling.md#using-suspens Restored rows do not prove current authorization. Isolate SQLite data by user or tenant. If an authorization failure must hide cached rows, block the view separately: a remote failure can open the Collection from restored rows. +### On-demand Collections offline + +An on-demand Collection loads rows for each query demand: the filter, order, and limit that a query asks the source for. With `initialRender`, a demand can settle on its rows from SQLite when the network does not answer. This includes Electric's `syncMode: 'progressive'`, which loads query subsets on demand until its full sync completes. + +```ts +type Issue = { id: string; title: string } + +const issues = createCollection( + persistedCollectionOptions({ + ...electricCollectionOptions({ + id: 'issues', + shapeOptions: { url: 'http://localhost:3000/v1/shape', params: { table: 'issues' } }, + syncMode: 'progressive', + getKey: (issue) => issue.id, + }), + persistence: syncedPersistence, + initialRender: { strategy: 'network-first', networkTimeoutMs: 3_000 }, + }), +) +``` + +The rule applies to each demand separately: + +- A demand is covered after a load of that exact demand settles online and its rows are saved in SQLite. Another filter, order, or limit is a different demand, even when its rows overlap. +- A later load of a covered demand first shows its rows from SQLite. If the network does not answer within `networkTimeoutMs`, or it fails, the load settles on those rows. `isReady`, `preload()`, and Suspense then proceed as they do after a network load. Only the load settles from SQLite; the on-demand Collection's own status does not change. +- The network load keeps running. Its rows replace the restored rows when they arrive. If it fails after the load settled, the query stays ready on the rows it shows. +- A demand that never settled online is not covered. Offline, it waits for the network. A query that has never run therefore stays loading offline unless a completed progressive full sync already holds every row. +- A source truncate, such as an Electric must-refetch, ends coverage for every demand, and so does a snapshot transaction (`begin({ snapshot: true })`), which replaces every row. A new cache generation starts without coverage. +- Scoped recovery also ends coverage. For the rest of that sync run, no load settles from SQLite and no load records coverage. +- A load that is released before it settles records no coverage, even if its source finishes it afterward. +- Coverage also ends when SQLite holds fewer rows for the demand than it held when the demand settled. A row that the source deletes, or that Query DB retention removes after `gcTime`, therefore ends coverage until the demand loads online again, unless another row of the demand reached SQLite meanwhile, for example through another query. Coverage counts the rows of an unordered demand; it does not record which rows they were. +- An ordered window with a `limit` stays covered only while SQLite holds the same rows for it, in any order. A row that leaves, joins, or replaces a row of the window ends its coverage. +- A load with `refetch: true` always waits for the network. +- Only the default single-process coordinator supports this. If you pass a multi-tab coordinator, such as `BrowserCollectionCoordinator`, on-demand loads do not settle from SQLite. They wait for the network, as they did before this option. + +Coverage proves that the rows were complete when the earlier load settled. It does not prove that they are current. The wait changes when readiness arrives, not which rows the source has. A cursor page or a demand built from other rows' values is covered only if the restored rows produce the same demand again. + +For a query that joins eager and on-demand sources, the query becomes ready when every eager source is ready, which network-first allows from its restored rows after the deadline, and every demand that the query reaches on an on-demand source has settled. A network failure alone does not settle an on-demand demand that is not covered. + ## Browser tabs and Electron renderers The browser package uses single-tab coordination by default. If tabs share one OPFS database, give the persistence instance a `BrowserCollectionCoordinator`. It elects an owner for each Collection and routes writes between tabs. Replace the initial database and persistence setup with this code in each tab: diff --git a/packages/angular-db/tests/injectLiveQuery-persisted-on-demand.test.ts b/packages/angular-db/tests/injectLiveQuery-persisted-on-demand.test.ts new file mode 100644 index 0000000000..597fc839d2 --- /dev/null +++ b/packages/angular-db/tests/injectLiveQuery-persisted-on-demand.test.ts @@ -0,0 +1,137 @@ +import { DatabaseSync } from 'node:sqlite' +import { EnvironmentInjector, createEnvironmentInjector } from '@angular/core' +import { TestBed } from '@angular/core/testing' +import { describe, expect, it, vi } from 'vitest' +import { eq } from '@tanstack/db' +import { SQLiteCorePersistenceAdapter } from '../../db-sqlite-persistence-core/src' +import { injectLiveQuery } from '../src/index' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import { + A1, + A2, + FRAMEWORK_HISTORY_IDS, + coverageRecords, + createReceivingSource, + receivingHistory, + recordCheckpoints, + session2SourceOptions, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + Session1Record, + Session2Record, + SourceEvent, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { QueryObservation } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +/** + * # Does an Angular live query settle offline on a covered on-demand demand? + * + * Receiving witness for persisted demand coverage. The law, the histories, and + * the checkpoint records are the receiving contract's + * (`db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * and the shared judge (`db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts`) + * judges each record. The law's authority is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`. + * + * Histories: `FRAMEWORK_HISTORY_IDS` (`covered`, `never`, and + * `covered-upstream-fails-at-once`). Session 1 is online and answers its load + * at once through the source, or loads nothing. Session 2 is offline on the + * same `node:sqlite` database, with the history's upstream behavior. Each + * session runs in its own environment injector, which stands in for a + * component's lifetime and is destroyed after the session. + * + * Production path: `injectLiveQuery` -> `createLiveQueryObserver` (wholesale + * mode) -> the live query's on-demand `loadSubset` -> the persisted wrapper, + * which waits for the network-first wait and then settles a covered demand from + * its restored rows. + * + * Observation: at each checkpoint of session 2, the signals `isReady` and + * `data`. + * + * Limits: this witness checks the wrapper's settlement through query + * readiness and rows. It uses one filter, one injection per session, and no template rendering. + */ + +function groupA( + injector: EnvironmentInjector, + collection: ReturnType[`collection`], +) { + return injector.runInContext(() => + injectLiveQuery((q) => + q.from({ row: collection }).where(({ row }) => eq(row.group, `a`)), + ), + ) +} + +function observeQuery(query: ReturnType): QueryObservation { + if (!query.isReady()) return { status: `loading` } + return { + status: `ready`, + rows: query + .data() + .map((row) => row.id) + .sort(), + } +} + +describe(`Angular injectLiveQuery over a persisted on-demand Collection offline`, () => { + it.each(FRAMEWORK_HISTORY_IDS.map(receivingHistory))( + `$description`, + async (history) => { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) + const parent = TestBed.inject(EnvironmentInjector) + const driver = createDriver(new DatabaseSync(`:memory:`)) + const persistence = () => ({ + adapter: new SQLiteCorePersistenceAdapter({ driver }), + }) + + const session1Events: Array = [] + if (history.session1.length > 0) { + const firstInjector = createEnvironmentInjector([], parent) + const first = createReceivingSource({ + persistence: persistence(), + waitMs: 0, + behavior: `answer-at-once`, + events: session1Events, + answerRows: [A1, A2], + }) + const query = groupA(firstInjector, first.collection) + await vi.waitFor(() => expect(query.isReady()).toBe(true)) + firstInjector.destroy() + await new Promise((resolve) => setTimeout(resolve, 50)) + await first.collection.cleanup() + } + const session1: Session1Record = { events: session1Events } + + const secondInjector = createEnvironmentInjector([], parent) + const second = createReceivingSource({ + persistence: persistence(), + ...session2SourceOptions(history), + events: [], + }) + const loadStartedAt = Date.now() + const query = groupA(secondInjector, second.collection) + try { + const checkpoints = await recordCheckpoints( + () => observeQuery(query), + () => second.received.length, + loadStartedAt, + ) + const session2: Session2Record = { + sent: [...second.received], + ...checkpoints, + } + // Positive witness: session 2 asked the offline source for group a. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + } finally { + secondInjector.destroy() + await second.collection.cleanup() + } + }, + ) +}) diff --git a/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.html b/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.html new file mode 100644 index 0000000000..80477eb2e7 --- /dev/null +++ b/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.html @@ -0,0 +1,12 @@ + + + + + + Persisted on-demand coverage witness + + + running + + + diff --git a/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.spec.ts b/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.spec.ts new file mode 100644 index 0000000000..30c9a708d6 --- /dev/null +++ b/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.spec.ts @@ -0,0 +1,84 @@ +/** + * Chromium receiving spec for persisted demand coverage, over real OPFS with + * the wa-sqlite worker driver. The law, the histories, the production path, + * and the observations are the receiving contract's + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * and the page `persisted-on-demand-coverage.opfs.ts` drives them. Each + * history is one page load for session 1 (online) and one later page load over + * the same OPFS file for session 2 (offline, upstream as the history says). + * + * The spec states no expected results. It checks that each run completed, that + * session 2 asked the offline source (the positive witness), and that the + * shared judge, which runs in the session 2 page, reports no violations at + * either checkpoint. The judge cannot load in Playwright's Node process (the + * `@tanstack/db` path reaches TypeScript sources that Playwright cannot + * transform), so the page runs it. + * + * The coordinated scenario passes `BrowserCollectionCoordinator`. The law does + * not cover it: the guide (`docs/guides/sqlite-persistence.md`) says only the + * default single-process coordinator supports this. Its run is recorded and not + * judged against the law. + */ +import { expect, test } from '@playwright/test' +import { RECEIVING_HISTORIES } from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-histories' +import type { Session1Record } from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { Session2Result } from './persisted-on-demand-coverage.opfs' + +const SCENARIOS = [ + ...RECEIVING_HISTORIES.map((history) => ({ + id: history.id, + name: history.description, + coordinator: `none` as const, + })), + { + id: `covered`, + name: `covered demand with a multi-tab coordinator (guide: out of scope)`, + coordinator: `browser` as const, + }, +] + +for (const scenario of SCENARIOS) { + test(scenario.name, async ({ page }) => { + test.setTimeout(120_000) + const pageErrors: Array = [] + page.on(`pageerror`, (error) => pageErrors.push(error.message)) + const databaseName = `pod-${crypto.randomUUID().slice(0, 8)}.sqlite` + const url = (phase: `1` | `2`) => + `/e2e/persisted-on-demand-coverage.opfs.html?phase=${phase}&history=${scenario.id}&coordinator=${scenario.coordinator}&db=${databaseName}` + + await page.goto(url(`1`)) + await page.waitForFunction( + () => window.__persistedOnDemandSession1 !== undefined, + undefined, + { timeout: 60_000 }, + ) + const session1 = await page.evaluate( + () => window.__persistedOnDemandSession1, + ) + expect(session1).toBeDefined() + expect(session1).not.toHaveProperty(`error`) + expect((session1 as Session1Record).events.length).toBeGreaterThanOrEqual(0) + + await page.goto(url(`2`)) + await page.waitForFunction( + () => window.__persistedOnDemandSession2 !== undefined, + undefined, + { timeout: 60_000 }, + ) + const session2 = (await page.evaluate( + () => window.__persistedOnDemandSession2, + )) as Session2Result | undefined + expect(session2).toBeDefined() + expect(session2).not.toHaveProperty(`error`) + expect(pageErrors).toEqual([]) + if (!session2 || !(`record` in session2)) return + // Positive witness: session 2 asked the offline source for the demand. + expect(session2.record.sent.length).toBeGreaterThan(0) + if (scenario.coordinator === `none`) { + expect(session2.violations).toEqual({ + beforeDeadline: [], + afterDeadline: [], + }) + } + }) +} diff --git a/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.ts b/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.ts new file mode 100644 index 0000000000..bf62433bcc --- /dev/null +++ b/packages/browser-db-sqlite-persistence/e2e/persisted-on-demand-coverage.opfs.ts @@ -0,0 +1,145 @@ +/** + * Chromium receiving driver for persisted demand coverage, over real OPFS with + * the wa-sqlite worker driver. It runs one session per page load over the same + * OPFS database. The Playwright spec (`persisted-on-demand-coverage.opfs.spec.ts`) + * navigates the page from session 1 to session 2, so session 2 is a later page + * over the same storage. + * + * # Law, authority, and histories + * + * The law and the histories are the receiving contract's + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`). + * The law is `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`; + * the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This page is the browser host. + * + * # Production path + * + * Each session opens `openBrowserWASQLiteOPFSDatabase` and + * `createBrowserWASQLitePersistence`, and the contract builds the Collection + * with `persistedCollectionOptions`. The coordinated scenario also passes a + * `BrowserCollectionCoordinator`. Session 1 closes its database (and + * coordinator, if any) before the page navigates. + * + * # Observations + * + * The page records the session 1 source log, and the session 2 loads the + * source received with its two checkpoints. Session 2 then runs the shared + * judge (`persisted-demand-coverage-model-oracle.ts`) over those records and + * reports its violations. The page states no expected result itself. + * + * # Limits + * + * One driver (wa-sqlite over OPFS in Chromium, one tab at a time). It does not + * cover Firefox or WebKit, real multi-tab coordination, or the Electron and + * Node drivers (the node witness covers better-sqlite3). + */ +import { + BrowserCollectionCoordinator, + createBrowserWASQLitePersistence, + openBrowserWASQLiteOPFSDatabase, +} from '../src/index' +import { + coverageRecords, + receivingHistory, + recordSession1, + recordSession2, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + ReceivingSession, + Session1Record, + Session2Record, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' + +// Session 2's record, and the shared judge's violations at each checkpoint. +// `violations` is absent for the coordinated run, which the law does not cover. +export type Session2Result = + | { + record: Session2Record + violations?: { + beforeDeadline: Array + afterDeadline: Array + } + } + | { error: string } + +declare global { + interface Window { + __persistedOnDemandSession1?: Session1Record | { error: string } + __persistedOnDemandSession2?: Session2Result + } +} + +// Session 1's record reaches session 2 through sessionStorage, which the tab +// keeps across the spec's navigation. +const storageKey = (databaseName: string) => + `persisted-demand-coverage:${databaseName}` + +// Opens one session's storage. `close` releases the coordinator and the +// database so that the next page can open the same OPFS file. +async function openSession( + databaseName: string, + coordinated: boolean, +): Promise { + const database = await openBrowserWASQLiteOPFSDatabase({ databaseName }) + // Without a coordinator the wrapper uses the default single-process + // coordinator, the only one the law covers (guide: sqlite-persistence.md). + const coordinator = coordinated + ? new BrowserCollectionCoordinator({ dbName: databaseName }) + : undefined + const persistence = createBrowserWASQLitePersistence({ + database, + ...(coordinator ? { coordinator } : {}), + schemaMismatchPolicy: `reset`, + }) + return { + persistence, + close: async () => { + coordinator?.dispose() + await database.close?.() + }, + } +} + +async function main(): Promise { + const params = new URLSearchParams(window.location.search) + const phase = params.get(`phase`) + const history = receivingHistory(params.get(`history`)!) + const coordinated = params.get(`coordinator`) === `browser` + const databaseName = params.get(`db`) + if (!databaseName) throw new Error(`missing db parameter`) + const session = await openSession(databaseName, coordinated) + try { + if (phase === `1`) { + const record = await recordSession1(session.persistence, history) + sessionStorage.setItem(storageKey(databaseName), JSON.stringify(record)) + window.__persistedOnDemandSession1 = record + return + } + const session1 = JSON.parse( + sessionStorage.getItem(storageKey(databaseName)) ?? `null`, + ) as Session1Record + const record = await recordSession2(session.persistence, history) + if (coordinated) { + window.__persistedOnDemandSession2 = { record } + return + } + const records = coverageRecords(history, session1, record) + window.__persistedOnDemandSession2 = { + record, + violations: { + beforeDeadline: coverageViolations(records.beforeDeadline), + afterDeadline: coverageViolations(records.afterDeadline), + }, + } + } catch (error) { + if (phase === `1`) + window.__persistedOnDemandSession1 = { error: String(error) } + else window.__persistedOnDemandSession2 = { error: String(error) } + } finally { + await session.close() + } +} + +void main() diff --git a/packages/browser-db-sqlite-persistence/playwright.opfs.config.ts b/packages/browser-db-sqlite-persistence/playwright.opfs.config.ts index f7f7eacc71..db2fc5a79d 100644 --- a/packages/browser-db-sqlite-persistence/playwright.opfs.config.ts +++ b/packages/browser-db-sqlite-persistence/playwright.opfs.config.ts @@ -16,6 +16,7 @@ export default defineConfig({ `leader-close.opfs.spec.ts`, `opfs-request-order-oracle.opfs.spec.ts`, `reload-class-oracle.opfs.spec.ts`, + `persisted-on-demand-coverage.opfs.spec.ts`, ], timeout: 60_000, fullyParallel: false, diff --git a/packages/capacitor-db-sqlite-persistence/tests/capacitor-persisted-demand-coverage.test.ts b/packages/capacitor-db-sqlite-persistence/tests/capacitor-persisted-demand-coverage.test.ts new file mode 100644 index 0000000000..d555b61fe6 --- /dev/null +++ b/packages/capacitor-db-sqlite-persistence/tests/capacitor-persisted-demand-coverage.test.ts @@ -0,0 +1,75 @@ +/** + * # Receiving witness: persisted demand coverage over the Capacitor driver + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * holds the histories and records the sessions. The law is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the Capacitor host. It checks the same law + * through the wrapper `createCapacitorSQLitePersistence`, over the better-sqlite3 stand-in in `tests/helpers/capacitor-sqlite-test-db.ts`. + * + * Histories: every history of `RECEIVING_HISTORIES`, each with its own database + * file. Session 1 is online and takes the history's steps. Session 2 is offline + * and opens a new driver on the same file, with its upstream as the history says. + * + * Production path: each session builds a `persistedCollectionOptions` + * Collection over `createCapacitorSQLitePersistence` and runs the query's preload. Session 1 closes + * its driver before session 2 opens. + * + * Observations: the records of both sessions, judged by the shared judge + * (`persisted-demand-coverage-model-oracle.ts`) at each session 2 checkpoint. + * + * Limits: It runs in this package's existing unit suite, which already exercises the wrapper over that stand-in. It does not run the Capacitor runtime on a device or simulator; the device e2e (`e2e/`, run with `test:e2e:ios` or `test:e2e:android`) needs an iOS simulator or an Android emulator, and `adb` is not installed here, so that path is not covered. The sessions are two driver instances in one process over one file, not an app restart. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { createCapacitorSQLitePersistence } from '../src' +import { + RECEIVING_HISTORIES, + coverageRecords, + recordReceivingHistory, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import { createCapacitorSQLiteTestDatabase } from './helpers/capacitor-sqlite-test-db' + +const dirs: Array = [] + +afterEach(() => { + for (const dir of dirs.splice(0)) + rmSync(dir, { recursive: true, force: true }) +}) + +const DESCRIBE = `capacitor persisted demand coverage (receiving witness)` + +describe(DESCRIBE, () => { + it.each(RECEIVING_HISTORIES)( + `$description`, + async (history) => { + const dir = mkdtempSync(join(tmpdir(), `capacitor-demand-coverage-`)) + dirs.push(dir) + const filename = join(dir, `state.sqlite`) + + const { session1, session2 } = await recordReceivingHistory( + history, + () => { + const database = createCapacitorSQLiteTestDatabase({ filename }) + return Promise.resolve({ + persistence: createCapacitorSQLitePersistence({ database }), + close: async () => { + await database.close() + }, + }) + }, + ) + // Positive witness: session 2 asked the offline source for the demand. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + }, + 60_000, + ) +}) diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-do-runtime-bridge.e2e.test.ts b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-do-runtime-bridge.e2e.test.ts index 0f9d5b7118..abd5300505 100644 --- a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-do-runtime-bridge.e2e.test.ts +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-do-runtime-bridge.e2e.test.ts @@ -1,244 +1,16 @@ import { mkdtempSync, rmSync } from 'node:fs' -import { createRequire } from 'node:module' import { tmpdir } from 'node:os' -import { dirname, join } from 'node:path' -import { setTimeout as delay } from 'node:timers/promises' -import { fileURLToPath } from 'node:url' -import { spawn } from 'node:child_process' +import { join } from 'node:path' import { describe, expect, it } from 'vitest' import { runRuntimeBridgeE2EContractSuite } from '../../db-sqlite-persistence-core/tests/contracts/runtime-bridge-e2e-contract' +import { postJson, startWranglerRuntime } from './fixtures/wrangler-runtime' import type { RuntimeBridgeE2EContractError, RuntimeBridgeE2EContractHarness, RuntimeBridgeE2EContractHarnessFactory, RuntimeBridgeE2EContractTodo, } from '../../db-sqlite-persistence-core/tests/contracts/runtime-bridge-e2e-contract' - -type RuntimeProcessHarness = { - baseUrl: string - restart: () => Promise - stop: () => Promise - getOutput: () => { - stderr: string - stdout: string - } -} - -type WranglerRuntimeResponse = - | { - ok: true - rows?: TPayload - } - | { - ok: false - error: RuntimeBridgeE2EContractError - } - -const packageDirectory = dirname(fileURLToPath(import.meta.url)) -const wranglerConfigPath = join(packageDirectory, `fixtures`, `wrangler.toml`) -const require = createRequire(import.meta.url) -const wranglerBinPath = require.resolve(`wrangler/bin/wrangler.js`) - -async function getAvailablePort(): Promise { - const netModule = await import('node:net') - return new Promise((resolve, reject) => { - const server = netModule.createServer() - server.listen(0, `127.0.0.1`, () => { - const address = server.address() - if (!address || typeof address === `string`) { - server.close() - reject(new Error(`Unable to allocate an available local port`)) - return - } - const selectedPort = address.port - server.close((error) => { - if (error) { - reject(error) - return - } - resolve(selectedPort) - }) - }) - server.on(`error`, reject) - }) -} - -function createRuntimeError( - message: string, - stderr: string, - stdout: string, -): Error { - return new Error([message, `stderr=${stderr}`, `stdout=${stdout}`].join(`\n`)) -} - -async function stopWranglerProcess( - child: ReturnType | undefined, -): Promise { - if (!child || child.exitCode !== null) { - return - } - - child.kill(`SIGTERM`) - const closed = await Promise.race([ - new Promise((resolve) => { - child.once(`close`, () => resolve(true)) - }), - delay(5_000).then(() => false), - ]) - - if (closed) { - return - } - - child.kill(`SIGKILL`) - await new Promise((resolve) => { - child.once(`close`, () => resolve()) - }) -} - -async function startWranglerRuntime(options: { - persistPath: string - syncEnabled?: boolean - schemaVersion?: number - collectionId?: string -}): Promise { - let child: ReturnType | undefined - let stdoutBuffer = `` - let stderrBuffer = `` - const port = await getAvailablePort() - - const spawnProcess = async (): Promise => { - const runtimeVarEntries = [ - [ - `PERSISTENCE_WITH_SYNC`, - options.syncEnabled !== undefined - ? String(options.syncEnabled) - : undefined, - ], - [ - `PERSISTENCE_SCHEMA_VERSION`, - options.schemaVersion !== undefined - ? String(options.schemaVersion) - : undefined, - ], - [`PERSISTENCE_COLLECTION_ID`, options.collectionId], - ].filter((entry): entry is [string, string] => entry[1] !== undefined) - - const wranglerArgs = [ - `dev`, - `--local`, - `--ip`, - `127.0.0.1`, - `--port`, - String(port), - `--persist-to`, - options.persistPath, - `--config`, - wranglerConfigPath, - ...runtimeVarEntries.flatMap(([key, value]) => [ - `--var`, - `${key}:${value}`, - ]), - ] - - child = spawn(process.execPath, [wranglerBinPath, ...wranglerArgs], { - cwd: packageDirectory, - env: { - ...process.env, - CI: `1`, - WRANGLER_SEND_METRICS: `false`, - }, - stdio: [`ignore`, `pipe`, `pipe`], - }) - - if (!child.stdout || !child.stderr) { - throw new Error(`Unable to capture wrangler dev process output streams`) - } - - child.stdout.on(`data`, (chunk: Buffer) => { - stdoutBuffer += chunk.toString() - }) - child.stderr.on(`data`, (chunk: Buffer) => { - stderrBuffer += chunk.toString() - }) - - const baseUrl = `http://127.0.0.1:${String(port)}` - const startAt = Date.now() - while (Date.now() - startAt < 45_000) { - if (child.exitCode !== null) { - throw createRuntimeError( - `Wrangler dev exited before becoming healthy`, - stderrBuffer, - stdoutBuffer, - ) - } - - try { - const healthResponse = await fetch(`${baseUrl}/health`) - if (healthResponse.ok) { - return - } - } catch { - // Runtime may still be starting. - } - - await delay(250) - } - - throw createRuntimeError( - `Timed out waiting for wrangler dev runtime`, - stderrBuffer, - stdoutBuffer, - ) - } - - await spawnProcess() - - return { - baseUrl: `http://127.0.0.1:${String(port)}`, - restart: async () => { - await stopWranglerProcess(child) - stdoutBuffer = `` - stderrBuffer = `` - await spawnProcess() - }, - stop: async () => { - await stopWranglerProcess(child) - }, - getOutput: () => ({ - stderr: stderrBuffer, - stdout: stdoutBuffer, - }), - } -} - -async function postJson( - runtime: RuntimeProcessHarness, - path: string, - body: unknown, -): Promise> { - let response: Response - try { - response = await fetch(`${runtime.baseUrl}${path}`, { - method: `POST`, - headers: { - 'content-type': `application/json`, - }, - body: JSON.stringify(body), - signal: AbortSignal.timeout(30_000), - }) - } catch (error) { - const output = runtime.getOutput() - throw createRuntimeError( - `Timed out waiting for wrangler dev runtime response: ${String(error)}`, - output.stderr, - output.stdout, - ) - } - - const parsed = (await response.json()) as WranglerRuntimeResponse - return parsed -} +import type { WranglerRuntimeResponse } from './fixtures/wrangler-runtime' function assertRuntimeError( response: WranglerRuntimeResponse, diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-persisted-demand-coverage.e2e.test.ts b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-persisted-demand-coverage.e2e.test.ts new file mode 100644 index 0000000000..6f394214d9 --- /dev/null +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/cloudflare-persisted-demand-coverage.e2e.test.ts @@ -0,0 +1,101 @@ +/** + * # Receiving witness: persisted demand coverage over Cloudflare Durable Objects + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * holds the histories and records the sessions. The law is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the Cloudflare host: it checks the + * same law through the Durable Object persistence + * (`createCloudflareDOSQLitePersistence`) inside a real workerd runtime started + * by `wrangler dev --local --persist-to`. The primary oracle does not run workerd. + * + * Histories: every history of `RECEIVING_HISTORIES`. Each history has its own + * wrangler persist directory. + * + * Production path: the test starts wrangler and posts session 1 to the Durable + * Object (`/demand-session`, phase `session1`). The Durable Object runs the + * contract's recorder in `e2e/fixtures/demand-coverage-session.mjs` over its + * own storage. The test stops wrangler, starts it again over the same persist + * directory, and posts session 2 (phase `session2`). + * + * Observations: the session 1 source log, and the session 2 loads and two + * checkpoints, read inside the Durable Object. + * + * Limits: local workerd with the Durable Object's SQLite storage, one Durable + * Object name, and a process restart between the sessions. It does not cover + * deployed Cloudflare, the browser or React adapter, or multi-object + * coordination. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' +import { + RECEIVING_HISTORIES, + coverageRecords, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import { postJson, startWranglerRuntime } from './fixtures/wrangler-runtime' +import type { + Session1Record, + Session2Record, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' + +describe(`cloudflare durable object persisted demand coverage (wrangler local, receiving witness)`, () => { + it.each(RECEIVING_HISTORIES)( + `$description`, + async (history) => { + const tempDirectory = mkdtempSync(join(tmpdir(), `db-cloudflare-demand-`)) + const persistPath = join(tempDirectory, `wrangler-state`) + try { + // Session 1 is online. It runs in one wrangler process, then stops. + const first = await startWranglerRuntime({ persistPath }) + let session1Record: Session1Record | undefined + try { + const session1 = await postJson<{ record: Session1Record }>( + first, + `/demand-session`, + { phase: `session1`, history: history.id }, + ) + if (!session1.ok) { + throw new Error(`${session1.error.name}: ${session1.error.message}`) + } + session1Record = session1.rows?.record + } finally { + await first.stop() + } + + // Session 2 is a later process over the same persist directory. + const second = await startWranglerRuntime({ persistPath }) + try { + const session2 = await postJson<{ record: Session2Record }>( + second, + `/demand-session`, + { phase: `session2`, history: history.id }, + ) + if (!session2.ok) { + throw new Error(`${session2.error.name}: ${session2.error.message}`) + } + const session2Record = session2.rows?.record + // Positive witness: session 2 asked the offline source. + expect(session2Record?.sent.length ?? 0).toBeGreaterThan(0) + expect(session1Record).toBeDefined() + const records = coverageRecords( + history, + session1Record as Session1Record, + session2Record as Session2Record, + ) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + } finally { + await second.stop() + } + } finally { + rmSync(tempDirectory, { recursive: true, force: true }) + } + }, + 90_000, + ) +}) diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/demand-coverage-session.mjs b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/demand-coverage-session.mjs new file mode 100644 index 0000000000..de74f20aa3 --- /dev/null +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/demand-coverage-session.mjs @@ -0,0 +1,35 @@ +// @ts-nocheck +/** + * Durable Object side of the persisted demand coverage witness + * (`e2e/cloudflare-persisted-demand-coverage.e2e.test.ts`). The worker runs + * one session per request, against the Durable Object's own storage, through + * the real `createCloudflareDOSQLitePersistence`. The test restarts wrangler + * between session 1 and session 2, so session 2 reads what workerd persisted. + * The histories and the session runners are the receiving contract's + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * imported here into the worker bundle. This module records what it observes; + * the test owns the judgment, through the shared judge once it is published. + */ +import { + receivingHistory, + recordSession1, + recordSession2, +} from '../../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts' +import { createCloudflareDOSQLitePersistence } from '../../dist/esm/index.js' + +// Runs one session. `phase` is `session1` (online, takes the history's steps) +// or `session2` (offline, returns the record). Errors reject to the caller. +export async function runDemandCoverageSession(storage, body) { + const persistence = createCloudflareDOSQLitePersistence({ storage }) + const history = receivingHistory(body.history) + if (body.phase === `session1`) { + return { + phase: `session1`, + record: await recordSession1(persistence, history), + } + } + return { + phase: `session2`, + record: await recordSession2(persistence, history), + } +} diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/worker.mjs b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/worker.mjs index b401358b43..7cd1e550ba 100644 --- a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/worker.mjs +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/worker.mjs @@ -9,6 +9,7 @@ import { createCloudflareDOSQLitePersistence, persistedCollectionOptions, } from '../../dist/esm/index.js' +import { runDemandCoverageSession } from './demand-coverage-session.mjs' const DEFAULT_COLLECTION_ID = `todos` const DEFAULT_SCHEMA_VERSION = 1 @@ -170,6 +171,13 @@ export class PersistenceObject extends DurableObject { const requestBody = await request.json() const collectionId = requestBody.collectionId ?? this.collectionId + if (request.method === `POST` && url.pathname === `/demand-session`) { + return jsonResponse(200, { + ok: true, + rows: await runDemandCoverageSession(this.ctx.storage, requestBody), + }) + } + if (request.method === `POST` && url.pathname === `/write-todo`) { if (collectionId !== this.collectionId) { throw createUnknownCollectionError(collectionId) diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/wrangler-runtime.ts b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/wrangler-runtime.ts new file mode 100644 index 0000000000..b439440d59 --- /dev/null +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/e2e/fixtures/wrangler-runtime.ts @@ -0,0 +1,238 @@ +import { spawn } from 'node:child_process' +import { createRequire } from 'node:module' +import { dirname, join } from 'node:path' +import { setTimeout as delay } from 'node:timers/promises' +import { fileURLToPath } from 'node:url' +import type { RuntimeBridgeE2EContractError } from '../../../db-sqlite-persistence-core/tests/contracts/runtime-bridge-e2e-contract' + +/** + * Wrangler dev process helpers shared by the Cloudflare Durable Object e2e + * witnesses. `startWranglerRuntime` runs the fixture worker in local workerd + * with `--persist-to`, so a runtime that is stopped and started again over + * the same directory reads the same Durable Object storage. + */ +export type RuntimeProcessHarness = { + baseUrl: string + restart: () => Promise + stop: () => Promise + getOutput: () => { + stderr: string + stdout: string + } +} + +export type WranglerRuntimeResponse = + | { + ok: true + rows?: TPayload + } + | { + ok: false + error: RuntimeBridgeE2EContractError + } + +const packageDirectory = dirname(fileURLToPath(import.meta.url)) +const wranglerConfigPath = join(packageDirectory, `wrangler.toml`) +const require = createRequire(import.meta.url) +const wranglerBinPath = require.resolve(`wrangler/bin/wrangler.js`) + +async function getAvailablePort(): Promise { + const netModule = await import('node:net') + return new Promise((resolve, reject) => { + const server = netModule.createServer() + server.listen(0, `127.0.0.1`, () => { + const address = server.address() + if (!address || typeof address === `string`) { + server.close() + reject(new Error(`Unable to allocate an available local port`)) + return + } + const selectedPort = address.port + server.close((error) => { + if (error) { + reject(error) + return + } + resolve(selectedPort) + }) + }) + server.on(`error`, reject) + }) +} + +function createRuntimeError( + message: string, + stderr: string, + stdout: string, +): Error { + return new Error([message, `stderr=${stderr}`, `stdout=${stdout}`].join(`\n`)) +} + +async function stopWranglerProcess( + child: ReturnType | undefined, +): Promise { + if (!child || child.exitCode !== null) { + return + } + + child.kill(`SIGTERM`) + const closed = await Promise.race([ + new Promise((resolve) => { + child.once(`close`, () => resolve(true)) + }), + delay(5_000).then(() => false), + ]) + + if (closed) { + return + } + + child.kill(`SIGKILL`) + await new Promise((resolve) => { + child.once(`close`, () => resolve()) + }) +} + +export async function startWranglerRuntime(options: { + persistPath: string + syncEnabled?: boolean + schemaVersion?: number + collectionId?: string +}): Promise { + let child: ReturnType | undefined + let stdoutBuffer = `` + let stderrBuffer = `` + const port = await getAvailablePort() + + const spawnProcess = async (): Promise => { + const runtimeVarEntries = [ + [ + `PERSISTENCE_WITH_SYNC`, + options.syncEnabled !== undefined + ? String(options.syncEnabled) + : undefined, + ], + [ + `PERSISTENCE_SCHEMA_VERSION`, + options.schemaVersion !== undefined + ? String(options.schemaVersion) + : undefined, + ], + [`PERSISTENCE_COLLECTION_ID`, options.collectionId], + ].filter((entry): entry is [string, string] => entry[1] !== undefined) + + const wranglerArgs = [ + `dev`, + `--local`, + `--ip`, + `127.0.0.1`, + `--port`, + String(port), + `--persist-to`, + options.persistPath, + `--config`, + wranglerConfigPath, + ...runtimeVarEntries.flatMap(([key, value]) => [ + `--var`, + `${key}:${value}`, + ]), + ] + + child = spawn(process.execPath, [wranglerBinPath, ...wranglerArgs], { + cwd: packageDirectory, + env: { + ...process.env, + CI: `1`, + WRANGLER_SEND_METRICS: `false`, + }, + stdio: [`ignore`, `pipe`, `pipe`], + }) + + if (!child.stdout || !child.stderr) { + throw new Error(`Unable to capture wrangler dev process output streams`) + } + + child.stdout.on(`data`, (chunk: Buffer) => { + stdoutBuffer += chunk.toString() + }) + child.stderr.on(`data`, (chunk: Buffer) => { + stderrBuffer += chunk.toString() + }) + + const baseUrl = `http://127.0.0.1:${String(port)}` + const startAt = Date.now() + while (Date.now() - startAt < 45_000) { + if (child.exitCode !== null) { + throw createRuntimeError( + `Wrangler dev exited before becoming healthy`, + stderrBuffer, + stdoutBuffer, + ) + } + + try { + const healthResponse = await fetch(`${baseUrl}/health`) + if (healthResponse.ok) { + return + } + } catch { + // Runtime may still be starting. + } + + await delay(250) + } + + throw createRuntimeError( + `Timed out waiting for wrangler dev runtime`, + stderrBuffer, + stdoutBuffer, + ) + } + + await spawnProcess() + + return { + baseUrl: `http://127.0.0.1:${String(port)}`, + restart: async () => { + await stopWranglerProcess(child) + stdoutBuffer = `` + stderrBuffer = `` + await spawnProcess() + }, + stop: async () => { + await stopWranglerProcess(child) + }, + getOutput: () => ({ + stderr: stderrBuffer, + stdout: stdoutBuffer, + }), + } +} + +export async function postJson( + runtime: RuntimeProcessHarness, + path: string, + body: unknown, +): Promise> { + let response: Response + try { + response = await fetch(`${runtime.baseUrl}${path}`, { + method: `POST`, + headers: { + 'content-type': `application/json`, + }, + body: JSON.stringify(body), + signal: AbortSignal.timeout(30_000), + }) + } catch (error) { + const output = runtime.getOutput() + throw createRuntimeError( + `Timed out waiting for wrangler dev runtime response: ${String(error)}`, + output.stderr, + output.stdout, + ) + } + + const parsed = (await response.json()) as WranglerRuntimeResponse + return parsed +} diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 239a64af79..d5f170b0f5 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -812,7 +812,10 @@ class PersistedReadinessTracker { private snapshot: PersistedReadinessSnapshot = { status: `loading` } private readonly listeners = new Set<() => void>() - constructor(readonly networkTimeoutMs: number) {} + constructor( + readonly networkTimeoutMs: number, + readonly onDemand: boolean, + ) {} getSnapshot = (): PersistedReadinessSnapshot => this.snapshot @@ -840,6 +843,20 @@ export type PersistedInitialRenderOptions = { } const DEFAULT_NETWORK_TIMEOUT_MS = 3_000 +// Persisted demand coverage: a load of this exact demand key reached applied +// settlement with its rows durably committed in this cache generation. +const DEMAND_COVERAGE_PREFIX = `persisted:coverage:` +const demandCoverageKey = (options: LoadSubsetOptions): string => + `${DEMAND_COVERAGE_PREFIX}${getLoadSubsetDemandKey(options)}` +// A window covers exactly the rows it held (maintainer decision, +// 2026-10-26), so its coverage also records their keys, in any order. +const windowKeys = ( + options: LoadSubsetOptions, + rows: ReadonlyArray<{ key: string | number }>, +): string | undefined => + options.limit === undefined + ? undefined + : JSON.stringify(rows.map(({ key }) => key).sort()) const CACHE_CLAIM_RENEW_WINDOW_MS = 60_000 const MAX_TIMER_TIMEOUT_MS = 2_147_483_647 @@ -2218,6 +2235,17 @@ class PersistedCollectionRuntime< : undefined } + // A reset or a new lifecycle makes an in-flight load obsolete (law 6 in + // `ARCHITECTURE.md`), so its settlement cannot record coverage. + captureResetEpoch(): () => boolean { + const { lifecycleGeneration, resetSequence, sourceTruncateGeneration } = + this + return () => + lifecycleGeneration === this.lifecycleGeneration && + resetSequence === this.resetSequence && + sourceTruncateGeneration === this.sourceTruncateGeneration + } + getLifecycleGeneration(): number { return this.lifecycleGeneration } @@ -2947,6 +2975,7 @@ class PersistedCollectionRuntime< onHydrationStarted?: () => void, isDemandRetained?: () => boolean, ): Promise { + const startedAt = Date.now() this.throwIfTerminal() await this.ensureCacheClaimActive() // The caller can retire demand while claim renewal or rotation is awaited. @@ -3099,12 +3128,132 @@ class PersistedCollectionRuntime< } if (upstreamLoadSubset) { - try { - await upstreamLoadSubset(options) - } catch (error) { - this.noteUpstreamLoadFailure(options, error) - throw error + const upstream = (async () => { + try { + await upstreamLoadSubset(options) + } catch (error) { + this.noteUpstreamLoadFailure(options, error) + throw error + } + })() + await this.settleCoveredDemand(options, upstream, startedAt) + } + } + + // The persisted subset restore of this demand is visible. A covered demand + // of an opted-in on-demand Collection settles once the wait since the load + // started has passed or its upstream load has failed. The upstream load + // keeps running; its rows are ordinary source writes, and a later failure + // is reported but leaves the demand settled. + // + // A refetch starts a new acquisition that waits for its own rows + // (`ARCHITECTURE.md`, "Demand grouping and ownership"), so coverage never + // settles it. + private async settleCoveredDemand( + options: LoadSubsetOptions, + upstream: Promise, + startedAt: number, + ): Promise { + // The coverage read below can outlast a fast upstream failure. + void upstream.catch(() => undefined) + if ( + !this.persistedReadiness?.onDemand || + options.refetch === true || + !(await this.isDemandCovered(options)) + ) { + return upstream + } + // A released load owns no wait, even if its source ignores the abort. + if (options.signal?.aborted) return + const deadlineAt = startedAt + this.persistedReadiness.networkTimeoutMs + return new Promise((resolve) => { + const timer = setTimeout(resolve, Math.max(0, deadlineAt - Date.now())) + const settle = () => { + clearTimeout(timer) + resolve() + } + options.signal?.addEventListener(`abort`, settle, { once: true }) + void upstream.catch(() => undefined).then(settle) + }) + } + + // Coverage belongs to one persisted cache generation. A rotation changes + // the storage ID, so a record that memory still holds from the previous + // generation does not count. + private currentCoverageRecord( + options: LoadSubsetOptions, + ): { rows: number; keys?: string } | undefined { + const record = this.syncControls.metadata?.collection.get( + demandCoverageKey(options), + ) as { generation?: unknown; rows?: unknown; keys?: string } | undefined + return record?.generation === this.storageCollectionId && + typeof record.rows === `number` + ? { rows: record.rows, keys: record.keys } + : undefined + } + + // The rows that this demand reads from SQLite in the current generation. + // The read waits behind every source transaction already committed, so it + // sees their rows durably. Coverage is bookkeeping: a failed read proves + // nothing and returns undefined, and the load keeps its own outcome. A + // read under an expired cache claim also starts claim recovery. Scoped + // recovery leaves the rest of its sync run uncertified, and its persisted + // subset restore publishes no rows, so no read proves coverage then. + private async readDemandRows( + options: LoadSubsetOptions, + ): Promise | undefined> { + if (this.scopedRecovery) return undefined + try { + return await this.applyMutex.run(() => + this.persistence.adapter.loadSubset(this.storageTargetFor(), options, { + requiredIndexSignatures: this.getRequiredIndexSignatures(), + cacheGenerationClaimId: this.cacheGenerationClaim?.claimId, + }), + ) + } catch { + if (this.hasExpiredCacheClaim()) { + this.scheduleCacheClaimRecovery(this.lifecycleGeneration) } + return undefined + } + } + + // A row deleted after the covered load settled, by the source or by + // retention, leaves fewer rows than coverage recorded, so it no longer + // proves the demand complete (maintainer decision, 2026-10-26). A window + // must also hold the same rows. + private async isDemandCovered(options: LoadSubsetOptions): Promise { + const record = this.currentCoverageRecord(options) + if (record === undefined) return false + const rows = await this.readDemandRows(options) + return ( + rows !== undefined && + rows.length >= record.rows && + windowKeys(options, rows) === record.keys + ) + } + + // The coverage record for a load that just settled, or undefined when the + // current record already says the same. + async nextCoverageRecord(options: LoadSubsetOptions): Promise< + | { + settledAt: number + generation: string + rows: number + keys?: string + } + | undefined + > { + const rows = await this.readDemandRows(options) + if (rows === undefined) return undefined + const keys = windowKeys(options, rows) + const current = this.currentCoverageRecord(options) + if (current?.rows === rows.length && current.keys === keys) return undefined + return { + settledAt: Date.now(), + generation: this.storageCollectionId, + rows: rows.length, + keys, } } @@ -3137,14 +3286,18 @@ class PersistedCollectionRuntime< return true } - return result.then( - () => { - this.queueRemoteSubsetEnsure(options) - }, - (error: unknown) => { - this.noteUpstreamLoadFailure(options, error) - throw error - }, + return this.settleCoveredDemand( + options, + result.then( + () => { + this.queueRemoteSubsetEnsure(options) + }, + (error: unknown) => { + this.noteUpstreamLoadFailure(options, error) + throw error + }, + ), + Date.now(), ) } @@ -6343,9 +6496,18 @@ function createWrappedSyncConfig< // Network-first initial rendering: once the deadline passes, or the // remote fails, a completed restore that installed rows opens the // Collection. An empty restore keeps it loading, so an empty list never - // stands for data the network has not answered yet. + // stands for data the network has not answered yet. An on-demand + // Collection restores per demand instead: each covered load settles + // from SQLite after its own wait (`settleCoveredDemand`), and query + // readiness follows its loads. const openFromDisk = () => { - if (!persistedReadiness || isCleanedUp() || getTerminalFailure()) return + if ( + !persistedReadiness || + persistedReadiness.onDemand || + isCleanedUp() || + getTerminalFailure() + ) + return if (persistedReadiness.getSnapshot().status !== `ready`) return if (params.collection.status !== `loading`) return if (params.collection.size === 0) return @@ -6598,6 +6760,15 @@ function createWrappedSyncConfig< } const reportUpstreamError = (error: unknown) => afterStartup(() => failUpstream(error)) + // Persisted demand coverage ends with the rows it described, at a + // truncate and at a snapshot transaction, which replaces every row. + const endDemandCoverage = (transaction: OpenSyncTransaction) => { + for (const { key } of wrappedParams.metadata?.collection.list( + DEMAND_COVERAGE_PREFIX, + ) ?? []) { + transaction.collectionMetadataWrites.set(key, { type: `delete` }) + } + } const wrappedParams = { ...params, fail: reportUpstreamError, @@ -6643,6 +6814,7 @@ function createWrappedSyncConfig< queuedBecauseHydrating: false, ...(terminalFailure === undefined ? {} : { terminalFailure }), } + if (options?.snapshot) endDemandCoverage(transaction) transactionStack.push(transaction) }, write: (message: ChangeMessageOrDeleteKeyMessage) => { @@ -6861,6 +7033,7 @@ function createWrappedSyncConfig< // Callers (for example electric resume/reset handling) may stage // collection-scoped metadata before truncating row data, and those // writes must commit atomically with the truncate transaction. + endDemandCoverage(openTransaction) openTransaction.truncate = true openTransaction.truncateOptions = options ?? {} }, @@ -7096,6 +7269,30 @@ function createWrappedSyncConfig< } } + // The load's rows committed first; this metadata-only source + // transaction persists after them, so coverage never outlives a failed + // row write. + // Only a current load records coverage. A reset makes the record + // obsolete before its commit could be aborted, so a current record + // fails only with a durable failure that already fails the runtime. + // A load that core aborted decides nothing, even if its source + // resolves it. + const recordCoverage = async ( + demand: LoadSubsetOptions, + isCurrent: () => boolean, + ): Promise => { + const current = () => isCurrent() && !demand.signal?.aborted + if (!current()) return + const record = await runtime.nextCoverageRecord(demand) + if (!record || !current()) return + wrappedParams.begin() + wrappedParams.metadata?.collection.set( + demandCoverageKey(demand), + record, + ) + await wrappedParams.commit() + } + let sourceResult: SyncConfigRes = {} let sourceResultSettled = false let activeSourceSyncEntry: Promise | undefined @@ -7302,7 +7499,16 @@ function createWrappedSyncConfig< try { // Returning a promise transfers its lease even if it rejects. // Only a synchronous throw leaves no upstream lease to release. - return currentSource.loadSubset(loadOptions) + const loaded = currentSource.loadSubset(loadOptions) + if (!persistedReadiness?.onDemand) return loaded + const isCurrent = runtime.captureResetEpoch() + // A synchronous answer stays synchronous. Its coverage record + // follows its rows in commit order without holding the load. + if (loaded === true) { + void recordCoverage(options, isCurrent).catch(() => undefined) + return true + } + return loaded.then(() => recordCoverage(options, isCurrent)) } catch (error) { acquisition.forwarded = false acquisition.source = undefined @@ -7546,14 +7752,12 @@ export function persistedCollectionOptions< } const networkTimeoutMs = getNetworkTimeoutMs(options.initialRender) - if (networkTimeoutMs !== undefined && options.syncMode === `on-demand`) { - throw new InvalidPersistedCollectionConfigError( - `network-first initial rendering requires eager persistence syncMode`, - ) - } const persistedReadinessTracker = networkTimeoutMs !== undefined - ? new PersistedReadinessTracker(networkTimeoutMs) + ? new PersistedReadinessTracker( + networkTimeoutMs, + options.syncMode === `on-demand`, + ) : undefined let claimed = false diff --git a/packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts b/packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts new file mode 100644 index 0000000000..2a54548693 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts @@ -0,0 +1,507 @@ +/** + * # Receiving contract: persisted demand coverage + * + * Law (`docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * glossary entry "persisted demand coverage"): with + * `initialRender: { strategy: 'network-first', networkTimeoutMs }` on a + * persisted `syncMode: 'on-demand'` Collection, a demand settles from storage + * in a later offline session when all three hold: + * + * 1. the persisted subset restore has published its rows in the current sync + * run; + * 2. the exact demand has persisted demand coverage, recorded by an earlier + * online load of that demand in the same cache generation; + * 3. the configured wait has passed since the load started, or the demand's + * upstream load has failed. + * + * Authority: the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * and the campaign oracle `persisted-demand-coverage-campaign-oracle.test.ts` + * own the law. The judge that compares a record with the law is shared, and + * this module holds no expected result. + * + * ## What this module holds + * + * - `RECEIVING_HISTORIES`: each history as data. Session 1 is online and takes + * the listed steps. Session 2 setup is the wait and the upstream behavior + * (premise control, not an outcome). + * - `createReceivingSource`: the on-demand source. It records every row write, + * delete, and settled load into an event log, and it controls the upstream + * behavior the history asks for. + * - `recordSession1` and `recordSession2`: the production entry points. They + * run one session over a host's persistence and return what was recorded. + * They never judge the record. + * - `recordReceivingHistory`: both sessions in one process, for in-process + * hosts. + * + * Out-of-process hosts (the OPFS page, the Electron renderer, the Cloudflare + * Durable Object) import this module into their bundle, call the recorders, + * and report the records back. Their test files pass each record to the + * shared judge. + * + * This module is platform-neutral: it imports only `@tanstack/db` and the + * persistence core, and it uses no test framework. + * + * ## Production path and observations + * + * Each session builds a Collection with `persistedCollectionOptions` over the + * host's persistence and runs the query's `preload()` with the history's wait. + * Session 1 answers its loads through the source, so the rows in storage come + * only from the wrapper. + * + * Session 2 records two checkpoints of the query: `beforeDeadline` 450 ms + * after the upstream load is forwarded, and `afterDeadline` after polling up + * to 2 500 ms for the query to leave `loading`. Each checkpoint carries its + * elapsed time since the forward, so the judge can decide whether the wait had + * passed. + * + * ## Limits + * + * The law covers only the default single-process coordinator, so every host + * here uses it. A framework host observes its own signal (a hook's readiness + * or rendered output) and records only what that signal shows. The histories + * cover one filter and one ordered window. + */ +import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import { persistedCollectionOptions } from '../../src' +import { A1, A2, A3 } from './persisted-demand-coverage-receiving-histories' +import type { Collection, LoadSubsetOptions } from '@tanstack/db' +import type { PersistedCollectionPersistence } from '../../src' +import type { + CoverageRecord, + OfflineUpstream, + QueryObservation, + QuerySpec, + SourceEvent, +} from '../persisted-demand-coverage-model-oracle' +import type { + ReceivingHistory, + Row, + Session2Upstream, + Shape, +} from './persisted-demand-coverage-receiving-histories' + +// The before-wait checkpoint. It is under half the ordinary wait, so the wait +// has not passed there, and it is late enough for a framework to show a +// settlement that an upstream failure produced: React Suspense holds a fallback +// reveal for about 300 ms. +const CHECKPOINT_MS = 450 +const AFTER_DEADLINE_POLL_MS = 2500 +const FORWARD_TIMEOUT_MS = 10_000 +const COLLECTION_ID = `persisted-demand-coverage-receiver` + +export * from './persisted-demand-coverage-receiving-histories' +// The model types a receiving driver records with. +export type { + CoverageRecord, + QueryObservation, + SourceEvent, +} from '../persisted-demand-coverage-model-oracle' + +export type UpstreamBehavior = `hold` | `reject` | `answer-at-once` + +export type ReceivingSource = { + collection: Collection + // Loads the source received, in arrival order. + received: Array + pending: () => number + // Writes rows in one sync transaction, then settles every held load. + answer: (rows: ReadonlyArray) => Promise + write: (rows: ReadonlyArray) => Promise + remove: (rows: ReadonlyArray) => Promise +} + +// One checkpoint of the offline query, and the milliseconds since its load +// started (the `preload()` call). +export type ReceivingCheckpoint = { + observed: QueryObservation + sinceLoadStartMs: number +} + +export const QUERY_NAME = `demand` + +export type Session1Record = { events: ReadonlyArray } + +export type Session2Record = { + // The loads the offline source received for the query, in arrival order. + sent: ReadonlyArray + beforeDeadline: ReceivingCheckpoint + afterDeadline: ReceivingCheckpoint +} + +export type ReceivingSession = { + persistence: PersistedCollectionPersistence + close: () => Promise +} + +export const sleep = (ms: number) => + new Promise((resolve) => setTimeout(resolve, ms)) + +export async function until( + predicate: () => boolean, + timeoutMs: number, +): Promise { + const deadline = Date.now() + timeoutMs + while (!predicate()) { + if (Date.now() > deadline) return false + await sleep(10) + } + return true +} + +/** + * The upstream behavior a session 2 history asks for. `hold` keeps each load + * pending, as an offline network does. `reject` rejects it at once, as a + * fetch does when the device is offline. + */ +export function sourceBehaviorFor( + upstream: Session2Upstream, +): UpstreamBehavior { + return upstream === `never-answers` ? `hold` : `reject` +} + +/** + * An on-demand source whose upstream the host controls. Every row write, + * delete, and settled load goes into `events`, so the judge can replay what + * the source did. `answer-at-once` writes `answerRows` and resolves, for a + * framework session that renders through its own hook. + */ +export function createReceivingSource(options: { + persistence: PersistedCollectionPersistence + waitMs: number + behavior: UpstreamBehavior + events: Array + answerRows?: ReadonlyArray +}): ReceivingSource { + const received: Array = [] + const pending: Array<{ load: LoadSubsetOptions; resolve: () => void }> = [] + let apply: + | ((type: `insert` | `delete`, rows: ReadonlyArray) => unknown) + | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: COLLECTION_ID, + getKey: (row) => row.id, + syncMode: `on-demand`, + initialRender: { + strategy: `network-first`, + networkTimeoutMs: options.waitMs, + }, + persistence: options.persistence, + sync: { + sync: (params) => { + apply = (type, rows) => { + params.begin() + for (const value of rows) params.write({ type, value }) + for (const row of rows) { + options.events.push( + type === `insert` + ? { + type: `rows`, + rows: [[row.id, { group: row.group, rank: row.rank }]], + } + : { type: `delete`, id: row.id }, + ) + } + return params.commit() + } + return { + loadSubset: (load: LoadSubsetOptions) => { + received.push(plainLoadSnapshot(load)) + if (options.behavior === `reject`) { + return Promise.reject(new Error(`upstream offline`)) + } + if (options.behavior === `answer-at-once`) { + return Promise.resolve( + apply?.(`insert`, options.answerRows ?? []), + ).then(() => { + options.events.push({ + type: `settled`, + load: plainLoadSnapshot(load), + current: true, + }) + }) + } + return new Promise((resolve) => { + pending.push({ load, resolve }) + }) + }, + } + }, + }, + }), + ) + return { + collection, + received, + pending: () => pending.length, + answer: async (rows) => { + await apply?.(`insert`, rows) + for (const { load, resolve } of pending.splice(0)) { + options.events.push({ + type: `settled`, + load: plainLoadSnapshot(load), + current: true, + }) + resolve() + } + }, + write: async (rows) => { + await apply?.(`insert`, rows) + }, + remove: async (rows) => { + await apply?.(`delete`, rows) + }, + } +} + +// The query's public signal, in the judge's vocabulary. A status the judge +// does not name is a driver error, so it throws rather than being guessed. +export function observeQuery( + query: ReturnType, +): QueryObservation { + if (query.status === `ready`) { + return { status: `ready`, rows: rowIds(query.toArray) } + } + if (query.status === `error` || query.status === `loading`) { + return { status: query.status } + } + throw new Error(`unexpected query status ${query.status}`) +} + +export function queryFor(shape: Shape, source: Collection) { + return shape === `filter` + ? createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => eq(row.group, `a`)), + ) + : createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.group, `a`)) + .orderBy(({ row }) => row.rank) + .limit(2), + ) +} + +/** The ids of rows, sorted: the only row form the record keeps. */ +export function rowIds(rows: ReadonlyArray<{ id: string }>): Array { + return rows.map((row) => row.id).sort() +} + +// Session 1 is online. Each load is answered by the source, so the rows in +// storage come only from the wrapper. +async function loadAndAnswer( + source: ReceivingSource, + query: ReturnType, + rows: ReadonlyArray, +): Promise { + const preloaded = query.preload() + const state = { settled: false } + void preloaded.then( + () => (state.settled = true), + () => (state.settled = true), + ) + for (let i = 0; i < 100 && !state.settled; i++) { + if (source.pending() > 0) await source.answer(rows) + await sleep(10) + } + await preloaded + // Premise witness: session 1 really loaded the demand online, so the + // history has coverage to test. This is not a verdict on the law. + const held = rowIds(query.toArray).join(`,`) + if (held !== `a1,a2`) { + throw new Error(`session 1 held ${held}, expected a1,a2`) + } +} + +/** + * Runs session 1 of a history over `persistence` and returns its source log. + * The caller owns the persistence and closes it afterwards. + */ +export async function recordSession1( + persistence: PersistedCollectionPersistence, + history: ReceivingHistory, +): Promise { + const events: Array = [] + const source = createReceivingSource({ + persistence, + waitMs: 0, + behavior: `hold`, + events, + }) + const query = queryFor(history.shape, source.collection) + const loadRows = history.shape === `filter` ? [A1, A2] : [A1, A2, A3] + try { + for (const step of history.session1) { + if (step.kind === `load`) await loadAndAnswer(source, query, loadRows) + if (step.kind === `remove`) await source.remove([step.row]) + if (step.kind === `write`) await source.write([step.row]) + } + await sleep(50) + return { events } + } finally { + await query.cleanup() + await source.collection.cleanup() + } +} + +/** + * A transport-safe copy of a load, for hosts that send records out of their + * bundle (a page, a worker, or a renderer). The IR can hold cycles, which JSON + * rejects, so the copy keeps each object once per branch and drops a back + * reference. The demand key and the judge read only the fields the copy keeps. + */ +export function plainLoadSnapshot( + value: T, + ancestors = new Set(), +): T { + if (value === null || typeof value !== `object`) return value + if (ancestors.has(value)) return undefined as T + ancestors.add(value) + const copy = Array.isArray(value) + ? value.map((item) => plainLoadSnapshot(item, ancestors)) + : Object.fromEntries( + Object.entries(value) + .filter(([, item]) => typeof item !== `function`) + .map(([key, item]) => [key, plainLoadSnapshot(item, ancestors)]), + ) + ancestors.delete(value) + return copy as T +} + +/** + * The production configuration of a session 2 source: the wait and the + * upstream behavior the history asks for. Every host builds its session 2 + * source from this, so the configuration is changed in one place. + */ +export function session2SourceOptions(history: ReceivingHistory): { + waitMs: number + behavior: UpstreamBehavior +} { + return { + waitMs: history.session2.waitMs, + behavior: sourceBehaviorFor(history.session2.upstream), + } +} + +/** + * Records the checkpoints of a session 2 query. `observe` reads the host's + * public signal: the query status and rows for a query host, or the rendered + * output for a framework. `forwarded` counts the loads the source received. + * Each checkpoint carries the milliseconds since `loadStartedAt`, the moment + * the query's load began. + */ +export async function recordCheckpoints( + observe: () => QueryObservation, + forwarded: () => number, + loadStartedAt: number, +): Promise<{ + beforeDeadline: ReceivingCheckpoint + afterDeadline: ReceivingCheckpoint +}> { + if (!(await until(() => forwarded() > 0, FORWARD_TIMEOUT_MS))) { + throw new Error(`session 2 never forwarded an upstream load`) + } + const take = (): ReceivingCheckpoint => ({ + observed: observe(), + sinceLoadStartMs: Date.now() - loadStartedAt, + }) + await sleep(CHECKPOINT_MS) + const beforeDeadline = take() + await until(() => observe().status !== `loading`, AFTER_DEADLINE_POLL_MS) + const afterDeadline = take() + return { beforeDeadline, afterDeadline } +} + +/** Runs session 2 of a history over `persistence` and returns its record. */ +export async function recordSession2( + persistence: PersistedCollectionPersistence, + history: ReceivingHistory, +): Promise { + const source = createReceivingSource({ + persistence, + ...session2SourceOptions(history), + events: [], + }) + const query = queryFor(history.shape, source.collection) + try { + const loadStartedAt = Date.now() + void query.preload().catch(() => undefined) + const checkpoints = await recordCheckpoints( + () => observeQuery(query), + () => source.received.length, + loadStartedAt, + ) + return { sent: [...source.received], ...checkpoints } + } finally { + await query.cleanup() + await source.collection.cleanup() + } +} + +/** + * Runs both sessions in one process, each over its own session from + * `openSession`. Session 1 closes before session 2 opens, so session 2 reads + * what session 1 wrote through storage alone. + */ +export async function recordReceivingHistory( + history: ReceivingHistory, + openSession: () => Promise, +): Promise<{ session1: Session1Record; session2: Session2Record }> { + const first = await openSession() + let session1: Session1Record + try { + session1 = await recordSession1(first.persistence, history) + } finally { + await first.close() + } + const second = await openSession() + try { + return { + session1, + session2: await recordSession2(second.persistence, history), + } + } finally { + await second.close() + } +} + +/** The query the history's session 2 reads, as the judge's `QuerySpec`. */ +export function querySpecFor(history: ReceivingHistory): QuerySpec { + return history.shape === `filter` + ? { group: `a` } + : { group: `a`, orderByRank: true, limit: 2 } +} + +const UPSTREAM: Record = { + 'never-answers': `never answers`, + 'fails-at-once': `fails at once`, +} + +/** + * The judge's records for a history: one per session 2 checkpoint. The + * premise `waitPassed` is the history's declared wait measured from the load + * start, so it does not depend on the production configuration. A host run + * with a different wait is then judged against the declared law. + */ +export function coverageRecords( + history: ReceivingHistory, + session1: Session1Record, + session2: Session2Record, +): { beforeDeadline: CoverageRecord; afterDeadline: CoverageRecord } { + const build = (checkpoint: ReceivingCheckpoint): CoverageRecord => ({ + events: session1.events, + queries: { + [QUERY_NAME]: { + spec: querySpecFor(history), + sent: session2.sent, + observed: checkpoint.observed, + }, + }, + upstream: UPSTREAM[history.session2.upstream], + waitPassed: checkpoint.sinceLoadStartMs >= history.session2.waitMs, + }) + return { + beforeDeadline: build(session2.beforeDeadline), + afterDeadline: build(session2.afterDeadline), + } +} diff --git a/packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-histories.ts b/packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-histories.ts new file mode 100644 index 0000000000..2d9c5f9722 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-histories.ts @@ -0,0 +1,109 @@ +/** + * The receiving histories as plain data: the rows, the session 1 steps, and the + * session 2 setup of each history. Playwright loads this module, and it must + * not import `@tanstack/db` (its tsconfig path reaches the TypeScript source, + * which Playwright cannot transform), so it imports nothing at all. + * + * `persisted-demand-coverage-receiving-contract.ts` re-exports everything here + * and holds the runners. The histories carry no expected results. The shared + * judge (`persisted-demand-coverage-model-oracle.ts`) judges the records. + */ + +export type Row = { id: string; group: string; rank: number; value: string } +export type Shape = `filter` | `window` + +export const A1: Row = { id: `a1`, group: `a`, rank: 1, value: `v1` } +export const A2: Row = { id: `a2`, group: `a`, rank: 2, value: `v1` } +export const A3: Row = { id: `a3`, group: `a`, rank: 3, value: `v1` } +export const A15: Row = { id: `a15`, group: `a`, rank: 1.5, value: `v1` } + +// The wait of an ordinary session 2 (premise). +export const SESSION_2_WAIT_MS = 1000 +// The wait of the upstream-failure history (premise). +export const FAILURE_WAIT_MS = 60_000 + +export type Session1Step = + { kind: `load` } | { kind: `remove`; row: Row } | { kind: `write`; row: Row } + +export type Session2Upstream = `never-answers` | `fails-at-once` + +export type ReceivingHistory = { + id: string + description: string + shape: Shape + session1: ReadonlyArray + session2: { waitMs: number; upstream: Session2Upstream } +} + +const LOAD: Session1Step = { kind: `load` } + +export const RECEIVING_HISTORIES: ReadonlyArray = [ + { + id: `covered`, + description: `covered demand settles offline with its restored rows`, + shape: `filter`, + // Session 1 loads D and the source answers a1 and a2. + session1: [LOAD], + session2: { waitMs: SESSION_2_WAIT_MS, upstream: `never-answers` }, + }, + { + id: `never`, + description: `never-loaded demand stays loading offline`, + shape: `filter`, + // Session 1 never asks for D. + session1: [], + session2: { waitMs: SESSION_2_WAIT_MS, upstream: `never-answers` }, + }, + { + id: `deleted`, + description: `covered demand stays loading after a source delete of one row`, + shape: `filter`, + // Session 1 loads D, then the source deletes a1. + session1: [LOAD, { kind: `remove`, row: A1 }], + session2: { waitMs: SESSION_2_WAIT_MS, upstream: `never-answers` }, + }, + { + id: `window-covered`, + description: `covered ordered window settles offline`, + shape: `window`, + // Session 1 loads `orderBy rank limit 2` with a1, a2, a3 in the source. + session1: [LOAD], + session2: { waitMs: SESSION_2_WAIT_MS, upstream: `never-answers` }, + }, + { + id: `window-replaced`, + description: `covered ordered window stays loading after a row replaced below the boundary`, + shape: `window`, + // After the load, the source deletes a1 and inserts a15 at rank 1.5. + session1: [LOAD, { kind: `remove`, row: A1 }, { kind: `write`, row: A15 }], + session2: { waitMs: SESSION_2_WAIT_MS, upstream: `never-answers` }, + }, + { + id: `covered-upstream-fails-at-once`, + description: `covered demand settles before the wait when its upstream load fails at once`, + shape: `filter`, + // As `covered`, but session 2's upstream load rejects as soon as it is + // forwarded, and the wait is long enough to be distinguished from it. + session1: [LOAD], + session2: { waitMs: FAILURE_WAIT_MS, upstream: `fails-at-once` }, + }, +] + +/** + * The histories a framework host runs. A hook observes readiness and rows + * only, so it runs the filter histories without source removals. + */ +export const FRAMEWORK_HISTORY_IDS = [ + `covered`, + `never`, + `covered-upstream-fails-at-once`, +] as const + +/** Looks up a history by its protocol id, for out-of-process hosts. */ +export function receivingHistory(id: string): ReceivingHistory { + const history = RECEIVING_HISTORIES.find((candidate) => candidate.id === id) + if (!history) { + throw new Error(`unknown persisted demand coverage history "${id}"`) + } + return history +} diff --git a/packages/db-sqlite-persistence-core/tests/helpers/demand-coverage-driver.ts b/packages/db-sqlite-persistence-core/tests/helpers/demand-coverage-driver.ts new file mode 100644 index 0000000000..861b219e3e --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/helpers/demand-coverage-driver.ts @@ -0,0 +1,330 @@ +import { vi } from 'vitest' +import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import { persistedCollectionOptions } from '../../src/index.js' +import type { Collection, LoadSubsetOptions } from '@tanstack/db' +import type { + SQLiteCorePersistenceAdapter, + SQLiteDriver, +} from '../../src/index.js' +import type { SourceEvent } from '../persisted-demand-coverage-model-oracle.js' + +/** + * The production driver shared by the persisted demand coverage oracle + * (`tests/persisted-demand-coverage-oracle.test.ts`) and its generated + * campaign (`tests/persisted-demand-coverage-campaign-oracle.test.ts`). It + * drives the real persisted wrapper over a generic on-demand source and + * records the source's own log for the shared judgment; it holds no model and + * no expected result. + */ + +export type Row = { id: string; group: string; rank: number; value: string } + +// Session 2 reads through this wrapper, which answers each read on a later +// macrotask, as a worker-backed or OPFS driver does. A driver that answers +// within one microtask turn hides any work that happens between the +// wrapper's own awaits, such as an upstream failure during a coverage read. +// `setImmediate` stays real under the fake clock, so the drains below yield +// to it. +export function withMacrotaskReads(driver: SQLiteDriver): SQLiteDriver { + return { + ...driver, + query: (sql, params) => + new Promise((resolve) => setImmediate(resolve)).then(() => + driver.query(sql, params), + ), + } +} + +export const realMacrotask = () => + new Promise((resolve) => setImmediate(resolve)) + +// ORC-010: run a check, then every cleanup. A cleanup failure never replaces +// the primary failure; both survive in one AggregateError. +export async function withCleanup( + check: () => Promise, + cleanups: ReadonlyArray<() => unknown>, +): Promise { + let result: T | undefined + let primary: { error: unknown } | undefined + try { + result = await check() + } catch (error) { + primary = { error } + } + const cleanupFailures: Array = [] + for (const cleanup of cleanups) { + try { + await cleanup() + } catch (error) { + cleanupFailures.push(error) + } + } + if (primary) { + if (cleanupFailures.length === 0) throw primary.error + throw new AggregateError( + [primary.error, ...cleanupFailures], + `Demand-coverage check and cleanup both failed`, + { cause: primary.error }, + ) + } + if (cleanupFailures.length === 1) throw cleanupFailures[0] + if (cleanupFailures.length > 1) { + throw new AggregateError(cleanupFailures, `Demand-coverage cleanup failed`) + } + return result as T +} + +export function useFakeClock(): void { + vi.useFakeTimers({ + toFake: [ + `setTimeout`, + `clearTimeout`, + `setInterval`, + `clearInterval`, + `Date`, + ], + }) +} + +export async function drain(): Promise { + for (let i = 0; i < 25; i++) { + await vi.advanceTimersByTimeAsync(0) + await realMacrotask() + } +} + +export type Upstream = { + calls: Array + // The row IDs the Collection showed when each load was forwarded. + visibleAtCall: Array> + pendingCount: () => number + answer: (rows: ReadonlyArray) => Promise + fail: (error: unknown) => void + truncate: () => Promise + // Commits a snapshot transaction of these rows: it replaces every row. + snapshot: (rows: ReadonlyArray) => Promise + // When set, each later load rejects as soon as it is called. + rejectAtOnce: boolean + // When set, the next load truncates synchronously inside the source's + // `loadSubset` call, after the source received the load. + truncateInNextLoad: boolean + // Writes rows without settling any load. + write: (rows: ReadonlyArray) => Promise + // Settles only the oldest pending load, without writing rows. + settleOldest: () => void + startScopedRecovery: () => Promise + // Deletes rows in one sync transaction. + remove: (rows: ReadonlyArray) => Promise + // Updates rows in one sync transaction. + update: (rows: ReadonlyArray) => Promise + // The source's own log of what it did, for the shared judgment in + // `tests/persisted-demand-coverage-model-oracle.ts`. Each source is one + // sync run. A load is current when no truncate, snapshot, or generation + // change came between the source receiving it and settling it, and aborted + // when core aborted its signal before the source settled it. + events: Array + // When the source received each load (`Date.now()`, so the fake clock + // where a history installs one). + arrivedAt: Map +} + +const rowEvent = (rows: ReadonlyArray): SourceEvent => ({ + type: `rows`, + rows: rows.map((row) => [row.id, { group: row.group, rank: row.rank }]), +}) + +// A generic on-demand source. Each load waits until the test answers or +// fails it. An answer writes the rows in one sync transaction and settles +// every pending load after its applied receipt. +export function onDemandSource( + adapter: SQLiteCorePersistenceAdapter, + waitMs: number, + extra: { onInsert?: () => Promise } = {}, +): { collection: Collection; upstream: Upstream } { + const pending: Array<{ + resolve: () => void + reject: (error: unknown) => void + options: LoadSubsetOptions + epoch: number + }> = [] + // How many truncates and generation changes the source has seen. + let epoch = 0 + const settle = (load: (typeof pending)[number]) => { + upstream.events.push({ + type: `settled`, + load: load.options, + current: load.epoch === epoch, + ...(load.options.signal?.aborted ? { aborted: true } : {}), + }) + load.resolve() + } + let write: ((rows: ReadonlyArray) => unknown) | undefined + let truncate: (() => unknown) | undefined + let snapshot: ((rows: ReadonlyArray) => unknown) | undefined + let startScopedRecovery: (() => Promise) | undefined + let remove: ((rows: ReadonlyArray) => unknown) | undefined + let update: ((rows: ReadonlyArray) => unknown) | undefined + const upstream: Upstream = { + calls: [], + events: [{ type: `sync run starts` }], + arrivedAt: new Map(), + visibleAtCall: [], + rejectAtOnce: false, + truncateInNextLoad: false, + pendingCount: () => pending.length, + answer: async (rows) => { + await write?.(rows) + upstream.events.push(rowEvent(rows)) + for (const load of pending.splice(0)) settle(load) + }, + fail: (error) => { + for (const load of pending.splice(0)) { + upstream.events.push({ type: `failed`, load: load.options }) + load.reject(error) + } + }, + truncate: async () => { + epoch++ + await truncate?.() + upstream.events.push({ type: `truncate` }) + }, + snapshot: async (rows) => { + epoch++ + await snapshot?.(rows) + upstream.events.push({ + type: `snapshot`, + rows: rows.map((row) => [row.id, { group: row.group, rank: row.rank }]), + }) + }, + write: async (rows) => { + await write?.(rows) + upstream.events.push(rowEvent(rows)) + }, + settleOldest: () => { + const load = pending.shift() + if (load) settle(load) + }, + // Production starts recovery synchronously, so the log records it first. + startScopedRecovery: async () => { + epoch++ + // Premise: the sync run is active, so the recovery really starts. + if (!startScopedRecovery) { + throw new Error(`scoped recovery needs an active sync run`) + } + upstream.events.push({ type: `scoped recovery starts` }) + await startScopedRecovery() + }, + remove: async (rows) => { + await remove?.(rows) + for (const row of rows) + upstream.events.push({ type: `delete`, id: row.id }) + }, + update: async (rows) => { + await update?.(rows) + upstream.events.push(rowEvent(rows)) + }, + } + const collection = createCollection( + persistedCollectionOptions({ + id: `demand-coverage-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + initialRender: { strategy: `network-first`, networkTimeoutMs: waitMs }, + persistence: { adapter }, + ...(extra.onInsert ? { onInsert: extra.onInsert } : {}), + sync: { + sync: (params) => { + write = (rows) => { + params.begin() + for (const value of rows) params.write({ type: `insert`, value }) + return params.commit() + } + truncate = () => { + params.begin() + params.truncate() + return params.commit() + } + snapshot = (rows) => { + params.begin({ snapshot: true }) + for (const value of rows) params.write({ type: `insert`, value }) + return params.commit() + } + startScopedRecovery = () => { + const start = params.metadata?.persistence?.startScopedRecovery + if (!start) throw new Error(`the wrapper offers no scoped recovery`) + return start() + } + remove = (rows) => { + params.begin() + for (const value of rows) params.write({ type: `delete`, value }) + return params.commit() + } + update = (rows) => { + params.begin() + for (const value of rows) params.write({ type: `update`, value }) + return params.commit() + } + return { + loadSubset: (options: LoadSubsetOptions) => { + upstream.calls.push(options) + upstream.arrivedAt.set(options, Date.now()) + upstream.visibleAtCall.push( + collection.toArray.map((row) => row.id).sort(), + ) + if (upstream.rejectAtOnce) { + upstream.events.push({ type: `failed`, load: options }) + return Promise.reject(new Error(`offline at once`)) + } + const loaded = new Promise((resolve, reject) => { + pending.push({ resolve, reject, options, epoch }) + }) + if (upstream.truncateInNextLoad) { + upstream.truncateInNextLoad = false + epoch++ + void truncate?.() + upstream.events.push({ type: `truncate` }) + } + return loaded + }, + } + }, + }, + }), + ) + return { collection, upstream } +} + +export function filterQuery( + source: Collection, + group: string, +) { + return createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => eq(row.group, group)), + ) +} + +export function trackPreload(query: Collection) { + const state = { value: `pending` as `pending` | `resolved` | `rejected` } + void query.preload().then( + () => (state.value = `resolved`), + () => (state.value = `rejected`), + ) + return () => state.value +} + +export async function untilUpstreamCalled(upstream: Upstream, count = 1) { + for (let i = 0; i < 200 && upstream.calls.length < count; i++) { + await vi.advanceTimersByTimeAsync(0) + await realMacrotask() + } + if (upstream.calls.length < count) { + throw new Error( + `checkpoint not reached: upstream load ${count} never started`, + ) + } +} + +export function quietConsole(): void { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) +} diff --git a/packages/db-sqlite-persistence-core/tests/helpers/node-sqlite-driver.ts b/packages/db-sqlite-persistence-core/tests/helpers/node-sqlite-driver.ts new file mode 100644 index 0000000000..248c2ef1ee --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/helpers/node-sqlite-driver.ts @@ -0,0 +1,71 @@ +import type { DatabaseSync } from 'node:sqlite' +import type { SQLiteDriver } from '../../src/index.js' + +function toBinding(value: unknown): string | number | bigint | null { + if (value === null || value === undefined) return null + if (typeof value === `boolean`) return value ? 1 : 0 + if ( + typeof value === `string` || + typeof value === `number` || + typeof value === `bigint` + ) { + return value + } + return String(value) +} + +/** + * A `node:sqlite` driver for tests that meets the driver contract's + * serialization rule (`tests/contracts/sqlite-driver-contract.ts`): work + * outside a transaction waits behind an active transaction, so several + * Collections can share one connection. Work inside a transaction uses the + * transaction driver directly. + */ +export function createSerializedNodeSqliteDriver( + database: DatabaseSync, +): SQLiteDriver { + let tail: Promise = Promise.resolve() + const serialize = (task: () => Promise): Promise => { + const result = tail.then(task, task) + tail = result.then( + () => undefined, + () => undefined, + ) + return result + } + const direct: SQLiteDriver = { + exec: (sql) => { + database.exec(sql) + return Promise.resolve() + }, + query: (sql, params = []) => + Promise.resolve( + database + .prepare(sql) + .all(...params.map(toBinding)) + .map((row) => ({ ...row })) as Array, + ), + run: (sql, params = []) => { + database.prepare(sql).run(...params.map(toBinding)) + return Promise.resolve() + }, + transaction: (transaction) => transaction(direct), + } + return { + exec: (sql) => serialize(() => direct.exec(sql)), + query: (sql, params) => serialize(() => direct.query(sql, params)), + run: (sql, params) => serialize(() => direct.run(sql, params)), + transaction: (transaction) => + serialize(async () => { + database.exec(`BEGIN IMMEDIATE`) + try { + const result = await transaction(direct) + database.exec(`COMMIT`) + return result + } catch (error) { + database.exec(`ROLLBACK`) + throw error + } + }), + } +} diff --git a/packages/db-sqlite-persistence-core/tests/on-demand-offline-readiness-witness.test.ts b/packages/db-sqlite-persistence-core/tests/on-demand-offline-readiness-witness.test.ts new file mode 100644 index 0000000000..83225409b8 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/on-demand-offline-readiness-witness.test.ts @@ -0,0 +1,156 @@ +import { describe, expect, it, onTestFinished, vi } from 'vitest' +import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import { createDeferred } from '../../db/src/deferred.js' +import { persistedCollectionOptions } from '../src/index.js' +import type { Collection, LoadSubsetOptions } from '@tanstack/db' +import type { PersistenceAdapter } from '../src/index.js' + +/** + * # What does an on-demand persisted query show while the network is pending? + * + * This file records current behavior; it does not state a contract. It + * gathers evidence for a design question: whether `initialRender: + * network-first` should extend to on-demand sources, where the core's + * readiness evidence is a settled subset request rather than Collection + * readiness (live-query ARCHITECTURE.md, "Initial readiness"). + * + * The source is a generic on-demand sync, not Electric. Its upstream + * `loadSubset` either stays pending (an offline network) or settles at once + * without writing (a source that already holds complete local rows, as + * ARCHITECTURE.md describes for a complete resume). The SQLite adapter holds + * a complete, partial, or empty cache for the queried subset. The wrapper + * cannot tell a complete cache from a partial one; both read back as rows. + * + * Checkpoint: after the local read settles and the upstream load has started, + * then again after a delay longer than any plausible microtask scheduling. + */ + +type Row = { id: string; group: string } + +function storageAdapter(local: ReadonlyArray) { + const adapter: PersistenceAdapter = { + loadSubset: () => + Promise.resolve(local.map((value) => ({ key: value.id, value }))), + loadResumeSnapshot: () => + Promise.resolve({ + rows: [], + keySet: { status: `consistent` }, + collectionMetadata: [], + latestTerm: 0, + latestSeq: 0, + latestRowVersion: 0, + resetEpoch: 0, + }), + applyCommittedTx: async () => {}, + ensureIndex: async () => {}, + } + return adapter +} + +type UpstreamMode = `offline` | `settles-at-once` + +function persistedOnDemand(adapter: PersistenceAdapter, mode: UpstreamMode) { + const calls: Array = [] + const collection = createCollection( + persistedCollectionOptions({ + id: `offline-witness-${Math.random().toString(36).slice(2)}`, + getKey: (row) => row.id, + persistence: { adapter }, + syncMode: `on-demand`, + sync: { + sync: () => ({ + loadSubset: (options: LoadSubsetOptions) => { + calls.push(options) + return mode === `offline` + ? createDeferred().promise + : Promise.resolve() + }, + }), + }, + }), + ) + onTestFinished(() => collection.cleanup()) + return { collection, calls } +} + +function groupA(source: Collection) { + const query = createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => eq(row.group, `a`)), + ) + onTestFinished(() => query.cleanup()) + return query +} + +function observe( + source: Collection, + query: Collection, +) { + return { + queryStatus: query.status, + queryReady: query.isReady(), + queryRows: query.toArray.map((row: Row) => row.id).sort(), + sourceStatus: source.status, + sourceRows: source.toArray.map((row) => row.id).sort(), + } +} + +async function runHistory(local: ReadonlyArray, mode: UpstreamMode) { + const { collection: source, calls } = persistedOnDemand( + storageAdapter(local), + mode, + ) + const query = groupA(source) + let preloadSettled = false + void query.preload().then( + () => (preloadSettled = true), + () => (preloadSettled = true), + ) + await vi.waitFor(() => expect(calls).toHaveLength(1)) + const afterUpstreamStart = { ...observe(source, query), preloadSettled } + await new Promise((resolve) => setTimeout(resolve, 50)) + const afterDelay = { ...observe(source, query), preloadSettled } + return { afterUpstreamStart, afterDelay } +} + +const complete = [ + { id: `a1`, group: `a` }, + { id: `a2`, group: `a` }, +] +const partial = [{ id: `a1`, group: `a` }] + +describe(`on-demand persisted query with a pending network (current behavior)`, () => { + it(`offline upstream, complete local subset`, async () => { + const { afterDelay } = await runHistory(complete, `offline`) + console.info(`offline/complete`, afterDelay) + expect(afterDelay.queryReady).toBe(false) + expect(afterDelay.preloadSettled).toBe(false) + }) + + it(`offline upstream, partial local subset`, async () => { + const { afterDelay } = await runHistory(partial, `offline`) + console.info(`offline/partial`, afterDelay) + expect(afterDelay.queryReady).toBe(false) + expect(afterDelay.preloadSettled).toBe(false) + }) + + it(`offline upstream, empty local cache`, async () => { + const { afterDelay } = await runHistory([], `offline`) + console.info(`offline/empty`, afterDelay) + expect(afterDelay.queryReady).toBe(false) + expect(afterDelay.preloadSettled).toBe(false) + }) + + it(`upstream that settles at once, complete local subset`, async () => { + const { afterDelay } = await runHistory(complete, `settles-at-once`) + console.info(`settles/complete`, afterDelay) + expect(afterDelay.queryReady).toBe(true) + expect(afterDelay.queryRows).toEqual([`a1`, `a2`]) + }) + + it(`upstream that settles at once, empty local cache`, async () => { + const { afterDelay } = await runHistory([], `settles-at-once`) + console.info(`settles/empty`, afterDelay) + expect(afterDelay.queryReady).toBe(true) + expect(afterDelay.queryRows).toEqual([]) + }) +}) diff --git a/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-campaign-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-campaign-oracle.test.ts new file mode 100644 index 0000000000..1a0e077d15 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-campaign-oracle.test.ts @@ -0,0 +1,1250 @@ +import { DatabaseSync } from 'node:sqlite' +import { fc } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { + IR, + createCollection, + createLiveQueryCollection, + eq, +} from '@tanstack/db' +import { + oraclePropertyOptions, + oracleReplayRequested, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + SQLiteCorePersistenceAdapter, + persistedCollectionOptions, +} from '../src/index.js' +import { createSerializedNodeSqliteDriver as createDriver } from './helpers/node-sqlite-driver.js' +import { + drain, + filterQuery, + onDemandSource, + quietConsole, + realMacrotask, + useFakeClock, + withCleanup, + withMacrotaskReads, +} from './helpers/demand-coverage-driver.js' +import { coverageViolations } from './persisted-demand-coverage-model-oracle.js' +import type { Row } from './helpers/demand-coverage-driver.js' +import type { + CoverageReach, + CoverageRecord, + OfflineUpstream, + QueryObservation, + QuerySpec, + SourceEvent, +} from './persisted-demand-coverage-model-oracle.js' +import type { Collection, LoadSubsetOptions, SyncConfig } from '@tanstack/db' +import type { SQLiteDriver } from '../src/index.js' + +/** + * # Generated histories for persisted demand coverage + * + * Law: the persisted demand coverage law of + * `tests/persisted-demand-coverage-oracle.test.ts` (its opening prose and + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`). + * That file checks fixed histories. This one generates longer online + * histories, so that loads, live source writes, releases, and truncates + * interleave in orders that no fixed history names, and checks one + * consequence of the law in a later offline session: + * + * An offline load of a demand settles from SQLite exactly when an earlier + * current load of that exact demand settled online, no truncate ended that + * coverage, and SQLite still holds at least as many of the demand's rows as + * when the load settled. It then shows the rows SQLite holds for the demand. + * Every other demand waits for its upstream load. + * + * ## History grammar + * + * Rows `r1` to `r4` each belong to group `a` or `b` with a rank from 1 to 3, + * so ranks can tie, or do not exist. Three live queries read demands: + * `group = 'a'`, `group = 'b'`, and the window `w`, `group = 'a' orderBy + * rank limit 2`, which core loads as a limited load and a tie-boundary load. + * Session 1 is online and runs a generated command sequence: + * + * - `start d`: a live query starts reading demand d; its load waits. + * - `answer d`: the source writes its rows of d and settles d's loads. + * - `settle-oldest d`: the source settles d's oldest waiting load without + * writing rows. After a truncate that is the obsolete load, so this + * separates an obsolete load from the current one. + * - `fail d`: the source rejects d's loads; the query is then released. + * - `release d`: the live query of d is cleaned up, which aborts its load. + * - `write id group rank`: the source inserts, moves, or re-ranks a row, + * without a load. + * - `delete id`: the source deletes a row, without a load. + * - `truncate`: the source truncates. Active demands load again. + * - `snapshot`: the source commits a snapshot transaction of every row it + * holds, which replaces every synced row as a truncate at its start does. + * + * Each history also chooses how the source treats an aborted load: + * + * - `rejects`: it drops the load and rejects it. + * - `ignores`: it keeps the load waiting, so a load that the wrapper aborted + * at a truncate can still settle. That obsolete load could wrongly record + * coverage. + * - `stops and resolves`: it stops installing rows and resolves the load at + * once, as the abort contract allows ("async adapters should stop before + * installing more request-scoped rows"). The load is over for core, so it + * must record nothing, even though it fulfilled. + * + * Session 2 reopens the same database offline and reads all three demands. + * Each history also chooses its wait, 0 or 1000 ms, and whether its source + * never answers or fails every load at once. + * + * ## Model + * + * The model replays the source's log of what happened: the rows it wrote or + * deleted, its truncates, and each load it settled. It keeps the rows SQLite + * holds and each demand's recorded row count. A source write reaches SQLite + * whether or not a load asked for it. A load is obsolete when a truncate came + * between the source receiving it and settling it (law 6 in + * `packages/db/src/query/live/ARCHITECTURE.md`); it records nothing. A load + * that core aborted before the source settled it records nothing either. A + * truncate or a snapshot also ends every coverage record and empties SQLite; + * the snapshot's rows then reach SQLite. A current load + * that settles records how many rows SQLite holds for its exact demand, and + * for a limited load which rows. A failed or released load records nothing. + * Offline, a query is ready exactly when every load it sent in session 2 is + * covered. When the wrapper asks for a demand again after a truncate, and + * when core sends a window's tie-boundary load, is scheduling that the law + * leaves open, so the model reads both from the logs instead of predicting + * them. + * + * ## Observation and checkpoint + * + * Each live query's status and its rows (the window's in order), at 300 ms + * of real time and, after a 1000 ms wait, again at 1500 ms. The checkpoints + * sit well clear of the deadline, so the wait premise at each is certain. A + * covered load is loading before its wait and ready after it, or ready at + * once when its upstream fails; an uncovered load stays loading, or fails + * with its upstream. At each checkpoint the on-demand Collection's own + * status is observed too: settling loads from SQLite must leave it `loading`, + * because the offline source never calls `markReady()`. + * + * Declared reach (ORC-015): histories in which a demand is covered offline, + * uncovered offline, covered but short of rows, ended by a truncate, ended + * by a snapshot, settled through an obsolete load, settled through an + * aborted load, settled only by an aborted load (so its session-2 check + * separates "records nothing" from "records coverage"), moved between + * demands, and recorded twice; a window covered offline, a window that holds + * other rows than it recorded, and a window whose tie-boundary load is + * uncovered. Teardown drops every waiting load before it aborts them, so + * only a release during the commands settles an aborted load. The fixed and random + * campaigns run the same property with 40 histories each. + * + * Parts 2 and 3 sweep a storage failure over every adapter call and a + * release over every step of a load; their laws are stated beside them. + * + * Limits: a session-1 wait long enough that no session-1 load settles from + * SQLite except on failure; real timers in session 1, so checkpoints are + * coarse; no cursor or offset loads (page moves are in the fixed oracle); + * one process and the default coordinator; `node:sqlite` only; no scoped + * recovery (Part 2b of the fixed oracle owns it). + */ + +type Group = `a` | `b` +const IDS = [`r1`, `r2`, `r3`, `r4`] as const +type Id = (typeof IDS)[number] +type Rank = 1 | 2 | 3 +type RowState = { group: Group; rank: Rank } +// The demands that live queries read: each group unordered, and the window +// `group = 'a' orderBy rank limit 2`. +type Demand = `a` | `b` | `w` +const DEMANDS: ReadonlyArray = [`a`, `b`, `w`] +const groupOf = (demand: Demand): Group => (demand === `b` ? `b` : `a`) +const WINDOW_LIMIT = 2 + +type Command = + | { type: `start`; demand: Demand } + | { type: `answer`; demand: Demand } + | { type: `settle-oldest`; demand: Demand } + | { type: `fail`; demand: Demand } + | { type: `release`; demand: Demand } + | { type: `write`; id: Id; group: Group; rank: Rank } + | { type: `delete`; id: Id } + | { type: `truncate` } + | { type: `snapshot` } + +// What the source does with a load that core aborts. +type AbortBehavior = `rejects` | `ignores` | `stops and resolves` + +type History = { + abort: AbortBehavior + // The offline session's wait and how its source answers. + offline: { waitMs: 0 | 1000; upstream: OfflineUpstream } + initial: Partial> + commands: ReadonlyArray +} + +const SPECS: Record = { + a: { group: `a` }, + b: { group: `b` }, + w: { group: `a`, orderByRank: true, limit: WINDOW_LIMIT }, +} + +// --------------------------------------------------------------------------- +// Model and comparison +// +// The judgment lives in `tests/persisted-demand-coverage-model-oracle.ts`, +// shared with every receiving driver. It replays the online session's source +// log and judges each offline query from the loads it sent. This file only +// records that log and those observations at the real wrapper. +// --------------------------------------------------------------------------- + +// --------------------------------------------------------------------------- +// Production driver +// --------------------------------------------------------------------------- + +const rowFor = ( + id: Id, + group: Group, + rank: number = IDS.indexOf(id) + 1, +): Row => ({ id, group, rank, value: `v1` }) + +// The live query a demand label names. +function queryFor(source: Collection, demand: Demand) { + if (demand !== `w`) return filterQuery(source, demand) + return createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.group, `a`)) + .orderBy(({ row }) => row.rank) + .limit(WINDOW_LIMIT), + ) +} + +// Which live query a load serves: the window's loads carry a limit or a +// rank bound; the others filter one group. +function demandOf(options: LoadSubsetOptions): Demand { + const text = JSON.stringify(options.where) + if (options.limit !== undefined || text.includes(`"rank"`)) return `w` + if (text.includes(`"value":"a"`)) return `a` + if (text.includes(`"value":"b"`)) return `b` + throw new Error(`unexpected demand ${text}`) +} + +type SyncParams = Parameters[`sync`]>[0] +type Pending = { + resolve: () => void + reject: (error: unknown) => void + options: LoadSubsetOptions + // The source's truncate count when it received the load. + epoch: number +} + +// A generic on-demand source whose loads wait per demand. It writes only +// what the driver tells it to; the driver tracks which rows it wrote since +// the last truncate, so it never deletes a row the Collection lacks. +function campaignSource( + driver: SQLiteDriver, + online: boolean | OfflineUpstream, + waitMs: number, + abort: AbortBehavior = `rejects`, +) { + const pending = new Map>() + // When the source received each load. The wrapper starts a load before it + // forwards it, so a load's wait has surely passed once this much time has. + const arrivedAt = new Map() + const sent: Record> = { a: [], b: [], w: [] } + const events: Array = [] + let epoch = 0 + const settled = (load: Pending) => { + events.push({ + type: `settled`, + load: load.options, + current: load.epoch === epoch, + ...(load.options.signal?.aborted ? { aborted: true } : {}), + }) + load.resolve() + } + let params: SyncParams | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: `demand-coverage-campaign`, + getKey: (row) => row.id, + syncMode: `on-demand`, + initialRender: { strategy: `network-first`, networkTimeoutMs: waitMs }, + persistence: { adapter: new SQLiteCorePersistenceAdapter({ driver }) }, + sync: { + sync: (syncParams) => { + params = syncParams + return { + loadSubset: (options: LoadSubsetOptions) => { + const demand = demandOf(options) + sent[demand].push(options) + arrivedAt.set(options, Date.now()) + if (online === `fails at once`) { + return Promise.reject(new Error(`offline`)) + } + if (online !== true) return new Promise(() => {}) + return new Promise((resolve, reject) => { + const entry = { resolve, reject, options, epoch } + pending.set(demand, [...(pending.get(demand) ?? []), entry]) + options.signal?.addEventListener( + `abort`, + () => { + // A load the driver already dropped is not the source's. + const waiting = pending.get(demand) ?? [] + if (abort === `ignores` || !waiting.includes(entry)) return + pending.set( + demand, + (pending.get(demand) ?? []).filter((e) => e !== entry), + ) + if (abort === `stops and resolves`) settled(entry) + else reject(new DOMException(`aborted`, `AbortError`)) + }, + { once: true }, + ) + }) + }, + } + }, + }, + }), + ) + const written = new Set() + const commit = async (apply: (sync: SyncParams) => void): Promise => { + const sync = params! + sync.begin() + apply(sync) + await sync.commit() + } + const writeRows = async (rows: ReadonlyArray) => { + await commit((sync) => { + for (const [id, state] of rows) { + sync.write({ + type: written.has(id) ? `update` : `insert`, + value: rowFor(id, state.group, state.rank), + }) + written.add(id) + } + }) + events.push({ type: `rows`, rows }) + } + return { + collection, + sent, + arrivedAt, + events, + started: () => params !== undefined, + hasPending: (demand: Demand) => (pending.get(demand)?.length ?? 0) > 0, + async answer(demand: Demand, rows: ReadonlyArray) { + await writeRows(rows) + for (const load of pending.get(demand)?.splice(0) ?? []) settled(load) + }, + // A released demand's loads stop at the source either way. + forget(demand: Demand) { + pending.delete(demand) + }, + settleOldest(demand: Demand) { + const load = pending.get(demand)?.shift() + if (load) settled(load) + }, + fail(demand: Demand) { + for (const load of pending.get(demand)?.splice(0) ?? []) { + load.reject(new Error(`offline`)) + } + }, + write: (id: Id, state: RowState) => writeRows([[id, state]]), + async remove(id: Id) { + if (!written.has(id)) return + written.delete(id) + await commit((sync) => + sync.write({ type: `delete`, value: rowFor(id, `a`) }), + ) + events.push({ type: `delete`, id }) + }, + async truncate() { + written.clear() + epoch++ + await commit((sync) => sync.truncate()) + events.push({ type: `truncate` }) + }, + // Every row the source holds, in one snapshot transaction. + async snapshot(rows: ReadonlyArray) { + written.clear() + epoch++ + const sync = params! + sync.begin({ snapshot: true }) + for (const [id, state] of rows) { + sync.write({ + type: `insert`, + value: rowFor(id, state.group, state.rank), + }) + written.add(id) + } + await sync.commit() + events.push({ type: `snapshot`, rows }) + }, + } +} + +const settle = (ms = 20) => new Promise((resolve) => setTimeout(resolve, ms)) + +// One record per offline checkpoint: before the wait (with a nonzero wait) +// and after it. +async function runHistory( + history: History, +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const truth = new Map( + Object.entries(history.initial) as Array<[Id, RowState]>, + ) + let events: ReadonlyArray = [] + + // Session 1 is online. Its wait is long, so a load settles from SQLite + // only when its upstream load fails. + { + const source = campaignSource(driver, true, 60_000, history.abort) + events = source.events + const queries = new Map>() + // A reacquired load, or a window's tie-boundary load, reaches the source + // shortly after an earlier load settles, so wait a little before a no-op. + const untilPending = async (demand: Demand) => { + if (!queries.has(demand)) return + for (let i = 0; i < 10 && !source.hasPending(demand); i++) await settle(5) + } + const release = async (demand: Demand) => { + const query = queries.get(demand) + queries.delete(demand) + await query?.cleanup() + source.forget(demand) + await settle() + } + await withCleanup(async () => { + // An on-demand Collection starts its sync at the first preload, so a + // source write before any load has a sync to write through. + // The source never marks the Collection ready, so do not await it. + void source.collection.preload().catch(() => undefined) + for (let i = 0; i < 50 && !source.started(); i++) await settle(5) + expect(source.started()).toBe(true) + for (const command of history.commands) { + switch (command.type) { + case `start`: { + if (queries.has(command.demand)) break + const query = queryFor(source.collection, command.demand) + queries.set(command.demand, query) + void query.preload().catch(() => undefined) + for (let i = 0; i < 50 && !source.hasPending(command.demand); i++) { + await settle(5) + } + // Positive witness: the demand reached the source. + expect(source.hasPending(command.demand)).toBe(true) + break + } + case `answer`: { + await untilPending(command.demand) + if (!source.hasPending(command.demand)) break + // A source may return more rows than a load asks for: it + // answers with every row of the demand's group. + const group = groupOf(command.demand) + await source.answer( + command.demand, + [...truth].filter(([, state]) => state.group === group), + ) + await settle() + break + } + case `settle-oldest`: + await untilPending(command.demand) + source.settleOldest(command.demand) + await settle() + break + case `fail`: + if (!queries.has(command.demand)) break + source.fail(command.demand) + await settle() + await release(command.demand) + break + case `release`: + await release(command.demand) + break + case `write`: { + const state = { group: command.group, rank: command.rank } + truth.set(command.id, state) + await source.write(command.id, state) + await settle() + break + } + case `delete`: + truth.delete(command.id) + await source.remove(command.id) + await settle() + break + case `truncate`: { + await source.truncate() + // Active demands load again after the truncate; when is the + // wrapper's scheduling, so the driver only gives it a moment. + for (let i = 0; i < 50; i++) { + if ([...queries.keys()].every((d) => source.hasPending(d))) break + await settle(5) + } + break + } + case `snapshot`: { + await source.snapshot([...truth]) + await settle() + break + } + } + } + await settle(50) + // Teardown aborts every waiting load; the source drops them first, so + // only a release during the commands settles an aborted load. + for (const demand of DEMANDS) source.forget(demand) + }, [ + ...DEMANDS.map((demand) => () => queries.get(demand)?.cleanup()), + () => source.collection.cleanup(), + ]) + } + + // Session 2 is offline. Checkpoints at 300 ms and, after a nonzero wait, + // at 1500 and 2600 ms. A window's tie-boundary load arrives after its + // limited load settles, so at 1500 ms its own wait may still run. + const { waitMs, upstream } = history.offline + const source = campaignSource(driver, upstream, waitMs) + const queries = DEMANDS.map((demand) => queryFor(source.collection, demand)) + const observed = (index: number): QueryObservation => { + const query = queries[index]! + return query.status === `ready` + ? { status: `ready`, rows: query.toArray.map((row) => row.id) } + : { status: query.status === `error` ? `error` : `loading` } + } + const checkpoint = (at = Date.now()): CoverageRecord => ({ + events, + queries: Object.fromEntries( + DEMANDS.map((demand, index) => [ + demand, + { + spec: SPECS[demand], + sent: [...source.sent[demand]], + observed: observed(index), + }, + ]), + ), + upstream, + waitPassed: (load) => + at - (source.arrivedAt.get(load) ?? Infinity) >= waitMs, + // The offline source never calls `markReady()`. + collection: { status: source.collection.status, sourceMarkedReady: false }, + }) + return withCleanup(async () => { + for (const query of queries) void query.preload().catch(() => undefined) + const records: Array = [] + await settle(300) + records.push(checkpoint()) + if (waitMs > 0) { + await settle(1200) + records.push(checkpoint()) + await settle(1100) + records.push(checkpoint()) + } + return records + }, [ + ...queries.map((query) => () => query.cleanup()), + () => source.collection.cleanup(), + ]) +} + +// --------------------------------------------------------------------------- +// Part 2: swept storage failures +// +// One history, with a storage failure swept over every call. Session 1 +// covers D (`group = 'a'`) online with `a1` and `a2`; session 2 reads D +// offline. The adapter fails exactly one asynchronous call, the Nth call of +// one session, either before the call takes effect or after it took effect +// (the acknowledgement is lost). N runs over every call that the history +// makes without a failure, so every storage boundary on the path fails once. +// +// Laws: +// - Safety, for every N: if session 2 settles D from SQLite, it shows +// exactly `a1` and `a2`. A failure never makes an incomplete restore look +// complete. +// - A failed coverage read in session 1 (a read after the answer) keeps the +// load's outcome: the session-1 query is ready on `a1` and `a2`, and with +// no record session 2 waits. +// - A failed row write in session 1, before it takes effect, leaves no +// coverage: session 2 waits. +// - A failed coverage check in session 2 (a read after the restore +// published) proves nothing: session 2 waits and does not fail. +// - A failed coverage write in session 1 (a metadata-only commit after the +// answer) is a terminal local persistence failure, before or after its +// effect, so the Collection crashes, as for any durable write (the +// glossary's "crash"). Whether the load itself fulfilled no longer decides +// anything. +// Other failures, such as a startup read, a claim, or an acknowledgement +// lost after a row write applied, are held only to safety. +// --------------------------------------------------------------------------- + +type FailureTiming = `before effect` | `after effect` +type StorageCall = { + session: 1 | 2 + method: string + // A read after the answer (session 1) or after the restore (session 2). + late: boolean + rows: boolean +} + +function failingAdapter( + adapter: SQLiteCorePersistenceAdapter, + session: 1 | 2, + log: Array, + late: () => boolean, + failAt: { index: number; timing: FailureTiming } | undefined, +): SQLiteCorePersistenceAdapter { + return new Proxy(adapter, { + get(target, property) { + const value: unknown = Reflect.get(target, property, target) + // The adapter's one synchronous method is not a storage call. + if (typeof value !== `function` || property === `getCacheGenerationNow`) { + return typeof value === `function` ? value.bind(target) : value + } + return (...args: Array) => { + const call = (): unknown => Reflect.apply(value, target, args) + const index = log.length + const tx = args[1] as { mutations?: Array } | undefined + const entry: StorageCall = { + session, + method: String(property), + late: late(), + rows: + property === `applyCommittedTx` && (tx?.mutations?.length ?? 0) > 0, + } + log.push(entry) + if (failAt?.index !== index) return call() + const failure = new Error(`injected storage failure`) + if (failAt.timing === `before effect`) return Promise.reject(failure) + return Promise.resolve(call()).then(() => Promise.reject(failure)) + } + }, + }) +} + +// Cleanup may report the injected failure, for example from a claim release. +// That is the failure doing its job; any other cleanup error still fails. +function fromInjectedFailure(error: unknown): boolean { + for (let cause = error; cause instanceof Error; cause = cause.cause) { + if (cause.message.includes(`injected storage failure`)) return true + } + return false +} +const tolerateInjected = (cleanup: () => unknown) => async () => { + try { + await cleanup() + } catch (error) { + if (!fromInjectedFailure(error)) throw error + } +} + +type FaultObservation = { + session1: { status: string; rows: Array } + session2: { status: string; rows: Array } +} + +async function runWithFault( + failure: { session: 1 | 2; index: number; timing: FailureTiming } | undefined, +): Promise }> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const calls: Array = [] + const failAtFor = (session: 1 | 2) => + failure?.session === session ? failure : undefined + const observe = (query: ReturnType) => ({ + status: query.status, + rows: query.toArray.map((row) => row.id).sort(), + }) + let session1: FaultObservation[`session1`] + { + const phase = { answered: false } + const sessionCalls: Array = [] + const { collection, upstream } = onDemandSource( + failingAdapter( + new SQLiteCorePersistenceAdapter({ driver }), + 1, + sessionCalls, + () => phase.answered, + failAtFor(1), + ), + 0, + ) + const query = filterQuery(collection, `a`) + session1 = await withCleanup(async () => { + void query.preload().catch(() => undefined) + for (let i = 0; i < 60 && upstream.pendingCount() === 0; i++) { + await settle(5) + } + phase.answered = true + if (upstream.pendingCount() > 0) { + await upstream + .answer([rowFor(`r1`, `a`), rowFor(`r2`, `a`)]) + .catch(() => undefined) + } + await settle(150) + return observe(query) + }, [ + tolerateInjected(() => query.cleanup()), + tolerateInjected(() => collection.cleanup()), + ]) + calls.push(...sessionCalls) + } + const holder: { collection?: { toArray: Array } } = {} + const sessionCalls: Array = [] + const { collection } = onDemandSource( + failingAdapter( + new SQLiteCorePersistenceAdapter({ driver }), + 2, + sessionCalls, + () => (holder.collection?.toArray.length ?? 0) > 0, + failAtFor(2), + ), + 0, + ) + holder.collection = collection + const query = filterQuery(collection, `a`) + const session2 = await withCleanup(async () => { + void query.preload().catch(() => undefined) + await settle(300) + return observe(query) + }, [ + tolerateInjected(() => query.cleanup()), + tolerateInjected(() => collection.cleanup()), + ]) + calls.push(...sessionCalls) + return { session1, session2, calls } +} + +// The law for one failure, given the call that failed. +function checkFault( + failed: StorageCall, + timing: FailureTiming, + observed: FaultObservation, + reach: (state: FaultReach) => void, +): void { + if (observed.session2.status === `ready`) { + expect(observed.session2.rows).toEqual([`r1`, `r2`]) + } + if (failed.session === 1 && failed.method === `loadSubset` && failed.late) { + reach(`coverage read failed`) + expect(observed.session1).toEqual({ status: `ready`, rows: [`r1`, `r2`] }) + expect(observed.session2.status).toBe(`loading`) + } + if (failed.session === 1 && failed.rows && timing === `before effect`) { + reach(`row write failed`) + expect(observed.session2.status).toBe(`loading`) + } + if (failed.session === 2 && failed.method === `loadSubset` && failed.late) { + reach(`coverage check failed`) + expect(observed.session2.status).toBe(`loading`) + } + if ( + failed.session === 1 && + failed.method === `applyCommittedTx` && + !failed.rows && + failed.late + ) { + reach(`coverage write failed`) + expect(observed.session1.status).toBe(`crashed`) + } +} + +type FaultReach = + | `coverage read failed` + | `row write failed` + | `coverage check failed` + | `coverage write failed` + +// --------------------------------------------------------------------------- +// Part 3: swept releases +// +// A release can land at any await of a load. Session 1 covers D online. In +// session 2, with a 1000 ms wait and a fake clock, the live query of D is +// released after K steps, for every K up to the step at which the load would +// have settled; each step is one zero-length clock turn and one real +// macrotask, and reads answer on a later macrotask. Laws (a released load +// owns no wait; law 6 in `ARCHITECTURE.md`): +// +// - after the release, the source Collection is not loading a subset, and +// it stays so when the clock passes the wait; +// - every upstream load that the release left behind was aborted, and no +// upstream load starts after the release. +// +// Declared reach: a release before the load reached the source, while it +// waited on the source, and while a coverage read was in flight. +// --------------------------------------------------------------------------- + +type ReleaseReach = + | `released before the source load` + | `released while the source load waited` + | `released during a coverage read` + +// Session 1 online: D loads `r1` and `r2` and records coverage. +async function coverGroupA(driver: SQLiteDriver): Promise { + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = filterQuery(collection, `a`) + await withCleanup(async () => { + const preload = query.preload() + for (let i = 0; i < 60 && upstream.pendingCount() === 0; i++) { + await settle(5) + } + await upstream.answer([rowFor(`r1`, `a`), rowFor(`r2`, `a`)]) + await preload + await settle(50) + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +async function runReleaseAt( + driver: SQLiteDriver, + steps: number, + reach: (state: ReleaseReach) => void, +): Promise<{ loadingAfterRelease: boolean; loadingAfterWait: boolean }> { + useFakeClock() + const reads = { restored: false, inFlight: 0 } + const holder: { collection?: { toArray: Array } } = {} + const base = new SQLiteCorePersistenceAdapter({ + driver: withMacrotaskReads(driver), + }) + const adapter = new Proxy(base, { + get(target, property) { + if (property === `loadSubset`) { + return async (...args: Parameters) => { + const late = (holder.collection?.toArray.length ?? 0) > 0 + if (late) reads.inFlight++ + try { + // A read after the restore spans three macrotasks, so that some + // step of the sweep releases while it is in flight. + if (late) { + await realMacrotask() + await realMacrotask() + } + return await target.loadSubset(...args) + } finally { + if (late) reads.inFlight-- + } + } + } + const value: unknown = Reflect.get(target, property, target) + return typeof value === `function` ? value.bind(target) : value + }, + }) + const { collection, upstream } = onDemandSource(adapter, 1000) + holder.collection = collection + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + for (let i = 0; i < steps; i++) { + await vi.advanceTimersByTimeAsync(0) + await realMacrotask() + } + if (upstream.calls.length === 0) reach(`released before the source load`) + else reach(`released while the source load waited`) + if (reads.inFlight > 0) reach(`released during a coverage read`) + const callsAtRelease = upstream.calls.length + await query.cleanup() + await drain() + const loadingAfterRelease = collection.isLoadingSubset + // Every upstream load left behind was aborted. + expect(upstream.calls.filter((call) => !call.signal?.aborted)).toEqual([]) + await vi.advanceTimersByTimeAsync(2000) + await drain() + // A released demand starts no upstream load later. + expect(upstream.calls.length).toBe(callsAtRelease) + return { loadingAfterRelease, loadingAfterWait: collection.isLoadingSubset } + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +// --------------------------------------------------------------------------- +// Generators and campaigns +// --------------------------------------------------------------------------- + +const groupArb = fc.constantFrom(`a`, `b`) +const demandArb = fc.constantFrom(`a`, `b`, `w`) +const rankArb = fc.constantFrom(1, 2, 3) +const idArb = fc.constantFrom(...IDS) +const onDemand = (type: T, weight: number) => ({ + arbitrary: fc.record({ type: fc.constant(type), demand: demandArb }), + weight, +}) +const commandArb = fc.oneof( + onDemand(`start`, 4), + onDemand(`answer`, 4), + onDemand(`settle-oldest`, 1), + onDemand(`fail`, 1), + onDemand(`release`, 2), + { + arbitrary: fc.record({ + type: fc.constant(`write` as const), + id: idArb, + group: groupArb, + rank: rankArb, + }), + weight: 3, + }, + { + arbitrary: fc.record({ type: fc.constant(`delete` as const), id: idArb }), + weight: 2, + }, + { arbitrary: fc.constant({ type: `truncate` as const }), weight: 1 }, + { arbitrary: fc.constant({ type: `snapshot` as const }), weight: 1 }, +) as fc.Arbitrary + +// A full load cycle answers twice, so that a window's tie-boundary load, +// which follows its limited load, settles too. +const cycle = (demand: Demand): ReadonlyArray => [ + { type: `start`, demand }, + { type: `answer`, demand }, + { type: `answer`, demand }, + { type: `release`, demand }, +] + +// Compound steps make the rarer orders likely: a covered demand that then +// loses, gains, or moves a row; a full load cycle, which a later cycle of +// the same demand records again; and a truncate while a load waits, +// answered or settled alone. Each is only a sequence of the commands above. +const stepArb: fc.Arbitrary> = fc.oneof( + { arbitrary: commandArb.map((command) => [command]), weight: 6 }, + { + // Cover the window over two rows, then rank a third row below them, so + // the window would hold other rows than it recorded. + arbitrary: fc + .tuple( + fc.shuffledSubarray([...IDS], { minLength: 3, maxLength: 3 }), + rankArb, + ) + .map(([[first, second, third], rank]): ReadonlyArray => [ + { type: `write`, id: first!, group: `a`, rank: 2 }, + { type: `write`, id: second!, group: `a`, rank: 3 }, + ...cycle(`w`), + { type: `write`, id: third!, group: `a`, rank: rank === 3 ? 1 : rank }, + ]), + weight: 2, + }, + { + arbitrary: fc + .tuple( + demandArb, + idArb, + rankArb, + fc.constantFrom(`move`, `delete`, `rerank`), + ) + .map(([demand, id, rank, change]): ReadonlyArray => [ + { type: `write`, id, group: groupOf(demand), rank }, + ...cycle(demand), + change === `delete` + ? { type: `delete`, id } + : { + type: `write`, + id, + group: + change === `move` + ? groupOf(demand) === `a` + ? `b` + : `a` + : groupOf(demand), + rank: rank === 1 ? 3 : 1, + }, + ]), + weight: 3, + }, + { arbitrary: demandArb.map(cycle), weight: 2 }, + { + // Cover a demand, then replace every row with a snapshot. + arbitrary: demandArb.map((demand): ReadonlyArray => [ + ...cycle(demand), + { type: `snapshot` }, + ]), + weight: 1, + }, + { + // Release a demand while its load waits at the source. A source that + // stops and resolves then settles an aborted load. + arbitrary: demandArb.map((demand): ReadonlyArray => [ + { type: `start`, demand }, + { type: `release`, demand }, + ]), + weight: 3, + }, + { + arbitrary: demandArb.map((demand): ReadonlyArray => [ + { type: `start`, demand }, + { type: `truncate` }, + { type: `answer`, demand }, + ]), + weight: 1, + }, + { + arbitrary: demandArb.map((demand): ReadonlyArray => [ + { type: `start`, demand }, + { type: `truncate` }, + { type: `settle-oldest`, demand }, + ]), + weight: 1, + }, +) +const historyArb: fc.Arbitrary = fc.record({ + abort: fc.constantFrom( + `rejects`, + `ignores`, + `stops and resolves`, + ), + offline: fc.record({ + waitMs: fc.constantFrom<0 | 1000>(0, 1000), + upstream: fc.constantFrom( + `never answers`, + `fails at once`, + ), + }), + initial: fc + .record( + Object.fromEntries( + IDS.map((id) => [ + id, + fc.option(fc.record({ group: groupArb, rank: rankArb }), { + nil: undefined, + }), + ]), + ) as Record>, + ) + .map( + (initial) => + Object.fromEntries( + Object.entries(initial).filter(([, state]) => state !== undefined), + ) as Partial>, + ), + commands: fc + .array(stepArb, { minLength: 2, maxLength: 8 }) + .map((steps) => steps.flat()), +}) + +const PROPERTY = `persisted-demand-coverage.history` +const RUNS = 40 +const REACH_STATES: ReadonlyArray = [ + `covered offline`, + `uncovered offline`, + `covered but short of rows`, + `coverage ended by a truncate`, + `coverage ended by a snapshot`, + `obsolete load settled`, + `aborted load settled`, + `demand settled only by an aborted load`, + `row moved between demands`, + `coverage recorded twice`, + `window covered offline`, + `window holds other rows`, + `tie-boundary load uncovered`, + `covered load waits for the deadline`, + `uncovered load fails with its upstream`, +] + +async function campaign(options: Parameters[1]) { + const reach = createOracleReach(PROPERTY, REACH_STATES) + await fc.assert( + fc.asyncProperty(historyArb, async (history) => { + reach.beginHistory() + for (const record of await runHistory(history)) { + expect( + coverageViolations(record, (state) => reach.reach(state)), + ).toEqual([]) + } + }), + options, + ) + reach.assertReached() + return reach.counts() +} + +// The release law: a released load owns no wait, so the source Collection is +// not loading a subset after the release, nor after the wait would have +// passed. +function checkRelease( + steps: number, + observed: { loadingAfterRelease: boolean; loadingAfterWait: boolean }, +): void { + expect({ steps, ...observed }).toEqual({ + steps, + loadingAfterRelease: false, + loadingAfterWait: false, + }) +} + +declareOracle({ + check: [coverageViolations, checkFault, checkRelease], + grammar: [historyArb], +}) + +describe(`persisted demand coverage over generated histories`, () => { + it(`keeps the law when any one storage call fails`, async () => { + quietConsole() + const baseline = await runWithFault(undefined) + // Positive witness: without a failure, session 2 settles D offline. + expect(baseline.session2).toEqual({ status: `ready`, rows: [`r1`, `r2`] }) + const reach = createOracleReach( + `persisted demand coverage storage failures`, + [ + `coverage read failed`, + `row write failed`, + `coverage check failed`, + `coverage write failed`, + ], + ) + for (const session of [1, 2] as const) { + const count = baseline.calls.filter((c) => c.session === session).length + for (let index = 0; index < count; index++) { + for (const timing of [`before effect`, `after effect`] as const) { + reach.beginHistory() + const run = await runWithFault({ session, index, timing }) + const failed = run.calls.filter((c) => c.session === session)[index] + if (!failed) continue + try { + checkFault(failed, timing, run, (state) => reach.reach(state)) + } catch (error) { + throw new Error( + `failure at session ${session} call ${index} ` + + `(${failed.method}, ${timing}, late: ${failed.late})`, + { cause: error }, + ) + } + } + } + } + reach.assertReached() + }, 600_000) + + it(`owns no wait after a release at any step of a load`, async () => { + quietConsole() + const driver = createDriver(new DatabaseSync(`:memory:`)) + await coverGroupA(driver) + const reach = createOracleReach( + `persisted demand coverage releases`, + [ + `released before the source load`, + `released while the source load waited`, + `released during a coverage read`, + ], + ) + for (let steps = 0; steps <= 24; steps++) { + reach.beginHistory() + checkRelease( + steps, + await runReleaseAt(driver, steps, (state) => reach.reach(state)), + ) + } + reach.assertReached() + }, 300_000) + + it(`rejects observations that ignore lost rows, obsolete loads, or other window rows`, () => { + // Checker calibration (ORC-006): the shared judgment must reject a wrong + // observation of each kind. Each record below states what production + // would wrongly show; the comparison must report it. + const loadA: LoadSubsetOptions = { + where: new IR.Func(`eq`, [new IR.PropRef([`group`]), new IR.Value(`a`)]), + } + const loadW = { + ...loadA, + orderBy: [ + { + expression: new IR.PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: WINDOW_LIMIT, + } as unknown as LoadSubsetOptions + const rows: SourceEvent = { + type: `rows`, + rows: [ + [`r1`, { group: `a`, rank: 1 }], + [`r2`, { group: `a`, rank: 2 }], + ], + } + const covered: Array = [ + rows, + { type: `settled`, load: loadA, current: true }, + { type: `settled`, load: loadW, current: true }, + ] + const ready = (ids: Array): QueryObservation => ({ + status: `ready`, + rows: ids, + }) + const record = ( + events: ReadonlyArray, + a: QueryObservation, + w: QueryObservation, + ): CoverageRecord => ({ + events, + // The offline session's own loads of the same demands. + queries: { + a: { spec: SPECS.a, sent: [{ ...loadA }], observed: a }, + w: { spec: SPECS.w, sent: [{ ...loadW }], observed: w }, + }, + upstream: `never answers`, + waitPassed: true, + }) + // The right observations pass. + expect( + coverageViolations( + record(covered, ready([`r1`, `r2`]), ready([`r1`, `r2`])), + ), + ).toEqual([]) + // A demand that lost a row, settled anyway. + expect( + coverageViolations( + record([...covered, { type: `delete`, id: `r1` }], ready([`r2`]), { + status: `loading`, + }), + ), + ).toHaveLength(1) + // An obsolete load treated as coverage. + expect( + coverageViolations( + record( + [rows, { type: `settled`, load: loadA, current: false }], + ready([`r1`, `r2`]), + { status: `loading` }, + ), + ), + ).toHaveLength(1) + // A window that holds another row, settled anyway. + expect( + coverageViolations( + record( + [ + ...covered, + { type: `rows`, rows: [[`r3`, { group: `a`, rank: 0 }]] }, + ], + ready([`r1`, `r2`, `r3`]), + ready([`r3`, `r1`]), + ), + ), + ).toHaveLength(1) + // A window shown out of order. + expect( + coverageViolations( + record(covered, ready([`r1`, `r2`]), ready([`r2`, `r1`])), + ), + ).toHaveLength(1) + }) + + it.skipIf(oracleReplayRequested())( + `refines the model over a fixed campaign`, + async () => { + quietConsole() + await campaign({ seed: 2102, numRuns: oracleRuns(RUNS) }) + }, + 120_000, + ) + + it(`refines the model over a random or replayed campaign`, async () => { + quietConsole() + await campaign(oraclePropertyOptions(RUNS, PROPERTY)) + }, 120_000) +}) diff --git a/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts new file mode 100644 index 0000000000..6e92d27111 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts @@ -0,0 +1,557 @@ +import { declareOracle } from '../../db/tests/oracle-declaration' +import type { CollectionStatus, LoadSubsetOptions } from '@tanstack/db' + +/** + * # The persisted demand coverage model + * + * Law (`docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * the glossary's "persisted demand coverage" and "network-first initial + * render", and law 6 in `packages/db/src/query/live/ARCHITECTURE.md`): an + * opted-in persisted wrapper over an on-demand source settles a load from + * SQLite only when its exact demand has coverage. A current load of that + * demand recorded coverage when it settled; no truncate or generation change + * ended it since; SQLite still holds at least as many of the demand's rows + * as it recorded; and a limited load still holds the same rows. A covered + * load settles from SQLite once its wait has passed or its upstream load has + * failed. A load with `refetch: true` is never covered. A snapshot + * transaction replaces every synced row, as a truncate at its start does, so + * it ends coverage too (the glossary's "snapshot transaction"). A load that + * settles after core aborted it records nothing: its later outcome decides + * nothing (the glossary's "abort"). Scoped recovery ends coverage and does + * not certify the new generation for offline reuse + * (`docs/collections/electric-collection.md`), so for the rest of that sync + * run no load records coverage and none settles from SQLite; the persisted + * subset restore publishes no rows then. A source whose full sync completed, + * such as Electric's progressive mode, holds every row locally: it resumes + * from those rows and settles every later load at once, so a query that + * never ran is ready on its restored rows (`ARCHITECTURE.md`: "A source that + * resumes from complete local rows settles its loads at once"), until a + * truncate or generation change. Any other load waits + * for its upstream load, and fails with it. A query is ready when every load + * it sent has settled; a query over an eager and an on-demand source is ready + * on the same condition for the on-demand loads, once the eager source has + * opened from its restored rows. Settling loads from SQLite never changes the + * on-demand Collection's own status: it stays as its source set it, so an + * offline source that never called `markReady()` keeps it `loading` (the + * guide's "only the load settles from SQLite"). + * + * This module is the one judgment that the fixed oracle, the generated + * campaign, and every receiving driver share (ORC-014, ORC-016). A driver + * records what happened at its own production boundary and passes that record + * here. No driver states an expected result. + * + * ## The record a driver supplies + * + * - The source log of every session so far, in order: the start of each + * sync run, the rows each source wrote or deleted, its truncates and + * snapshot transactions, the start of scoped recovery, the ends of a + * persisted cache generation (a schema reset), and each load it settled or + * failed. A settled load is current when no truncate, snapshot, or + * generation change came between the source receiving it and settling it, + * and obsolete otherwise; it is aborted when core aborted its signal before + * the source settled it. A driver that injected a failed coverage read + * marks the settled load whose coverage read failed. + * - For each live query at the checkpoint: what it reads (`QuerySpec`), the + * loads the source received for it in this session, and what it showed. + * A driver that injected a failed coverage check marks the query. + * - Optionally, the on-demand Collection's status at the checkpoint, and + * whether its source called `markReady()` in this session. + * - Premises at the checkpoint: whether each load's wait has passed since + * that load started, and, for loads the log does not settle or fail, how + * the source answers. Each load measures its own wait: a window's + * tie-boundary load starts only after its limited load settles, so its + * wait ends later. + * + * The model reads which loads a session sent from that record rather than + * predicting them. When core sends a window's tie-boundary load, and when the + * wrapper asks for a demand again after a reset, is scheduling that the law + * leaves open. + * + * ## Abstractions (ORC-009) + * + * - The model's SQLite rows for a Collection are every row its source wrote + * since the last truncate or generation change. A source write reaches + * SQLite whether or not a load asked for it. + * - To read a load's rows, the model evaluates the load's own filter, cursor, + * order, and limit over those rows. Ties break by key, as the SQLite + * adapter's `ORDER BY ..., key ASC` does. + * - A record holds a row count and, for a limited load, its row keys. The + * production record holds the same facts; the model computes them itself. + * - Two loads have the same demand when their request data asks for the same + * rows: the same filter, order, limit, cursor, and offset, with an offset + * of 0 the same as none. The model compares that request data itself; it + * does not use core's demand key. + * - Scoped recovery combines two production facts: it ends the persisted + * cache generation, and it leaves the rest of its sync run uncertified. + * - `full sync completes` is a source fact, not a wrapper record: the source + * reached its first complete sync of every row (Electric's progressive + * `up-to-date`). The model predicts the source's later settlements from it + * rather than reading them from the log. + * - Coverage is judged with the records at the checkpoint, not at the instant + * the load checked. No supplied history records a demand while another load + * of it waits to check. + * + * Limits: a join reads one eager and one on-demand Collection, and the + * evaluator knows only the expression forms these histories send. + */ + +export type RowState = { group?: string; rank?: number } & Readonly< + Record +> + +/** The Collection a fact belongs to; most records have one, `source`. */ +type InCollection = { collection?: string } + +/** One fact from a source. */ +export type SourceEvent = InCollection & + ( + | { type: `rows`; rows: ReadonlyArray } + | { type: `delete`; id: string } + | { type: `truncate` } + | { + type: `snapshot` + rows: ReadonlyArray + } + | { type: `generation ends` } + | { type: `scoped recovery starts` } + | { type: `sync run starts` } + | { type: `full sync completes` } + | { + type: `settled` + load: LoadSubsetOptions + current: boolean + aborted?: boolean + coverage?: `recorded` | `read fails` + } + | { type: `failed`; load: LoadSubsetOptions } + ) + +/** + * What a live query reads: one group, optionally ordered by rank, and a + * window of `limit` rows from `offset`. + */ +export type FilterSpec = { + group: string + orderByRank?: boolean + limit?: number + offset?: number + collection?: string +} +/** An eager parent Collection joined with an on-demand child Collection. */ +export type JoinSpec = { + join: { parents: string; children: string; on: string } + select: ( + parent: Readonly>, + child: Readonly>, + ) => Record +} +export type QuerySpec = FilterSpec | JoinSpec + +/** How a source answers the loads that its log neither settles nor fails. */ +export type OfflineUpstream = `never answers` | `fails at once` + +export type QueryObservation = { + status: `ready` | `loading` | `error` + rows?: ReadonlyArray>> +} + +export type CoverageReach = + | `covered offline` + | `uncovered offline` + | `covered but short of rows` + | `coverage ended by a truncate` + | `coverage ended by a snapshot` + | `aborted load settled` + | `demand settled only by an aborted load` + | `covered during scoped recovery` + | `settled from a completed full sync` + | `obsolete load settled` + | `row moved between demands` + | `coverage recorded twice` + | `window covered offline` + | `window holds other rows` + | `tie-boundary load uncovered` + | `covered load waits for the deadline` + | `uncovered load fails with its upstream` + +/** Whether a load's wait has passed: for every load, or per load. */ +export type WaitPremise = boolean | ((load: LoadSubsetOptions) => boolean) + +export type CoverageQuery = { + spec: QuerySpec + sent: ReadonlyArray + observed: QueryObservation + /** Overrides the record's premise for this query. */ + waitPassed?: WaitPremise + /** The driver failed this query's coverage check read. */ + checkFails?: boolean +} + +/** The on-demand Collection's own status, and what its source did. */ +export type CollectionObservation = { + status: CollectionStatus + sourceMarkedReady: boolean +} + +export type CoverageRecord = { + events: ReadonlyArray + queries: Readonly> + collection?: CollectionObservation + upstream?: OfflineUpstream + waitPassed: WaitPremise +} + +// The expression forms these loads carry (`packages/db/src/query/ir.ts`). +type Expr = { + type?: string + name?: string + args?: ReadonlyArray + path?: ReadonlyArray + value?: unknown +} +type ModelRow = { id: string } & RowState + +function evaluate(expr: Expr, row: ModelRow): unknown { + if (expr.type === `ref`) return row[expr.path![expr.path!.length - 1]!] + if (expr.type === `val`) return expr.value + const args = (expr.args ?? []).map((arg) => evaluate(arg, row)) + const [left, right] = args as [never, never] + switch (expr.name) { + case `eq`: + return left === right + case `in`: + return (right as ReadonlyArray).includes(left) + case `gt`: + return left > right + case `gte`: + return left >= right + case `lt`: + return left < right + case `lte`: + return left <= right + case `and`: + return args.every(Boolean) + case `or`: + return args.some(Boolean) + case `not`: + return !left + } + throw new Error(`unmodelled expression ${JSON.stringify(expr)}`) +} + +const byKey = (x: ModelRow, y: ModelRow) => + x.id < y.id ? -1 : x.id > y.id ? 1 : 0 +const byRankThenKey = (x: ModelRow, y: ModelRow) => + Number(x.rank) - Number(y.rank) || byKey(x, y) +const modelRows = (rows: ReadonlyMap): Array => + [...rows].map(([id, state]) => ({ ...state, id })) + +// The rows a load reads from SQLite, in its order. +function readLoad( + load: LoadSubsetOptions, + rows: ReadonlyMap, +): Array { + // A cursor and an offset describe the same continuation; a load that + // carries both is read by its cursor (`LoadSubsetOptions`). + const where = load.where as Expr | undefined + const from = load.cursor?.whereFrom as Expr | undefined + const matching = modelRows(rows).filter( + (row) => + (where === undefined || evaluate(where, row) === true) && + (from === undefined || evaluate(from, row) === true), + ) + const orderBy = (load.orderBy ?? []) as unknown as ReadonlyArray<{ + expression: Expr + compareOptions?: { direction?: `asc` | `desc` } + }> + matching.sort((x, y) => { + for (const { expression, compareOptions } of orderBy) { + const a = evaluate(expression, x) as number + const b = evaluate(expression, y) as number + if (a !== b) { + return ( + (a < b ? -1 : 1) * (compareOptions?.direction === `desc` ? -1 : 1) + ) + } + } + return byKey(x, y) + }) + const ids = matching.map((row) => row.id) + const offset = from === undefined ? (load.offset ?? 0) : 0 + return load.limit === undefined + ? ids.slice(offset) + : ids.slice(offset, offset + load.limit) +} + +// The rows a live query shows over the restored rows. +function readQuery( + spec: QuerySpec, + sqlite: (collection: string) => ReadonlyMap, +): Array> { + if (`join` in spec) { + const { parents, children, on } = spec.join + const childRows = modelRows(sqlite(children)) + return modelRows(sqlite(parents)) + .sort(byKey) + .flatMap((parent) => + childRows + .filter((child) => child[on] === parent.id) + .sort(byKey) + .map((child) => spec.select(parent, child)), + ) + } + const ids = modelRows(sqlite(spec.collection ?? `source`)) + .filter((row) => row.group === spec.group) + .sort(spec.orderByRank ? byRankThenKey : byKey) + .map((row) => row.id) + const offset = spec.offset ?? 0 + return spec.limit === undefined + ? ids.slice(offset) + : ids.slice(offset, offset + spec.limit) +} + +// The demand identity the law uses: two loads cover each other only when +// their request data asks for the same rows. An offset of 0 skips no rows. +const demandKey = (load: LoadSubsetOptions): string => + JSON.stringify([ + load.where, + load.orderBy, + load.limit, + load.offset || undefined, + load.cursor, + ]) +const rowsKey = (ids: ReadonlyArray) => JSON.stringify([...ids].sort()) +const isWindow = (spec: QuerySpec) => + !(`join` in spec) && spec.limit !== undefined +const loadsOf = (spec: QuerySpec) => + `join` in spec ? spec.join.children : (spec.collection ?? `source`) + +/** + * The model: replay the source log, then judge each query at the checkpoint + * from the loads it sent. + */ +export function expectedObservations( + record: CoverageRecord, + reach: (state: CoverageReach) => void = () => {}, +): Record { + const sqlite = new Map>() + const rowsIn = (collection = `source`) => { + let rows = sqlite.get(collection) + if (!rows) sqlite.set(collection, (rows = new Map())) + return rows + } + const records = new Map() + const recordKey = (collection: string, load: LoadSubsetOptions) => + `${collection} ${demandKey(load)}` + const settledByUpstream = new Set() + const failed = new Set() + // The Collections whose current sync run is in scoped recovery, and the + // demands that settled online during it, which record nothing. + const recovering = new Set() + const settledWhileRecovering = new Set() + // The Collections whose source completed a full sync of every row. + const complete = new Set() + // Demands whose only settlement since their coverage ended was aborted. + const settledOnlyAborted = new Set() + const writeRows = ( + collection: string, + rows: ReadonlyArray, + ) => { + for (const [id, state] of rows) { + const before = rowsIn(collection).get(id) + if (before !== undefined && before.group !== state.group) { + reach(`row moved between demands`) + } + rowsIn(collection).set(id, state) + } + } + // A truncate, a snapshot, and a generation change empty SQLite and end + // every coverage record of the Collection. + const endCoverage = (collection: string, by: CoverageReach) => { + if ([...records.keys()].some((key) => key.startsWith(`${collection} `))) { + reach(by) + } + rowsIn(collection).clear() + complete.delete(collection) + for (const keys of [records, settledOnlyAborted]) { + for (const key of [...keys.keys()]) { + if (key.startsWith(`${collection} `)) keys.delete(key) + } + } + } + for (const event of record.events) { + const collection = event.collection ?? `source` + switch (event.type) { + case `sync run starts`: + recovering.delete(collection) + break + case `full sync completes`: + complete.add(collection) + break + case `rows`: + writeRows(collection, event.rows) + break + case `delete`: + rowsIn(collection).delete(event.id) + break + case `truncate`: + case `generation ends`: + endCoverage(collection, `coverage ended by a truncate`) + break + case `scoped recovery starts`: + endCoverage(collection, `coverage ended by a truncate`) + recovering.add(collection) + break + case `snapshot`: + endCoverage(collection, `coverage ended by a snapshot`) + writeRows(collection, event.rows) + break + case `settled`: { + settledByUpstream.add(event.load) + if (!event.current) { + reach(`obsolete load settled`) + break + } + const key = recordKey(collection, event.load) + if (event.aborted) { + reach(`aborted load settled`) + if (!records.has(key)) settledOnlyAborted.add(key) + break + } + settledOnlyAborted.delete(key) + if (recovering.has(collection)) { + settledWhileRecovering.add(key) + break + } + if (event.coverage === `read fails`) break + if (records.has(key)) reach(`coverage recorded twice`) + const rows = readLoad(event.load, rowsIn(collection)) + records.set(key, { + count: rows.length, + rows: event.load.limit === undefined ? undefined : rowsKey(rows), + }) + break + } + case `failed`: + failed.add(event.load) + } + } + const expected: Record = {} + for (const [name, query] of Object.entries(record.queries)) { + const collection = loadsOf(query.spec) + const covered = (load: LoadSubsetOptions): boolean => { + if (load.refetch === true || query.checkFails) return false + if (recovering.has(collection)) { + if (settledWhileRecovering.has(recordKey(collection, load))) { + reach(`covered during scoped recovery`) + } + return false + } + const stored = records.get(recordKey(collection, load)) + if (stored === undefined) { + if (settledOnlyAborted.has(recordKey(collection, load))) { + reach(`demand settled only by an aborted load`) + } + return false + } + const rows = readLoad(load, rowsIn(collection)) + if (rows.length < stored.count) { + reach(`covered but short of rows`) + return false + } + if (stored.rows !== undefined && stored.rows !== rowsKey(rows)) { + reach(`window holds other rows`) + return false + } + return true + } + // Positive witness: the query asked its source for data. + if (query.sent.length === 0) { + throw new Error(`no load was sent for query ${name}`) + } + const premise = query.waitPassed ?? record.waitPassed + const waitPassed = (load: LoadSubsetOptions) => + typeof premise === `function` ? premise(load) : premise + const loads = query.sent.map((load) => { + if (complete.has(collection) && load.refetch !== true) { + reach(`settled from a completed full sync`) + return `settled` as const + } + if (settledByUpstream.has(load)) return `settled` as const + const upstreamFailed = + failed.has(load) || record.upstream === `fails at once` + const isCovered = covered(load) + if (isCovered) { + reach( + isWindow(query.spec) ? `window covered offline` : `covered offline`, + ) + if (upstreamFailed || waitPassed(load)) return `settled` as const + reach(`covered load waits for the deadline`) + return `waiting` as const + } + reach(`uncovered offline`) + if (upstreamFailed) { + reach(`uncovered load fails with its upstream`) + return `failed` as const + } + return `waiting` as const + }) + if ( + isWindow(query.spec) && + loads[0] === `settled` && + loads.includes(`waiting`) + ) { + reach(`tie-boundary load uncovered`) + } + expected[name] = loads.includes(`failed`) + ? { status: `error` } + : loads.every((load) => load === `settled`) + ? { status: `ready`, rows: readQuery(query.spec, rowsIn) } + : { status: `loading` } + } + return expected +} + +/** + * The comparison: each query's observed status, and its rows when ready. An + * unordered query's rows compare as a set; an ordered one's in order. + */ +export function coverageViolations( + record: CoverageRecord, + reach?: (state: CoverageReach) => void, +): Array { + const expected = expectedObservations(record, reach) + const violations: Array = [] + // The Collection's status is the source's: `ready` only after its + // `markReady()`. These histories never fail the source's sync itself; that + // is the source-health law's. + if (record.collection) { + const { status, sourceMarkedReady } = record.collection + const want = sourceMarkedReady ? `ready` : `loading` + if (status !== want) { + violations.push( + `on-demand Collection: expected status ${want}, observed ${status}`, + ) + } + } + for (const [name, query] of Object.entries(record.queries)) { + const want = expected[name]! + const got = query.observed + const ordered = !(`join` in query.spec) && query.spec.orderByRank === true + const rows = (observation: QueryObservation) => { + const list = (observation.rows ?? []).map((row) => JSON.stringify(row)) + return JSON.stringify(ordered ? list : list.sort()) + } + const same = + want.status === got.status && + (want.status !== `ready` || rows(want) === rows(got)) + if (!same) { + violations.push( + `query ${name}: expected ${JSON.stringify(want)}, observed ${JSON.stringify(got)}`, + ) + } + } + return violations +} + +declareOracle({ check: [coverageViolations], model: [expectedObservations] }) diff --git a/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts new file mode 100644 index 0000000000..39ef83f68d --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts @@ -0,0 +1,2731 @@ +import { DatabaseSync } from 'node:sqlite' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { + BTreeIndex, + IR, + createCollection, + createLiveQueryCollection, + eq, + getLoadSubsetDemandKey, +} from '@tanstack/db' +import { + SQLiteCorePersistenceAdapter, + persistedCollectionOptions, +} from '../src/index.js' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createSerializedNodeSqliteDriver as createDriver } from './helpers/node-sqlite-driver.js' +import { + drain, + filterQuery, + onDemandSource, + quietConsole, + trackPreload, + untilUpstreamCalled, + useFakeClock, + withCleanup, + withMacrotaskReads, +} from './helpers/demand-coverage-driver.js' +import { coverageViolations } from './persisted-demand-coverage-model-oracle.js' +import type { Row, Upstream } from './helpers/demand-coverage-driver.js' +import type { + CoverageReach, + CoverageRecord, + QueryObservation, + QuerySpec, + SourceEvent, + WaitPremise, +} from './persisted-demand-coverage-model-oracle.js' +import type { Collection, LoadSubsetOptions } from '@tanstack/db' +import type { SQLiteDriver } from '../src/index.js' + +/** + * # When may an on-demand query settle on rows from SQLite? + * + * Law (issue #2102 item 1, + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the persisted-wrapper paragraph in + * `packages/db/src/query/live/ARCHITECTURE.md`): an opted-in persisted + * wrapper over an on-demand source may settle a demand's load from SQLite + * when all three hold: + * + * 1. the persisted subset restore for the demand has published its rows in + * the current sync run, possibly none; + * 2. the exact demand has persisted demand coverage in the current persisted + * cache generation, because an earlier load of the same demand key reached + * applied settlement with its rows durably committed; + * 3. the configured wait has passed since that load started, or the load's + * upstream load has failed; + * 4. the restore returned at least as many rows as SQLite held for the demand + * when the covered load settled (maintainer decision, 2026-10-26). A row + * deleted afterwards, by the source or by retention, ends coverage until + * the demand loads online again. An ordered window must hold the same + * rows it held then, in any order. + * + * Otherwise the load waits for its upstream load, as it does without + * persistence. After a settlement from SQLite the upstream load keeps + * running, and its rows publish when they apply; a later upstream failure + * does not undo the settlement. Releasing the demand aborts its upstream + * load. Coverage survives release: a later load of the same demand starts + * its own wait and may settle from SQLite again. + * + * Sources combine for network-first initial rendering: a query may render + * from local data when each source permits it. An eager source permits it + * after its persisted restore; an on-demand source permits it after the + * query's demands on it have settled. + * + * This file uses a generic on-demand source over the real persisted wrapper + * and `SQLiteCorePersistenceAdapter` on `node:sqlite`. Each history has two + * sessions over one database. Session 1 is online and creates coverage only + * through real loads; nothing seeds coverage metadata. Session 2 controls + * the upstream load and the clock. + * + * Coverage is bookkeeping: a coverage read that fails, or a release during + * it, does not change the load's own outcome or lifetime (Part 2c). + * + * Two judgments live here. Part 1 is a bounded enumeration with its own trace + * model, which predicts status, `preload()` state, and rows at five + * checkpoints. Parts 2 to 4 are fixed histories judged by the model shared + * with the generated campaign and every receiving driver, + * `tests/persisted-demand-coverage-model-oracle.ts`. They record the + * source's log and each query's loads and observation, and state no expected + * answer (ORC-016). Two pinned checks whose laws other owners hold live in + * `persisted-demand-coverage-supplemental.test.ts`. + * + * Limits: Electric, Query DB, React, and Solid have receiving witnesses in + * their packages, not here. Resets are a source truncate, scoped recovery + * (also followed by an online load in the same sync run), a schema reset, a + * refetch, and cache-claim expiry; Electric's must-refetch is in its + * witness, and snapshot transactions and aborted loads that resolve are in + * the generated campaign. There is no coordinator other than the default + * single-process one (multi-tab is out of scope, maintainer decision + * 2026-10-26), and no native host. A failure before the persisted subset + * restore publishes is not a legal history (Part 1). Every block is a + * bounded enumeration or fixed history, not a generated campaign, so ORC-007 + * and ORC-015 do not apply here. + */ + +// =========================================================================== +// Part 1: one demand, two sessions +// =========================================================================== + +// --------------------------------------------------------------------------- +// History grammar +// +// Session 1 decides whether demand D (`group = 'a'`, no order or limit) has +// coverage and which rows SQLite holds for it: +// +// - `not-requested`: session 1 never asks for D. +// - `failed`: D's upstream load rejects, so it never reaches applied +// settlement. +// - `overlapping-demand`: a different filter (`id = 'a1'`) settles. SQLite +// holds `a1`, which matches D, but D has no coverage. +// - `other-window`: the same filter with `orderBy rank, limit 1` settles, +// along with its tie-boundary load. SQLite holds `a1`. Neither demand key +// is D's key, so D has no coverage. +// - `other-order`: the same filter with `orderBy rank` and no limit settles. +// SQLite holds every row that matches D, but the ordered demand key is not +// D's key, so D has no coverage. Exact keys are deliberately stricter than +// row completeness. +// - `durable-write-failed`: D's upstream load answers with `a1` and `a2`, +// but SQLite rejects the transaction that writes those rows. Neither the +// rows nor coverage may become durable, so D has no coverage. +// - `settled-empty`: D settles with no rows. The empty answer is complete. +// - `settled`: D settles with `a1` and `a2`. +// - `truncated`: D settles with `a1` and `a2`, then the source truncates. +// Coverage ends with the rows it described, so D has no coverage and +// SQLite holds no rows. The truncate replays D; that load never answers. +// - `truncated-in-flight`: D's load writes `a1` and `a2`, the source +// truncates, and only then does that load settle. The truncate made the +// load obsolete (law 6 in `ARCHITECTURE.md`: an obsolete acquisition cannot +// satisfy current demand), so its settlement records no coverage. SQLite +// holds no rows, and the replayed load never answers. +// - `recovered-in-flight`: D's load writes `a1` and `a2`, the source starts +// scoped recovery, which rotates the persisted cache generation, and only +// then does that load settle. The obsolete load records no coverage in the +// new generation, which holds no rows; the reacquired load never answers. +// - `settled-then-row-deleted`: D settles with `a1` and `a2`, then the +// source deletes `a1`. SQLite holds `a2`, fewer rows than D had when it +// settled, so D has no coverage. +// - `recovering-in-flight`: as `recovered-in-flight`, but the load settles +// while scoped recovery is still pending. Recording coverage must not +// change the outcome of a load whose rows applied: without the coverage +// record, this history leaves the session-1 query ready, and the driver +// checks that it does not enter `error`. +// +// Session 2 asks for D again with a configured wait of 0 ms or 1000 ms. Its +// upstream load follows one schedule: +// +// - `pending`: the network never answers (offline). +// - `fails-at-once`: the load rejects as soon as the wrapper forwards it, as +// a fetch does when the device is offline, so it lands before the +// `restored` checkpoint. Session 2's reads answer on a later macrotask, so +// the rejection arrives while the wrapper still reads coverage. The wrapper +// must observe it there: an unhandled rejection fails the run. +// - `fails-early`: the load rejects before the wait passes. +// - `answers-early`: the load answers before the wait passes. +// - `answers-late`: the load answers after the wait passes. +// - `answers-empty-late`: the load answers after the wait with no rows. +// - `fails-late`: the load rejects after the wait passes. +// +// With a 0 ms wait, "early" events still follow the restore checkpoint, so +// they arrive after the wait has passed. +// +// Ablation (ORC-004): each value is needed by a case below. +// `not-requested` and `failed` reject "any local read counts"; +// `overlapping-demand` rejects "coverage per Collection" and "matching rows +// count"; `other-window` rejects "coverage keyed by filter only"; +// `other-order` rejects "coverage ignores order"; +// `durable-write-failed` rejects "coverage written before the rows commit"; +// `settled-empty` rejects "nonempty rows count"; `truncated` rejects +// "coverage survives truncate"; `truncated-in-flight` and +// `recovered-in-flight` reject "any settled load records coverage", including +// a load that a reset made obsolete; `recovering-in-flight` rejects "a failed +// coverage record fails the load"; `settled-then-row-deleted` rejects +// "coverage ignores rows deleted after it settled"; `settled` carries every positive case. The 1000 ms wait rejects "wait ignored" and "half deadline". +// `fails-early` rejects "failure errors a covered demand"; `fails-at-once` +// rejects "a failure during the coverage check goes unobserved"; `answers-late` +// rejects "upstream aborted after SQLite"; `fails-late` rejects "late +// failure undoes a settlement"; `answers-empty-late` checks that an empty +// answer settles an uncovered demand and leaves a covered one unchanged; +// `answers-early` checks that the network wins before the deadline. +// Excluded: a session-2 event before the persisted subset restore publishes. +// The wrapper forwards a load upstream only after that restore is visible, so +// no upstream answer or failure can precede it. The driver checks this: the +// first session-2 load must see every restored row when it is forwarded. +// Also excluded, because it is not a +// legal history: a source truncate transaction kept open while a later load +// of D hydrates and records coverage. A truncate clears hydrated demand, so +// that load must hydrate, and the wrapper rejects a source transaction that +// crosses a hydration cycle (`InvalidPersistedCollectionConfigError`). +// --------------------------------------------------------------------------- + +const COVERAGE_HISTORIES = [ + `not-requested`, + `failed`, + `overlapping-demand`, + `other-window`, + `other-order`, + `durable-write-failed`, + `settled-empty`, + `settled`, + `truncated`, + `truncated-in-flight`, + `recovered-in-flight`, + `recovering-in-flight`, + `settled-then-row-deleted`, +] as const +type CoverageHistory = (typeof COVERAGE_HISTORIES)[number] + +const UPSTREAM_SCHEDULES = [ + `pending`, + `fails-at-once`, + `fails-early`, + `answers-early`, + `answers-late`, + `answers-empty-late`, + `fails-late`, +] as const +type UpstreamSchedule = (typeof UPSTREAM_SCHEDULES)[number] + +const WAITS = [0, 1000] as const +type Wait = (typeof WAITS)[number] + +type Cell = { + history: CoverageHistory + waitMs: Wait + upstream: UpstreamSchedule +} + +const A1: Row = { id: `a1`, group: `a`, rank: 1, value: `v1` } +const A2: Row = { id: `a2`, group: `a`, rank: 2, value: `v1` } + +const SESSION_1_ROWS: Record> = { + 'not-requested': [], + failed: [], + 'overlapping-demand': [A1], + 'other-window': [A1], + 'other-order': [A1, A2], + // The upstream answer was A1 and A2, but no row became durable. + 'durable-write-failed': [], + 'settled-empty': [], + settled: [A1, A2], + truncated: [], + 'truncated-in-flight': [], + 'recovered-in-flight': [], + 'recovering-in-flight': [], + 'settled-then-row-deleted': [A2], +} + +// Session 2's upstream answer updates a1 and adds a3. It omits a2; an +// on-demand load reports rows, not deletions, so a2 stays. +const SESSION_2_ANSWER: ReadonlyArray = [ + { id: `a1`, group: `a`, rank: 1, value: `v2` }, + { id: `a3`, group: `a`, rank: 3, value: `v2` }, +] + +// --------------------------------------------------------------------------- +// Observations and checkpoints +// +// The query reads only D, so its initial-query readiness is D's load +// settlement. The model therefore predicts three public facts together: +// `query.status`, the state of the session-2 `preload()` promise, and, when +// ready, the exact rows. Rows are not compared while loading; this law does +// not govern partial results before settlement. +// +// Checkpoints, in order: +// +// - `restored`: the upstream load has started and microtasks have drained, +// before fake time advances. The persisted subset restore has had every +// chance to publish. A slower publication fails the ready cells here; it +// cannot pass them. +// - `afterEarlyEvent`: after an early upstream answer or failure. +// - `beforeDeadline`: fake time is 1 ms before the wait passes. +// - `afterDeadline`: fake time has reached the wait. +// - `afterLateEvent`: after a late upstream answer or failure. +// --------------------------------------------------------------------------- + +type Observation = { + queryStatus: `loading` | `ready` | `error` + preload: `pending` | `resolved` | `rejected` + rows?: ReadonlyArray +} + +type Trace = { + restored: Observation + afterEarlyEvent: Observation + beforeDeadline: Observation + afterDeadline: Observation + afterLateEvent: Observation +} + +const CHECKPOINTS = [ + `restored`, + `afterEarlyEvent`, + `beforeDeadline`, + `afterDeadline`, + `afterLateEvent`, +] as const satisfies ReadonlyArray + +function rowLabels(rows: Iterable): Array { + return Array.from(rows, (row) => `${row.id}:${row.value}`).sort() +} + +// --------------------------------------------------------------------------- +// Independent model +// +// The model keeps four facts: whether D is covered, the rows visible to D, +// whether D's load has settled, and whether it failed. Coverage comes from +// the session-1 history label alone; the model never inspects SQLite or the +// wrapper. Each fact changes a later observation: +// +// - coverage decides whether the wait or an upstream failure settles D; +// - visible rows are the exact rows at settlement and after a later answer; +// - settlement separates `ready` from `loading`; +// - failure separates `error` from `loading` for an unsettled demand. +// +// Vocabulary (ORC-009): `covered` is persisted demand coverage; the initial +// visible rows are the persisted subset restore; `settled` is D's load +// settlement, observed through initial-query readiness and `preload()`. +// +// `Design` lets the calibration below express plausible wrong designs as +// variants of the same rule. The law is `LAW`. +// --------------------------------------------------------------------------- + +type Design = { + name: string + isCovered: (history: CoverageHistory) => boolean + // `false` settles a covered demand at restore, ignoring the wait. + honorsWait: boolean + // Fraction of the wait after which a covered demand settles. + deadlineFraction: number + // A covered demand settles when its upstream load fails. + failureSettlesCovered: boolean + // Upstream rows still publish after a settlement from SQLite. + upstreamContinues: boolean + // A later upstream failure leaves a settled demand ready. + settlementSurvivesFailure: boolean + // Settles before the persisted subset restore publishes its rows. + settlesBeforeRestore: boolean +} + +const LAW: Design = { + name: `persisted demand coverage`, + isCovered: (history) => history === `settled` || history === `settled-empty`, + honorsWait: true, + deadlineFraction: 1, + failureSettlesCovered: true, + upstreamContinues: true, + settlementSurvivesFailure: true, + settlesBeforeRestore: false, +} + +function expectedTrace(cell: Cell, design: Design = LAW): Trace { + const covered = design.isCovered(cell.history) + const visible = new Map( + SESSION_1_ROWS[cell.history].map((row) => [row.id, row]), + ) + const state = { + settled: false, + settledFromSqlite: false, + failed: false, + // Set only by the wrong design that settles before the restore publishes. + rowsAtEarlySettle: undefined as Array | undefined, + } + + const observe = (): Observation => { + if (state.failed) return { queryStatus: `error`, preload: `rejected` } + if (!state.settled) return { queryStatus: `loading`, preload: `pending` } + return { + queryStatus: `ready`, + preload: `resolved`, + rows: state.rowsAtEarlySettle ?? rowLabels(visible.values()), + } + } + const settleFromSqlite = () => { + if (state.settled || state.failed) return + state.settled = true + state.settledFromSqlite = true + } + const answer = (rows: ReadonlyArray) => { + if (state.failed) return + if (state.settledFromSqlite && !design.upstreamContinues) return + for (const row of rows) visible.set(row.id, row) + state.rowsAtEarlySettle = undefined + state.settled = true + } + const fail = () => { + if (state.failed) return + if (!state.settled) { + if (covered && design.failureSettlesCovered) settleFromSqlite() + else state.failed = true + } else if (!design.settlementSurvivesFailure) { + state.failed = true + } + } + + // restored (t = 0) + if (covered && (cell.waitMs === 0 || !design.honorsWait)) settleFromSqlite() + if (cell.upstream === `fails-at-once`) fail() + if (design.settlesBeforeRestore && covered) state.rowsAtEarlySettle = [] + const restored = observe() + + // early event (t = 0) + if (cell.upstream === `answers-early`) answer(SESSION_2_ANSWER) + if (cell.upstream === `fails-early`) fail() + const afterEarlyEvent = observe() + + // t = wait - 1 + if ( + covered && + cell.waitMs > 0 && + design.deadlineFraction * cell.waitMs <= cell.waitMs - 1 + ) { + settleFromSqlite() + } + const beforeDeadline = cell.waitMs === 0 ? afterEarlyEvent : observe() + + // t = wait + if (covered) settleFromSqlite() + const afterDeadline = observe() + + // late event + if (cell.upstream === `answers-late`) answer(SESSION_2_ANSWER) + if (cell.upstream === `answers-empty-late`) answer([]) + if (cell.upstream === `fails-late`) fail() + const afterLateEvent = observe() + + return { + restored, + afterEarlyEvent, + beforeDeadline, + afterDeadline, + afterLateEvent, + } +} + +// --------------------------------------------------------------------------- +// Refinement check +// +// Production refines the model when every checkpoint agrees on status, +// preload state, and, when the model is ready, the exact sorted rows. The +// recorder keeps duplicates and order, so the comparison can reject both. +// --------------------------------------------------------------------------- + +function sameObservation(expected: Observation, actual: Observation): boolean { + if (expected.queryStatus !== actual.queryStatus) return false + if (expected.preload !== actual.preload) return false + if (expected.rows === undefined) return true + return ( + actual.rows !== undefined && + actual.rows.length === expected.rows.length && + expected.rows.every((row, index) => actual.rows![index] === row) + ) +} + +function firstMismatch( + expected: Trace, + actual: Partial, +): + | { checkpoint: keyof Trace; expected: Observation; actual?: Observation } + | undefined { + for (const checkpoint of CHECKPOINTS) { + const observed = actual[checkpoint] + if (!observed || !sameObservation(expected[checkpoint], observed)) { + return { checkpoint, expected: expected[checkpoint], actual: observed } + } + } + return undefined +} + +const ALL_CELLS: ReadonlyArray = COVERAGE_HISTORIES.flatMap((history) => + WAITS.flatMap((waitMs) => + UPSTREAM_SCHEDULES.map((upstream) => ({ history, waitMs, upstream })), + ), +) + +function cellName(cell: Cell): string { + return `${cell.history}, wait ${cell.waitMs} ms, upstream ${cell.upstream}` +} + +// --------------------------------------------------------------------------- +// Production driver +// --------------------------------------------------------------------------- +// The generic source, clock, and cleanup helpers live in +// `tests/helpers/demand-coverage-driver.ts`. + +function session1Query( + source: Collection, + history: CoverageHistory, +) { + if (history === `overlapping-demand`) { + return createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => eq(row.id, `a1`)), + ) + } + if (history === `other-window`) { + // A numeric order keeps the window expressible, so the wrapper sends a + // limited load and a tie-boundary load instead of a full-source load. + return createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.group, `a`)) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + } + if (history === `other-order`) { + return createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.group, `a`)) + .orderBy(({ row }) => row.rank), + ) + } + return filterQuery(source, `a`) +} + +// An adapter that rejects every committed transaction that writes rows. +// Transactions with only metadata pass, so an implementation that writes +// coverage separately from the rows makes coverage durable without them. +function rejectingRowWrites(adapter: SQLiteCorePersistenceAdapter) { + const rejected = { count: 0 } + const proxy = new Proxy(adapter, { + get(target, property) { + if (property === `applyCommittedTx`) { + return (...args: Parameters) => { + if (args[1].mutations.length > 0) { + rejected.count++ + return Promise.reject(new Error(`injected row-write failure`)) + } + return target.applyCommittedTx(...args) + } + } + const value: unknown = Reflect.get(target, property, target) + return typeof value === `function` ? value.bind(target) : value + }, + }) + return { adapter: proxy, rejected } +} + +// Runs session 1 online and returns the demand keys it sent upstream. +async function settleSession1( + driver: SQLiteDriver, + history: CoverageHistory, +): Promise> { + if (history === `not-requested`) return [] + const base = new SQLiteCorePersistenceAdapter({ driver }) + const writes = + history === `durable-write-failed` ? rejectingRowWrites(base) : undefined + const { collection, upstream } = onDemandSource(writes?.adapter ?? base, 0) + const query = session1Query(collection, history) + return withCleanup(async () => { + const preloadState = { settled: false } + const isSettled = () => preloadState.settled + const preload = query.preload().finally(() => (preloadState.settled = true)) + // A failing history rejects before the loop below inspects the promise. + void preload.catch(() => undefined) + // Answer each upstream load in turn until preload settles; a window + // sends its limited load and then its tie-boundary load. + for (;;) { + await vi.waitFor(() => + expect(isSettled() || upstream.pendingCount() > 0).toBe(true), + ) + if (isSettled()) break + if (history === `failed`) { + upstream.fail(new Error(`session 1 upstream failed`)) + } else if ( + history === `truncated-in-flight` || + history === `recovered-in-flight` + ) { + const loadsBefore = upstream.calls.length + await upstream.write([A1, A2]) + if (history === `truncated-in-flight`) await upstream.truncate() + else await upstream.startScopedRecovery() + // Positive witness: the reset reacquired D while its first load was + // still pending. + await vi.waitFor(() => + expect(upstream.calls.length).toBeGreaterThan(loadsBefore), + ) + upstream.settleOldest() + await new Promise((resolve) => setTimeout(resolve, 200)) + break + } else if (history === `recovering-in-flight`) { + await upstream.write([A1, A2]) + const recovery = upstream.startScopedRecovery() + upstream.settleOldest() + await recovery + await new Promise((resolve) => setTimeout(resolve, 200)) + expect(query.status).not.toBe(`error`) + break + } else if (history === `durable-write-failed`) { + await upstream.answer([A1, A2]).catch(() => undefined) + // The load may reject or stay pending after a durable failure; the + // law only needs the rows and coverage to stay out of SQLite. + await Promise.race([ + preload.catch(() => undefined), + new Promise((resolve) => setTimeout(resolve, 200)), + ]) + // Positive witness: SQLite really rejected the row write. + expect(writes?.rejected.count).toBeGreaterThan(0) + break + } else { + // This history deletes a1 only after D settles with both rows. + await upstream.answer( + history === `settled-then-row-deleted` + ? [A1, A2] + : SESSION_1_ROWS[history], + ) + } + await new Promise((resolve) => setTimeout(resolve, 0)) + } + if ( + history === `settled-empty` || + history === `settled` || + history === `truncated` || + history === `settled-then-row-deleted` + ) { + await preload + } else if ( + history !== `truncated-in-flight` && + history !== `recovered-in-flight` && + history !== `recovering-in-flight` + ) { + // The in-flight histories leave the reacquired load pending, so their + // preload never settles. + await preload.catch(() => undefined) + } + if (history === `settled-then-row-deleted`) { + await upstream.remove([A1]) + // Positive witness: the delete reached the Collection. + await vi.waitFor(() => + expect(collection.toArray.map((row) => row.id)).toEqual([`a2`]), + ) + await new Promise((resolve) => setTimeout(resolve, 50)) + } + if (history === `truncated`) { + const loadsBefore = upstream.calls.length + await upstream.truncate() + // Positive witness: the truncate replayed D, and that load stays + // pending, so it cannot record coverage again. + await vi.waitFor(() => + expect(upstream.calls.length).toBeGreaterThan(loadsBefore), + ) + expect(upstream.pendingCount()).toBeGreaterThan(0) + } + return upstream.calls.map(getLoadSubsetDemandKey) + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +function observeQuery( + query: Collection, + preload: () => Observation[`preload`], +): Observation { + const status = query.status + const queryStatus = + status === `ready` || status === `error` ? status : `loading` + return { + queryStatus, + preload: preload(), + ...(queryStatus === `ready` ? { rows: rowLabels(query.toArray) } : {}), + } +} + +async function runCell(cell: Cell): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1Keys = await settleSession1(driver, cell.history) + + // Session 2. Configuration throws here on an implementation that rejects + // network-first for on-demand sources; that is a setup failure, not a + // checkpoint mismatch. + useFakeClock() + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver: withMacrotaskReads(driver) }), + cell.waitMs, + ) + if (cell.upstream === `fails-at-once`) upstream.rejectAtOnce = true + const query = filterQuery(collection, `a`) + const preload = trackPreload(query) + const observe = () => observeQuery(query, preload) + const actual: Partial = {} + return withCleanup(async () => { + await untilUpstreamCalled(upstream) + // Positive witness: the session-1 history really did or did not send + // D's exact demand key. + const keyD = getLoadSubsetDemandKey(upstream.calls[0]!) + expect(session1Keys.includes(keyD)).toBe( + cell.history === `settled` || + cell.history === `settled-empty` || + cell.history === `truncated` || + cell.history === `truncated-in-flight` || + cell.history === `recovered-in-flight` || + cell.history === `recovering-in-flight` || + cell.history === `settled-then-row-deleted` || + cell.history === `failed` || + cell.history === `durable-write-failed`, + ) + // The wrapper forwarded the load only after its restore was visible. + expect(upstream.visibleAtCall[0]).toEqual( + expect.arrayContaining(SESSION_1_ROWS[cell.history].map((row) => row.id)), + ) + await drain() + // Checkpoint witness: the persisted subset restore has published every + // durable row that matches D. Missing rows mean the checkpoint was not + // reached, not that the law was violated. + const restoredIds = new Set( + collection.toArray + .filter((row) => row.group === `a`) + .map((row) => row.id), + ) + const missing = SESSION_1_ROWS[cell.history].filter( + (row) => !restoredIds.has(row.id), + ) + if (missing.length > 0) { + throw new Error( + `checkpoint not reached: restore had not published ${missing.map((row) => row.id).join(`, `)}`, + ) + } + actual.restored = observe() + + if (cell.upstream === `answers-early`) { + await upstream.answer(SESSION_2_ANSWER) + } + if (cell.upstream === `fails-early`) upstream.fail(new Error(`offline`)) + await drain() + actual.afterEarlyEvent = observe() + + if (cell.waitMs > 0) { + await vi.advanceTimersByTimeAsync(cell.waitMs - 1) + await drain() + } + actual.beforeDeadline = observe() + + await vi.advanceTimersByTimeAsync(cell.waitMs === 0 ? 0 : 1) + await drain() + actual.afterDeadline = observe() + + if (cell.upstream === `answers-late`) { + await upstream.answer(SESSION_2_ANSWER) + } + if (cell.upstream === `answers-empty-late`) await upstream.answer([]) + if (cell.upstream === `fails-late`) upstream.fail(new Error(`offline`)) + await drain() + actual.afterLateEvent = observe() + return actual + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +// =========================================================================== +// Part 2: load lifecycle within one session +// +// Parts 2 to 4 are fixed histories. Each records what happened at the real +// wrapper (the source's log, each query's sent loads and observation, and the +// premises it injected) and passes that record to the shared judgment in +// `tests/persisted-demand-coverage-model-oracle.ts`. None states an expected +// answer (ORC-016); the model predicts each observation from the record. +// +// Two covered demands, D (`group = 'a'`) and E (`group = 'b'`), and an +// offline network in session 2 with a 1000 ms wait. Fixed history: +// +// - t = 0: query 1 starts D's load. This also starts the sync run. +// - t = 300: query 1 is cleaned up, releasing D. Its upstream load aborts. +// - t = 400: query 2 starts a new load of D. +// - t = 600: query 3 starts E's load. +// +// Each load measures its wait from its own start, so query 2 settles at 1400 +// and query 3 at 1600, and release keeps coverage. Checkpoints sit 1 ms +// either side of 1000, 1400, and 1600, so a deadline measured from the sync +// run, carried over from the released load, or dropped with coverage on +// release changes an observation the judgment checks. Each query's rows are +// compared when it is ready, so a load cannot settle on rows from a sibling +// demand's restore. +// =========================================================================== + +// The shared comparison. Every fixed history below is an input to it. +function judge(records: ReadonlyArray): void { + expect(records.length).toBeGreaterThan(0) + for (const record of records) expect(coverageViolations(record)).toEqual([]) +} + +function observed(query: Collection): QueryObservation { + return query.status === `ready` + ? { + status: `ready`, + rows: (query.toArray as Array).map((row) => row.id), + } + : { status: query.status === `error` ? `error` : `loading` } +} + +// A load's wait has passed once this much time passed since its source +// received it; the wrapper starts the load before it forwards it. +function waitPassedAt( + upstream: { arrivedAt: ReadonlyMap }, + waitMs: number, + at = Date.now(), +): WaitPremise { + return (load) => at - (upstream.arrivedAt.get(load) ?? Infinity) >= waitMs +} + +const D: QuerySpec = { group: `a` } +const E: QuerySpec = { group: `b` } + +// Marks the source's last settled load as one whose coverage read the +// history failed, a premise the driver injected. +function coverageReadFailed(events: Array): void { + for (let index = events.length - 1; index >= 0; index--) { + const event = events[index]! + if (event.type === `settled`) { + event.coverage = `read fails` + return + } + } + throw new Error(`no settled load to mark`) +} + +const LIFECYCLE_CHECKPOINTS = [999, 1000, 1399, 1400, 1599, 1600] as const + +async function runLifecycle(): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + let session1: ReadonlyArray = [] + { + const adapter = new SQLiteCorePersistenceAdapter({ driver }) + const { collection, upstream } = onDemandSource(adapter, 0) + session1 = upstream.events + const d = filterQuery(collection, `a`) + const e = filterQuery(collection, `b`) + await withCleanup(async () => { + const loads = Promise.all([d.preload(), e.preload()]) + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(2)) + await upstream.answer([ + A1, + A2, + { id: `b1`, group: `b`, rank: 1, value: `v1` }, + ]) + await loads + }, [() => d.cleanup(), () => e.cleanup(), () => collection.cleanup()]) + } + + useFakeClock() + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 1000, + ) + const query1 = filterQuery(collection, `a`) + let query2: ReturnType | undefined + let query3: ReturnType | undefined + const records: Array = [] + return withCleanup(async () => { + let now = 0 + const advanceTo = async (t: number) => { + await vi.advanceTimersByTimeAsync(t - now) + now = t + await drain() + } + void query1.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, 1) + await advanceTo(300) + await query1.cleanup() + await drain() + // Positive witness: the release aborted query 1's upstream load. + expect(upstream.calls[0]!.signal?.aborted).toBe(true) + await advanceTo(400) + query2 = filterQuery(collection, `a`) + void query2.preload() + await untilUpstreamCalled(upstream, 2) + await advanceTo(600) + query3 = filterQuery(collection, `b`) + void query3.preload() + await untilUpstreamCalled(upstream, 3) + for (const t of LIFECYCLE_CHECKPOINTS) { + await advanceTo(t) + records.push({ + events: [...session1, ...upstream.events], + queries: { + query2: { + spec: D, + sent: [upstream.calls[1]!], + observed: observed(query2), + }, + query3: { + spec: E, + sent: [upstream.calls[2]!], + observed: observed(query3), + }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(upstream, 1000), + // The offline source never calls `markReady()`. + collection: { status: collection.status, sourceMarkedReady: false }, + }) + } + return records + }, [ + () => vi.useRealTimers(), + () => query2?.cleanup(), + () => query3?.cleanup(), + () => collection.cleanup(), + ]) +} + +// =========================================================================== +// Part 2b: resets and refetch within one session +// +// Session 1 covers D online. Then, in session 2: +// +// - Rotation: a load of D settles from SQLite, its query is released, and +// the source starts scoped recovery, which ends the persisted cache +// generation. A new load of D must wait for the network. +// - Refetch: a load of D with `refetch: true`, which never settles from +// SQLite (`ARCHITECTURE.md`, "Demand grouping and ownership"). +// - Schema reset: session 2 opens the database with another `schemaVersion`, +// which deletes the rows and their metadata: a generation change. +// - Cache-claim expiry before the coverage record, and before the coverage +// check. A read under an expired claim proves nothing, so the first load +// records nothing yet settles as its upstream load did, and the second +// load waits for the network. Both are premises the driver injects and +// marks in the record. +// +// - Recovery reload: after scoped recovery, D loads online again and its +// query is released; then D is asked for once more while the network is +// silent. Scoped recovery leaves the rest of its sync run uncertified, and +// its persisted subset restore publishes no rows, so the online load +// records nothing and the last load must wait for the network. +// +// - Truncate inside the load: the source truncates synchronously inside its +// own `loadSubset` call for D, then settles only that obsolete load. A +// truncate came between the source receiving the load and settling it, so +// the load records nothing, and session 2 waits. +// +// An empty covered demand isolates the generation check in the rotation +// history: no row count can end its coverage, so only the generation does. +// =========================================================================== + +async function coverD( + driver: SQLiteDriver, + rows: ReadonlyArray = [A1, A2], +): Promise<{ + key: ReturnType + events: Array +}> { + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + const preload = query.preload() + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(1)) + await upstream.answer(rows) + await preload + return { + key: getLoadSubsetDemandKey(upstream.calls[0]!), + events: upstream.events, + } + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +async function runSessionReset( + kind: `rotation` | `refetch`, + rows: ReadonlyArray = [A1, A2], +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1 = await coverD(driver, rows) + useFakeClock() + const waitMs = kind === `rotation` ? 0 : 1000 + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + waitMs, + ) + const events = () => [...session1.events, ...upstream.events] + const first = filterQuery(collection, `a`) + let second: ReturnType | undefined + let subscription: { unsubscribe: () => void } | undefined + const records: Array = [] + return withCleanup(async () => { + void first.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, 1) + await drain() + await vi.advanceTimersByTimeAsync(waitMs) + await drain() + records.push({ + events: events(), + queries: { + first: { + spec: D, + sent: [upstream.calls[0]!], + observed: observed(first), + }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(upstream, waitMs), + }) + const callsBefore = upstream.calls.length + const later = { settled: false } + if (kind === `rotation`) { + await first.cleanup() + await drain() + await upstream.startScopedRecovery() + await drain() + second = filterQuery(collection, `a`) + void second.preload().then( + () => (later.settled = true), + () => (later.settled = true), + ) + } else { + const changes = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription = changes + ;( + changes as unknown as { + requestSnapshot: (options: object) => unknown + } + ).requestSnapshot({ + where: new IR.Func(`eq`, [ + new IR.PropRef([`group`]), + new IR.Value(`a`), + ]), + refetch: true, + onLoadSubsetResult: (result: true | Promise) => { + if (result === true) later.settled = true + else + void result.then( + () => (later.settled = true), + () => (later.settled = true), + ) + }, + }) + } + await vi.waitFor(() => + expect(upstream.calls.length).toBeGreaterThan(callsBefore), + ) + if (kind === `refetch`) { + // Positive witness: the refetch reached the source as D's demand. + const refetchLoad = upstream.calls[upstream.calls.length - 1]! + expect(refetchLoad.refetch).toBe(true) + expect(getLoadSubsetDemandKey(refetchLoad)).toBe(session1.key) + } + await drain() + await vi.advanceTimersByTimeAsync(waitMs + 1) + await drain() + records.push({ + events: events(), + queries: { + later: { + spec: D, + sent: upstream.calls.slice(callsBefore), + observed: later.settled + ? { + status: `ready`, + rows: collection.toArray + .filter((row) => row.group === `a`) + .map((row) => row.id), + } + : { status: `loading` }, + }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(upstream, waitMs), + }) + return records + }, [ + () => vi.useRealTimers(), + () => subscription?.unsubscribe(), + () => first.cleanup(), + () => second?.cleanup(), + () => collection.cleanup(), + ]) +} + +async function runRecoveryReload(): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1 = await coverD(driver) + useFakeClock() + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const queries: Array> = [] + const ask = async () => { + const query = filterQuery(collection, `a`) + queries.push(query) + const calls = upstream.calls.length + void query.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, calls + 1) + await drain() + return query + } + return withCleanup(async () => { + // A first load of D starts the sync run, and so lets recovery start. + const first = await ask() + await vi.advanceTimersByTimeAsync(1) + await drain() + await first.cleanup() + await drain() + await upstream.startScopedRecovery() + await drain() + // The online load during scoped recovery. + const online = await ask() + await upstream.answer([A1, A2]) + await drain() + // Positive witness: the online load settled. + expect(online.status).toBe(`ready`) + await online.cleanup() + await drain() + const callsBefore = upstream.calls.length + const last = await ask() + await vi.advanceTimersByTimeAsync(1) + await drain() + const record: CoverageRecord = { + events: [...session1.events, ...upstream.events], + queries: { + last: { + spec: D, + sent: upstream.calls.slice(callsBefore), + observed: observed(last), + }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(upstream, 0), + } + // Positive witness: the model saw D settle during scoped recovery. + const reached = new Set() + coverageViolations(record, (state) => reached.add(state)) + expect(reached.has(`covered during scoped recovery`)).toBe(true) + return [record] + }, [ + () => vi.useRealTimers(), + ...[0, 1, 2].map((index) => () => queries[index]?.cleanup()), + () => collection.cleanup(), + ]) +} + +async function runTruncateInsideLoad(): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1: Array = [] + { + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + upstream.truncateInNextLoad = true + const query = filterQuery(collection, `a`) + await withCleanup(async () => { + void query.preload().catch(() => undefined) + await vi.waitFor(() => expect(upstream.pendingCount()).toBeGreaterThan(0)) + // Positive witness: the truncate ran inside the source's load call. + expect(upstream.truncateInNextLoad).toBe(false) + await upstream.write([A1, A2]) + upstream.settleOldest() + await new Promise((resolve) => setTimeout(resolve, 200)) + session1.push(...upstream.events) + }, [() => query.cleanup(), () => collection.cleanup()]) + } + useFakeClock() + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, 1) + await drain() + await vi.advanceTimersByTimeAsync(1) + await drain() + return [ + { + events: [...session1, ...upstream.events], + queries: { + d: { spec: D, sent: [...upstream.calls], observed: observed(query) }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(upstream, 0), + }, + ] + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +async function runSchemaReset(): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + // An empty covered demand: no row count can end its coverage, so only the + // reset's removal of collection metadata does. + const session1 = await coverD(driver, []) + useFakeClock() + // The SQLite adapter owns the schema version; a new one resets the rows. + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver, schemaVersion: 99 }), + 0, + ) + const query = filterQuery(collection, `a`) + void query.preload().catch(() => undefined) + return withCleanup(async () => { + await untilUpstreamCalled(upstream, 1) + await drain() + await vi.advanceTimersByTimeAsync(1) + await drain() + return [ + { + // The new schema version is the premise: a generation change. + events: [ + ...session1.events, + { type: `generation ends` }, + ...upstream.events, + ], + queries: { + d: { spec: D, sent: upstream.calls, observed: observed(query) }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(upstream, 0), + }, + ] + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +async function runClaimExpiredBeforeRecord(): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const clock = { now: Date.now() } + const adapter = new SQLiteCorePersistenceAdapter({ + driver, + now: () => clock.now, + cacheGenerationClaimTtlMs: 60_000, + }) + const { collection, upstream } = onDemandSource(adapter, 0) + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(1)) + await upstream.write([A1, A2]) + // The claim expires after the rows committed and before the load settles. + clock.now += 10 * 60_000 + upstream.settleOldest() + coverageReadFailed(upstream.events) + await new Promise((resolve) => setTimeout(resolve, 300)) + return [ + { + events: upstream.events, + queries: { + d: { + spec: D, + sent: upstream.calls.slice(0, 1), + observed: observed(query), + }, + }, + waitPassed: true, + }, + ] + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +async function runClaimExpiredBeforeCheck( + rows: ReadonlyArray = [A1, A2], +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1 = await coverD(driver, rows) + const clock = { now: Date.now() } + const base = new SQLiteCorePersistenceAdapter({ + driver, + now: () => clock.now, + cacheGenerationClaimTtlMs: 60_000, + }) + // The restore reads D under a live claim, and the wrapper forwards D's + // load upstream only once that restore is visible. The claim expires just + // before the next read of D, the coverage check. An empty covered demand + // isolates the generation check: after claim recovery rotates the cache, + // no row count can end the old record, so only its generation does. + const holder: { upstream?: Upstream } = {} + const reads = { afterRestore: 0 } + const adapter = new Proxy(base, { + get(target, property) { + if (property === `loadSubset`) { + return (...args: Parameters) => { + if ((holder.upstream?.calls.length ?? 0) > 0) { + if (++reads.afterRestore === 1) clock.now += 10 * 60_000 + } + return target.loadSubset(...args) + } + } + const value: unknown = Reflect.get(target, property, target) + return typeof value === `function` ? value.bind(target) : value + }, + }) + const { collection, upstream } = onDemandSource(adapter, 0) + holder.upstream = upstream + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await vi.waitFor(() => expect(upstream.calls.length).toBeGreaterThan(0)) + await new Promise((resolve) => setTimeout(resolve, 300)) + // Positive witnesses: the coverage check read D after the claim expired, + // and claim recovery, which rotates the cache, loaded D again. + expect(reads.afterRestore).toBeGreaterThan(0) + expect(upstream.calls.length).toBeGreaterThan(1) + return [ + { + events: [...session1.events, ...upstream.events], + queries: { + d: { + spec: D, + sent: upstream.calls, + observed: observed(query), + checkFails: true, + }, + }, + upstream: `never answers`, + waitPassed: true, + }, + ] + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +// =========================================================================== +// Part 2c: coverage bookkeeping +// +// Coverage is bookkeeping beside a load, so it must not change the load's +// own outcome or lifetime (the law record's rules on applied loads): +// +// - A coverage read that fails after the load's rows applied records no +// coverage. The load still settles as its upstream load did. +// - A coverage read that fails during the coverage check proves nothing, so +// the load waits for its upstream load, as an uncovered load does. +// - A source that answers synchronously writes its rows before it returns. +// Its coverage must count those rows, so a later deletion of one ends it. +// The durable write lands a few macrotasks late, so a coverage read that +// does not wait behind it would count too few rows. +// - An unordered demand that loses one row and gains another keeps its row +// count, so it stays covered (maintainer decision, 2026-10-26). +// - A truncate during an empty load's coverage read leaves no coverage. +// - A coverage read or a cache-claim renewal that outlasts the wait lets D +// settle as soon as it returns: the wait runs from the load's start. +// +// The release laws (a released load owns no wait and starts no upstream +// load) are swept over every step of a load in Part 3 of the campaign file. +// =========================================================================== + +// An adapter whose reads the test can fail or hold. `next` applies to the +// first read after `armed()` returns true. +function controlledReads( + adapter: SQLiteCorePersistenceAdapter, + armed: () => boolean, + next: `fail` | `hold`, +) { + const reads = { failed: 0, held: 0, release: () => {} } + let used = false + const proxy = new Proxy(adapter, { + get(target, property) { + if (property === `loadSubset`) { + return (...args: Parameters) => { + if (used || !armed()) return target.loadSubset(...args) + used = true + if (next === `fail`) { + reads.failed++ + return Promise.reject(new Error(`injected read failure`)) + } + reads.held++ + return new Promise((resolve) => { + reads.release = resolve + }).then(() => target.loadSubset(...args)) + } + } + const value: unknown = Reflect.get(target, property, target) + return typeof value === `function` ? value.bind(target) : value + }, + }) + return { adapter: proxy, reads } +} + +// Each durable write lands a few macrotasks late, as with a worker driver, +// so a coverage read that does not wait behind it would count too few rows. +function slowDurableWrites(adapter: SQLiteCorePersistenceAdapter) { + return new Proxy(adapter, { + get(target, property) { + if (property === `applyCommittedTx`) { + return async (...args: Parameters) => { + for (let i = 0; i < 3; i++) { + await new Promise((resolve) => setImmediate(resolve)) + } + return target.applyCommittedTx(...args) + } + } + const value: unknown = Reflect.get(target, property, target) + return typeof value === `function` ? value.bind(target) : value + }, + }) +} + +async function runCoverageReadFailsAfterLoad(): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const state = { armed: false } + const { adapter, reads } = controlledReads( + new SQLiteCorePersistenceAdapter({ driver }), + () => state.armed, + `fail`, + ) + const { collection, upstream } = onDemandSource(adapter, 0) + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(1)) + // The next read after the answer is the coverage read. + state.armed = true + await upstream.answer([A1, A2]) + coverageReadFailed(upstream.events) + await new Promise((resolve) => setTimeout(resolve, 300)) + // Positive witness: the coverage read failed. + expect(reads.failed).toBe(1) + return [ + { + events: upstream.events, + queries: { + d: { spec: D, sent: upstream.calls, observed: observed(query) }, + }, + waitPassed: true, + }, + ] + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +async function runCoverageReadFailsDuringCheck(): Promise< + Array +> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1 = await coverD(driver) + const holder: { collection?: Collection } = {} + // The first read after the restore published D's rows is the check. + const { adapter, reads } = controlledReads( + new SQLiteCorePersistenceAdapter({ driver }), + () => (holder.collection?.toArray.length ?? 0) > 0, + `fail`, + ) + const { collection, upstream } = onDemandSource(adapter, 0) + holder.collection = collection + const query = filterQuery(collection, `a`) + const record = (): CoverageRecord => ({ + events: [...session1.events, ...upstream.events], + queries: { + d: { + spec: D, + sent: upstream.calls.slice(0, 1), + observed: observed(query), + checkFails: true, + }, + }, + waitPassed: true, + }) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(1)) + await new Promise((resolve) => setTimeout(resolve, 300)) + const beforeAnswer = record() + await upstream.answer([A1, A2]) + await new Promise((resolve) => setTimeout(resolve, 100)) + // Positive witness: the coverage check read failed. + expect(reads.failed).toBe(1) + return [beforeAnswer, record()] + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +// A source that answers D synchronously, then does nothing, deletes `a1`, +// or truncates at once, while the coverage read of that answer still waits +// behind the slow durable write. A synchronous answer's load has already +// settled, so core cannot abort it; only the reset check stops a coverage +// record of an empty answer that a truncate made obsolete. +async function runSynchronousAnswer( + rows: ReadonlyArray, + after: `nothing` | `delete` | `truncate`, +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + // This source answers synchronously, so the driver records its log here. + const session1: Array = [] + const records: Array = [] + { + const handles: { + remove?: (row: Row) => void + truncate?: () => void + } = {} + const sent: Array = [] + // Same ID as `onDemandSource`, so session 2 restores these rows. + const collection = createCollection( + persistedCollectionOptions({ + id: `demand-coverage-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + initialRender: { strategy: `network-first`, networkTimeoutMs: 0 }, + persistence: { + adapter: slowDurableWrites( + new SQLiteCorePersistenceAdapter({ driver }), + ), + }, + sync: { + sync: ({ begin, write, commit, truncate }) => { + handles.remove = (row) => { + begin() + write({ type: `delete`, value: row }) + commit() + session1.push({ type: `delete`, id: row.id }) + } + handles.truncate = () => { + begin() + truncate() + commit() + session1.push({ type: `truncate` }) + } + return { + loadSubset: (options: LoadSubsetOptions) => { + sent.push(options) + // After a truncate, core replays D; the source leaves that + // load waiting, so only the reset check decides coverage. + if (after === `truncate` && sent.length > 1) { + return new Promise(() => {}) + } + begin() + for (const value of rows) write({ type: `insert`, value }) + commit() + session1.push( + { + type: `rows`, + rows: rows.map( + (row) => + [row.id, { group: row.group, rank: row.rank }] as const, + ), + }, + { type: `settled`, load: options, current: true }, + ) + return true + }, + } + }, + }, + }), + ) + const query = filterQuery(collection, `a`) + await withCleanup(async () => { + await query.preload() + // Session 1's checkpoint precedes any truncate: a truncate replay keeps + // an already ready query ready, which the model does not judge. + const checkpoint = () => + records.push({ + events: [...session1], + queries: { + d: { spec: D, sent: [...sent], observed: observed(query) }, + }, + waitPassed: true, + }) + if (after === `truncate`) { + checkpoint() + // The truncate lands while the answer's coverage read still waits. + handles.truncate!() + await new Promise((resolve) => setTimeout(resolve, 100)) + } else { + await new Promise((resolve) => setTimeout(resolve, 100)) + checkpoint() + } + if (after === `delete`) { + handles.remove!(A1) + await new Promise((resolve) => setTimeout(resolve, 100)) + } + }, [() => query.cleanup(), () => collection.cleanup()]) + } + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = filterQuery(collection, `a`) + void query.preload().catch(() => undefined) + return withCleanup(async () => { + await vi.waitFor(() => expect(upstream.calls.length).toBeGreaterThan(0)) + await new Promise((resolve) => setTimeout(resolve, 300)) + records.push({ + events: [...session1, ...upstream.events], + queries: { + d: { spec: D, sent: upstream.calls, observed: observed(query) }, + }, + upstream: `never answers`, + waitPassed: true, + }) + return records + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +// Session 1 of D, followed by tail events from the source while D is still +// active, then an offline session 2 with a wait of 0. The judgment checks D +// at the end of session 1's answer and again offline. +async function runTailHistory( + rows: ReadonlyArray, + tail: (upstream: Upstream) => Promise, + options: { holdCoverageRead?: boolean } = {}, +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const records: Array = [] + const state = { armed: false } + const { adapter, reads } = controlledReads( + new SQLiteCorePersistenceAdapter({ driver }), + () => state.armed, + `hold`, + ) + let session1: ReadonlyArray = [] + { + const { collection, upstream } = onDemandSource( + options.holdCoverageRead + ? adapter + : new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + session1 = upstream.events + const query = filterQuery(collection, `a`) + await withCleanup(async () => { + void query.preload().catch(() => undefined) + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(1)) + if (options.holdCoverageRead) { + // The next read after the answer is the coverage read. + state.armed = true + await upstream.answer(rows) + await vi.waitFor(() => expect(reads.held).toBe(1)) + } else { + await upstream.answer(rows) + await new Promise((resolve) => setTimeout(resolve, 50)) + records.push({ + events: [...upstream.events], + queries: { + d: { + spec: D, + sent: upstream.calls.slice(0, 1), + observed: observed(query), + }, + }, + waitPassed: true, + }) + } + await tail(upstream) + if (options.holdCoverageRead) reads.release() + await new Promise((resolve) => setTimeout(resolve, 100)) + }, [() => query.cleanup(), () => collection.cleanup()]) + } + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = filterQuery(collection, `a`) + void query.preload().catch(() => undefined) + return withCleanup(async () => { + await vi.waitFor(() => expect(upstream.calls.length).toBeGreaterThan(0)) + await new Promise((resolve) => setTimeout(resolve, 300)) + records.push({ + events: [...session1, ...upstream.events], + queries: { + d: { spec: D, sent: upstream.calls, observed: observed(query) }, + }, + upstream: `never answers`, + waitPassed: true, + }) + return records + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +// An unordered demand that loses `a1` and gains `a3` while it is active. +const runUnorderedRowReplaced = () => + runTailHistory([A1, A2], async (upstream) => { + await upstream.remove([A1]) + await upstream.write([{ id: `a3`, group: `a`, rank: 3, value: `v1` }]) + }) + +// A truncate while the coverage read of an empty load is in flight. The +// truncate commits behind the held read, so the history does not await it +// until the read is released. +const runTruncateDuringCoverageRead = () => + runTailHistory( + [], + async (upstream) => { + void upstream.truncate() + await Promise.resolve() + }, + { holdCoverageRead: true }, + ) + +// A held step in session 2 (the coverage read, or the load's cache-claim +// renewal) that outlasts the 1000 ms wait. Once it returns, D must settle. +async function runHeldPastTheWait( + held: `coverage read` | `claim renewal`, +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1 = await coverD(driver) + useFakeClock() + const holder: { collection?: Collection } = {} + const hold = { count: 0, release: () => {} } + const reads = controlledReads( + new SQLiteCorePersistenceAdapter({ + driver, + // A claim that lasts 30 s is inside the 60 s renewal window from the + // start, so the load renews it. + ...(held === `claim renewal` + ? { cacheGenerationClaimTtlMs: 30_000 } + : {}), + }), + () => + held === `coverage read` && (holder.collection?.toArray.length ?? 0) > 0, + `hold`, + ) + // Startup renews the claim before any load; the load's own renewal is the + // next one, so hold that. + let renewalsSeen = 0 + const adapter = new Proxy(reads.adapter, { + get(target, property) { + if ( + held === `claim renewal` && + property === `renewCacheGenerationClaim` + ) { + return ( + ...args: Parameters< + NonNullable + > + ) => { + if (++renewalsSeen !== 2) + return target.renewCacheGenerationClaim(...args) + hold.count++ + return new Promise((resolve) => { + hold.release = resolve + }).then(() => target.renewCacheGenerationClaim(...args)) + } + } + const value: unknown = Reflect.get(target, property, target) + return typeof value === `function` ? value.bind(target) : value + }, + }) + const { collection, upstream } = onDemandSource(adapter, 1000) + holder.collection = collection + const query = filterQuery(collection, `a`) + void query.preload().catch(() => undefined) + return withCleanup(async () => { + const isHeld = () => + (held === `coverage read` ? reads.reads.held : hold.count) > 0 + for (let i = 0; i < 200 && !isHeld(); i++) await drain() + // Positive witness: the load started and is held at the named step. + expect(isHeld()).toBe(true) + expect(collection.isLoadingSubset).toBe(true) + await vi.advanceTimersByTimeAsync(1000) + if (held === `coverage read`) reads.reads.release() + else hold.release() + await drain() + return [ + { + events: [...session1.events, ...upstream.events], + queries: { + d: { spec: D, sent: upstream.calls, observed: observed(query) }, + }, + upstream: `never answers`, + // The load started at t = 0, before the held step; the clock is now + // past 1000 ms. + waitPassed: true, + }, + ] + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +// =========================================================================== +// Part 3: joined sources +// +// An eager source P restored from SQLite and an on-demand source C. Offline, +// P's Collection opens from its restored rows once its wait passes. The +// joined query is ready only when its demand on C has settled from SQLite, +// so the same joined query must have covered that demand in session 1. +// +// The shared judgment reads C's loads from session 2's record: the query is +// ready exactly when every one of them has settled, and it then shows P's +// rows joined with C's restored rows. The wrong design "an on-demand opt-in +// permits fallback on its own" makes the query ready without coverage; each +// never-requested history feeds that wrong observation to the judgment as a +// calibration. +// +// Session 2 also reads the joined query through an outer live query +// (`nested`), which must show the joined row when it is ready. +// +// A query can reach C through more than one live query. In `two-readers` the +// outer query reads the joined query and also joins C itself on another key, +// so it has two demands on C. The query is ready only when every demand it +// reaches on C has settled. The history `covered by the inner query only` +// covers just the joined query's demand, so the query must stay loading. It +// rejects the wrong design "the first live query reached on C stands for +// all of them". +// +// Source failures are not histories here. How a source error reaches a +// query's status is the source-health law, and its oracle owns that law +// (`packages/db/tests/source-health-oracle.test.ts`). +// =========================================================================== + +type ChildHistory = + | `covered by the same joined query` + | `covered by the inner query only` + | `never requested` +type JoinedShape = `flat` | `nested` | `two-readers` +// What the joined query reads: P's rows joined with C's rows on `parentId`. +const JOINED: QuerySpec = { + join: { parents: `parents`, children: `children`, on: `parentId` }, + select: (parent, child) => ({ + id: parent.id, + parentValue: parent.value, + childValue: child.value, + }), +} + +type Parent = { id: string; value: string } +type Child = { id: string; parentId: string; value: string } + +function eagerParents(adapter: SQLiteCorePersistenceAdapter, online: boolean) { + // The source's own log, for the shared judgment. + const events: Array = [] + const collection = createCollection( + persistedCollectionOptions({ + id: `demand-coverage-parents`, + getKey: (row) => row.id, + initialRender: { strategy: `network-first`, networkTimeoutMs: 0 }, + persistence: { adapter }, + sync: { + sync: ({ begin, write, commit, markReady }) => { + if (online) { + begin() + write({ type: `insert`, value: { id: `p1`, value: `parent` } }) + commit() + events.push({ + type: `rows`, + collection: `parents`, + rows: [[`p1`, { value: `parent` }]], + }) + markReady() + } + return {} + }, + }, + }), + ) + return { collection, events } +} + +function onDemandChildren( + adapter: SQLiteCorePersistenceAdapter, + online: boolean, +) { + const calls: Array = [] + // The source's own log, for the shared judgment, and when it received + // each load. + const events: Array = [] + const arrivedAt = new Map() + const collection = createCollection( + persistedCollectionOptions({ + id: `demand-coverage-children`, + getKey: (row) => row.id, + syncMode: `on-demand`, + initialRender: { strategy: `network-first`, networkTimeoutMs: 0 }, + persistence: { adapter }, + sync: { + sync: ({ begin, write, commit }) => { + return { + loadSubset: (options: LoadSubsetOptions) => { + calls.push(options) + arrivedAt.set(options, Date.now()) + if (!online) return new Promise(() => {}) + // The source may return more rows than requested. + begin() + write({ + type: `insert`, + value: { id: `c1`, parentId: `p1`, value: `child` }, + }) + return Promise.resolve(commit()).then(() => { + events.push( + { + type: `rows`, + collection: `children`, + rows: [[`c1`, { parentId: `p1`, value: `child` }]], + }, + { + type: `settled`, + collection: `children`, + load: options, + current: true, + }, + ) + }) + }, + } + }, + }, + }), + ) + return { collection, calls, events, arrivedAt } +} + +function shapedQuery( + shape: JoinedShape, + parents: Collection, + children: Collection, +) { + const inner = joined(parents, children) + if (shape === `flat`) return { inner, query: inner } + const query = + shape === `nested` + ? createLiveQueryCollection((q) => + q.from({ row: inner }).select(({ row }) => ({ + id: row.id, + parentValue: row.parentValue, + childValue: row.childValue, + })), + ) + : // A second demand on C, keyed by value instead of parentId. + createLiveQueryCollection((q) => + q + .from({ row: inner }) + .join({ again: children }, ({ row, again }) => + eq(again.value, row.childValue), + ) + .select(({ row }) => ({ + id: row.id, + parentValue: row.parentValue, + childValue: row.childValue, + })), + ) + return { inner, query } +} + +function joined( + parents: Collection, + children: Collection, +) { + return createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .join({ child: children }, ({ parent, child }) => + eq(child.parentId, parent.id), + ) + .select(({ parent, child }) => ({ + id: parent.id, + parentValue: parent.value, + childValue: child.value, + })), + ) +} + +async function runJoined( + history: ChildHistory, + shape: JoinedShape, +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const session1: Array = [] + + // Session 1 is online. Both Collections share one adapter, as an + // application does. The children load only through the session-1 query. + { + const adapter = new SQLiteCorePersistenceAdapter({ driver }) + const eagerOnline = eagerParents(adapter, true) + const parents = eagerOnline.collection + const children = onDemandChildren(adapter, true) + const session1Query = + history === `never requested` + ? undefined + : history === `covered by the inner query only` + ? shapedQuery(`flat`, parents, children.collection) + : shapedQuery(shape, parents, children.collection) + await withCleanup(async () => { + if (session1Query) await session1Query.query.preload() + else await parents.preload() + // Positive witness: session 1 sent each demand that the history + // covers, and only those. + const keys = new Set(children.calls.map(getLoadSubsetDemandKey)) + expect(keys.size).toBe( + history === `never requested` + ? 0 + : shape === `two-readers` && + history === `covered by the same joined query` + ? 2 + : 1, + ) + session1.push(...eagerOnline.events, ...children.events) + }, [ + () => session1Query?.query.cleanup(), + () => session1Query?.inner.cleanup(), + () => children.collection.cleanup(), + () => parents.cleanup(), + ]) + } + + useFakeClock() + const adapter = new SQLiteCorePersistenceAdapter({ driver }) + const parents = eagerParents(adapter, false).collection + const children = onDemandChildren(adapter, false) + const { inner, query } = shapedQuery(shape, parents, children.collection) + + const subscription = query.subscribeChanges(() => {}) + // Virtual row properties (`$key`, `$synced`, ...) are not part of the law. + const visibleRows = () => + (query.toArray as Array).map((row) => + Object.fromEntries( + Object.entries(row).filter(([key]) => !key.startsWith(`$`)), + ), + ) + const observe = (): CoverageRecord => ({ + events: [...session1, ...children.events], + queries: { + joined: { + spec: JOINED, + sent: [...children.calls], + observed: + query.status === `ready` + ? { + status: `ready`, + rows: visibleRows() as QueryObservation[`rows`], + } + : { status: query.status === `error` ? `error` : `loading` }, + }, + }, + upstream: `never answers`, + waitPassed: waitPassedAt(children, 0), + // The offline source never calls `markReady()`. + collection: { + status: children.collection.status, + sourceMarkedReady: false, + }, + }) + return withCleanup(async () => { + for (let i = 0; i < 200 && children.calls.length === 0; i++) { + await vi.advanceTimersByTimeAsync(0) + } + // Positive witness: session 2 reached the on-demand side's demand. + expect(children.calls.length).toBeGreaterThan(0) + await drain() + await vi.advanceTimersByTimeAsync(1) + await drain() + const afterWait = observe() + // A late checkpoint: nothing else is scheduled to change the outcome. + await vi.advanceTimersByTimeAsync(10_000) + await drain() + return [afterWait, observe()] + }, [ + () => vi.useRealTimers(), + () => subscription.unsubscribe(), + () => query.cleanup(), + () => inner.cleanup(), + () => children.collection.cleanup(), + () => parents.cleanup(), + ]) +} + +// =========================================================================== +// Part 4: ordered windows +// +// W is the window `group = 'a' orderBy rank limit 2`. An ordered window +// reaches the source as a limited load plus a tie-boundary load whose key is +// built from the restored boundary row, so both must be covered for W to +// settle from SQLite. The judgment reads both loads from session 2's record +// and judges each against its own demand's record: a limited load must hold +// the same rows (the window law, maintainer decision 2026-10-26); the +// unordered tie-boundary load must hold at least as many rows. +// +// Session 1 loads W online, and one tail event follows while W is active: +// +// - `unchanged`: nothing. +// - `row-deleted`: the source deletes `a1`. +// - `row-moved-out`: the source moves `a2` to rank 10. +// - `row-moved-in`: the source adds `a0` at rank 0. +// - `rows-swapped`: the source swaps the ranks of `a1` and `a2`. +// - `row-replaced`: the source deletes `a1` and adds `a15` at rank 1.5, so +// W keeps its size and its boundary row `a2`. +// - `row-gained-under-limit`: session 1 has only `a1`, so W is not full. +// The source then adds `a0` at rank 0, and `a1` stays the boundary row. +// - Ties at the boundary: session 1 has `a2` and `a2b` at rank 2, so W holds +// one of them and its tie-boundary demand holds both. The source then +// deletes the tie row outside W, deletes the one inside W, or adds `a2c` at +// rank 2. +// +// The last two plain histories keep the boundary row, so the tie-boundary +// key does not change; they reject "a window counts only rows" by the row +// count alone. The tie histories reject "a window ignores its tie-boundary +// load" and "every demand must hold the keys it held". +// +// Two more histories move the window to its next page offline. Page 2 is +// covered only when session 1 loaded it too. Each runs with and without an +// index on `rank`. With one, core continues the window with a cursor from the +// restored boundary; without one, it reloads the longer prefix `limit 4`. The +// wrong designs "a demand key ignores offset and cursor" and "a demand key +// ignores its limit" would then settle a page that session 1 never loaded. A +// positive witness checks which load the move sent. +// =========================================================================== + +const A0: Row = { id: `a0`, group: `a`, rank: 0, value: `v1` } +const A15: Row = { id: `a15`, group: `a`, rank: 1.5, value: `v1` } +const A2B: Row = { id: `a2b`, group: `a`, rank: 2, value: `v1` } +const A2C: Row = { id: `a2c`, group: `a`, rank: 2, value: `v1` } +const A3: Row = { id: `a3`, group: `a`, rank: 3, value: `v1` } +const A4: Row = { id: `a4`, group: `a`, rank: 4, value: `v1` } + +const W: QuerySpec = { group: `a`, orderByRank: true, limit: 2 } + +// Each history: the rows session 1 answers W with, and the tail event. A tie +// tail receives the rows W showed, to pick the tie row inside or outside it. +const WINDOW_HISTORIES = { + unchanged: { rows: [A1, A2, A3], tail: async () => {} }, + 'row-deleted': { + rows: [A1, A2, A3], + tail: (upstream: Upstream) => upstream.remove([A1]), + }, + 'row-moved-out': { + rows: [A1, A2, A3], + tail: (upstream: Upstream) => upstream.update([{ ...A2, rank: 10 }]), + }, + 'row-moved-in': { + rows: [A1, A2, A3], + tail: (upstream: Upstream) => upstream.write([A0]), + }, + 'rows-swapped': { + rows: [A1, A2, A3], + tail: (upstream: Upstream) => + upstream.update([ + { ...A1, rank: 2 }, + { ...A2, rank: 1 }, + ]), + }, + 'row-replaced': { + rows: [A1, A2, A3], + tail: async (upstream: Upstream) => { + await upstream.remove([A1]) + await upstream.write([A15]) + }, + }, + 'row-gained-under-limit': { + rows: [A1], + tail: (upstream: Upstream) => upstream.write([A0]), + }, + 'tie-unchanged': { rows: [A1, A2, A2B, A3], tail: async () => {} }, + 'tie-row-outside-deleted': { + rows: [A1, A2, A2B, A3], + tail: (upstream: Upstream, shown: ReadonlyArray) => + upstream.remove([A2, A2B].filter((row) => !shown.includes(row.id))), + }, + 'tie-row-inside-deleted': { + rows: [A1, A2, A2B, A3], + tail: (upstream: Upstream, shown: ReadonlyArray) => + upstream.remove([A2, A2B].filter((row) => shown.includes(row.id))), + }, + 'tie-row-added': { + rows: [A1, A2, A2B, A3], + tail: (upstream: Upstream) => upstream.write([A2C]), + }, +} satisfies Record< + string, + { + rows: ReadonlyArray + tail: (upstream: Upstream, shown: ReadonlyArray) => Promise + } +> +type WindowHistory = keyof typeof WINDOW_HISTORIES + +function windowQuery(source: Collection) { + return createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.group, `a`)) + .orderBy(({ row }) => row.rank) + .limit(2), + ) +} + +// Session 1 answers each load with every row of group a until `done` +// settles; a source may return more rows than requested. +async function answerUntil( + upstream: Upstream, + rows: ReadonlyArray, + done: Promise, +) { + const state = { settled: false } + void Promise.resolve(done).then( + () => (state.settled = true), + () => (state.settled = true), + ) + for (let i = 0; i < 100 && !state.settled; i++) { + if (upstream.pendingCount() > 0) await upstream.answer(rows) + await new Promise((resolve) => setTimeout(resolve, 10)) + } + await done +} + +async function runWindow( + history: WindowHistory, +): Promise> { + const { rows, tail } = WINDOW_HISTORIES[history] + const driver = createDriver(new DatabaseSync(`:memory:`)) + const records: Array = [] + let session1: ReadonlyArray = [] + { + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + session1 = upstream.events + const query = windowQuery(collection) + await withCleanup(async () => { + await answerUntil(upstream, rows, query.preload()) + records.push({ + events: [...upstream.events], + queries: { + w: { spec: W, sent: [...upstream.calls], observed: observed(query) }, + }, + waitPassed: true, + }) + // Positive witness: when the window ends at a tie, W sent a + // tie-boundary load. + if (rows.includes(A2B)) { + expect(upstream.calls.some((call) => call.limit === undefined)).toBe( + true, + ) + } + await new Promise((resolve) => setTimeout(resolve, 50)) + await tail( + upstream, + query.toArray.map((row) => row.id), + ) + await new Promise((resolve) => setTimeout(resolve, 50)) + }, [() => query.cleanup(), () => collection.cleanup()]) + } + useFakeClock() + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = windowQuery(collection) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, 1) + await drain() + await vi.advanceTimersByTimeAsync(1) + await drain() + records.push({ + events: [...session1, ...upstream.events], + queries: { + w: { spec: W, sent: upstream.calls, observed: observed(query) }, + }, + upstream: `never answers`, + waitPassed: true, + }) + return records + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +async function runNextPage( + session1LoadsPageTwo: boolean, + indexed: boolean, +): Promise> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const all = [A1, A2, A3, A4] + const PAGE_TWO: QuerySpec = { ...W, offset: 2 } + const records: Array = [] + let session1: ReadonlyArray = [] + { + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + session1 = upstream.events + if (indexed) { + collection.createIndex((row) => row.rank, { indexType: BTreeIndex }) + } + const query = windowQuery(collection) + await withCleanup(async () => { + await answerUntil(upstream, all, query.preload()) + if (session1LoadsPageTwo) { + await answerUntil( + upstream, + all, + Promise.resolve(query.utils.setWindow({ offset: 2, limit: 2 })), + ) + records.push({ + events: [...upstream.events], + queries: { + w: { + spec: PAGE_TWO, + sent: [...upstream.calls], + observed: observed(query), + }, + }, + waitPassed: true, + }) + } + await new Promise((resolve) => setTimeout(resolve, 50)) + }, [() => query.cleanup(), () => collection.cleanup()]) + } + useFakeClock() + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + if (indexed) { + collection.createIndex((row) => row.rank, { indexType: BTreeIndex }) + } + const query = windowQuery(collection) + const events = () => [...session1, ...upstream.events] + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, 1) + await drain() + await vi.advanceTimersByTimeAsync(1) + await drain() + records.push({ + events: events(), + queries: { + w: { spec: W, sent: [...upstream.calls], observed: observed(query) }, + }, + upstream: `never answers`, + waitPassed: true, + }) + const callsBefore = upstream.calls.length + const moved = { settled: false } + void Promise.resolve(query.utils.setWindow({ offset: 2, limit: 2 })).then( + () => (moved.settled = true), + () => (moved.settled = true), + ) + await drain() + // Positive witness: with an index, the move asked the source for page 2 + // with a cursor; without one, for the longer prefix `limit 4`. + expect( + upstream.calls + .slice(callsBefore) + .some((call) => + indexed + ? call.cursor !== undefined + : call.cursor === undefined && call.limit === 4, + ), + ).toBe(true) + await vi.advanceTimersByTimeAsync(1) + await drain() + // The move's settlement is the observation for page 2. + records.push({ + events: events(), + queries: { + w: { + spec: PAGE_TWO, + sent: [...upstream.calls], + observed: moved.settled ? observed(query) : { status: `loading` }, + }, + }, + upstream: `never answers`, + waitPassed: true, + }) + return records + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +afterEach(() => { + vi.useRealTimers() + vi.restoreAllMocks() +}) + +declareOracle({ + check: [coverageViolations, firstMismatch, sameObservation], + model: [expectedTrace], + grammar: [ALL_CELLS], +}) + +describe(`persisted demand coverage oracle`, () => { + // ------------------------------------------------------------------------- + // Calibration: the comparison rejects wrong answers, and each block + // distinguishes its law from plausible wrong designs at a named case + // (ORC-006, ORC-013). These checks judge the models and comparisons; + // production mutants run separately against an implementation. + // ------------------------------------------------------------------------- + + it(`rejects wrong observations at a checkpoint`, () => { + const ready = { + queryStatus: `ready`, + preload: `resolved`, + rows: [`a1:v1`, `a2:v1`], + } as const + expect(sameObservation(ready, ready)).toBe(true) + expect(sameObservation(ready, { ...ready, rows: [`a1:v1`] })).toBe(false) + expect(sameObservation(ready, { ...ready, rows: [`a2:v1`, `a1:v1`] })).toBe( + false, + ) + expect(sameObservation(ready, { ...ready, rows: [`a1:v1`, `a1:v1`] })).toBe( + false, + ) + expect(sameObservation(ready, { ...ready, preload: `pending` })).toBe(false) + expect( + sameObservation(ready, { queryStatus: `loading`, preload: `pending` }), + ).toBe(false) + expect( + sameObservation( + { queryStatus: `loading`, preload: `pending` }, + { queryStatus: `ready`, preload: `resolved`, rows: [] }, + ), + ).toBe(false) + }) + + const wrongDesigns: ReadonlyArray<{ design: Design; witness: Cell }> = [ + { + design: { ...LAW, name: `any local read counts`, isCovered: () => true }, + witness: { history: `not-requested`, waitMs: 0, upstream: `pending` }, + }, + { + design: { + ...LAW, + name: `nonempty rows count`, + isCovered: (history) => SESSION_1_ROWS[history].length > 0, + }, + witness: { history: `settled-empty`, waitMs: 0, upstream: `pending` }, + }, + { + // A Collection-wide flag set by any settled load, or rows that match D. + design: { + ...LAW, + name: `coverage per Collection`, + isCovered: (history) => + history !== `not-requested` && history !== `failed`, + }, + witness: { + history: `overlapping-demand`, + waitMs: 0, + upstream: `pending`, + }, + }, + { + design: { + ...LAW, + name: `coverage keyed by filter only`, + isCovered: (history) => + LAW.isCovered(history) || + history === `other-window` || + history === `other-order`, + }, + witness: { history: `other-window`, waitMs: 0, upstream: `pending` }, + }, + { + // Honors limit but drops `orderBy` from the key. + design: { + ...LAW, + name: `coverage ignores order`, + isCovered: (history) => + LAW.isCovered(history) || history === `other-order`, + }, + witness: { history: `other-order`, waitMs: 0, upstream: `pending` }, + }, + { + design: { + ...LAW, + name: `coverage written before the rows commit`, + isCovered: (history) => + LAW.isCovered(history) || history === `durable-write-failed`, + }, + witness: { + history: `durable-write-failed`, + waitMs: 0, + upstream: `pending`, + }, + }, + { + design: { + ...LAW, + name: `coverage survives truncate`, + isCovered: (history) => + LAW.isCovered(history) || history === `truncated`, + }, + witness: { history: `truncated`, waitMs: 0, upstream: `pending` }, + }, + { + design: { + ...LAW, + name: `any settled load records coverage`, + isCovered: (history) => + LAW.isCovered(history) || + history === `truncated-in-flight` || + history === `recovered-in-flight`, + }, + witness: { + history: `truncated-in-flight`, + waitMs: 0, + upstream: `pending`, + }, + }, + { + design: { + ...LAW, + name: `coverage ignores rows deleted after it settled`, + isCovered: (history) => + LAW.isCovered(history) || history === `settled-then-row-deleted`, + }, + witness: { + history: `settled-then-row-deleted`, + waitMs: 0, + upstream: `pending`, + }, + }, + { + design: { ...LAW, name: `wait ignored`, honorsWait: false }, + witness: { history: `settled`, waitMs: 1000, upstream: `pending` }, + }, + { + design: { ...LAW, name: `half deadline`, deadlineFraction: 0.5 }, + witness: { history: `settled`, waitMs: 1000, upstream: `pending` }, + }, + { + design: { + ...LAW, + name: `failure errors a covered demand`, + failureSettlesCovered: false, + }, + witness: { history: `settled`, waitMs: 1000, upstream: `fails-early` }, + }, + { + design: { + ...LAW, + name: `upstream aborted after SQLite`, + upstreamContinues: false, + }, + witness: { history: `settled`, waitMs: 0, upstream: `answers-late` }, + }, + { + design: { + ...LAW, + name: `late failure undoes a settlement`, + settlementSurvivesFailure: false, + }, + witness: { history: `settled`, waitMs: 0, upstream: `fails-late` }, + }, + { + design: { + ...LAW, + name: `settles before restore`, + settlesBeforeRestore: true, + }, + witness: { history: `settled`, waitMs: 0, upstream: `pending` }, + }, + ] + + it.each(wrongDesigns.map((entry) => [entry.design.name, entry] as const))( + `distinguishes the law from the wrong design: %s`, + (_name, { design, witness }) => { + expect(ALL_CELLS).toContainEqual(witness) + expect( + firstMismatch(expectedTrace(witness), expectedTrace(witness, design)), + ).toBeDefined() + }, + ) + + it(`enumerates every declared cell once`, () => { + const names = ALL_CELLS.map(cellName) + expect(new Set(names).size).toBe(names.length) + expect(names).toHaveLength( + COVERAGE_HISTORIES.length * WAITS.length * UPSTREAM_SCHEDULES.length, + ) + }) + + // ------------------------------------------------------------------------- + // Refinement against production. + // ------------------------------------------------------------------------- + + it.each(ALL_CELLS.map((cell) => [cellName(cell), cell] as const))( + `refines the model: %s`, + async (_name, cell) => { + quietConsole() + const actual = await runCell(cell) + expect(firstMismatch(expectedTrace(cell), actual)).toBeUndefined() + }, + ) + + // ------------------------------------------------------------------------- + // Parts 2 to 4: fixed histories judged by the shared model. + // ------------------------------------------------------------------------- + + it(`gives each load of a covered demand its own wait and keeps coverage across release`, async () => { + quietConsole() + const records = await runLifecycle() + judge(records) + // Checker calibration: a deadline measured from the sync run would show + // query 2 ready at 1000 ms. The judgment must reject that observation. + const at1000 = records[LIFECYCLE_CHECKPOINTS.indexOf(1000)]! + const early = { + ...at1000, + queries: { + ...at1000.queries, + query2: { + ...at1000.queries.query2!, + observed: { status: `ready`, rows: [`a1`, `a2`] }, + }, + }, + } satisfies CoverageRecord + expect(coverageViolations(early)).toHaveLength(1) + }) + + it.each([ + ...([`flat`, `nested`, `two-readers`] as const).flatMap( + (shape: JoinedShape) => + ([`covered by the same joined query`, `never requested`] as const).map( + (history: ChildHistory) => [history, shape] as const, + ), + ), + // The inner-only history differs from full coverage only when the query + // reaches C through two live queries. + [`covered by the inner query only`, `two-readers`] as const, + ])( + `makes a joined query ready only when its on-demand demands settle: %s, %s`, + async (history, shape) => { + quietConsole() + const records = await runJoined(history, shape) + judge(records) + // Checker calibration: an opt-in that permits fallback alone would + // release a never-requested demand. The judgment must reject it. + if (history === `never requested`) { + const last = records[records.length - 1]! + const released = { + ...last, + queries: { + joined: { + ...last.queries.joined!, + observed: { status: `ready`, rows: [] }, + }, + }, + } satisfies CoverageRecord + expect(coverageViolations(released)).toHaveLength(1) + } + }, + ) + + it.each([ + [`a rotation`, () => runSessionReset(`rotation`)], + [`a rotation of an empty demand`, () => runSessionReset(`rotation`, [])], + [`a refetch`, () => runSessionReset(`refetch`)], + [`a refetch of an empty demand`, () => runSessionReset(`refetch`, [])], + [`a load after scoped recovery reloaded the demand`, runRecoveryReload], + [`a truncate inside the source's load call`, runTruncateInsideLoad], + [`a schema reset`, runSchemaReset], + [ + `a cache claim that expires before the coverage record`, + runClaimExpiredBeforeRecord, + ], + [ + `a cache claim that expires before the coverage check`, + () => runClaimExpiredBeforeCheck(), + ], + [ + `a cache claim that expires before an empty demand's coverage check`, + () => runClaimExpiredBeforeCheck([]), + ], + ] as const)(`judges a reset in the same session: %s`, async (_name, run) => { + quietConsole() + judge(await run()) + }) + + it.each([ + [ + `a coverage read that fails after the load`, + runCoverageReadFailsAfterLoad, + ], + [`a coverage check read that fails`, runCoverageReadFailsDuringCheck], + [`a synchronous answer`, () => runSynchronousAnswer([A1, A2], `nothing`)], + [ + `a synchronous answer, then a deletion`, + () => runSynchronousAnswer([A1, A2], `delete`), + ], + [ + `an empty synchronous answer, then a truncate during its coverage read`, + () => runSynchronousAnswer([], `truncate`), + ], + [ + `an unordered demand that loses a row and gains another`, + runUnorderedRowReplaced, + ], + [ + `a truncate during an empty load's coverage read`, + runTruncateDuringCoverageRead, + ], + [ + `a coverage read that outlasts the wait`, + () => runHeldPastTheWait(`coverage read`), + ], + [ + `a cache-claim renewal that outlasts the wait`, + () => runHeldPastTheWait(`claim renewal`), + ], + ] as const)(`judges coverage bookkeeping: %s`, async (_name, run) => { + quietConsole() + judge(await run()) + }) + + it.each(Object.keys(WINDOW_HISTORIES) as Array)( + `settles an ordered window from SQLite only while its demands are covered: %s`, + async (history) => { + quietConsole() + judge(await runWindow(history)) + }, + ) + + it.each([ + [true, true], + [true, false], + [false, true], + [false, false], + ] as const)( + `moves a window offline to a page only when that page is covered (session 1 loaded page 2: %s, indexed: %s)`, + async (session1LoadsPageTwo, indexed) => { + quietConsole() + judge(await runNextPage(session1LoadsPageTwo, indexed)) + }, + ) +}) diff --git a/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-supplemental.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-supplemental.test.ts new file mode 100644 index 0000000000..de1c94ebd4 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-supplemental.test.ts @@ -0,0 +1,218 @@ +import { DatabaseSync } from 'node:sqlite' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { IR, createCollection, getLoadSubsetDemandKey } from '@tanstack/db' +import { + SQLiteCorePersistenceAdapter, + persistedCollectionOptions, +} from '../src/index.js' +import { createSerializedNodeSqliteDriver as createDriver } from './helpers/node-sqlite-driver.js' +import { + drain, + filterQuery, + onDemandSource, + quietConsole, + untilUpstreamCalled, + useFakeClock, + withCleanup, +} from './helpers/demand-coverage-driver.js' +import type { Row } from './helpers/demand-coverage-driver.js' +import type { SQLiteDriver } from '../src/index.js' + +/** + * Supplemental checks beside the persisted demand coverage oracle + * (`persisted-demand-coverage-oracle.test.ts`). These pin two behaviors that + * the opt-in must leave as they were, and whose laws other owners hold, so + * they are ordinary examples rather than inputs to the coverage judgment + * (ORC-016): + * + * - An optimistic write on a demand settled from SQLite follows the + * optimistic overlay rules (`packages/db/tests/optimistic-history-oracle.ts` + * owns them): the row shows while its handler runs and leaves when it + * rejects, and the query stays ready throughout. + * - Opting in keeps a synchronous answer synchronous: a second request for a + * demand already hydrated in this sync run reports `true` to core, as it + * did before the opt-in existed. + */ + +const A1: Row = { id: `a1`, group: `a`, rank: 1, value: `v1` } +const A2: Row = { id: `a2`, group: `a`, rank: 2, value: `v1` } + +async function coverD( + driver: SQLiteDriver, + rows: ReadonlyArray = [A1, A2], +): Promise> { + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + ) + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + const preload = query.preload() + await vi.waitFor(() => expect(upstream.pendingCount()).toBe(1)) + await upstream.answer(rows) + await preload + return getLoadSubsetDemandKey(upstream.calls[0]!) + }, [() => query.cleanup(), () => collection.cleanup()]) +} + +type OptimisticObservation = { + settledBeforeWrite: Array + duringWrite: Array + afterRollback: Array + readyThroughout: boolean +} + +async function runOptimisticWrite(): Promise { + const driver = createDriver(new DatabaseSync(`:memory:`)) + await coverD(driver) + useFakeClock() + let rejectWrite!: (error: unknown) => void + const { collection, upstream } = onDemandSource( + new SQLiteCorePersistenceAdapter({ driver }), + 0, + { + onInsert: () => + new Promise((_resolve, reject) => { + rejectWrite = reject + }), + }, + ) + const query = filterQuery(collection, `a`) + return withCleanup(async () => { + void query.preload().catch(() => undefined) + await untilUpstreamCalled(upstream, 1) + await drain() + await vi.advanceTimersByTimeAsync(1) + await drain() + const ids = () => query.toArray.map((row) => row.id).sort() + let readyThroughout = query.status === `ready` + const settledBeforeWrite = ids() + const transaction = collection.insert({ + id: `a9`, + group: `a`, + rank: 9, + value: `local`, + }) + void transaction.isPersisted.promise.catch(() => undefined) + await drain() + readyThroughout &&= query.status === `ready` + const duringWrite = ids() + rejectWrite(new Error(`offline write`)) + await drain() + readyThroughout &&= query.status === `ready` + return { + settledBeforeWrite, + duringWrite, + afterRollback: ids(), + readyThroughout, + } + }, [ + () => vi.useRealTimers(), + () => query.cleanup(), + () => collection.cleanup(), + ]) +} + +async function runSynchronousAnswer(): Promise<{ + sourceCalls: number + secondResult: `true` | `promise` | `none` +}> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + let sourceCalls = 0 + const collection = createCollection( + persistedCollectionOptions({ + id: `demand-coverage-synchronous`, + getKey: (row) => row.id, + syncMode: `on-demand`, + initialRender: { strategy: `network-first`, networkTimeoutMs: 0 }, + persistence: { adapter: new SQLiteCorePersistenceAdapter({ driver }) }, + sync: { + sync: ({ begin, write, commit }) => ({ + loadSubset: () => { + sourceCalls++ + begin() + for (const value of [A1, A2]) write({ type: `insert`, value }) + commit() + return true + }, + }), + }, + }), + ) + const first = filterQuery(collection, `a`) + let subscription: { unsubscribe: () => void } | undefined + return withCleanup(async () => { + await first.preload() + await new Promise((resolve) => setTimeout(resolve, 50)) + const callsBefore = sourceCalls + const second = { result: `none` as `true` | `promise` | `none` } + const changes = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription = changes + ;( + changes as unknown as { requestSnapshot: (options: object) => unknown } + ).requestSnapshot({ + where: new IR.Func(`eq`, [new IR.PropRef([`group`]), new IR.Value(`a`)]), + onLoadSubsetResult: (result: true | Promise) => { + second.result = result === true ? `true` : `promise` + }, + }) + return { + sourceCalls: sourceCalls - callsBefore, + secondResult: second.result, + } + }, [ + () => subscription?.unsubscribe(), + () => first.cleanup(), + () => collection.cleanup(), + ]) +} + +afterEach(() => { + vi.useRealTimers() + vi.restoreAllMocks() +}) + +describe(`persisted demand coverage beside other owners' laws`, () => { + it(`keeps an optimistic write's overlay rules on a demand settled from SQLite`, async () => { + quietConsole() + expect(await runOptimisticWrite()).toEqual({ + settledBeforeWrite: [`a1`, `a2`], + duringWrite: [`a1`, `a2`, `a9`], + afterRollback: [`a1`, `a2`], + readyThroughout: true, + }) + }) + + it(`keeps a synchronous answer for an already hydrated demand synchronous`, async () => { + quietConsole() + // Positive witness: the second request reached the source. + expect(await runSynchronousAnswer()).toEqual({ + sourceCalls: 1, + secondResult: `true`, + }) + }) +}) + +describe(`demand coverage test driver`, () => { + // The harness keeps an oracle mismatch as the primary failure even when + // cleanup fails too (ORC-010). + it(`preserves a primary failure when cleanup also fails`, async () => { + const mismatch = new Error(`wrong rows`) + const cleanupFailure = new Error(`cleanup failed`) + await expect( + withCleanup( + () => Promise.reject(mismatch), + [ + () => { + throw cleanupFailure + }, + ], + ), + ).rejects.toMatchObject({ + cause: mismatch, + errors: [mismatch, cleanupFailure], + }) + }) +}) diff --git a/packages/db-sqlite-persistence-core/tests/persisted-readiness-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-readiness-oracle.test.ts index 0ce84fe894..f94331d878 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-readiness-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-readiness-oracle.test.ts @@ -89,7 +89,8 @@ import type { Call, Restore } from './persisted-readiness-support.js' * * The adapter is a controlled in-memory persistence adapter, not native * SQLite. Restarts, cross-tab coordination, and on-demand persistence are - * outside this owner (on-demand rejects network-first configuration). Calls + * outside this owner (`persisted-demand-coverage-oracle.test.ts` owns on-demand + * network-first restores). Calls * that break the adapter contract, and the option and deadline examples, are * fixed examples in persisted-readiness.test.ts, not generated. */ diff --git a/packages/db-sqlite-persistence-core/tests/persisted-readiness.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-readiness.test.ts index 00b0e3e8dd..6f8e0e62df 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-readiness.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-readiness.test.ts @@ -106,19 +106,6 @@ describe(`persisted readiness examples`, () => { } }) - it(`rejects network-first initial rendering for on-demand persistence`, () => { - const fixture = controlledAdapter() - expect(() => - persistedCollectionOptions({ - id: `local-on-demand`, - getKey: (row) => row.id, - syncMode: `on-demand`, - initialRender: { strategy: `network-first`, networkTimeoutMs: 100 }, - persistence: { adapter: fixture.adapter }, - }), - ).toThrow(/network-first.*eager/i) - }) - it.each([-1, Number.NaN, Number.POSITIVE_INFINITY, 2_147_483_648])( `rejects an invalid network timeout (%s)`, (networkTimeoutMs) => { diff --git a/packages/db-sqlite-persistence-core/tsconfig.json b/packages/db-sqlite-persistence-core/tsconfig.json index 4696d0d30a..4e726cca3f 100644 --- a/packages/db-sqlite-persistence-core/tsconfig.json +++ b/packages/db-sqlite-persistence-core/tsconfig.json @@ -17,6 +17,6 @@ "@tanstack/store": ["../store/src"] } }, - "include": ["src", "tests", "vite.config.ts"], + "include": ["src", "tests", "vite.config.ts", "vitest.mutants.config.ts"], "exclude": ["node_modules", "dist"] } diff --git a/packages/db-sqlite-persistence-core/vitest.mutants.config.ts b/packages/db-sqlite-persistence-core/vitest.mutants.config.ts new file mode 100644 index 0000000000..f7918d4efa --- /dev/null +++ b/packages/db-sqlite-persistence-core/vitest.mutants.config.ts @@ -0,0 +1,44 @@ +import { appendFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { mergeConfig } from 'vitest/config' +import baseConfig from './vite.config' +import type { Plugin } from 'vite' + +/** + * Applies one mutant from `scripts/demand-coverage-mutants.mjs` while vitest + * transforms the module, so no file on disk changes and other test runs in + * the same checkout see unmutated code. The mutant arrives as JSON in + * `TANSTACK_DB_MUTANT`; each transform that applies it appends to the file in + * `TANSTACK_DB_MUTANT_MARK`, so the harness can tell a surviving mutant from + * one that never reached the code. + */ +function mutantPlugin(): Plugin { + const raw = process.env.TANSTACK_DB_MUTANT + const mutant = raw + ? (JSON.parse(raw) as { file: string; find: string; replace: string }) + : undefined + const target = mutant ? resolve(__dirname, `../..`, mutant.file) : undefined + return { + name: `tanstack-db-mutant`, + enforce: `pre`, + transform(code, id) { + if (!mutant || id.split(`?`)[0] !== target) return undefined + const count = code.split(mutant.find).length - 1 + if (count !== 1) { + throw new Error(`mutant target occurs ${count} times in ${mutant.file}`) + } + if (process.env.TANSTACK_DB_MUTANT_MARK) { + appendFileSync(process.env.TANSTACK_DB_MUTANT_MARK, `${id}\n`) + } + return code.replace(mutant.find, () => mutant.replace) + }, + } +} + +export default mergeConfig(baseConfig, { + plugins: [mutantPlugin()], + test: { + coverage: { enabled: false }, + typecheck: { enabled: false }, + }, +}) diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 5deefe163d..0ae984528a 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -897,6 +897,22 @@ is a rejection: a failed load. Buffering, snapshot tokens, shape offsets, Collec transactions, and local indexes are source-specific ways to satisfy it; they are not materializer state. +A persisted wrapper is a source at this boundary. Over an opted-in on-demand +source, it may establish a request's baseline from durable rows instead of the +network. That exact demand must have persisted demand coverage in the current +persisted cache generation. The request then settles once its persisted subset +restore is visible and the configured wait has passed, or its upstream load +has failed. The upstream load continues as work the wrapper owns. Its later +rows are ordinary source writes. Releasing the demand, or cleanup, aborts it. +Coverage proves completeness when the covered load settled. It proves neither +freshness nor broader extent than the exact demand (law 7). A demand without +coverage settles only when its upstream load does. A load that a reset made +obsolete records no coverage (law 6), a `refetch` load never settles from +coverage, and coverage counts only while the restore returns at least as many +rows as the covered load left. This settlement applies only under the +default single-process coordinator; under a multi-tab coordinator, demand +routes through the coordinator and waits for its upstream load. + Lazy-demand growth requests only uncovered keys. When contraction or churn would leave fragmented coverage, the controller requests one complete current key union while retaining every established acquisition that still intersects @@ -1511,7 +1527,12 @@ query can hold rows that the source does not yet update. The source contract, not this law, decides when it delivers those changes; Electric applies them at its atomic swap. A source that resumes from complete local rows settles its loads at once, so its queries are ready on those rows before the network -responds. +responds. A persisted wrapper applies the same rule to a single demand that has +persisted demand coverage: after its wait, that demand settles on its restored +rows. Initial readiness, ordered-window publication, and later demand +therefore proceed offline on covered demands without a separate rule. A +demand built from other rows' values, such as a cursor page, is covered only +when the restored rows produce the same demand key. **Deferred acquisition:** a live-query Collection reads local memory and starts no provider work on its own. Until it has a subscriber or a preload in @@ -1741,7 +1762,9 @@ every row before the top-K (a recorded limit). 7. **Applied settlement:** a successful subset load settles only after its establishing sync transactions are visible; a source must not add queue priority merely to force the load to settle. Settlement proves no broader - source extent than the exact request. + source extent than the exact request. A persisted subset restore of a + demand with persisted demand coverage is an establishing transaction for + that demand. 8. **Nested propagation:** every materialized relation consumes the fully materialized output relation of its children. 9. **Publication:** reads, events, and downstream queries observe the same diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 8e6fdac1a7..8b77f8feb6 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -156,6 +156,7 @@ const staticOracleProperties = [ `sqlite-resume.startup-generation`, `sqlite-resume.managed-claim-read`, `sqlite-resume.retired-storage`, + `persisted-demand-coverage.history`, `sqlite-reload.class`, `browser.multi-tab-reserved-history`, `sqlite-reload.class.electron`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index fd46c89917..8418d5d773 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -387,6 +387,11 @@ const ownerGroups: ReadonlyArray< `sqlite-resume`, `managed-claim-read retired-storage`, ], + [ + `db-sqlite-persistence-core/tests/persisted-demand-coverage-campaign-oracle.test.ts`, + `persisted-demand-coverage`, + `history`, + ], [ `trailbase-db-collection/tests/lifecycle-oracle.property.test.ts`, `trailbase`, diff --git a/packages/electric-db-collection/tests/electric-progressive-offline-readiness-witness.test.ts b/packages/electric-db-collection/tests/electric-progressive-offline-readiness-witness.test.ts new file mode 100644 index 0000000000..1cd616422b --- /dev/null +++ b/packages/electric-db-collection/tests/electric-progressive-offline-readiness-witness.test.ts @@ -0,0 +1,455 @@ +import { DatabaseSync } from 'node:sqlite' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import { ShapeStream } from '@electric-sql/client' +import { + SQLiteCorePersistenceAdapter, + persistedCollectionOptions, +} from '../../db-sqlite-persistence-core/src' +import { electricCollectionOptions } from '../src/electric' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + CoverageRecord, + QuerySpec, + SourceEvent, +} from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { Message, Row } from '@electric-sql/client' +import type { Collection, LoadSubsetOptions, SyncConfig } from '@tanstack/db' +import type { SQLiteDriver } from '../../db-sqlite-persistence-core/src' +import type { ElectricCollectionUtils } from '../src/electric' + +/** + * # What does a persisted progressive Electric query show while offline? + * + * The first block records current behavior without network-first; it does + * not state a contract. The second block is a receiving witness for persisted + * demand coverage with network-first opted in. The file is the + * Electric companion to the generic on-demand witness in + * `db-sqlite-persistence-core/tests/on-demand-offline-readiness-witness.test.ts`. + * + * Each history runs two sessions against one real SQLite database. Session 1 + * either completes a progressive full sync, loads one query subset without + * completing the full sync, or does nothing. Session 2 restarts with a stream + * that never delivers a message and snapshot requests that never settle: an + * offline network. The checkpoint is a delay after session 2's live query + * starts. It records query readiness, visible rows, and whether session 2 + * resumed the stream from a persisted offset. + * + * The network-first block is a production driver for the shared judgment in + * `persisted-demand-coverage-model-oracle.ts`; it states no expected result + * (ORC-014). Session 2 reads either group a, which session 1 loaded, or + * group b, which no query of session 1 read. After a completed full sync the + * model expects both ready: Electric holds every row and settles each later + * load at once. Without one, only a covered subset settles. + */ + +type Item = Row & { id: number; group: string } +type Subscriber = (messages: Array>) => void + +let subscribers: Array = [] +let fetchSnapshot: () => Promise = () => new Promise(() => {}) +let requestSnapshot: () => Promise = () => new Promise(() => {}) + +vi.mock(`@electric-sql/client`, async () => ({ + ...(await vi.importActual(`@electric-sql/client`)), + ShapeStream: vi.fn(() => ({ + subscribe: (subscriber: Subscriber) => { + subscribers.push(subscriber) + return () => {} + }, + unsubscribeAll: () => {}, + requestSnapshot: () => requestSnapshot(), + fetchSnapshot: () => fetchSnapshot(), + forceDisconnectAndRefresh: () => Promise.resolve(), + isUpToDate: false, + shapeHandle: `shape-1`, + lastOffset: `10_0`, + })), +})) + +const rows: Array = [ + { id: 1, group: `a` }, + { id: 2, group: `a` }, + { id: 3, group: `b` }, +] + +function insert(value: Item): Message { + return { + key: String(value.id), + value, + headers: { operation: `insert`, offset: `10_0` } as never, + } +} + +// What Electric received and settled, for the shared judgment. +type SourceLog = { + events: Array + sent: Array +} + +// Records each load at Electric's own `loadSubset` boundary. +function recordingSync( + sync: SyncConfig, + log: SourceLog, +): SyncConfig { + return { + ...sync, + sync: (params) => { + const result = sync.sync(params) + if (!result || typeof result === `function` || !result.loadSubset) { + return result + } + const loadSubset = result.loadSubset + return { + ...result, + loadSubset: (options: LoadSubsetOptions) => { + log.sent.push(options) + const settled = (): void => { + log.events.push({ type: `settled`, load: options, current: true }) + } + const loaded = loadSubset(options) + if (loaded === true) { + settled() + return loaded + } + return loaded.then(settled) + }, + } + }, + } +} + +function progressive( + driver: SQLiteDriver, + networkFirst: boolean, + syncMode: `progressive` | `on-demand` = `progressive`, + log?: SourceLog, +) { + const electric = electricCollectionOptions({ + id: `${syncMode}-offline-witness`, + shapeOptions: { url: `http://test-url`, params: { table: `items` } }, + syncMode, + getKey: (item: Item) => item.id, + }) + return createCollection( + persistedCollectionOptions< + Item, + string | number, + never, + ElectricCollectionUtils + >({ + ...electric, + ...(log ? { sync: recordingSync(electric.sync, log) } : {}), + ...(networkFirst + ? { initialRender: { strategy: `network-first`, networkTimeoutMs: 0 } } + : {}), + persistence: { adapter: new SQLiteCorePersistenceAdapter({ driver }) }, + }), + ) +} + +function groupA(source: Collection) { + return groupQuery(source, `a`) +} + +function groupQuery( + source: Collection, + group: `a` | `b`, +) { + return createLiveQueryCollection((q) => + q.from({ item: source }).where(({ item }) => eq(item.group, group)), + ) +} + +const rowsEvent = (items: ReadonlyArray): SourceEvent => ({ + type: `rows`, + rows: items.map((item) => [String(item.id), { group: item.group }]), +}) + +async function settle(ms = 50) { + await new Promise((resolve) => setTimeout(resolve, ms)) +} + +type Session1 = `full-sync` | `subset-only` | `nothing` + +async function runHistory(session1: Session1, networkFirst: boolean) { + const driver = createDriver(new DatabaseSync(`:memory:`)) + + if (session1 !== `nothing`) { + subscribers = [] + fetchSnapshot = () => + Promise.resolve({ + metadata: {}, + data: rows.filter((row) => row.group === `a`).map(insert), + }) + const first = progressive(driver, networkFirst) + const firstQuery = groupA(first) + await firstQuery.preload() + if (session1 === `full-sync`) { + for (const subscriber of subscribers) { + subscriber([ + ...rows.map(insert), + { headers: { control: `up-to-date` } }, + ]) + } + await vi.waitFor(() => expect(first.status).toBe(`ready`)) + } + await settle() + await firstQuery.cleanup() + await first.cleanup() + } + + // Session 2: the network is offline. + subscribers = [] + fetchSnapshot = () => new Promise(() => {}) + vi.mocked(ShapeStream).mockClear() + const second = progressive(driver, networkFirst) + const query = groupA(second) + let preloaded = false + void query.preload().then( + () => (preloaded = true), + () => (preloaded = true), + ) + await settle() + const streamOptions = vi.mocked(ShapeStream).mock.calls.at(-1)?.[0] as + { offset?: string; handle?: string } | undefined + const observation = { + resumedFrom: streamOptions + ? { offset: streamOptions.offset, handle: streamOptions.handle } + : undefined, + sourceStatus: second.status, + queryReady: query.isReady(), + preloaded, + queryRows: query.toArray.map((item) => item.id).sort(), + } + await query.cleanup() + await second.cleanup() + return observation +} + +afterEach(() => { + subscribers = [] + fetchSnapshot = () => new Promise(() => {}) + requestSnapshot = () => new Promise(() => {}) +}) + +describe(`persisted progressive Electric query while offline (current behavior)`, () => { + // Observed on main at 31dad2dd3; these pin current behavior, not a law. + it.each([ + [`full-sync`, { resumed: true, ready: true, rows: [1, 2] }], + [`subset-only`, { resumed: false, ready: false, rows: [1, 2] }], + [`nothing`, { resumed: false, ready: false, rows: [] }], + ] as const)(`session 1: %s`, async (session1, expected) => { + const observation = await runHistory(session1, false) + expect(observation.sourceStatus).toBe(`loading`) + expect({ + resumed: observation.resumedFrom?.offset !== undefined, + ready: observation.queryReady, + rows: observation.queryRows, + }).toEqual(expected) + expect(observation.preloaded).toBe(expected.ready) + }) +}) + +// Receiving driver for persisted demand coverage +// (`db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts`) +// through Electric's progressive mode, with network-first opted in. The driver +// records Electric's own log: the rows it delivered, each load it received +// and settled, and the completion of its full sync. The shared model judges +// session 2's query from that log. +async function recordHistory( + session1: Session1, + group: `a` | `b`, +): Promise { + const driver = createDriver(new DatabaseSync(`:memory:`)) + const events: Array = [{ type: `sync run starts` }] + if (session1 !== `nothing`) { + subscribers = [] + const groupRows = rows.filter((row) => row.group === `a`) + fetchSnapshot = () => { + events.push(rowsEvent(groupRows)) + return Promise.resolve({ metadata: {}, data: groupRows.map(insert) }) + } + const log: SourceLog = { events, sent: [] } + const first = progressive(driver, true, `progressive`, log) + const firstQuery = groupA(first) + await firstQuery.preload() + if (session1 === `full-sync`) { + events.push(rowsEvent(rows)) + for (const subscriber of subscribers) { + subscriber([ + ...rows.map(insert), + { headers: { control: `up-to-date` } }, + ]) + } + await vi.waitFor(() => expect(first.status).toBe(`ready`)) + events.push({ type: `full sync completes` }) + } + await settle() + await firstQuery.cleanup() + await first.cleanup() + } + + // Session 2: the network is offline. + subscribers = [] + fetchSnapshot = () => new Promise(() => {}) + const log: SourceLog = { events: [{ type: `sync run starts` }], sent: [] } + const second = progressive(driver, true, `progressive`, log) + const query = groupQuery(second, group) + void query.preload().catch(() => undefined) + await settle() + const record: CoverageRecord = { + events: [...events, ...log.events], + queries: { + [group]: { + spec: { group } satisfies QuerySpec, + sent: [...log.sent], + observed: + query.status === `ready` + ? { + status: `ready`, + rows: query.toArray.map((item) => String(item.id)), + } + : { status: query.status === `error` ? `error` : `loading` }, + }, + }, + upstream: `never answers`, + waitPassed: true, + } + await query.cleanup() + await second.cleanup() + return record +} + +describe(`persisted progressive Electric query while offline with network-first`, () => { + it.each( + ([`full-sync`, `subset-only`, `nothing`] as const).flatMap((session1) => + ([`a`, `b`] as const).map((group) => [session1, group] as const), + ), + )(`session 1: %s; session 2 reads group %s`, async (session1, group) => { + const record = await recordHistory(session1, group) + const reached = new Set() + expect(coverageViolations(record, (state) => reached.add(state))).toEqual( + [], + ) + // Positive witness: a completed full sync reached the model's rule that + // the source settles every later load at once. + expect(reached.has(`settled from a completed full sync`)).toBe( + session1 === `full-sync`, + ) + }) +}) + +// Electric's own load after a settlement from SQLite. A progressive load +// fetches its snapshot without a cancel signal, so after the wrapper settles +// the demand from SQLite the Electric load can still be waiting. Releasing +// the demand aborts it. A snapshot that answers after that must not write its +// rows (law 6 in `ARCHITECTURE.md`: a conforming source publishes no +// request-scoped rows after cancellation). +describe(`Electric's own load after a settlement from SQLite`, () => { + it(`writes no rows from a snapshot that answers after release`, async () => { + const driver = createDriver(new DatabaseSync(`:memory:`)) + subscribers = [] + fetchSnapshot = () => + Promise.resolve({ + metadata: {}, + data: rows.filter((row) => row.group === `a`).map(insert), + }) + const first = progressive(driver, true) + const firstQuery = groupA(first) + await firstQuery.preload() + await settle() + await firstQuery.cleanup() + await first.cleanup() + + subscribers = [] + let answerLate!: (value: unknown) => void + fetchSnapshot = () => + new Promise((resolve) => { + answerLate = resolve + }) + const second = progressive(driver, true) + const query = groupA(second) + await query.preload() + // Positive witness: the demand settled from SQLite while Electric's own + // snapshot is still pending. + expect(query.isReady()).toBe(true) + expect(typeof answerLate).toBe(`function`) + await query.cleanup() + await settle() + answerLate({ metadata: {}, data: [insert({ id: 9, group: `a` })] }) + await settle() + expect(second.toArray.map((item) => item.id).sort()).toEqual([1, 2]) + await second.cleanup() + }) +}) + +// Must-refetch in on-demand mode. An Electric must-refetch truncates the +// source, which ends persisted demand coverage for every demand. Session 1 +// loads group a through `requestSnapshot`, whose rows arrive as stream +// messages. In the second history a must-refetch and an up-to-date follow, and +// the reacquired snapshot never answers. Session 2 is offline: the covered +// subset settles on its restored rows, and the refetched one keeps loading. +describe(`on-demand Electric must-refetch with network-first`, () => { + it.each([ + [`loaded group a`, { ready: true, rows: [1, 2] }], + [`loaded group a, then must-refetch`, { ready: false, rows: [] }], + ] as const)(`session 1: %s`, async (session1, expected) => { + const driver = createDriver(new DatabaseSync(`:memory:`)) + subscribers = [] + requestSnapshot = () => { + for (const subscriber of subscribers) { + subscriber([ + ...rows.filter((row) => row.group === `a`).map(insert), + // The SDK ends an injected subset with `subset-end`, the point at + // which Electric commits its rows. + { headers: { control: `subset-end` } } as never, + ]) + } + return Promise.resolve(undefined) + } + const first = progressive(driver, true, `on-demand`) + const firstQuery = groupA(first) + const preload = firstQuery.preload() + await settle(20) + for (const subscriber of subscribers) { + subscriber([{ headers: { control: `up-to-date` } }]) + } + await preload + // Positive witness: session 1 loaded group a. + expect(firstQuery.toArray.map((item) => item.id).sort()).toEqual([1, 2]) + if (session1 === `loaded group a, then must-refetch`) { + requestSnapshot = () => new Promise(() => {}) + for (const subscriber of subscribers) { + subscriber([{ headers: { control: `must-refetch` } } as never]) + } + await settle() + for (const subscriber of subscribers) { + subscriber([{ headers: { control: `up-to-date` } }]) + } + await settle(200) + // Positive witness: the must-refetch truncated the rows. + expect(first.toArray).toEqual([]) + } + await firstQuery.cleanup() + await first.cleanup() + + subscribers = [] + requestSnapshot = () => new Promise(() => {}) + const second = progressive(driver, true, `on-demand`) + const query = groupA(second) + let preloaded = false + void query.preload().then( + () => (preloaded = true), + () => undefined, + ) + await settle(300) + expect({ + ready: preloaded, + rows: query.toArray.map((item) => item.id).sort(), + }).toEqual(expected) + await query.cleanup() + await second.cleanup() + }) +}) diff --git a/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/main.ts b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/main.ts new file mode 100644 index 0000000000..46d416e909 --- /dev/null +++ b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/main.ts @@ -0,0 +1,96 @@ +/** + * Real Electron main process for the persisted demand coverage witness + * (`tests/persisted-demand-coverage.electron.e2e.test.ts`). It serves the real + * `exposeElectronSQLitePersistence` handler over the real `ipcMain`, with the + * real Node persistence over a `node:sqlite` connection. Electron's ABI cannot + * load better-sqlite3's native module, so a small adapter gives `node:sqlite` + * the better-sqlite3 surface the Node driver uses (the same adapter the reload + * class fixture uses). The renderer asks for one database file per history + * (`demand:session`), so each history reads and writes its own file. + */ +import { DatabaseSync } from 'node:sqlite' +import { BrowserWindow, app, ipcMain } from 'electron' +import { createNodeSQLitePersistence } from '../../../../node-db-sqlite-persistence/src' +import { exposeElectronSQLitePersistence } from '../../../src/main' +import type { BetterSqlite3Database } from '../../../../node-db-sqlite-persistence/src' + +export const RESULT_PREFIX = `__TANSTACK_DB_DEMAND_COVERAGE__:` + +function binding(value: unknown): string | number | bigint | null { + if (value == null) return null + if (typeof value === `boolean`) return value ? 1 : 0 + if ( + typeof value === `string` || + typeof value === `number` || + typeof value === `bigint` + ) + return value + return String(value) +} + +// The better-sqlite3 surface that BetterSqlite3SQLiteDriver uses. +function betterSqliteSurface(database: DatabaseSync): BetterSqlite3Database { + return { + prepare: (sql: string) => { + const statement = database.prepare(sql) + return { + all: (...params: Array) => + statement.all(...params.map(binding)).map((row) => ({ ...row })), + run: (...params: Array) => + statement.run(...params.map(binding)), + } + }, + exec: (sql: string) => { + database.exec(sql) + }, + close: () => database.close(), + } as unknown as BetterSqlite3Database +} + +let window: BrowserWindow | undefined +let session: { close: () => void } | undefined + +// One session per history file: the previous session closes first, so the +// renderer's next session is a fresh main-process persistence over that file. +ipcMain.handle(`demand:session`, (_event, filename: string) => { + session?.close() + const database = new DatabaseSync(filename) + const dispose = exposeElectronSQLitePersistence({ + ipcMain, + persistence: createNodeSQLitePersistence({ + database: betterSqliteSurface(database), + }), + }) + session = { + close: () => { + dispose() + database.close() + }, + } +}) +ipcMain.handle(`demand:close-session`, () => { + session?.close() + session = undefined +}) +// Quit only after stdout has flushed: a result can be larger than the pipe's +// buffer, and an early quit truncates it. +ipcMain.handle(`demand:result`, (_event, result: string) => { + process.stdout.write(`${RESULT_PREFIX}${result}\n`, () => app.quit()) +}) + +void app.whenReady().then(() => { + window = new BrowserWindow({ + show: false, + webPreferences: { + preload: process.env.DEMAND_COVERAGE_PRELOAD, + contextIsolation: true, + sandbox: false, + }, + }) + void window.loadFile(process.env.DEMAND_COVERAGE_PAGE!, { + query: { + phase: process.env.DEMAND_COVERAGE_PHASE!, + dir: process.env.DEMAND_COVERAGE_DIR!, + }, + }) +}) diff --git a/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/page.html b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/page.html new file mode 100644 index 0000000000..61c0bd68f1 --- /dev/null +++ b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/page.html @@ -0,0 +1,10 @@ + + + + + Persisted demand coverage (Electron) + + + + + diff --git a/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/preload.cjs b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/preload.cjs new file mode 100644 index 0000000000..25141e2e27 --- /dev/null +++ b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/preload.cjs @@ -0,0 +1,6 @@ +// Exposes the one IPC call the persisted demand coverage driver needs. +const { contextBridge, ipcRenderer } = require(`electron`) + +contextBridge.exposeInMainWorld(`demandBridge`, { + invoke: (channel, ...args) => ipcRenderer.invoke(channel, ...args), +}) diff --git a/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/renderer.ts b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/renderer.ts new file mode 100644 index 0000000000..8724d25f9f --- /dev/null +++ b/packages/electron-db-sqlite-persistence/tests/e2e/persisted-demand-coverage/renderer.ts @@ -0,0 +1,102 @@ +/** + * Renderer driver for the persisted demand coverage witness in + * `tests/persisted-demand-coverage.electron.e2e.test.ts`. It runs in a real + * Electron renderer and calls the real main process over real IPC through + * `createElectronSQLitePersistence`. The test runs the app twice: phase `1` + * is an online session that takes each history's steps, then changes the + * source. Phase `2` is a later, separate app launch over the same database + * file, with its upstream load answered as the history says. + * + * The histories, the expected checkpoints, and the two session runners are + * the receiving contract's + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`). + * This driver only opens each history's main-process session and records what + * the runners observe; the test owns the assertions. + */ +import { createElectronSQLitePersistence } from '../../../src/renderer' +import { + RECEIVING_HISTORIES, + recordSession1, + recordSession2, +} from '../../../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { + ReceivingSession, + Session1Record, + Session2Record, +} from '../../../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { + ElectronPersistenceInvoke, + ElectronPersistenceResponseEnvelope, +} from '../../../src/protocol' + +type HistoryResult = { + name: string + error?: string + session1?: Session1Record + session2?: Session2Record +} + +type Bridge = { + invoke: (channel: string, ...args: Array) => Promise +} + +declare global { + interface Window { + demandBridge: Bridge + } +} + +const bridge = window.demandBridge +const invoke: ElectronPersistenceInvoke = (channel, request) => + bridge.invoke( + channel, + request, + ) as Promise + +// Opens the main-process session over one history's database file. The +// renderer's persistence then reaches that file only through IPC. +async function openSession(filename: string): Promise { + await bridge.invoke(`demand:session`, filename) + return { + persistence: createElectronSQLitePersistence({ invoke }), + close: async () => { + await bridge.invoke(`demand:close-session`) + }, + } +} + +async function run(): Promise { + const params = new URLSearchParams(location.search) + const phase = params.get(`phase`) + const dir = params.get(`dir`)! + const results: Array = [] + for (const [index, history] of RECEIVING_HISTORIES.entries()) { + const filename = `${dir}/h${index}.sqlite` + try { + const { persistence, close } = await openSession(filename) + try { + if (phase === `1`) { + results.push({ + name: history.id, + session1: await recordSession1(persistence, history), + }) + } else { + results.push({ + name: history.id, + session2: await recordSession2(persistence, history), + }) + } + } finally { + await close() + } + } catch (error) { + results.push({ + name: history.id, + error: error instanceof Error ? error.message : String(error), + }) + } + } + await bridge.invoke(`demand:result`, JSON.stringify({ phase, results })) +} + +void run() diff --git a/packages/electron-db-sqlite-persistence/tests/persisted-demand-coverage.electron.e2e.test.ts b/packages/electron-db-sqlite-persistence/tests/persisted-demand-coverage.electron.e2e.test.ts new file mode 100644 index 0000000000..b8ec6a3fd7 --- /dev/null +++ b/packages/electron-db-sqlite-persistence/tests/persisted-demand-coverage.electron.e2e.test.ts @@ -0,0 +1,227 @@ +/** + * # Receiving witness: persisted demand coverage over Electron + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * holds the histories and records the sessions. The law is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the Electron host. It runs the + * histories through the real Electron persistence path: the renderer's + * `createElectronSQLitePersistence` over real IPC to the main process's + * `exposeElectronSQLitePersistence`, with `node:sqlite` in main (Electron's + * ABI cannot load better-sqlite3). The node witness covers better-sqlite3. + * + * Histories: every history of `RECEIVING_HISTORIES`, each with its own + * database file, one per history. The test launches Electron twice. Phase 1 + * is an online app launch that records each history's session 1 log. Phase 2 + * is a later, separate app launch over the same files that records each + * history's session 2. Session 2's upstream behavior is the history's. + * + * Production path: the renderer driver + * (`tests/e2e/persisted-demand-coverage/renderer.ts`) calls the contract's + * recorders in a real renderer, and each history opens its main-process session + * over IPC. + * + * Observations: the records carry the source log, the loads the source + * received, and two session 2 checkpoints with their elapsed time since the + * forward. The shared judge compares them with the law. + * + * Limits: one driver (Electron renderer over IPC to a node:sqlite main + * process, one window, single-process coordinator). It does not cover + * `ElectronCollectionCoordinator`, multi-window coordination, or the + * better-sqlite3 driver (the node witness covers it). + */ +import { spawn } from 'node:child_process' +import { copyFileSync, existsSync, mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { dirname, join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { build } from 'esbuild' +import electronPath from 'electron' +import { expect, it, onTestFinished } from 'vitest' +import { + RECEIVING_HISTORIES, + coverageRecords, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + Session1Record, + Session2Record, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' + +const fixture = join( + dirname(fileURLToPath(import.meta.url)), + `e2e`, + `persisted-demand-coverage`, +) +const packages = resolve(fixture, `../../../..`) +const RESULT_PREFIX = `__TANSTACK_DB_DEMAND_COVERAGE__:` +const APP_TIMEOUT_MS = 240_000 + +type PhaseResult = { + phase: `1` | `2` + results: Array<{ + name: string + error?: string + session1?: Session1Record + session2?: Session2Record + }> +} + +async function bundle(out: string) { + const alias = { + '@tanstack/db': join(packages, `db/src`), + '@tanstack/db-ivm': join(packages, `db-ivm/src`), + '@tanstack/db-sqlite-persistence-core': join( + packages, + `db-sqlite-persistence-core/src`, + ), + // Electron's ABI cannot load the native module; main passes a database. + 'better-sqlite3': join( + fixture, + `..`, + `reload-class`, + `better-sqlite3-unavailable.cjs`, + ), + } + await build({ + entryPoints: [join(fixture, `main.ts`)], + outfile: join(out, `main.mjs`), + bundle: true, + platform: `node`, + format: `esm`, + external: [`electron`], + alias, + banner: { + js: `import { createRequire } from 'node:module'; const require = createRequire(import.meta.url);`, + }, + }) + await build({ + entryPoints: [join(fixture, `renderer.ts`)], + outfile: join(out, `renderer.js`), + bundle: true, + platform: `browser`, + format: `iife`, + alias, + }) + copyFileSync(join(fixture, `preload.cjs`), join(out, `preload.cjs`)) + copyFileSync(join(fixture, `page.html`), join(out, `page.html`)) +} + +// One real Electron app launch per phase. Phase 2 is a new process, so it +// shares nothing with phase 1 except the database files on disk. +function runPhase( + out: string, + databases: string, + phase: `1` | `2`, +): Promise { + return new Promise((resolveResult, reject) => { + // Launch as the reload class and runtime bridge e2e do: Linux CI has no + // display and no configured setuid sandbox. + const electronArgs = [ + `--disable-gpu`, + `--disable-dev-shm-usage`, + `--no-sandbox`, + join(out, `main.mjs`), + ] + const xvfbRun = `/usr/bin/xvfb-run` + const useXvfb = existsSync(xvfbRun) + const child = spawn( + useXvfb ? xvfbRun : (electronPath as unknown as string), + useXvfb + ? [ + `-a`, + `--server-args=-screen 0 1280x720x24`, + electronPath as unknown as string, + ...electronArgs, + ] + : electronArgs, + { + env: { + ...process.env, + ELECTRON_RUN_AS_NODE: undefined, + DEMAND_COVERAGE_PHASE: phase, + DEMAND_COVERAGE_PRELOAD: join(out, `preload.cjs`), + DEMAND_COVERAGE_PAGE: join(out, `page.html`), + DEMAND_COVERAGE_DIR: databases, + }, + }, + ) + let stdout = `` + let stderr = `` + child.stdout.on(`data`, (chunk) => (stdout += String(chunk))) + child.stderr.on(`data`, (chunk) => (stderr += String(chunk))) + // A hung app is reported with its stderr before the test times out. + const timer = setTimeout(() => { + child.kill() + reject(new Error(`phase ${phase} had no result: ${stderr.slice(-2_000)}`)) + }, APP_TIMEOUT_MS) + child.on(`error`, (error) => { + clearTimeout(timer) + reject(error) + }) + child.on(`exit`, (code) => { + clearTimeout(timer) + const line = stdout.split(`\n`).find((l) => l.startsWith(RESULT_PREFIX)) + if (!line) { + reject( + new Error( + `phase ${phase} had no result (exit ${code}): ${stderr.slice(-2_000)}`, + ), + ) + return + } + resolveResult(JSON.parse(line.slice(RESULT_PREFIX.length)) as PhaseResult) + }) + }) +} + +it(`real Electron records every persisted demand coverage history across two app launches`, async () => { + const out = mkdtempSync(join(tmpdir(), `demand-coverage-app-`)) + const databases = mkdtempSync(join(tmpdir(), `demand-coverage-db-`)) + onTestFinished(() => { + rmSync(out, { recursive: true, force: true }) + rmSync(databases, { recursive: true, force: true }) + }) + await bundle(out) + + const session1 = await runPhase(out, databases, `1`) + expect(session1.results.map(({ name }) => name)).toEqual( + RECEIVING_HISTORIES.map(({ id }) => id), + ) + expect(session1.results.filter(({ error }) => error !== undefined)).toEqual( + [], + ) + + const session2 = await runPhase(out, databases, `2`) + expect(session2.results.map(({ name }) => name)).toEqual( + RECEIVING_HISTORIES.map(({ id }) => id), + ) + expect(session2.results.filter(({ error }) => error !== undefined)).toEqual( + [], + ) + // Positive witness: each session 2 asked the offline source for the demand. + for (const result of session2.results) { + expect(result.session2?.sent.length ?? 0).toBeGreaterThan(0) + } + // Each history's session 1 log and session 2 record go to the shared judge, + // one judgment per checkpoint. + for (const history of RECEIVING_HISTORIES) { + const session1Record = session1.results.find( + ({ name }) => name === history.id, + )?.session1 + const session2Record = session2.results.find( + ({ name }) => name === history.id, + )?.session2 + expect(session1Record).toBeDefined() + expect(session2Record).toBeDefined() + const records = coverageRecords( + history, + session1Record as Session1Record, + session2Record as Session2Record, + ) + expect(coverageViolations(records.beforeDeadline), history.id).toEqual([]) + expect(coverageViolations(records.afterDeadline), history.id).toEqual([]) + } +}, 600_000) diff --git a/packages/electron-db-sqlite-persistence/tsconfig.json b/packages/electron-db-sqlite-persistence/tsconfig.json index fb7a248f30..41ad8a7598 100644 --- a/packages/electron-db-sqlite-persistence/tsconfig.json +++ b/packages/electron-db-sqlite-persistence/tsconfig.json @@ -30,6 +30,7 @@ "tests/**/*.test.ts", "tests/**/*.test-d.ts", "tests/e2e/reload-class/*.ts", + "tests/e2e/persisted-demand-coverage/*.ts", "vite.config.ts", "vitest.e2e.config.ts" ], diff --git a/packages/expo-db-sqlite-persistence/e2e/expo-persisted-demand-coverage.e2e.test.ts b/packages/expo-db-sqlite-persistence/e2e/expo-persisted-demand-coverage.e2e.test.ts new file mode 100644 index 0000000000..af3d6a4802 --- /dev/null +++ b/packages/expo-db-sqlite-persistence/e2e/expo-persisted-demand-coverage.e2e.test.ts @@ -0,0 +1,79 @@ +/** + * # Receiving witness: persisted demand coverage over the Expo driver + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * holds the histories and records the sessions. The law is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the Expo host. It checks the same law + * through the wrapper `createExpoSQLitePersistence`, over the expo-sqlite-shaped driver in `tests/helpers/expo-sqlite-test-db.ts`. + * + * Histories: every history of `RECEIVING_HISTORIES`, each with its own database + * file. Session 1 is online and takes the history's steps. Session 2 is offline + * and opens a new driver on the same file, with its upstream as the history says. + * + * Production path: each session builds a `persistedCollectionOptions` + * Collection over `createExpoSQLitePersistence` and runs the query's preload. Session 1 closes + * its driver before session 2 opens. + * + * Observations: the records of both sessions, judged by the shared judge + * (`persisted-demand-coverage-model-oracle.ts`) at each session 2 checkpoint. + * + * Limits: No device runs here. The helper backs the driver with better-sqlite3 unless a runtime factory is supplied (`TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE`). The emulator runtime lane (`TANSTACK_DB_EXPO_RUNTIME_PLATFORM`) skips this witness, as it skips the conformance suite, because that lane runs the native driver through its own app. The sessions are two driver instances in one process over one file, not an app restart. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { createExpoSQLitePersistence } from '../src' +import { createExpoSQLiteTestDatabase } from '../tests/helpers/expo-sqlite-test-db' +import { + RECEIVING_HISTORIES, + coverageRecords, + recordReceivingHistory, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +const dirs: Array = [] + +afterEach(() => { + for (const dir of dirs.splice(0)) + rmSync(dir, { recursive: true, force: true }) +}) + +const DESCRIBE = `expo persisted demand coverage (receiving witness)` + +// The emulator lane runs the native driver; this witness runs the shimmed driver. +const runtimePlatform = process.env.TANSTACK_DB_EXPO_RUNTIME_PLATFORM?.trim() +const describeShimmedDriver = runtimePlatform ? describe.skip : describe + +describeShimmedDriver(DESCRIBE, () => { + it.each(RECEIVING_HISTORIES)( + `$description`, + async (history) => { + const dir = mkdtempSync(join(tmpdir(), `expo-demand-coverage-`)) + dirs.push(dir) + const filename = join(dir, `state.sqlite`) + + const { session1, session2 } = await recordReceivingHistory( + history, + () => { + const database = createExpoSQLiteTestDatabase({ filename }) + return Promise.resolve({ + persistence: createExpoSQLitePersistence({ database }), + close: async () => { + await database.closeAsync() + }, + }) + }, + ) + // Positive witness: session 2 asked the offline source for the demand. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + }, + 60_000, + ) +}) diff --git a/packages/node-db-sqlite-persistence/tests/node-persisted-on-demand-coverage.test.ts b/packages/node-db-sqlite-persistence/tests/node-persisted-on-demand-coverage.test.ts new file mode 100644 index 0000000000..6020dd1e5e --- /dev/null +++ b/packages/node-db-sqlite-persistence/tests/node-persisted-on-demand-coverage.test.ts @@ -0,0 +1,83 @@ +/** + * # Receiving witness: persisted demand coverage over better-sqlite3 + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * states the law, owns the histories and their expected checkpoints, and runs + * the sessions. The law is `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`; + * the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the better-sqlite3 host. It checks + * the same law over the real better-sqlite3 driver and + * `createNodeSQLitePersistence`, which the primary oracle does not exercise + * (it runs `node:sqlite` through a test driver). + * + * Histories: every history of `RECEIVING_HISTORIES`, each with its own + * database file in a temp directory. Session 1 is online and takes the + * history's steps. Session 2 is offline and opens a new connection to the same + * file. Its upstream load never answers, or rejects at once, as the history says. + * + * Production path: each session builds a `persistedCollectionOptions` + * Collection over `createNodeSQLitePersistence({ database })` and runs the + * query's preload. Session 1 closes its database before session 2 opens. + * + * Observations and checkpoints (session 2 only, read by the contract): + * - `beforeDeadline`: 100 ms after the upstream load is forwarded. + * - `afterDeadline`: after polling up to 2 500 ms for the query to leave + * `loading`. + * Each history's `expected` holds the outcome the law gives at both + * checkpoints. The positive witness is `upstreamCalls > 0`: the offline source + * was asked before any settlement. + * + * Mutant evidence (run in a temporary copy, see the coverage map): with + * `SESSION_2_WAIT_MS = 60_000` in the contract, the covered histories stay + * loading at `afterDeadline` and fail. With `covered-upstream-fails-at-once` + * changed to `never-answers`, that history stays loading and fails. + * + * Limits: one driver (better-sqlite3, file database, single process + * coordinator). It does not cover the browser OPFS driver (see the browser + * witness), a second process, or multi-tab coordination. The bookkeeping is + * observed only through the public query status and rows. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import BetterSqlite3 from 'better-sqlite3' +import { createNodeSQLitePersistence } from '../src' +import { + RECEIVING_HISTORIES, + coverageRecords, + recordReceivingHistory, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +const dirs: Array = [] + +afterEach(() => { + for (const dir of dirs.splice(0)) + rmSync(dir, { recursive: true, force: true }) +}) + +describe(`node better-sqlite3 persisted demand coverage (receiving witness)`, () => { + it.each(RECEIVING_HISTORIES)(`$description`, async (history) => { + const dir = mkdtempSync(join(tmpdir(), `node-pod-`)) + dirs.push(dir) + const path = join(dir, `db.sqlite`) + + const { session1, session2 } = await recordReceivingHistory(history, () => { + const database = new BetterSqlite3(path) + return Promise.resolve({ + persistence: createNodeSQLitePersistence({ database }), + close: () => { + database.close() + return Promise.resolve() + }, + }) + }) + // Positive witness: session 2 asked the offline source for the demand. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + }) +}) diff --git a/packages/query-db-collection/tests/persisted-on-demand-offline-witness.test.ts b/packages/query-db-collection/tests/persisted-on-demand-offline-witness.test.ts new file mode 100644 index 0000000000..85ed7738cc --- /dev/null +++ b/packages/query-db-collection/tests/persisted-on-demand-offline-witness.test.ts @@ -0,0 +1,249 @@ +import { DatabaseSync } from 'node:sqlite' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { QueryClient } from '@tanstack/query-core' +import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import { + SQLiteCorePersistenceAdapter, + persistedCollectionOptions, +} from '../../db-sqlite-persistence-core/src' +import { queryCollectionOptions } from '../src/query' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import type { Collection, LoadSubsetOptions } from '@tanstack/db' +import type { SQLiteDriver } from '../../db-sqlite-persistence-core/src' +import type { QueryCollectionUtils } from '../src/query' + +/** + * # Does an offline on-demand Query DB query settle on its rows from SQLite? + * + * Receiving witness for persisted demand coverage. The oracle in + * `db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * states the law with a generic on-demand source. This file checks that Query + * DB, with its own row owners and retention metadata, supplies the premise + * that oracle controls: a subset load settles only after its rows are visible, + * so the wrapper records coverage after them. + * + * Each history runs two sessions over one `node:sqlite` database, each with a + * fresh QueryClient. Session 1 is online and loads either `group = 'a'` or + * nothing. Session 2 asks for `group = 'a'` with `networkTimeoutMs: 1000`, and + * its `queryFn` either never answers or rejects. The checkpoints are just + * before and at the wait on a fake clock, and after the network answers or + * fails. + * + * Expected, from the law: a covered demand stays loading before the wait, then + * becomes ready on its restored rows; a rejected `queryFn` settles a covered + * demand at once; a demand that session 1 never loaded keeps loading. + * + * Query DB retention can delete a released query's rows: once its retention + * time passes, Query DB removes rows that no other query owns. Coverage + * describes those rows, so it must end with them (maintainer decision, + * 2026-10-26). In the history `loaded group a, then retention expired`, + * session 1 releases the query with a 20 ms `gcTime`, and a positive witness + * checks that Query DB deleted the rows. Session 2 then behaves as if session + * 1 never loaded group a. Limits: + * one filter, no ordered window or cursor page, no Query cache persistence, and + * no SSR dehydration. + */ + +type Item = { id: string; group: string; value: string } + +const SERVER: ReadonlyArray = [ + { id: `a1`, group: `a`, value: `v1` }, + { id: `a2`, group: `a`, value: `v1` }, + { id: `b1`, group: `b`, value: `v1` }, +] + +type Network = `online` | `pending` | `rejects` + +function groupOf(options: LoadSubsetOptions | undefined): string | undefined { + const where = options?.where as + { name?: string; args?: Array<{ value?: unknown }> } | undefined + return where?.name === `eq` ? (where.args?.[1]?.value as string) : undefined +} + +function items(driver: SQLiteDriver, network: Network, gcTime = Infinity) { + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false, gcTime } }, + }) + const fetches: Array = [] + const collection = createCollection( + persistedCollectionOptions< + Item, + string | number, + never, + QueryCollectionUtils + >({ + ...queryCollectionOptions({ + id: `query-db-demand-coverage`, + queryClient, + syncMode: `on-demand`, + queryKey: (options) => [`items`, groupOf(options) ?? `all`], + queryFn: (context) => { + const group = groupOf(context.meta?.loadSubsetOptions) + fetches.push(group) + if (network === `pending`) return new Promise>(() => {}) + if (network === `rejects`) { + return Promise.reject(new Error(`offline`)) + } + return Promise.resolve(SERVER.filter((row) => row.group === group)) + }, + getKey: (row) => row.id, + }), + initialRender: { strategy: `network-first`, networkTimeoutMs: 1000 }, + persistence: { adapter: new SQLiteCorePersistenceAdapter({ driver }) }, + }), + ) + return { collection, fetches, queryClient } +} + +function groupQuery( + source: Collection, + group: string, +) { + return createLiveQueryCollection((q) => + q.from({ item: source }).where(({ item }) => eq(item.group, group)), + ) +} + +async function drain() { + for (let i = 0; i < 25; i++) await vi.advanceTimersByTimeAsync(0) +} + +type Observation = { ready: boolean; rows: Array } + +type Session1 = + `loaded group a` | `loaded group a, then retention expired` | `nothing` + +async function runHistory( + session1: Session1, + network: Exclude, +): Promise<{ + beforeWait: Observation + atWait: Observation + session1Fetches: Array + session2Fetches: Array +}> { + const driver = createDriver(new DatabaseSync(`:memory:`)) + let session1Fetches: Array = [] + if (session1 !== `nothing`) { + const expires = session1 === `loaded group a, then retention expired` + const first = items(driver, `online`, expires ? 20 : Infinity) + const query = groupQuery(first.collection, `a`) + try { + await query.preload() + session1Fetches = first.fetches + if (expires) { + await query.cleanup() + // Positive witness: retention expiry deleted the released rows. + await vi.waitFor( + () => + expect( + first.collection.toArray.filter((row) => row.group === `a`), + ).toEqual([]), + // Retention runs on Query DB's own timers, which a loaded CI + // runner can delay well past the 20 ms gcTime. + { timeout: 10_000 }, + ) + await new Promise((resolve) => setTimeout(resolve, 100)) + } + } finally { + await query.cleanup() + await first.collection.cleanup() + first.queryClient.clear() + } + } + + vi.useFakeTimers({ + toFake: [`setTimeout`, `clearTimeout`, `Date`], + }) + const second = items(driver, network) + const query = groupQuery(second.collection, `a`) + try { + void query.preload().catch(() => undefined) + for (let i = 0; i < 200 && second.fetches.length === 0; i++) { + await vi.advanceTimersByTimeAsync(0) + } + await drain() + const observe = (): Observation => ({ + ready: query.status === `ready`, + rows: query.toArray.map((row) => row.id).sort(), + }) + // A rejected queryFn settles a covered demand before the wait. + const beforeWait = network === `rejects` ? observe() : undefined + await vi.advanceTimersByTimeAsync(999) + await drain() + const pendingBeforeWait = observe() + await vi.advanceTimersByTimeAsync(1) + await drain() + return { + beforeWait: beforeWait ?? pendingBeforeWait, + atWait: observe(), + session1Fetches, + session2Fetches: second.fetches, + } + } finally { + vi.useRealTimers() + await query.cleanup() + await second.collection.cleanup() + second.queryClient.clear() + } +} + +afterEach(() => { + vi.useRealTimers() + vi.restoreAllMocks() +}) + +describe(`persisted on-demand Query DB query while offline with network-first`, () => { + it.each([ + [ + `loaded group a`, + `pending`, + { + beforeWait: { ready: false, rows: [`a1`, `a2`] }, + atWait: { ready: true, rows: [`a1`, `a2`] }, + }, + ], + [ + `loaded group a`, + `rejects`, + { + beforeWait: { ready: true, rows: [`a1`, `a2`] }, + atWait: { ready: true, rows: [`a1`, `a2`] }, + }, + ], + [ + `nothing`, + `pending`, + { + beforeWait: { ready: false, rows: [] }, + atWait: { ready: false, rows: [] }, + }, + ], + [ + `loaded group a, then retention expired`, + `pending`, + { + beforeWait: { ready: false, rows: [] }, + atWait: { ready: false, rows: [] }, + }, + ], + ] as const)( + `session 1: %s, session 2 network: %s`, + async (session1, network, expected) => { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) + const observed = await runHistory(session1, network) + // Positive witnesses: session 1 fetched exactly group a when it loaded, + // and session 2 asked the network for group a. + expect(observed.session1Fetches).toEqual( + session1 === `nothing` ? [] : [`a`], + ) + expect(observed.session2Fetches).toContain(`a`) + expect({ + beforeWait: observed.beforeWait, + atWait: observed.atWait, + }).toEqual(expected) + }, + 20_000, + ) +}) diff --git a/packages/react-db/tests/useLiveSuspenseQuery-persisted-on-demand.test.tsx b/packages/react-db/tests/useLiveSuspenseQuery-persisted-on-demand.test.tsx new file mode 100644 index 0000000000..bfca1c6ef5 --- /dev/null +++ b/packages/react-db/tests/useLiveSuspenseQuery-persisted-on-demand.test.tsx @@ -0,0 +1,148 @@ +import { DatabaseSync } from 'node:sqlite' +import { describe, expect, it, vi } from 'vitest' +import { render as renderComponent, waitFor } from '@testing-library/react' +import { Suspense } from 'react' +import { eq } from '@tanstack/db' +import { SQLiteCorePersistenceAdapter } from '../../db-sqlite-persistence-core/src' +import { useLiveSuspenseQuery } from '../src/useLiveSuspenseQuery' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import { + A1, + A2, + FRAMEWORK_HISTORY_IDS, + coverageRecords, + createReceivingSource, + receivingHistory, + recordCheckpoints, + session2SourceOptions, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + Session1Record, + Session2Record, + SourceEvent, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { QueryObservation } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +/** + * # Does React Suspense release offline on a covered on-demand demand? + * + * Receiving witness for persisted demand coverage. The law, the histories, and + * the checkpoint records are the receiving contract's + * (`db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * and the shared judge (`db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts`) + * judges each record. The law's authority is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`. + * + * Histories: `FRAMEWORK_HISTORY_IDS` (`covered`, `never`, and + * `covered-upstream-fails-at-once`). Session 1 is online and answers its load + * at once, through the source, or loads nothing. Session 2 is offline, on the + * same `node:sqlite` database, with the history's upstream behavior. + * + * Production path: `useLiveSuspenseQuery` under `Suspense`, over a real + * persisted on-demand Collection and `SQLiteCorePersistenceAdapter`. + * + * Observation: at each checkpoint of session 2, whether the fallback still + * renders (`loading`) or the rendered ids are shown (`ready`). + * + * Limits: one filter, one React render path, no StrictMode. Suspense releases at query + * readiness, so this host checks the readiness that the wrapper's settlement + * feeds. + */ + +function GroupA({ + collection, +}: { + collection: ReturnType[`collection`] +}) { + const { data } = useLiveSuspenseQuery((q) => + q.from({ row: collection }).where(({ row }) => eq(row.group, `a`)), + ) + return ( +
+ {data + .map((row) => row.id) + .sort() + .join(`,`) || `empty`} +
+ ) +} + +// Reads the Suspense output: the fallback while the query is loading, or the +// rendered ids once it is ready. +function renderedOutcome( + view: ReturnType, +): QueryObservation { + if (view.queryByText(`Loading`) !== null) return { status: `loading` } + const text = view.container.textContent + return { + status: `ready`, + rows: text === `empty` ? [] : text.split(`,`), + } +} + +describe(`React Suspense over a persisted on-demand Collection offline`, () => { + it.each(FRAMEWORK_HISTORY_IDS.map(receivingHistory))( + `$description`, + async (history) => { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) + const driver = createDriver(new DatabaseSync(`:memory:`)) + const persistence = () => ({ + adapter: new SQLiteCorePersistenceAdapter({ driver }), + }) + + const session1Events: Array = [] + if (history.session1.length > 0) { + const first = createReceivingSource({ + persistence: persistence(), + waitMs: 0, + behavior: `answer-at-once`, + events: session1Events, + answerRows: [A1, A2], + }) + const view = renderComponent( + Loading}> + + , + ) + await waitFor(() => expect(view.getByText(`a1,a2`)).toBeTruthy()) + view.unmount() + await new Promise((resolve) => setTimeout(resolve, 50)) + await first.collection.cleanup() + } + const session1: Session1Record = { events: session1Events } + + const second = createReceivingSource({ + persistence: persistence(), + ...session2SourceOptions(history), + events: [], + }) + const loadStartedAt = Date.now() + const view = renderComponent( + Loading}> + + , + ) + try { + const checkpoints = await recordCheckpoints( + () => renderedOutcome(view), + () => second.received.length, + loadStartedAt, + ) + const session2: Session2Record = { + sent: [...second.received], + ...checkpoints, + } + // Positive witness: session 2 asked the offline source for group a. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + } finally { + view.unmount() + await second.collection.cleanup() + } + }, + ) +}) diff --git a/packages/react-native-db-sqlite-persistence/e2e/react-native-persisted-demand-coverage.e2e.test.ts b/packages/react-native-db-sqlite-persistence/e2e/react-native-persisted-demand-coverage.e2e.test.ts new file mode 100644 index 0000000000..3dd79ce0c7 --- /dev/null +++ b/packages/react-native-db-sqlite-persistence/e2e/react-native-persisted-demand-coverage.e2e.test.ts @@ -0,0 +1,81 @@ +/** + * # Receiving witness: persisted demand coverage over the React Native driver + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * holds the histories and records the sessions. The law is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the React Native host. It checks the same law + * through the wrapper `createReactNativeSQLitePersistence`, over the op-sqlite-shaped driver in `tests/helpers/op-sqlite-test-db.ts`. + * + * Histories: every history of `RECEIVING_HISTORIES`, each with its own database + * file. Session 1 is online and takes the history's steps. Session 2 is offline + * and opens a new driver on the same file, with its upstream as the history says. + * + * Production path: each session builds a `persistedCollectionOptions` + * Collection over `createReactNativeSQLitePersistence` and runs the query's preload. Session 1 closes + * its driver before session 2 opens. + * + * Observations: the records of both sessions, judged by the shared judge + * (`persisted-demand-coverage-model-oracle.ts`) at each session 2 checkpoint. + * + * Limits: No device runs here. The helper backs the driver with better-sqlite3 unless a runtime factory is supplied (`TANSTACK_DB_MOBILE_SQLITE_FACTORY_MODULE`), in which case this witness uses that factory like the conformance suite. It does not run on a device or simulator, and it does not cover the native OP-SQLite binding unless a runtime factory is supplied. The sessions are two driver instances in one process over one file, not an app restart. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { createReactNativeSQLitePersistence } from '../src' +import { createOpSQLiteTestDatabase } from '../tests/helpers/op-sqlite-test-db' +import { + RECEIVING_HISTORIES, + coverageRecords, + recordReceivingHistory, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +const dirs: Array = [] + +afterEach(() => { + for (const dir of dirs.splice(0)) + rmSync(dir, { recursive: true, force: true }) +}) + +const DESCRIBE = `react native persisted demand coverage (receiving witness)` + +describe(DESCRIBE, () => { + it.each(RECEIVING_HISTORIES)( + `$description`, + async (history) => { + const dir = mkdtempSync(join(tmpdir(), `rn-demand-coverage-`)) + dirs.push(dir) + const filename = join(dir, `state.sqlite`) + + const { session1, session2 } = await recordReceivingHistory( + history, + () => { + const database = createOpSQLiteTestDatabase({ + filename, + resultShape: `statement-array`, + }) + return Promise.resolve({ + persistence: createReactNativeSQLitePersistence({ + database, + arrayResultMode: `statement-results`, + }), + close: async () => { + await database.close() + }, + }) + }, + ) + // Positive witness: session 2 asked the offline source for the demand. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + }, + 60_000, + ) +}) diff --git a/packages/solid-db/tests/useLiveQuery-persisted-on-demand.test.tsx b/packages/solid-db/tests/useLiveQuery-persisted-on-demand.test.tsx new file mode 100644 index 0000000000..6dedbb5651 --- /dev/null +++ b/packages/solid-db/tests/useLiveQuery-persisted-on-demand.test.tsx @@ -0,0 +1,142 @@ +import { DatabaseSync } from 'node:sqlite' +import { describe, expect, it, vi } from 'vitest' +import { render } from '@solidjs/testing-library' +import { Suspense } from 'solid-js' +import { eq } from '@tanstack/db' +import { SQLiteCorePersistenceAdapter } from '../../db-sqlite-persistence-core/src' +import { useLiveQuery } from '../src/useLiveQuery' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import { + A1, + A2, + FRAMEWORK_HISTORY_IDS, + coverageRecords, + createReceivingSource, + receivingHistory, + recordCheckpoints, + session2SourceOptions, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + Session1Record, + Session2Record, + SourceEvent, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { QueryObservation } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +/** + * # Does Solid Suspense release offline on a covered on-demand demand? + * + * Receiving witness for persisted demand coverage. The law, the histories, and + * the checkpoint records are the receiving contract's + * (`db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * and the shared judge (`db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts`) + * judges each record. The law's authority is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`. + * + * Histories: `FRAMEWORK_HISTORY_IDS` (`covered`, `never`, and + * `covered-upstream-fails-at-once`). Session 1 is online and answers its load + * at once through the source, or loads nothing. Session 2 is offline on the + * same `node:sqlite` database, with the history's upstream behavior. + * + * Production path: `useLiveQuery` under Solid `Suspense`, over a real persisted + * on-demand Collection and `SQLiteCorePersistenceAdapter`. + * + * Observation: at each checkpoint of session 2, whether the fallback still + * renders (`loading`) or the rows are rendered (`ready`). + * + * Limits: one filter and one Solid render path. Suspense releases at query + * readiness, so this host checks the readiness that the wrapper's settlement + * feeds. + */ + +function GroupA(props: { + collection: ReturnType[`collection`] +}) { + const query = useLiveQuery((q) => + q.from({ row: props.collection }).where(({ row }) => eq(row.group, `a`)), + ) + return ( +
+ {query() + .map((row) => row.id) + .sort() + .join(`,`) || `empty`} +
+ ) +} + +// Reads the Suspense output: the fallback while the query is loading, or the +// rendered rows once it is ready. +function renderedOutcome(view: ReturnType): QueryObservation { + if (view.queryByTestId(`waiting`) !== null) return { status: `loading` } + const text = view.queryByTestId(`rows`)?.textContent ?? `` + return { status: `ready`, rows: text === `empty` ? [] : text.split(`,`) } +} + +describe(`Solid Suspense over a persisted on-demand Collection offline`, () => { + it.each(FRAMEWORK_HISTORY_IDS.map(receivingHistory))( + `$description`, + async (history) => { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) + const driver = createDriver(new DatabaseSync(`:memory:`)) + const persistence = () => ({ + adapter: new SQLiteCorePersistenceAdapter({ driver }), + }) + + const session1Events: Array = [] + if (history.session1.length > 0) { + const first = createReceivingSource({ + persistence: persistence(), + waitMs: 0, + behavior: `answer-at-once`, + events: session1Events, + answerRows: [A1, A2], + }) + const view = render(() => ( + Loading}> + + + )) + const content = await view.findByTestId(`rows`) + await vi.waitFor(() => expect(content.textContent).toBe(`a1,a2`)) + view.unmount() + await new Promise((resolve) => setTimeout(resolve, 50)) + await first.collection.cleanup() + } + const session1: Session1Record = { events: session1Events } + + const second = createReceivingSource({ + persistence: persistence(), + ...session2SourceOptions(history), + events: [], + }) + const loadStartedAt = Date.now() + const view = render(() => ( + Loading}> + + + )) + try { + const checkpoints = await recordCheckpoints( + () => renderedOutcome(view), + () => second.received.length, + loadStartedAt, + ) + const session2: Session2Record = { + sent: [...second.received], + ...checkpoints, + } + // Positive witness: session 2 asked the offline source for group a. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + } finally { + view.unmount() + await second.collection.cleanup() + } + }, + ) +}) diff --git a/packages/svelte-db/tests/useLiveQuery-persisted-on-demand.svelte.test.ts b/packages/svelte-db/tests/useLiveQuery-persisted-on-demand.svelte.test.ts new file mode 100644 index 0000000000..4981854e06 --- /dev/null +++ b/packages/svelte-db/tests/useLiveQuery-persisted-on-demand.svelte.test.ts @@ -0,0 +1,135 @@ +import { DatabaseSync } from 'node:sqlite' +import { describe, expect, it, vi } from 'vitest' +import { eq } from '@tanstack/db' +import { SQLiteCorePersistenceAdapter } from '../../db-sqlite-persistence-core/src' +import { useLiveQuery } from '../src/useLiveQuery.svelte.js' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import { + A1, + A2, + FRAMEWORK_HISTORY_IDS, + coverageRecords, + createReceivingSource, + receivingHistory, + recordCheckpoints, + session2SourceOptions, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + Session1Record, + Session2Record, + SourceEvent, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { QueryObservation } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +/** + * # Does a Svelte live query settle offline on a covered on-demand demand? + * + * Receiving witness for persisted demand coverage. The law, the histories, and + * the checkpoint records are the receiving contract's + * (`db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * and the shared judge (`db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts`) + * judges each record. The law's authority is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`. + * + * Histories: `FRAMEWORK_HISTORY_IDS` (`covered`, `never`, and + * `covered-upstream-fails-at-once`). Session 1 is online and answers its load + * at once through the source, or loads nothing. Session 2 is offline on the + * same `node:sqlite` database, with the history's upstream behavior. + * + * Production path: `useLiveQuery` -> `createLiveQueryObserver` -> the live + * query's on-demand `loadSubset` -> the persisted wrapper, which waits for the + * network-first wait and then settles a covered demand from its restored rows. + * + * Observation: at each checkpoint of session 2, the primitive's `isReady` and + * `data`. + * + * Limits: this witness checks the wrapper's settlement through query + * readiness and rows. It uses one filter, one primitive instance per session, and no component lifecycle. + */ + +function queryGroupA( + collection: ReturnType[`collection`], +) { + return useLiveQuery((q) => + q.from({ row: collection }).where(({ row }) => eq(row.group, `a`)), + ) +} + +// `useLiveQuery` registers Svelte effects, so each query mounts in its own +// effect root; the returned `dispose` tears that root down. +function mountGroupA( + collection: ReturnType[`collection`], +) { + const mounted: { query?: ReturnType } = {} + const dispose = $effect.root(() => { + mounted.query = queryGroupA(collection) + }) + return { query: mounted.query!, dispose } +} + +function observeQuery(query: ReturnType): QueryObservation { + if (!query.isReady) return { status: `loading` } + return { + status: `ready`, + rows: query.data.map((row) => row.id).sort(), + } +} + +describe(`Svelte useLiveQuery over a persisted on-demand Collection offline`, () => { + it.each(FRAMEWORK_HISTORY_IDS.map(receivingHistory))( + `$description`, + async (history) => { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) + const driver = createDriver(new DatabaseSync(`:memory:`)) + const persistence = () => ({ + adapter: new SQLiteCorePersistenceAdapter({ driver }), + }) + + const session1Events: Array = [] + if (history.session1.length > 0) { + const first = createReceivingSource({ + persistence: persistence(), + waitMs: 0, + behavior: `answer-at-once`, + events: session1Events, + answerRows: [A1, A2], + }) + const { query, dispose } = mountGroupA(first.collection) + await vi.waitFor(() => expect(query.isReady).toBe(true)) + dispose() + await new Promise((resolve) => setTimeout(resolve, 50)) + await first.collection.cleanup() + } + const session1: Session1Record = { events: session1Events } + + const second = createReceivingSource({ + persistence: persistence(), + ...session2SourceOptions(history), + events: [], + }) + const loadStartedAt = Date.now() + const { query, dispose } = mountGroupA(second.collection) + try { + const checkpoints = await recordCheckpoints( + () => observeQuery(query), + () => second.received.length, + loadStartedAt, + ) + const session2: Session2Record = { + sent: [...second.received], + ...checkpoints, + } + // Positive witness: session 2 asked the offline source for group a. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + } finally { + dispose() + await second.collection.cleanup() + } + }, + ) +}) diff --git a/packages/tauri-db-sqlite-persistence/tests/tauri-persisted-demand-coverage.test.ts b/packages/tauri-db-sqlite-persistence/tests/tauri-persisted-demand-coverage.test.ts new file mode 100644 index 0000000000..a3a61f8a0e --- /dev/null +++ b/packages/tauri-db-sqlite-persistence/tests/tauri-persisted-demand-coverage.test.ts @@ -0,0 +1,75 @@ +/** + * # Receiving witness: persisted demand coverage over the Tauri driver + * + * Law and authority: the receiving contract + * (`packages/db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`) + * holds the histories and records the sessions. The law is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`, + * and the primary oracle `packages/db-sqlite-persistence-core/tests/persisted-demand-coverage-oracle.test.ts` + * owns it across its cells. This file is the Tauri host. It checks the same law + * through the wrapper `createTauriSQLitePersistence`, over the better-sqlite3 stand-in in `tests/helpers/tauri-sql-test-db.ts`. + * + * Histories: every history of `RECEIVING_HISTORIES`, each with its own database + * file. Session 1 is online and takes the history's steps. Session 2 is offline + * and opens a new driver on the same file, with its upstream as the history says. + * + * Production path: each session builds a `persistedCollectionOptions` + * Collection over `createTauriSQLitePersistence` and runs the query's preload. Session 1 closes + * its driver before session 2 opens. + * + * Observations: the records of both sessions, judged by the shared judge + * (`persisted-demand-coverage-model-oracle.ts`) at each session 2 checkpoint. + * + * Limits: It runs in this package's existing unit suite, which already exercises the wrapper over that stand-in. It does not run the Tauri runtime (`tauri-plugin-sql` in a Rust process). The Tauri e2e (`e2e/run-tauri-e2e.ts`) that would run the native path needs `cargo`, which is not installed here, so that path is not covered. The sessions are two driver instances in one process over one file, not an app restart. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { createTauriSQLitePersistence } from '../src' +import { + RECEIVING_HISTORIES, + coverageRecords, + recordReceivingHistory, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import { createTauriSQLiteTestDatabase } from './helpers/tauri-sql-test-db' + +const dirs: Array = [] + +afterEach(() => { + for (const dir of dirs.splice(0)) + rmSync(dir, { recursive: true, force: true }) +}) + +const DESCRIBE = `tauri persisted demand coverage (receiving witness)` + +describe(DESCRIBE, () => { + it.each(RECEIVING_HISTORIES)( + `$description`, + async (history) => { + const dir = mkdtempSync(join(tmpdir(), `tauri-demand-coverage-`)) + dirs.push(dir) + const filename = join(dir, `state.sqlite`) + + const { session1, session2 } = await recordReceivingHistory( + history, + () => { + const database = createTauriSQLiteTestDatabase({ filename }) + return Promise.resolve({ + persistence: createTauriSQLitePersistence({ database }), + close: async () => { + await database.close() + }, + }) + }, + ) + // Positive witness: session 2 asked the offline source for the demand. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + }, + 60_000, + ) +}) diff --git a/packages/vue-db/tests/useLiveQuery-persisted-on-demand.test.ts b/packages/vue-db/tests/useLiveQuery-persisted-on-demand.test.ts new file mode 100644 index 0000000000..44abc88852 --- /dev/null +++ b/packages/vue-db/tests/useLiveQuery-persisted-on-demand.test.ts @@ -0,0 +1,121 @@ +import { DatabaseSync } from 'node:sqlite' +import { describe, expect, it, vi } from 'vitest' +import { eq } from '@tanstack/db' +import { SQLiteCorePersistenceAdapter } from '../../db-sqlite-persistence-core/src' +import { useLiveQuery } from '../src/useLiveQuery' +import { createSerializedNodeSqliteDriver as createDriver } from '../../db-sqlite-persistence-core/tests/helpers/node-sqlite-driver' +import { + A1, + A2, + FRAMEWORK_HISTORY_IDS, + coverageRecords, + createReceivingSource, + receivingHistory, + recordCheckpoints, + session2SourceOptions, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import { coverageViolations } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' +import type { + Session1Record, + Session2Record, + SourceEvent, +} from '../../db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract' +import type { QueryObservation } from '../../db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle' + +/** + * # Does a Vue live query settle offline on a covered on-demand demand? + * + * Receiving witness for persisted demand coverage. The law, the histories, and + * the checkpoint records are the receiving contract's + * (`db-sqlite-persistence-core/tests/contracts/persisted-demand-coverage-receiving-contract.ts`), + * and the shared judge (`db-sqlite-persistence-core/tests/persisted-demand-coverage-model-oracle.ts`) + * judges each record. The law's authority is + * `docs/contributing/oracle-reviews/2026-10-20-on-demand-network-first-law.md`. + * + * Histories: `FRAMEWORK_HISTORY_IDS` (`covered`, `never`, and + * `covered-upstream-fails-at-once`). Session 1 is online and answers its load + * at once through the source, or loads nothing. Session 2 is offline on the + * same `node:sqlite` database, with the history's upstream behavior. + * + * Production path: `useLiveQuery` -> `createLiveQueryObserver` -> the live + * query's on-demand `loadSubset` -> the persisted wrapper, which waits for the + * network-first wait and then settles a covered demand from its restored rows. + * + * Observation: at each checkpoint of session 2, the composable's `isReady` and + * `data`. + * + * Limits: this witness checks the wrapper's settlement through query + * readiness and rows. It uses one filter, one composable instance per session, and no component lifecycle. + */ + +function groupA( + collection: ReturnType[`collection`], +) { + return useLiveQuery((q) => + q.from({ row: collection }).where(({ row }) => eq(row.group, `a`)), + ) +} + +function observeQuery(query: ReturnType): QueryObservation { + if (!query.isReady.value) return { status: `loading` } + return { + status: `ready`, + rows: query.data.value.map((row) => row.id).sort(), + } +} + +describe(`Vue useLiveQuery over a persisted on-demand Collection offline`, () => { + it.each(FRAMEWORK_HISTORY_IDS.map(receivingHistory))( + `$description`, + async (history) => { + vi.spyOn(console, `warn`).mockImplementation(() => {}) + vi.spyOn(console, `error`).mockImplementation(() => {}) + const driver = createDriver(new DatabaseSync(`:memory:`)) + const persistence = () => ({ + adapter: new SQLiteCorePersistenceAdapter({ driver }), + }) + + const session1Events: Array = [] + if (history.session1.length > 0) { + const first = createReceivingSource({ + persistence: persistence(), + waitMs: 0, + behavior: `answer-at-once`, + events: session1Events, + answerRows: [A1, A2], + }) + const query = groupA(first.collection) + await vi.waitFor(() => expect(query.isReady.value).toBe(true)) + await new Promise((resolve) => setTimeout(resolve, 50)) + await first.collection.cleanup({ dependents: true }) + } + const session1: Session1Record = { events: session1Events } + + const second = createReceivingSource({ + persistence: persistence(), + ...session2SourceOptions(history), + events: [], + }) + const loadStartedAt = Date.now() + const query = groupA(second.collection) + try { + const checkpoints = await recordCheckpoints( + () => observeQuery(query), + () => second.received.length, + loadStartedAt, + ) + const session2: Session2Record = { + sent: [...second.received], + ...checkpoints, + } + // Positive witness: session 2 asked the offline source for group a. + expect(session2.sent.length).toBeGreaterThan(0) + const records = coverageRecords(history, session1, session2) + expect(coverageViolations(records.beforeDeadline)).toEqual([]) + expect(coverageViolations(records.afterDeadline)).toEqual([]) + } finally { + await second.collection.cleanup({ dependents: true }) + } + }, + ) +}) diff --git a/scripts/demand-coverage-mutants.mjs b/scripts/demand-coverage-mutants.mjs new file mode 100644 index 0000000000..49d3bda754 --- /dev/null +++ b/scripts/demand-coverage-mutants.mjs @@ -0,0 +1,458 @@ +// Hostile mutants for persisted demand coverage. +// +// Each mutant replaces one exact piece of production code with a plausible +// wrong design. The harness applies it in memory through +// `packages/db-sqlite-persistence-core/vitest.mutants.config.ts`, runs the +// owner oracles, and expects every mutant to fail a named test unless the +// mutant is marked `equivalent` with the reason no observation can tell it +// apart. A mutant whose target no longer matches, or that never reached a +// transformed module, is an error: the list has drifted from the code. +// +// Usage: node scripts/demand-coverage-mutants.mjs [--only=P01,P02] [--list] +// The review record `docs/contributing/oracle-reviews/ +// 2026-10-21-persisted-demand-coverage-review.md` describes the sweep. +import { execFile } from 'node:child_process' +import { mkdtempSync, readFileSync, existsSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' + +const ROOT = resolve(import.meta.dirname, `..`) +const PACKAGE = join(ROOT, `packages/db-sqlite-persistence-core`) +const ORACLES = [ + `tests/persisted-demand-coverage-oracle.test.ts`, + `tests/persisted-demand-coverage-campaign-oracle.test.ts`, + `tests/persisted-readiness-oracle.test.ts`, +] +const TIMEOUT_MS = 15 * 60_000 + +const MUTANTS = [ + { + id: 'P38', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'const deadlineAt = startedAt + this.persistedReadiness.networkTimeoutMs', + replace: 'const deadlineAt = startedAt', + rationale: 'a covered load settles from SQLite without waiting', + }, + { + id: 'P01', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ': JSON.stringify(rows.map(({ key }) => key).sort())', + replace: ': JSON.stringify(rows.map(({ key }) => key))', + rationale: + 'windowKeys order-sensitive: window reorder without membership change uncovers', + }, + { + id: 'P02', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ': JSON.stringify(rows.map(({ key }) => key).sort())', + replace: ': undefined', + rationale: 'windowKeys ignored: ordered windows covered by row count only', + }, + { + id: 'P03', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: '`${DEMAND_COVERAGE_PREFIX}${getLoadSubsetDemandKey(options)}`', + replace: '`${getLoadSubsetDemandKey(options)}`', + rationale: + 'coverage key not namespaced: truncate prefix sweep no longer finds it', + }, + { + id: 'P04', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'sourceTruncateGeneration === this.sourceTruncateGeneration', + replace: 'true', + rationale: 'captureResetEpoch ignores source truncate generation', + }, + { + id: 'P05', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'resetSequence === this.resetSequence &&', + replace: 'true &&', + rationale: 'captureResetEpoch ignores resetSequence', + equivalent: + 'in the single-process scope only scoped recovery bumps resetSequence while a load or record exists (startup reclaim precedes loads; a peer reset is multi-tab), and during scoped recovery the coverage read proves nothing', + }, + { + id: 'P06', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' lifecycleGeneration === this.lifecycleGeneration &&\n resetSequence === this.resetSequence &&', + replace: ' resetSequence === this.resetSequence &&', + rationale: 'captureResetEpoch ignores lifecycle generation', + equivalent: + 'a replaced lifecycle has no live sync callbacks, so a late coverage record is a no-op', + }, + { + id: 'P07', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'if (!record || !current()) return', + replace: 'if (!record) return', + rationale: + 'recordCoverage: no check after the coverage read that a reset may have begun', + }, + { + id: 'P08', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'if (!current()) return\n const record = await runtime.nextCoverageRecord(demand)', + replace: 'const record = await runtime.nextCoverageRecord(demand)', + rationale: + 'recordCoverage: early isCurrent check removed (should be EQUIVALENT: second check still guards)', + equivalent: + 'the isCurrent check after the coverage read still guards the record', + }, + { + id: 'P09', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'if (current?.rows === rows.length && current.keys === keys) return undefined', + replace: 'if (false) return undefined', + rationale: + 'nextCoverageRecord: always rewrites record even when unchanged (should be EQUIVALENT)', + equivalent: 'rewriting an unchanged record changes no observation', + }, + { + id: 'P10', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'record?.generation === this.storageCollectionId &&', + replace: 'record !== undefined &&', + rationale: + 'currentCoverageRecord ignores storage generation: stale-generation record counts', + equivalent: + 'a stale-generation record can stay in memory only after an in-run rotation, and every in-run rotation (scoped recovery, claim recovery through startScopedRecovery) leaves the coverage read proving nothing', + }, + { + id: 'P11', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'rows.length >= record.rows &&', + replace: 'rows.length > record.rows &&', + rationale: 'isDemandCovered off-by-one: exact-count rows no longer cover', + }, + { + id: 'P12', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'rows.length >= record.rows &&', + replace: 'true &&', + rationale: 'isDemandCovered ignores row count (only window keys remain)', + }, + { + id: 'P13', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'windowKeys(options, rows) === record.keys', + replace: 'true', + rationale: + 'isDemandCovered ignores window keys: replaced row with same count still covered', + }, + { + id: 'P15', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'return await this.applyMutex.run(() =>', + replace: 'return await Promise.resolve().then(() =>', + rationale: + 'readDemandRows not serialized behind committed source transactions', + }, + { + id: 'P16', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' this.scheduleCacheClaimRecovery(this.lifecycleGeneration)\n }\n return undefined', + replace: ' void 0\n }\n return undefined', + rationale: 'readDemandRows: expired claim does not start claim recovery', + }, + { + id: 'P17', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'options.refetch === true ||\n !(await this.isDemandCovered(options))', + replace: '!(await this.isDemandCovered(options))', + rationale: + 'settleCoveredDemand: refetch check removed (refetch settled from coverage)', + }, + { + id: 'P18', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: '!this.persistedReadiness?.onDemand ||\n options.refetch', + replace: 'options.refetch', + rationale: + 'settleCoveredDemand: onDemand check removed (EQUIVALENT if records only exist for onDemand)', + equivalent: + 'persisted readiness exists only for an on-demand opt-in here, so the check is redundant', + }, + { + id: 'P19', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' if (options.signal?.aborted) return\n const deadlineAt', + replace: ' const deadlineAt', + rationale: 'settleCoveredDemand: released load still enters the wait', + }, + { + id: 'P20', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'const deadlineAt = startedAt + this.persistedReadiness.networkTimeoutMs', + replace: + 'const deadlineAt = Date.now() + this.persistedReadiness.networkTimeoutMs', + rationale: 'deadline anchored at coverage-check time instead of load start', + }, + { + id: 'P21', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'Math.max(0, deadlineAt - Date.now())', + replace: 'deadlineAt - Date.now()', + rationale: + 'timer delay without Math.max (EQUIVALENT: negative delay fires at once)', + equivalent: 'setTimeout treats a negative delay as zero', + }, + { + id: 'P22', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' options.signal?.addEventListener(`abort`, settle, { once: true })\n', + replace: '', + rationale: 'settleCoveredDemand: release does not end the covered wait', + }, + { + id: 'P23', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' return upstream\n }\n // A released load', + replace: ' return\n }\n // A released load', + rationale: 'uncovered load resolves without waiting for its upstream load', + }, + { + id: 'P24', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' const startedAt = Date.now()\n this.throwIfTerminal()\n await this.ensureCacheClaimActive()', + replace: + ' this.throwIfTerminal()\n await this.ensureCacheClaimActive()\n const startedAt = Date.now()', + rationale: + 'load start clock read after cache claim wait (wait anchored late)', + }, + { + id: 'P25', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' await this.settleCoveredDemand(options, upstream, startedAt)', + replace: + ' void this.settleCoveredDemand(options, upstream, startedAt)', + rationale: 'loadSubset resolves before its covered settlement', + }, + { + id: 'P26', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' await upstreamLoadSubset(options)\n } catch (error) {\n this.noteUpstreamLoadFailure(options, error)', + replace: + ' await upstreamLoadSubset(options)\n } catch (error) {', + rationale: 'loadSubset: upstream failure not noted', + equivalent: + 'observable only through the Query DB receiving witness, outside this harness', + }, + { + id: 'P27', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' truncateGeneration === this.sourceTruncateGeneration &&\n this.activeSubsets.get(subsetKey) === options\n ) {\n this.hydratedDemands.add', + replace: + ' true &&\n this.activeSubsets.get(subsetKey) === options\n ) {\n this.hydratedDemands.add', + rationale: + 'loadSubset marks demand hydrated even when a truncate intervened', + equivalent: + 'the hydrated mark after a truncate belongs to the hydration path; the truncate already ends coverage', + }, + { + id: 'P28', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' this.hasExpiredCacheClaim() ||\n !this.startupSettled ||', + replace: ' !this.startupSettled ||', + rationale: + 'loadHydratedSubset: settles from coverage under an expired cache claim', + equivalent: 'a coverage read under an expired claim already proves nothing', + }, + { + id: 'P29', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' this.isHydratingNow() ||\n this.canRouteRemoteDemandThroughCoordinator() ||\n options.refetch === true ||', + replace: + ' this.isHydratingNow() ||\n this.canRouteRemoteDemandThroughCoordinator() ||', + rationale: 'loadHydratedSubset: refetch check removed', + equivalent: 'settleCoveredDemand repeats the refetch guard', + }, + { + id: 'P30', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' options.refetch === true ||\n !this.hydratedDemands.has(getLoadSubsetDemandKey(options))', + replace: ' options.refetch === true', + rationale: + 'loadHydratedSubset: no restore-published-this-run gate (law condition 1)', + }, + { + id: 'P31', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'void recordCoverage(options, isCurrent).catch(() => undefined)\n return true', + replace: 'return true', + rationale: 'synchronous answer does not record coverage', + }, + { + id: 'P32', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'return loaded.then(() => recordCoverage(options, isCurrent))', + replace: 'return loaded', + rationale: 'asynchronous answer does not record coverage', + }, + { + id: 'P33', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'const isCurrent = runtime.captureResetEpoch()', + replace: 'const isCurrent = () => true', + rationale: + 'recordCoverage ignores reset epoch (stale load records coverage)', + }, + { + id: 'P34', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'acquisitions.get(options) !== acquisition\n ) {\n return true', + replace: 'false\n ) {\n return true', + rationale: + 'forwardUpstream: released acquisition still starts upstream lease', + equivalent: + 'hydration checks the abort signal at each step before it forwards a load', + }, + { + id: 'P35', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' endDemandCoverage(openTransaction)\n openTransaction.truncate = true', + replace: ' openTransaction.truncate = true', + rationale: 'source truncate does not delete demand coverage records', + }, + { + id: 'P36', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' } catch {\n if (this.hasExpiredCacheClaim()) {\n this.scheduleCacheClaimRecovery(this.lifecycleGeneration)\n }\n return undefined\n }', + replace: + ' } catch (error) {\n if (!this.hasExpiredCacheClaim()) throw error\n this.scheduleCacheClaimRecovery(this.lifecycleGeneration)\n return undefined\n }', + rationale: 'a failed coverage read fails the load', + }, + { + id: 'P37', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' options.limit === undefined\n ? undefined\n : JSON.stringify', + replace: ' JSON.stringify', + rationale: 'every demand records its row keys, not only a window', + }, + // The coverage key. The model judges demand identity from the request + // data itself, so a key that merges two pages is visible to it. (Mutating + // core's own demand key is not distinguishable here: core's subset + // deduplication uses the same key, so the merged key changes which loads + // core sends, and the model reads those loads from the record.) + { + id: 'K01', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: '`${DEMAND_COVERAGE_PREFIX}${getLoadSubsetDemandKey(options)}`', + replace: + '`${DEMAND_COVERAGE_PREFIX}${getLoadSubsetDemandKey({ ...options, cursor: undefined, offset: undefined })}`', + rationale: 'coverage keyed without cursor and offset: pages share coverage', + }, + { + id: 'P40', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' if (options?.snapshot) endDemandCoverage(transaction)\n', + replace: '', + rationale: 'a snapshot transaction keeps demand coverage', + }, + { + id: 'P41', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: 'const current = () => isCurrent() && !demand.signal?.aborted', + replace: 'const current = isCurrent', + rationale: + 'a load that core aborted records coverage when its source resolves', + }, + { + id: 'P42', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' if (this.scopedRecovery) return undefined\n', + replace: '', + rationale: 'scoped recovery reads, records, and settles coverage', + }, + { + id: 'P39', + file: 'packages/db-sqlite-persistence-core/src/persisted.ts', + find: ' !persistedReadiness ||\n persistedReadiness.onDemand ||\n', + replace: ' !persistedReadiness ||\n', + rationale: + 'an on-demand Collection opens from disk after the deadline, like an eager one', + }, +] + +function run(mutant) { + const mark = join(mkdtempSync(join(tmpdir(), `db-mutant-`)), `applied`) + return new Promise((done) => { + execFile( + `npx`, + [ + `vitest`, + `run`, + `--config`, + `vitest.mutants.config.ts`, + `--bail`, + `1`, + ...ORACLES, + ], + { + cwd: PACKAGE, + timeout: TIMEOUT_MS, + killSignal: `SIGKILL`, + maxBuffer: 256 * 1024 * 1024, + env: { + ...process.env, + CI: `1`, + TANSTACK_DB_MUTANT: JSON.stringify(mutant), + TANSTACK_DB_MUTANT_MARK: mark, + }, + }, + (error, stdout, stderr) => { + const output = `${stdout}\n${stderr}`.replace(/\x1b\[[0-9;]*m/g, ``) + const failed = output.match(/^\s*×\s.*$/m)?.[0].trim() + const applied = + existsSync(mark) && readFileSync(mark, `utf8`).trim() !== `` + if (!applied) + done({ outcome: `not applied`, detail: output.slice(-2000) }) + else if (error?.killed) done({ outcome: `timeout` }) + else if (!error) done({ outcome: `survived` }) + else if (failed) done({ outcome: `killed`, detail: failed }) + else done({ outcome: `error`, detail: output.slice(-2000) }) + }, + ) + }) +} + +const only = process.argv + .find((arg) => arg.startsWith(`--only=`)) + ?.slice(`--only=`.length) + .split(`,`) +const selected = MUTANTS.filter((mutant) => !only || only.includes(mutant.id)) +if (process.argv.includes(`--list`)) { + for (const m of selected) + console.log(`${m.id} ${m.equivalent ? `(equivalent) ` : ``}${m.rationale}`) + process.exit(0) +} + +// Validate every target before running anything. +const drifted = selected.filter( + (m) => readFileSync(join(ROOT, m.file), `utf8`).split(m.find).length !== 2, +) +if (drifted.length > 0) { + console.error( + `mutant targets no longer match: ${drifted.map((m) => m.id).join(`, `)}`, + ) + process.exit(1) +} + +const failures = [] +for (const mutant of selected) { + const started = Date.now() + const { outcome, detail } = await run(mutant) + const expected = mutant.equivalent ? `survived` : `killed` + const ok = outcome === expected || (mutant.equivalent && outcome === `killed`) + const seconds = Math.round((Date.now() - started) / 1000) + console.log( + `${ok ? `ok ` : `BAD`} ${mutant.id} ${outcome} (${seconds}s) ${mutant.rationale}${detail && outcome === `killed` ? ` | ${detail}` : ``}`, + ) + if (!ok) failures.push({ mutant, outcome, detail }) +} +for (const { mutant, outcome, detail } of failures) { + console.error( + `\n${mutant.id} ${outcome}: ${mutant.rationale}\n${detail ?? ``}`, + ) +} +if (failures.length > 0) process.exit(1)