From 20e2ad9f81c91c60e5d9c219892c6421ae12baca Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 10:24:01 -0600 Subject: [PATCH 01/11] fix(persistence): recover Electric caches after interleaved source commits Law: after successful recovery and source catch-up, every open eager Collection and durable rows equal the source head (coverage-map Law 2, #2209). Preserve arrival-order application and cached-row visibility. Reach: Browser and SQLite source-progress properties each run 80 fixed and 80 random histories. Browser restart/reload reached 25/13 fixed and 24/23 random. SQLite discontinuity/duplicate/migration reached 61/64/34 fixed and 53/61/39 random. Ten real SDK/OPFS cells each reached all four declared states. RED: before production edits on 82c5f999b, Browser source-prefix seed2281 path0:0:0:0 lost row1; SQLite predecessor oracle missed gap evidence; real SDK/OPFS matrix had four semantic failures and four passing drain controls. Commands and full chronology are in docs/contributing/oracle-reviews/issue-2281-investigation.md. GREEN: pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run tests/multi-tab-electric-oracle.test.ts --maxWorkers=2 --coverage.enabled=false: 312 passed,2 expected failures. SQLite resume owner61, Electric races49, bridge8, real SDK/OPFS14 passed. Browser tsc, lint:changed and diff check pass on base07bfc0eb2. Accepted test-driver simplification passed tsc and one real receiving cell. Mutant: dropping durable gap retention fails core, Browser restart/reload and real disjoint comparisons; rejecting duplicate endpoints fails core; bypassing startup provenance fails controlled and real legacy comparisons. All mutants restored. Limits: bounded same-version eager SQLite sources, controlled Node and real Chromium OPFS paths. No monotonic live ordering, mixed-version writes, on-demand, native-device or arbitrary scheduling claim. Separate #2209 permanent-close, #2267 resumed-update and seed-metadata gaps remain open. --- docs/contributing/glossary.md | 1 + docs/contributing/oracle-coverage.md | 74 +- .../issue-2281-investigation.md | 909 ++++++++++++++++++ ...electric-source-prefix-oracle.opfs.spec.ts | 246 +++++ .../e2e/electric-source-prefix.opfs.html | 10 + .../e2e/electric-source-prefix.opfs.ts | 178 ++++ .../playwright.opfs-electric.config.ts | 1 + .../tests/electric-convergence-oracle.ts | 49 + .../tests/multi-tab-electric-oracle.test.ts | 399 +++++++- .../src/persisted.ts | 52 + .../src/sqlite-core-adapter.ts | 70 +- .../sqlite-resume-snapshot-oracle.test.ts | 180 +++- packages/db/src/sync-persistence.ts | 12 + packages/db/src/types.ts | 15 + packages/db/tests/oracle-config.ts | 2 + packages/db/tests/sync-persistence.test.ts | 23 + .../electric-db-collection/src/electric.ts | 26 +- .../electric-resume-snapshot-races.test.ts | 13 + 18 files changed, 2205 insertions(+), 55 deletions(-) create mode 100644 docs/contributing/oracle-reviews/issue-2281-investigation.md create mode 100644 packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts create mode 100644 packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.html create mode 100644 packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts create mode 100644 packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts diff --git a/docs/contributing/glossary.md b/docs/contributing/glossary.md index e64d8c7214..61eb31cd5c 100644 --- a/docs/contributing/glossary.md +++ b/docs/contributing/glossary.md @@ -98,6 +98,7 @@ until cleanup ends the run. | durable anchor | The row version and reset epoch observed before one source sync transaction is sent to a coordinator. If both still match and the exact transaction ID is absent, the replacement leader may apply that transaction once. | | exact-ID reconciliation | The replacement leader checks one immutable source transaction ID under the SQLite writer lock after its RPC result becomes indeterminate. A present ID acknowledges the original commit; an absent ID needs an unchanged durable anchor. | | reconciliation reload | A peer notification after exact-ID reconciliation that reloads persisted rows even when the receiver has already observed the same stream position. Equal public rows do not publish a second change batch. | +| source progress token | A sync adapter's opaque identity for one complete materialization at a provider position. Re-delivering that position is idempotent. Its predecessor expectation lets persistence detect a discontinuity between interleaved source commits without ordering provider handles. A discontinuity invalidates source progress evidence until an authoritative replacement. It is separate from a durable row version and a sync-run generation. | ## Offline outbox terms diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 89890d7a4a..d6cc3dd4c9 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2693,6 +2693,18 @@ serialization, unavailable key evidence after raw loss, concurrent schema migration, WAL across processes, and arbitrary longer transaction histories remain with their existing owners or need explicit future witnesses. +The same owner checks the optional source-progress bridge with an independent +path-of-edges model. A discontinuity stays incompatible across matching and +empty commits; a repeated endpoint is idempotent; a full replacement repairs +continuity. Physical key integrity is checked separately after every transaction. +The fixed/random `sqlite-resume.source-progress` campaigns use 80 runs each, +seed 2281 and direct seed/path replay. Histories have 3–16 actions plus an initial +replacement, opaque positions, stale predecessors, duplicates and optional old +schema migration. Declared reaches require gaps, later matching commits, +replacement after a gap, duplicate endpoints and migration. The primary Browser +owner and real OPFS receiver supply source-head convergence evidence; the storage +model alone does not establish provider framing or cross-host scheduling. + The SQLite resume-snapshot owner also checks a cold, unique-key, non-delete full replacement with 205 rows. It compares exact durable rows, row metadata, key-set evidence, resume metadata, and applied position at transaction return; @@ -2914,8 +2926,11 @@ lagging stream applies v1 at `20_0`, and tab B closes before it reaches v2. Tab A then shows v1, and the durable `electric:resume` offset is back at `20_0`. Nothing redelivers v2 to tab A while it stays open, because its stream is already past it. A tab that opens later resumes from `20_0` and -repairs the cache. The convergence above therefore holds only while the -lagging tab keeps running or another tab restarts its stream. The +repairs the cache in that history. Restart formerly failed when a later commit advanced the resume position +over stale rows (#2281). The eager SQLite bridge now records source +continuity atomically and requires a fresh snapshot on reopen after a gap; +see the [repair evidence](oracle-reviews/issue-2281-investigation.md). +This preserves temporary arrival-order regressions and leaves #2209 open. The multi-tab Electric oracle in `browser-db-sqlite-persistence` pins this history with `it.fails` beside a passing control in which the lagging tab catches up. Ordering the copies would need a generic @@ -3958,7 +3973,7 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Generated histories in the same file interleave server changes, deploys, partial and complete deliveries, every stop except resume invalidation, restarts, and reloads across two or three eager tabs - (fixed seed 952051961 and a random seed, 60 runs each; the shared + (fixed seed 952051961 and a random seed, 80 runs each; the shared `TANSTACK_DB_ORACLE_SEED`, `_PATH`, and `_PROPERTY= browser.multi-tab-reserved-history` replay one). The model folds committed batches into the expected durable rows. After @@ -3990,17 +4005,54 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. A delivery may also commit only a prefix of what a stream owes, as a server answering at an earlier moment does, so lagging commits are generated. The final checkpoint requires the server head (Law 2). - Known gap #2209 is the one history that breaks it: a lagging tab + Known gap #2209 is one history that breaks it: a lagging tab commits an older row and closes for good. Generated histories never close a tab for good, and an `it.fails` replay through the same driver and checks pins that history. - A random CI campaign (seed `-1660880085`, path - `55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`) found a second Law 2 history that - reproduces on `main`: row 1 reaches v2 and both tabs catch up, then - tab B stops, tab A catches up, and tab A stops, and the durable rows - lose row 1. The maintainer chose to leave it open with #2281; an - `it.fails` replay through the same driver and checks pins it, and the - random campaign can still draw it. Owner: this oracle. + The [#2281 repair record](oracle-reviews/issue-2281-investigation.md) + retains the row-loss and resurrection counterexamples on `82c5f999b`. + A lagging source changed rows; another source advanced the shared resume + position over the missing change. The eager Electric/SQLite bridge now + detects a discontinuous predecessor atomically with rows and metadata. + Source continuity remains incompatible until an authoritative replacement; + the physical key ledger stays independent, so cached rows remain visible. + Reopening a missing, foreign, or discontinuous source token requests a + fresh snapshot. The original row-loss replay and both six-step membership + replays now pass. The generated `browser.electric-source-prefix` property + runs 80 histories with seed 2281 and 80 without a seed, with direct + seed/path replay. It crosses initial membership, one to three toggle + pairs, swapped source roles, empty/disjoint advancement, drain/no-drain, + tab reload or source failure followed by same-Collection restart, and + current/missing/foreign progress. Declared reach + requires all four divergent membership/advancement cells, a draining + control, missing/foreign provenance, and divergent recovery through both + actual lifecycle paths. Both Browser random registrations and the SQLite + source-progress registration stand aside during another property's path + replay. The SQLite owner separately + checks opaque predecessor paths, sticky gaps, duplicate endpoints, + replacement repair, and additive schema migration (80 fixed/random + histories each). Disabling gap retention fails both owners; rejecting + duplicate endpoints fails the SQLite law. Bypassing startup provenance + validation fails the Browser owner. + The real installed SDK/HTTP/PostgreSQL/Chromium OPFS receiving driver + `e2e/electric-source-prefix-oracle.opfs.spec.ts` shares the source-head + judgment. Ten cells cover both membership directions, empty/disjoint + advancement and drain/no-drain, plus missing legacy provenance with + disjoint advancement. Each declares held earlier response, stale commit, + later advancement, and reopened source comparison. All ten pass; gap and + legacy-validation mutants fail the matching real disjoint cells at that + comparison. This is bounded same-version eager Browser evidence, not + arbitrary scheduling, mixed client versions, native-device, Electron, + on-demand, optimistic, or monotonic-live-write coverage. + The final checkpoint now observes a reset-abandoned commit's source + failure, restarts it, and only then judges catch-up. The original reset + trace (seed `-1115798760`, path `1:4:3:3:3:3:3:3:3:3:3`) and the later + no-deploy reset trace (seed `-1371495859`, path + `75:0:0:0:0:0:0:1:1:0:0`) pass on unchanged production with that repaired + checkpoint. Both remain fixed inputs to the shared driver. A generated + cached-row regression also preserves intermediate observations during + source-continuity repair. #2209's permanent-close case and #2267's + resumed-update gap remain pinned expected failures under this owner. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md new file mode 100644 index 0000000000..ab797cd124 --- /dev/null +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -0,0 +1,909 @@ +# Issue #2281 investigation + +Investigated on 2026-10-29 against production commit +`82c5f999b32c9db827dd74e3f172604dfda28013`. The working branch is +`codex/investigate-2281`. +The investigation sections below preserve the original findings. The appended +implementation evidence records the subsequent repair and supersedes their +open-work status. + +This record describes an investigation, not a production repair or a claim +that the bug class is closed. Production and the generated campaign are +unchanged. The test additions retain two reduced counterexamples and four +passing controls through the existing model, driver, and comparison. + +Read the [issue and both comments](https://github.com/TanStack/db/issues/2281). +There are two different accounts of the multi-tab failures: + +1. The original reset/deploy trace reaches a premature convergence checkpoint. + Its final deliveries fail the sources; the driver does not recover them + afterward. Row 1 never leaves durable storage in this trace. +2. Both later multi-tab traces expose a production defect: a lagging source + changes the shared rows, then another source stores a newer resume position + without repairing those rows. A reopened source resumes beyond the missing + change. This is a concrete extension of the ordering problem in + [#2209](https://github.com/TanStack/db/issues/2209). + +The separate sync-metadata trace also reproduces. Its root cause was not +investigated here; no evidence joins it to the source-position defect. + +## Authority and observation + +Law 2 in `multi-tab-electric-oracle.test.ts` promises that durable rows and +every open eager tab equal the server head after the streams catch up. Its +authority is the persisted multi-tab eventual-consistency contract recorded +in the coverage map and #2209. Temporary regressions during interleaving are +allowed. Permanent divergence after recovery and catch-up is a violation. + +Law 5 in the same owner requires a peer reset to abandon older reservations. +Electric's next commit of such a transaction fails its source, and the app +recovers by restarting the tab. That maintainer decision on PR #2265 matters +to the original trace: a failed commit cannot establish catch-up. + +All experiments use the existing production driver: one SQLite file, a +better-sqlite3 connection per tab, the real Browser coordinator with Node +BroadcastChannel/Web Locks, the Browser persistence factory, persisted +Collections, and Electric with a controlled ShapeStream. The existing check +compares durable rows, public rows, and Collection status after each step +and at the final checkpoint. Temporary instrumentation additionally read +`collection_metadata['electric:resume']` through an observer connection and +recorded each recovered stream's requested handle/offset. + +## Original trace: the final delivery creates the failure + +Seed `-1115798760`, path `1:4:3:3:3:3:3:3:3:3:3` reproduces the issue's exact +final mismatch. Both tabs stage row 1's delete without committing, then the +peer reset abandons both reservations. Durable rows remain the initial rows +1 and 2 through every numbered step. There is no durable deletion followed +by a durable resurrection in this execution. + +The final loop checks `failedByReset` before `deliver()`. At that point it is +false: neither abandoned transaction has tried to commit yet. Delivery +encounters must-refetch and eventually attempts the abandoned transaction's +commit. It fails with `AbortError: Sync transaction was aborted before +application`. Both tabs stay ready with source errors, as Law 5 requires. +Only now does the model set `failedByReset = true`. The loop ends and checks +the server head, without restarting either failed source. + +A temporary driver intervention added the following immediately after the +final loop's existing `await deliver(index, 'all', true)`: + +```ts +if (models[index]!.failedByReset) { + await recover(index, `restart`) + await deliver(index, `all`, true) +} +``` + +With unchanged production, the original replay passes. The recorded original +history shows both restarted streams request offset `-1`; both accept the +fresh snapshot, and durable rows and both ready tabs contain only row 2, +with no source error. The intervention was removed after the experiment. +It establishes this trace's missing recovery, not a general driver repair. +A permanent checkpoint repair must retain the comparison of the failed +source state and then establish successful recovery before checking Law 2. + +## Later traces: rows and the resume position describe different prefixes + +The resurrection in the 2026-10-29 comment reproduces with seed `-699129038`, +path `78:2:0:3:9:12:12:13:12:13:13:13:13:14:14:14`: + +| Step | Durable row 3 | Durable offset | +| --- | --- | --- | +| A commits insert(3), delete(3) | absent | `2_0` | +| B commits only the older insert(3) | `v1` | `1_0` | +| A commits delete(1), insert(1) | still `v1` | `4_0` | +| B reopens, then A reopens | still `v1` | both request `4_0` | +| Both streams catch up with no remaining changes | still `v1` | `4_0` | + +The server deleted row 3 at offset `2_0`. B's late insert reintroduces it. +A's next batch touches only row 1, so it advances the global resume position +without removing row 3. Both new streams skip its delete. Unlike the original +reset trace, this reaches the final comparison with healthy, caught-up +streams and permanently wrong rows. + +The row-loss comment reproduces with seed `-1660880085`, path +`55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`. A commits delete(1) at `1_0`, then insert(1) +at `2_0`. B's older delete removes row 1 and sets the durable offset to `1_0`. +A then receives only up-to-date at its existing `2_0`: its metadata-only +transaction advances the durable offset to `2_0` while leaving row 1 absent. +Both tabs stop and restart from `2_0`, so neither receives the missing insert. +The loss happens at B's stale commit, before either stop. + +## Six-step reductions and controls + +After warming both tabs with rows 1 and 2, use this shared history: + +```ts +[ + { kind: `server change`, key, pick: 0 }, + { kind: `server change`, key, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `recover`, tab: 1, pick: 0 }, +] +``` + +For initially present key 1, the server changes are delete then insert. For +initially absent key 3, they are insert then delete. In each case A commits +through `2_0`, B commits only `1_0`, A stores `2_0` with an empty row batch, +and B reopens from `2_0`. Both cases fail the unchanged final comparison: +key 1 is missing, or key 3 survives with `label: 'v1'`. + +| Continuation after B's older commit | Key 1 | Key 3 | +| --- | --- | --- | +| A stores its cursor; B reopens | final assertion fails | final assertion fails | +| A stores its cursor; B drains its backlog before reopening | passes | passes | +| B reopens before A stores its cursor | passes | passes | + +These six executions are retained in the owner as two `it.fails` replays and +four passing controls. They contain no independent expected answers. The +existing generated grammar already has the necessary actions; its minimum +length is eight, so the six-step diagnostic reduction is shorter than its +generated domain. The reported longer counterexamples demonstrate generated +reach of the defect. No new generated reach state or sampling guarantee is +claimed by these fixed replays. + +## Production mechanism and repair boundary + +Relevant source locations at the investigated production commit: + +- `packages/electric-db-collection/src/electric.ts:2028`, + `stageResumeMetadata`: writes the current stream's handle and offset. +- `packages/electric-db-collection/src/electric.ts:2531`: an up-to-date with + no open row transaction still opens and commits a metadata transaction. +- `packages/db-sqlite-persistence-core/src/persisted.ts:4529`: forwards each + external source transaction to the coordinator with a local stream position. +- `packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts:2599`: orders + application by durable row version. Row mutations and collection metadata + are applied together, but no Electric server position fences stale copies. + +Atomic rows and metadata within each commit are insufficient: commits from +different stream positions are still composed in local arrival order. The +key-set ledger follows those same row mutations, so agreement with that +ledger does not prove that the rows equal the prefix named by the cursor. + +The accepted convergence law needs a production repair. #2209's proposed +source-position protocol is relevant: the writer must reject or reconcile +older source effects and associate a resumable cursor with the state that +actually establishes its prefix. Its design must also define source identity, +handle transitions, and settlement of a tab's dropped duplicate commits. +This investigation does not choose or implement that protocol. + +Two smaller changes are insufficient by themselves: + +- Keeping only the maximum cursor still allows B's stale row mutation. It + can leave storage already inconsistent with that maximum cursor. +- Suppressing metadata-only commits removes the reduced trigger but does + not repair the reported resurrection: A's ordinary row-1 transaction + advances to `4_0` while row 3 remains wrong. + +The primary oracle's per-step fold deliberately accepts the temporary stale +rows permitted by the contract. Its final server-head comparison correctly +rejects the later traces. Fixed campaigns did not force the combination of a +stale peer write, a later cursor advance, and recovery that skips the missing +change. Existing reach for durable rows behind the server does not establish +that combination. A repair needs a declared reach for that combination in +generated histories, including empty and disjoint-key advancing batches, +both membership directions, and the relevant public receiving paths. + +## Separate metadata comment + +The comment's seed-only command with `_PROPERTY` fails setup on this revision: +the replay parser also requires `_PATH`. That is not evidence of the metadata +law. Instead, the existing pinned replay was run by name. With `it.fails` +temporarily removed, it fails at step 6 (`peer`) with the exact reported law: + +```text +history: begin seed(1,bare) insertOverSeed(1) begin reinsert(1) commit peer(2,reload) +law: source reads its row metadata (metadata.row.get) +expected: {"1":"","2":{"peer":{"s":6}}} +actual: {"1":{"cached":true},"2":{"peer":{"s":6}}} +``` + +The test file was restored. The owning sync-metadata coverage entry already +records this gap. No production root cause or repair is claimed for it. + +## Evidence and replay + +- **Law:** eventual server-head convergence after actual catch-up (#2209 and + the coverage map); reset-abandoned commits fail until restart (PR #2265). +- **Reach:** all three supplied multi-tab traces executed and reached the + reported final mismatch. Six reduced/control histories executed: two + rejected stale final rows and four converged. These are finite witnesses, + not a new normal generated reach measurement. +- **RED:** all three supplied seed/path replays fail on unchanged production; + the two six-step reductions fail through the same comparison. +- **GREEN:** the four neighboring controls pass. The original reset trace + passes after temporary post-failure recovery in the driver. These are + diagnostic greens, not a green production repair. +- **Mutant:** none run; no closure claim. The unchanged defective production + supplies the two reduced wrong-result witnesses. +- **Limits:** eager mode, one Collection, one process, controlled Electric + SDK messages, Node coordinator APIs, and native SQLite. No live Electric + HTTP/SDK, Chromium OPFS scheduling, on-demand rows, optimistic mutations, + source-position protocol, or arbitrary interleaving closure is established. + The original checkpoint still needs repair; both production row defects + remain open, as does the distinct metadata gap. + +Dependencies installed successfully with `pnpm install --frozen-lockfile`; +no override or lockfile change was needed. Diagnostic runs disabled coverage +and typechecking. The final six replay cases also ran with package test +typechecking enabled: four runtime controls pass, two runtime cases are +expected failures, and typechecking reports no errors. + +From the repository root, replay a reported multi-tab case by setting its seed +and path above: + +```sh +TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history \ +TANSTACK_DB_ORACLE_SEED=-699129038 \ +TANSTACK_DB_ORACLE_PATH='78:2:0:3:9:12:12:13:12:13:13:13:13:14:14:14' \ +pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run \ + tests/multi-tab-electric-oracle.test.ts -t 'random or replayed' \ + --coverage.enabled=false --typecheck.enabled=false + +pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run \ + tests/multi-tab-electric-oracle.test.ts \ + -t 'cursor-only commit|lagging stream drains|reloads before the cursor advances' \ + --coverage.enabled=false + +pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest --run \ + tests/sync-metadata-session-persisted.test.ts \ + -t 'a reload after an inner re-insert' --coverage.enabled=false \ + --typecheck.enabled=false +``` + +Raw diagnostic logs and the temporary instrumentation patch are local artifacts +under `notes/issue-2281/` (ignored by Git). The retained tests and this record +contain the essential witnesses independently of those local artifacts. + +## Implementation pre-edit gate (2026-10-29) + +The user authorized implementation after the investigation. No production file +has changed at this gate. The reset checkpoint now checks the failed source +before restarting it and comparing the successful catch-up. Its original +seed/path passes on the unchanged production head. + +The selected repair preserves arrival-order application. An eager Electric +source supplies an opaque progress token and the token its transaction +continues. SQLite compares that predecessor under the same writer transaction +as the rows and resume metadata. A mismatch makes the existing resume key-set +evidence incompatible, sticky until an authoritative truncate. Reopen therefore +uses the existing full-snapshot repair. Tokens contain shape identity, opaque +handle, and offset; timestamps are not progress. No lexical handle ordering, +write suppression, new receipt outcome, or stronger live-ordering law is added. +The optional bridge is exposed only by adapters that implement the atomic check. + +Pre-edit receiving inventory: + +| Path | Observation cut and witness | +| --- | --- | +| Controlled Electric + Browser coordinator + native SQLite, leader/follower | Existing primary owner plus generated `browser.electric-source-prefix`: both membership directions, empty/disjoint advancement, swapped source roles, repeated toggles, reload/restart and draining control; final durable/public source-head equality. | +| Real installed Electric SDK + HTTP + PostgreSQL + Chromium OPFS + actual tab coordinator | New `electric-source-prefix-oracle.opfs.spec.ts`, using real responses held before delivery. It shares `sourceHeadViolation` with the primary owner. Every cell declares held older response, stale commit, later advancement and reopened comparison reach. Empty advancement requests an actual non-live catch-up from Electric. | +| SQLite atomic resume-evidence boundary | `sqlite-resume-snapshot-oracle.test.ts`, generated opaque predecessor edges, duplicate/empty commits and authoritative replacements. The model checks complete paths since the last replacement. Comparisons occur after every durable transaction. | +| Custom persistence adapter | Optional support flag and optional bridge method preserve compatibility; unsupported adapters do not advertise atomic progress checks. Their source-head repair is outside this SQLite claim. | + +The core change uses the same SQLite adapter and transaction semantics on its +other hosts. No native-device, Electron-process, progressive, or on-demand +Electric claim is transferred from these eager Browser witnesses. Those paths +are limits, not additional evidence from Node APIs. + +Pre-edit RED results on `82c5f999b`: + +- `browser.electric-source-prefix`, fixed seed 2281, path `0:0:0:0`: + present key, empty advancement, no drain, final source-head comparison loses + row 1 while both Collections are ready. +- `sqlite-resume.source-progress`, seed 2281, path `0:0:0:0:0:0`: + snapshot → advance → advance → stale predecessor still reports `consistent`. +- Real Chromium disjoint/lost-row cell: both Collections ready, no source error, + actual rows only `anchor:advanced`; PostgreSQL also has `target:new`. + Reopened SDK requests the later real LSN and the shared comparison rejects + the missing row. This is a semantic RED; an earlier recorder setup failure + was repaired before obtaining this result. + +Commands and raw output are in the ignored `notes/issue-2281/` logs. The full +real-host defect/control matrix is running before any production edit. + +The pre-edit real-host matrix completed: **4 semantic REDs and 4 GREEN controls** +(2.4 minutes). Every defect cell reached the final shared source-head comparison; +all draining controls declared each required state once and passed. Production +edits begin only after this completed gate. The matrix covered lost/resurrected +rows × empty/disjoint advancement × drain/no-drain. + +## Implementation evidence (2026-10-29) + +This entry supersedes the investigation-only status above. It preserves the +pre-edit gate and the earlier observations instead of rewriting their outcomes. +The reviewed executable state is identified by the manifest at the end of this +record; its parent is `82c5f999b32c9db827dd74e3f172604dfda28013`. + +### Repair and preserved freedom + +The optional bridge now supplies a predecessor token for each eager Electric +commit. Its next token is collection metadata in that same transaction. +Tokens encode shape identity, handle and offset; timestamp differences do not +create source progress. SQLite compares tokens under its writer transaction. +A matching predecessor or repeated complete endpoint preserves continuity. +A gap remains sticky across later commits until an authoritative truncate. +Startup resumes only when the stored token, resume record and atomic source +continuity evidence agree. Otherwise Electric requests a fresh full snapshot. + +The first implementation reused key-set incompatibility. A subsequent random +history rejected that design at an intermediate public-row checkpoint: cached +rows disappeared during reload even though the physical key set was intact. +Seed `-781606242`, path +`82:0:3:0:1:7:7:9:10:20:23:19:14:14:15:13:12:12:12:12:12:12:12:12:12:12:12:12` +now passes and its reduced history remains in the shared driver. Production now +stores **separate** `source_progress_incompatible` evidence. It does not +quarantine cached rows or reinterpret physical key integrity. The additive +migration checks both the setup fast path and the new column's initialization. +The generated SQLite owner reconstructs the old schema before reopening it. + +Arrival-order regressions, normal applied receipts and rows remain allowed. +No source ordering, write rejection, retry queue or new receipt outcome was +introduced. This repairs unsafe continuation after reopen, while #2209's +permanently closed lagging stream and #2267's resumed-update failure remain +tracked expected failures. The distinct seed-metadata failure also stays open. + +### New comment checked during implementation + +The [new comment](https://github.com/TanStack/db/issues/2281#issuecomment-6500845350) +adds seed `-1371495859`, path `75:0:0:0:0:0:0:1:1:0:0`. Its eight-step trace +reloads A, deletes row 1, advances both sources, posts a peer reset, advances +A again, then reinserts row 1. It has no deploy. + +The original production **and original driver** reproduce its exact CI result: +row 1 is `one` instead of `v2`; A is `error`, B is `ready`, both have source +errors, and A's preload rejects with the reported `AbortError`. Keeping original +production but using the repaired final checkpoint passes. The final delivery +is when the reset-abandoned transactions first fail; they need restart before +catch-up is established. This is an evaluation/checkpoint repair under the +adversarial-review repair-location table. No production expectation was relaxed. +The trace is retained as a fixed replay of the same model, driver and check. +Logs: `new-comment-parent-red.log`, `new-comment-checkpoint-green.log` and +`new-comment-current.log` under the ignored local notes directory. + +### Evidence block + +- **Law:** after successful recovery and actual catch-up, durable rows and each + open eager Collection equal the source head (Law 2, coverage-map contract and + #2209). Reset-abandoned commits fail until restart (PR #2265). The optional + bridge's predecessor continuity and idempotent endpoint contract is stated in + `SyncPersistenceCapabilityV1.resumeSnapshot` and tested independently of SQL. +- **Reach:** `browser.electric-source-prefix` uses 80 fixed-seed and 80 random + histories. It crosses membership, 1–3 toggle pairs, swapped fast/slow roles, + empty/disjoint advancement, drain/no-drain, reload/restart and + current/missing/foreign provenance. `sqlite-resume.source-progress` uses the + same 80/80 budget with 3–16 generated actions plus initial replacement. + Its actions are advance, stale, empty, snapshot and duplicate; half the + generated domain reconstructs an old schema. Measured counts follow below. + The real SDK/OPFS receiver ran 10 cells, each reaching all four declared + states once: held earlier response, stale commit, later advancement, + reopened source comparison. No fabricated messages or source positions enter + that driver. The original 8 cells cross membership × advancement × draining; + 2 more receive missing legacy provenance with disjoint advancement. +- **RED:** before production edits, the generated Browser owner at seed 2281, + path `0:0:0:0`, loses row 1 at its source-head comparison. The SQLite owner + rejects missing gap evidence. The real SDK/OPFS matrix produces 4 semantic + failures and 4 passing drain controls. Setup/recorder failures are not counted + as RED. The exact commands are the owner commands below on the parent + production files with the extended tests retained. +- **GREEN:** current production passes the full Browser owner (312 tests and + 2 tracked expected failures before the final new-comment replay was added), + SQLite resume owner (62 tests), persisted grammar (1188 tests, 49 tracked + expected failures, 2 todo), Electric resume-race suite (49 tests), bridge + validation (8 tests), and Browser TypeScript checking. These runs reported + no type errors. The full real SDK/OPFS run passes 14 tests: the 10 new cells + plus 2 existing recovery histories and 2 diagnostic tests. Final restored + mutant runs and the added replay are recorded below. +- **Mutant:** disabling the durable gap parameter fails the SQLite fixed + campaign at a stale predecessor (`2281`, path + `2:3:2:5:6:5:5:0:0:2:2:2:2`), the Browser fixed campaign at a lost row + (`2281`, `0:0:0:0`), and both real disjoint/no-drain cells. Removing the + duplicate-endpoint exception fails the SQLite comparison (`2281`, + `0:7:1:2:2:2:2:2:2`). Accepting missing/foreign startup tokens fails the + Browser fixed campaign (`2281`, `2:0:0`, missing token), and both real legacy + cells. Each rejection is an assertion failure at the intended evidence or + source-head comparison. The real tests' polling deadlines print the actual + persistent wrong rows; they reached the comparison, rather than timing out + during setup. All mutants were temporary and restored with `finally`. +- **Limits:** bounded, same-version eager Electric sources sharing SQLite. + The generic storage law does not prove provider framing. Real receiving + evidence is Chromium OPFS against the installed SDK and local Electric / + PostgreSQL, not a browser matrix or arbitrary network schedule. No claim is + made for mixed old/new client writes, Electron processes, mobile/Tauri, + multi-process WAL, custom adapters without the optional atomic capability, + on-demand subsets, optimistic mutations, or monotonic live source ordering. + The leftover-replay/reset scheduling handoff remains under the coverage-map + owner. Known #2209, #2267 and metadata counterexamples are not closed here. + +Measured normal-run reach (fixed seed 2281 / random): + +| Declared state | Browser histories | +| --- | ---: | +| present / empty | 14 / 10 | +| present / disjoint | 11 / 13 | +| absent / empty | 6 / 6 | +| absent / disjoint | 7 / 9 | +| drained before recovery | 42 / 42 | +| missing provenance at recovery | 26 / 31 | +| foreign provenance at recovery | 21 / 30 | + +| Declared state | SQLite histories | +| --- | ---: | +| discontinuous commit | 61 / 57 | +| matching commit after discontinuity | 37 / 41 | +| replacement after discontinuity | 44 / 34 | +| duplicate source position | 64 / 64 | +| legacy source progress column added | 34 / 35 | + +The unchanged broader history campaign also reached every declared state in +both normal runs. Its fixed/random counts were: must-refetch 47/44, peer commit +during reservation 12/13, stopped with open transaction 21/30, recovered 78/80, +durable behind server 64/65, peer commit before readiness 48/52, peer commit +after error/give-up/abort 2/9, 3/2, 3/2, reset abandoned transaction 7/9 and +reset-abandoned commit failed 4/7. Counts include attempted histories discarded +for the already recorded #2267 gap; the focused source-prefix campaign does +not rely on that gap's exclusion. + +### Oracle review (self-review) + +This is a self-review; no independent reviewer was dispatched. + +| Requirement | Outcome and evidence | +| --- | --- | +| ORC-001 | Law 2 and the optional bridge contract state authority and limits. Temporary stale rows remain permitted; only actual catch-up justifies source-head equality. | +| ORC-002 | The primary model is an independent server snapshot log. The core model is a path of opaque predecessor edges. Neither imports SQL continuation decisions. PostgreSQL supplies the real receiver's independent source head. | +| ORC-003 | The owners explain laws, legal actions, models, drivers and cuts beside executable code. The shared source-head checker names its authority and bounded projection. | +| ORC-004 | Both reported membership defects reconstruct as one toggle pair with no drain. Removing the stale commit or later advancement loses the defect; drain and earlier-reopen controls distinguish it. Extra pairs, swapped roles, empty/disjoint batches, recovery type and provenance retain distinct nearby cases. The core grammar reaches gap/duplicate/replacement/migration. Prefixes beyond the source log and a snapshot without authoritative replacement are excluded. | +| ORC-005 | Controlled production path uses real Browser coordination and SQLite, observing durable/public rows and status after each step and after recovery. Real tabs hold actual SDK HTTP responses. Source facts needed for final equality have arrived before each comparison; the new-comment experiment shows why a failed final commit does not meet that premise. | +| ORC-006 | Three hostile designs fail the intended comparisons, including real gap and legacy receivers. The checkpoint's original driver also rejects both reset traces before the repair. No setup failure is credited. | +| ORC-007 | Both new properties register in `oracle-config.ts`, run equal fixed/random budgets of 80, and use direct seed/path replay. The original broad grammar was not changed, preserving all supplied replay paths. Neither uses `fc.commands`. | +| ORC-008 | The core path distinguishes a coherent prefix from one with any missing edge; later matching commits cannot erase that distinction. Replacement resets the path. Duplicate endpoints add no edge. No production queue or recovery controller is copied into the model. | +| ORC-009 | The glossary defines source progress token. Model-only edges and the server snapshot log map to predecessor continuity and source states; they are not durable row versions or timestamps. | +| ORC-010 | Primary histories retain step/checkpoint and expected/actual observations. The real receiver retains its primary error as `cause`, appends observation/cleanup diagnostics and closes pages, coordinators, databases and temporary PostgreSQL tables. Core preserves primary failure during database cleanup. | +| ORC-011 | The concrete shared-fault hypothesis is trusting a wrong cursor as proof of rows. Source-head recomputation independently challenges the storage edge model; PostgreSQL rows challenge the controlled server/SDK premise. No additional duplicate model is needed for that hypothesis. | +| ORC-012 | This record gives each requirement's result, exact parent, executable manifest, witnessed boundaries, mutants and explicit remaining limits. No known in-scope counterexample remains in the bounded reopened-source class. | +| ORC-013 | Matching versus stale predecessor, repeated endpoint versus new position, sticky gap versus authoritative replacement, both memberships, empty/disjoint advancement and draining controls distinguish nearby wrong rules. Missing provenance cannot be rescued by intact key-set evidence. | +| ORC-014 | The real HTTP/SDK/OPFS receiver reproduces delayed old delivery followed by later cursor advancement. It shares `sourceHeadViolation` with the primary owner and reads PostgreSQL for the expected head. Ten finite cells bound the handoff. | +| ORC-015 | Both generated properties use failing `createOracleReach` checks; normal-run counts appear above. Each real receiving cell declares its own four reaches, all once. | +| ORC-016 | Fixed regressions replay `runGeneratedHistory`; the real cells call the shared judgment. Bridge validation examples stay in the ordinary `sync-persistence.test.ts`. No standalone type examples were added to oracle files. | + +The material formulation attack was a premature catch-up cut; the new trace +confirms that evaluation repair. The conceptual/composition attack was treating +source continuity and physical key integrity as one fact; the cached-row +counterexample required separate evidence. The suitability tradeoff is a full +snapshot after a discontinuity, instead of ordering opaque source handles. +That preserves the accepted arrival-order contract and uses existing recovery. +The tests establish correctness for those cuts, not optimal network work. + +Reproduce the main commands from the worktree root: + +```sh +pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest --run \ + tests/sqlite-resume-snapshot-oracle.test.ts \ + --coverage.enabled=false --disableConsoleIntercept +pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run \ + tests/multi-tab-electric-oracle.test.ts \ + --coverage.enabled=false --disableConsoleIntercept +pnpm --filter @tanstack/browser-db-sqlite-persistence exec playwright test \ + --config playwright.opfs-electric.config.ts \ + electric-source-prefix-oracle.opfs.spec.ts electric-resume-two-tab.opfs.spec.ts +``` + +Mutation commands used the fixed-seed tests selected by behavior name, with +coverage and typechecking disabled. The real gap mutant selected +`--grep 'after disjoint advancement$'`; the provenance mutant selected +`--grep 'from legacy progress'`. The normal typed owner runs above supply the +separate type evidence. Raw logs and the restoring mutation script remain local +in the ignored `notes/issue-2281/` directory. + +### Restored checks, code weight and cleanup refinement + +After restoring all three mutants, both new generated campaigns pass again. +The SQLite owner passes all 62 tests with no type errors. The Browser selected +run passes 8 reported tests, including the new-comment and cached-row replays, +with no type errors. All 6 real disjoint cells pass, including both legacy +cells that rejected the provenance mutant. The fixed reaches are unchanged; +that restored random run reached Browser states 11/19/11/7/32/36/22 and SQLite +states 56/46/40/64/44, in the table order above. + +Self-review found a separate recorder weakness: the existing Node driver +silently ignored Collection cleanup failures, and campaign `finally` blocks +could replace a primary mismatch. Cleanup now attempts every resource release, +retains errors, and aggregates a secondary cleanup error with the original +violation as `cause`. This changes test machinery only. The broader owner was +rerun after this refinement; its result is appended below. + +`pnpm size:compare --files --packages db,db-sqlite-persistence-core,electric-db-collection` +compares the parent merge-base with production minified bundles: + +| Package/scenario | Gzip change | Brotli change | +| --- | ---: | ---: | +| db | +37 B | -55 B | +| db-sqlite-persistence-core | +368 B | +321 B | +| electric-db-collection | +132 B | +112 B | +| three-package total | +537 B | +378 B | +| typical app scenario | +45 B | -60 B | + +The source delta is 156 inserted and 19 removed production lines across five +files. Test/config additions are separate: 1059 inserted and 23 removed lines +before this final evidence entry; none ships in the package bundles. + +Production grows because safe continuation needs durable provenance and an +atomic predecessor check that did not exist. The repair reuses existing +full-snapshot recovery instead of introducing a second recovery controller. +Collection metadata is deduplicated once and reused for both the source check +and its SQL writes. Reusing the old key-set flag was tried and rejected by the +cached-row counterexample: those two facts cannot be merged. Dropping duplicate +endpoint handling fails its hostile mutant. Dropping provenance validation fails +the legacy receivers. Stronger source ordering would require opaque-handle and +skipped-receipt semantics; it was not substituted for the accepted law. + +`pnpm lint:changed`, Browser `tsc --noEmit`, and `git diff --check` pass. +No oracle budget was lowered or removed. Installation used the lockfile as-is; +there is no dependency override or installation workaround in this diff. + + +### Final verification and reviewed state + +The full primary owner after cleanup refinement reports **314 passed and 2 +expected failures**, with no type errors (`multi-tab-close-final.log`). Every +normal fixed/random reach passed. A combined gap mutant plus a forced host +cleanup failure produced an `AggregateError` retaining the original final +source-head mismatch and the secondary cleanup failure; both source files were +restored. `cleanup-calibration.log` records that assertion-level rejection. + +All four supplied multi-tab seeds were then replayed directly on restored code: +`-1115798760`, `-1660880085`, `-699129038`, and the new `-1371495859`. Each +passed its exact reported shrink path through the unchanged broad grammar. +These are replay results, not substitutes for the normal generated campaigns. +Final lint and diff checks pass. No commit, push or pull request was created. +The local Electric/PostgreSQL test stack remains available for reproduction. + +Final primary-owner normal reaches (fixed / random): + +| Declared state | Histories | +| --- | ---: | +| present/empty | 14 / 6 | +| present/disjoint | 11 / 9 | +| absent/empty | 6 / 15 | +| absent/disjoint | 7 / 6 | +| drained before recovery | 42 / 44 | +| missing progress at recovery | 26 / 22 | +| foreign progress at recovery | 21 / 29 | +| must-refetch delivered | 47 / 49 | +| peer commit during an open reserved transaction | 12 / 14 | +| source stopped with an open transaction | 21 / 21 | +| tab recovered | 78 / 79 | +| durable rows behind the server | 64 / 62 | +| peer commit while a tab was not ready | 48 / 53 | +| peer commit after error without retry mid-transaction | 2 / 3 | +| peer commit after SDK gives up mid-transaction | 3 / 1 | +| peer commit after signal aborted mid-transaction | 3 / 5 | +| peer reset abandoned an open transaction | 7 / 15 | +| reset-abandoned commit put the Collection in error | 4 / 13 | + +This review covers the uncommitted working tree on parent +`82c5f999b32c9db827dd74e3f172604dfda28013`, branch `codex/investigate-2281`. +The SHA-256 identity of the source/test/coverage manifest is +`54d96ca70d3be37242cf060b7c7de6ae1e0eaf59d46059f4aa3bfa4660f4cb1a`. +Compute it from sorted records `path`, NUL, content SHA-256, newline. This +record excludes itself to avoid a circular identity; later edits need a new +append-only verification entry. + +| Reviewed file | Content SHA-256 | +| --- | --- | +| `docs/contributing/glossary.md` | `f8cec460111439ba9ee65b0c1ecf8a60e466b576c4d94d75af222a9ced4ed02d` | +| `docs/contributing/oracle-coverage.md` | `598fa6e4c3874eaafeb48174d3f13afd8626bc56e558d804a422d5bce3c19ba8` | +| `packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts` | `8fafec444efbcc0a171d8beab0a166b252941e84841e02d90b954ebd80517c81` | +| `packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.html` | `4734264a6227519aeecdf8b3a7eaeb42f917daa8ce784d12baf8a57cb83d4c00` | +| `packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts` | `a71db0fffc376a9b15c2d4669100876d6656df2c448134a17b890fa32f686c2f` | +| `packages/browser-db-sqlite-persistence/playwright.opfs-electric.config.ts` | `d18ef1f5dbbdec19e7db5b5498f039910d60cf43b7da48b617313c6f2ea8c62a` | +| `packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts` | `c3108fc39f4c8d7de91567011de9c829fb8e16fdb2a03d20231aca0a61ec27db` | +| `packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts` | `2868ead1bc73508ecdc685b0590a5a8f487b01db19fc846f6236ec2c0f843cc3` | +| `packages/db-sqlite-persistence-core/src/persisted.ts` | `1a3bc05bbf10646c8d75fb062799c8335a181a8d3a5e91d61a9244c1a4f5027a` | +| `packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts` | `406e4058b85055fc7972c18786f9461856aa46c90932f6346dc6faccd1106a0c` | +| `packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts` | `2ff7303909f08896844b2343f425a26eb87988fc41f1c1137bcf646198c6f779` | +| `packages/db/src/sync-persistence.ts` | `7052e177e444acbe47b57dc5d5bf7f57aae339c7ef6b7308765ea56f81bd4251` | +| `packages/db/src/types.ts` | `152fd447c615a76398e0f8e4f0714957df7879a77cb51f89b4fef40f83bcc367` | +| `packages/db/tests/oracle-config.ts` | `ef56c55cd1e7c759d056ed4d9be85e9cd79d2cdfc9743454170562240c50c908` | +| `packages/db/tests/sync-persistence.test.ts` | `ac531d2397b3a1e39be707c0e9d0656d968fd88cbd863a42f32e6fd0e31d230b` | +| `packages/electric-db-collection/src/electric.ts` | `f8eda904b0c02439196660844d41bafcd754404dabc4d9959caa8659b6c364b8` | +| `packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts` | `26f430ae9359d989fe221c747fc9f3aeceecd1c9e4137e5cca38e5a8ab257a46` | + +## Independent multi-agent review (2026-10-29) + +Four fresh reviewers examined the unchanged 17-file snapshot identified above: +SQLite model/API, Browser grammar/checkpoints, real SDK/OPFS receiving path, and +policy/evidence scope. They read the oracle-review skill, rubric, glossary and +applicable ownership records. No shared source or test files were changed. +The following findings supersede the earlier self-review's unconditional +conformance statements for the named requirements. These are oracle gaps; +no additional production defect was established. + +### P2: the focused recovery dimension never reaches restart + +`multi-tab-electric-oracle.test.ts:1477` generates a recover step with pick 0 or +1 while the lagging tab is still running. The branch at lines 1323–1328 always +selects reload in that state, so both sampled values close the tab and open a +new Collection. The focused source-prefix campaign does not establish a +same-Collection restart after stale prefix application and later advancement. +Two reviewers independently found this by control-flow inspection. + +The broader generated/storm owners do exercise restart after stops. The real +receiver exercises an actual close/reopen. Those witnesses do not supply this +missing combination. The earlier pre-edit inventory, coverage entry and +ORC-004 self-review overstate the focused reload/restart dimension. Existing +reach counters cannot detect its collapse because none names the actual +recovery path. + +The smallest general repair is to stop the slow source before the restart +branch, retain a running reload branch, and declare reaches from the actual +`how` path for divergent-source restart and reload. The owner remains this +Browser oracle. Until that witness and comparison evidence exist, the focused +restart claim remains open. This finding concerns ORC-004, ORC-005, ORC-012 and +ORC-015, including AGENTS.md's unreachable-claims rule. + +### P2: replay runs unrelated campaigns + +The Browser random registrations at lines 1514 and 1592 use plain `it`. With +the documented `-t 'random or replayed'` selector, both run even when +`TANSTACK_DB_ORACLE_PROPERTY` selects only one. `oraclePropertyOptions` applies +the shrink path to the matching property; it reseeds the other property's +normal 80-history campaign. + +A reviewer executed source-prefix seed 2281, path `0:0:0:0`: the requested +single history passed, then the unrelated broad campaign reported 80 recovered +histories and 49 must-refetch histories. The command reported 2 passed and took +21.74 seconds. This can delay replay or introduce an unrelated failure. + +The SQLite source-progress registration at +`sqlite-resume-snapshot-oracle.test.ts:961` has the same isolation defect, +assessed P3 in that file. Selecting `sqlite-resume.startup-generation`, seed +1659, path `0`, while filtering for source-progress tests still ran its normal +campaign (reach counts 55/32/34/64/40). The intended source-progress replay +itself passed. Existing other unguarded core properties are not evidence that +this new registration is correct. + +Use `oracleReplayTest(it, property)` or `oracleReplaySkips(property)` to stand +aside during another property's replay. Verify both property directions and +an unrelated selector. This is the missing ORC-007 selection check, separate +from successful execution of each property's own seed/path. + +### Other review results and limits + +The SQLite reviewer independently reconstructed stale predecessor, sticky gap, +duplicate endpoint and authoritative replacement behavior. Its fixed/random +campaigns passed with typechecking, as did direct duplicate-endpoint replay. +Browser reviewers passed both focused normal campaigns and new checkpoint +regressions. They found no weakened final source-head judgment or unsupported +removal of the known #2209/#2267 exclusions. + +The receiver reviewer ran both disjoint/no-drain real cells: each passed and +reached all four declared states. An isolated temporary probe held the reopened +page's first actual HTTP response. The Collection stayed loading and `open()` +stayed pending while stale cached rows remained visible. Releasing the response +reached the shared source-head comparison and passed. The commit counter was +already 1 because setup commits reset metadata; the existing status gate, not +the counter alone, prevented premature comparison. No readiness finding was +established. A suspected `pg.end()` cleanup issue was rejected after inspecting +the installed resolve-only completion implementation and finding no concrete +supported failure path. + +The evidence reviewer confirmed the recorded manifest and semantic RED/mutant +results. It found no supported production-first, model-hacking, quiet-exclusion, +pinned-only or missing-mutant shortcut. Historical logs do not hash production +at every execution boundary; they do not contradict the preserved pre-edit +gate. The two verified issues above remain unresolved in this reviewed snapshot. +No production mutants were independently rerun by the reviewers; the receiver +review did not independently repeat the other eight real cells or cleanup fault +injection. The earlier executed evidence retains those distinct claims. + + +## Review repairs and verification (2026-10-29) + +The user authorized both review fixes. The two findings above are resolved in +this state. No production file changed during this pass; all five production +content hashes still match the preceding reviewed manifest. The recorded bundle +measurements and unchanged real SDK/OPFS receiving evidence still apply. + +### Law, reach and failed controls + +The convergence law and its authority remain unchanged: once a source actually +recovers and catches up, durable/public rows equal the source head. The grammar +now distinguishes two legal recovery paths after the forced prefix divergence. +Recovery 0 stops the lagging source, then restarts the same Collection. Recovery +1 closes the tab and opens a new Collection. The callback receives the driver's +actual `how` after stream startup and quiescence. The new reach states count +that branch only when the baseline was divergent, rather than inferring it +from the generated pick. + +Before changing the grammar, those actual-path counters rejected the old +campaign: `divergent source restart: 0`, `divergent source reload: 38`. The fixed +80-history run failed its reach assertion (`review-restart-reach-red.log`). +This is a RED for the oracle's missing reach, not a production row failure. +The conditional stop then made the restart branch reachable. Both fixed/random +80-history campaigns pass at the original source-head comparison. + +The earlier pre-edit inventory's restart claim was inaccurate, as the independent +review records. The new restart evidence was obtained after the original +production repair. This entry does not retroactively attribute it to that gate. +This pass changes test machinery only. The original broad grammar and all four +issue replay paths remain unchanged; the focused recovery recipe now includes +the missing stop. + +### Replay selection + +The Browser source-prefix and broad-history random registrations, and the +SQLite source-progress random registration, now use `oracleReplayTest` with +their property names. Normal campaigns still run. During path replay, these +registrations stand aside unless their property is selected; fixed campaigns +continue to stand aside for every path replay. + +Before the guards, selecting the one-history source-prefix replay also ran the +broad 80-history campaign. Selecting startup-generation while filtering for +source-progress still ran the latter's normal campaign. Those successful test +exits were failed selection controls; their extra campaign output is the +violation (`review-replay-before.log`, `review-core-replay-before.log`). + +After the guards, an integration probe checked five selectors with coverage and +typechecking disabled: + +| Selector | Observed registration behavior | +| --- | --- | +| Browser source-prefix, 2281, `0:0:0:0` | Exactly one requested replay passed; actual restart reach was 1; no broad campaign. | +| Browser broad-history, -1371495859, `75:0:0:0:0:0:0:1:1:0:0` | Exactly one requested replay passed; no source-prefix campaign. | +| An unrelated registered property with both Browser random tests selected | Both registrations skipped; no campaign ran. | +| SQLite startup-generation with source-progress tests selected | Both source-progress registrations skipped. | +| SQLite source-progress duplicate-endpoint replay | Exactly one requested replay passed; fixed campaign skipped. | + +The Browser probe used the documented `-t 'random or replayed'` selector in both +directions. The SQLite probe used `-t 'preserves source progress evidence'`. +These checks target the reviewed registrations; they do not claim to repair +pre-existing replay-selection behavior in unrelated SQLite tests. + +### Mutants and restored GREEN + +A temporary SQL mutant that never retains a source gap failed the new restart +replay at `after every tab caught up`, with a missing source row. The same mutant +failed both existing cursor-only reload replays at that checkpoint. A temporary +startup-provenance bypass failed the missing-token restart replay (seed 2281, +path `2:0:0`) at the same source-head comparison. All were assertion failures +with concrete wrong rows, and the source files were restored in `finally`. +Logs: `review-mutant-gap-restart.log`, `review-mutant-gap-reload.log`, and +`review-mutant-legacy-restart.log`. + +After restoration, both Browser important generated properties ran their +normal fixed/random 80-history campaigns, alongside both cursor-only reload +replays. The selected command passed all 9 reported tests with no type errors +(`review-restored-campaigns-green.log`). The full SQLite owner also passed +(61 reported tests, no type errors, `review-core-green.log`). The changed +registration wrapper changes Vitest's type-test discovery count; runtime +campaigns and run budgets were not removed or lowered. + +Final focused source-prefix reaches (fixed / random): + +| Declared state | Histories | +| --- | ---: | +| present/empty | 14 / 8 | +| present/disjoint | 11 / 17 | +| absent/empty | 6 / 8 | +| absent/disjoint | 7 / 9 | +| drained before recovery | 42 / 38 | +| missing progress at recovery | 26 / 30 | +| foreign progress at recovery | 21 / 27 | +| divergent source restart | 25 / 23 | +| divergent source reload | 13 / 19 | + +Both normal broad-history campaigns also met all existing declared reaches. +The full SQLite source-progress reaches were 61/52 discontinuities, 37/35 later +matching commits, 44/31 replacement repairs, 64/66 duplicates and 34/42 legacy +column migrations. `pnpm lint:changed` and `git diff --check` pass. + +The original Browser reviewer re-reviewed the delta and independently ran both +replay directions. Each reported one passed replay, with the other campaign +absent; its source-prefix replay measured a real same-Collection restart. +The original SQLite reviewer confirmed its guard repair by inspection. Neither +reviewer found a remaining supported issue in these repairs. They did not +independently repeat this pass's normal campaigns or mutants. + +For the applicable rubric changes: ORC-004/005 now have an executable restart +history and its actual-path observation; ORC-007 has isolated selection checks; +ORC-015 has failing reach counters and measured normal counts; ORC-012 is updated +by this entry. Existing laws, independent models, failure handling, shared +receiving judgment and law-only file structure remain unchanged. The earlier +ORC-001–003/006/008–011/013–014/016 evidence still applies. + +The scope remains bounded same-version eager sources. The added in-page restart +interaction is a controlled SDK/Node Browser witness; the real Chromium matrix +continues to supply tab close/reopen evidence. No broader real in-page restart, +mixed-version, on-demand, native-device or monotonic-live-write claim is added. +The separate #2209, #2267 and metadata gaps remain open. No commit or push was +made during these fixes. + +The updated uncommitted 17-file manifest identity is `f326ed14d9464e8ecc7d62c1da21d53589e2a49f31cfa3dec162949f1325e62b`, on the same +parent `82c5f999b32c9db827dd74e3f172604dfda28013`. It uses the sorted path/NUL/ +content-hash/newline algorithm above and excludes this record. Only these +manifest entries differ from the previously reviewed snapshot: + +| Changed file | Content SHA-256 | +| --- | --- | +| `docs/contributing/oracle-coverage.md` | `ef76490d320e037e9362261feb245089f1901a54ce209237946af998cf04a531` | +| `packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts` | `eee52b0972a4920c573b4eb304e84ef256ed32b886583695cd35544415b7b746` | +| `packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts` | `16781c8eb12ef6698cf412108d705dccb5e345223dc319a53e0528d5e83d521b` | + + +## Prep-pr verification on updated main (2026-10-29) + +The task branch fast-forwarded to `07bfc0eb2584fd83a3efc14ec1fb452954a829ea` +before fresh review. Upstream changed release metadata and unrelated paths; +none overlapped this task's edits. The same 17-file manifest, excluding this +record, remains `f326ed14d9464e8ecc7d62c1da21d53589e2a49f31cfa3dec162949f1325e62b`. +No production, oracle, history grammar, or comparison changed during this pass. +Earlier RED, mutant, scope and ORC audit entries retain their stated provenance. + +Fresh verification against this base: + +- Full Browser multi-tab owner: 312 passed and 2 expected failures; no type errors. +- Full SQLite resume-snapshot owner: 61 reported passed; no type errors. +- Electric resume-snapshot races: 49 passed; no type errors. +- Persistence capability validation: 8 passed. +- Real SDK/PostgreSQL/Chromium OPFS: all 14 passed, including all ten prefix cells. +- Browser `tsc --noEmit`, `pnpm lint:changed`, and `git diff --check`: passed. + +The owner commands above used `--maxWorkers=2`; the real command used +`--workers=2`. Commands set `pnpm_config_verify_deps_before_run=false` after +pnpm tried to reinstall for upstream release-specifier changes. The installed +external dependencies used here did not change; no override or lockfile edit +was made. Local logs are `notes/issue-2281/prep-{browser,core,electric,bridge,real}-green.log`. + +The focused Browser fixed/random reaches, in the earlier table order, were +14/11, 11/10, 6/15, 7/11, 42/33, 26/35, 21/19, 25/24 and 13/23. +SQLite fixed/random reaches were gaps61/53, later matching37/42, +replacement44/36, duplicates64/61 and migration34/39. Every real prefix cell +reached its four declared states once. No budget was lowered. + +Fresh simplification and five independent review angles found no confirmed +behavioral defect. Evaluation also held the eager baseline read after initial +resume selection, then committed a discontinuous predecessor through SQLite. +Both a row-changing and a metadata-only commit advanced the row version while +physical key evidence stayed consistent. After release, the existing generation +fence rejected resume, persisted a reset marker and prevented queued publication. +Both focused probes passed; their temporary fixture copies were removed. +This refutes the suggested startup race at that controlled cut; it does not +establish new monotonic live-write or arbitrary-scheduling guarantees. +The existing finite bounds and separate open gaps remain unchanged. + +No commit, push or PR has been made at this checkpoint. Optional readability +and documentation suggestions remain for the prep-pr user walkthrough. + + +### Accepted test-driver simplification (2026-10-29) + +The user approved flattening the URL-input conversion in the real receiving +driver. Its native string, URL and Request forms retain the same URL result. +Browser `tsc --noEmit` passes. The real lost-row/disjoint-advancement cell +passes and reaches all four declared premises; no production, model, history +grammar or comparison changed. The 17-file manifest excluding this record is +now `dfdcaa048fc45d73c2b5750856ce91e50f812137e41c724235e7e4889637adad`, +on the same base `07bfc0eb2584fd83a3efc14ec1fb452954a829ea`. +Logs: `prep-s1-tsc.log` and `prep-s1-real.log` in the local evidence directory. + +### Restart evidence chronology clarification (2026-10-29) + +The restart reach correction in “Review repairs and verification” concerns the +focused `browser.electric-source-prefix` campaign. Its recipe originally +performed tab reload for both recovery choices; actual restart reach was added +after the production repair. That chronology remains unchanged. + +The earlier [row-loss trace](#later-traces-rows-and-the-resume-position-describe-different-prefixes) +already exercised same-Collection restart on unchanged production. Seed +`-1660880085`, path `55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`, stopped both sources +after the stale-prefix and metadata-only advancement. The original driver's +final loop restarted both existing Collections. The local `loss-trace.log` +records those restarts at lines 101 and 110, followed by the missing-row +source-head mismatch at line 119. This supplies the earlier restart defect +witness; it does not retroactively supply the focused campaign's missing reach. diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts new file mode 100644 index 0000000000..bf05850289 --- /dev/null +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts @@ -0,0 +1,246 @@ +/** + * Real receiving path for the source-prefix grammar in the multi-tab owner. + * A real Electric HTTP response containing the first membership change waits + * in tab B while A applies a later change. B then commits that older response. + * A advances with an empty batch or disjoint change, and B reopens. PostgreSQL + * supplies the head to the shared judgment; no expected answer is restated. + * The drain branch is the neighboring control. These finite real-host cells + * receive the generated Node law; they do not cover arbitrary browser schedules. + */ +import { expect, test } from '@playwright/test' +import { makePgClient } from '../../db-collection-e2e/support/global-setup' +import { sourceHeadViolation } from '../tests/electric-convergence-oracle' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { declareOracle } from '../../db/tests/oracle-declaration' +import type { Page } from '@playwright/test' +import type {} from './electric-source-prefix.opfs' + +const cells = [true, false] + .flatMap((present) => + ([`empty`, `disjoint`] as const).flatMap((advance) => + [false, true].map((drain) => ({ + present, + advance, + drain, + legacy: false, + })), + ), + ) + .concat( + [true, false].map((present) => ({ + present, + advance: `disjoint` as const, + drain: false, + legacy: true, + })), + ) +const observe = (page: Page) => + page.evaluate(() => window.__electricSourcePrefix!.observe()) +const rowIds = async (page: Page) => + (await observe(page)).rows.map(({ id }) => id) + +declareOracle({ check: [sourceHeadViolation], grammar: [cells] }) +for (const cell of cells) { + test(`recovers ${cell.present ? `lost` : `resurrected`} rows after ${cell.advance} advancement${cell.drain ? ` with peer drain` : ``}${cell.legacy ? ` from legacy progress` : ``}`, async ({ + context, + }) => { + const reach = createOracleReach( + `browser.electric-source-prefix.opfs`, + [ + `held earlier SDK response`, + `stale commit applied`, + `later advancement applied`, + `reopened source compared`, + ], + { replaying: false }, + ) + reach.beginHistory() + const token = crypto.randomUUID().replaceAll(`-`, ``).slice(0, 12) + const table = `electric_prefix_${token}` + const collectionId = `prefix-${token}` + const parameters = new URLSearchParams({ + table, + collectionId, + databaseId: collectionId, + }) + const url = `/e2e/electric-source-prefix.opfs.html?${parameters}` + const pg = makePgClient() + const pages: Array = [] + const diagnostics: Array = [] + const close = async (page: Page) => { + if (page.isClosed()) return + diagnostics.push( + ...(await page.evaluate( + () => window.__electricSourcePrefix?.cleanup() ?? [], + )), + ) + await page.close() + } + let primary: unknown + let connected = false + try { + await pg.connect() + connected = true + await pg.query( + `CREATE TABLE public.${table} (id TEXT PRIMARY KEY, label TEXT NOT NULL)`, + ) + await pg.query(`INSERT INTO public.${table} VALUES ('anchor', 'anchor')`) + if (cell.present) + await pg.query(`INSERT INTO public.${table} VALUES ('target', 'old')`) + const open = async () => { + const page = await context.newPage() + pages.push(page) + page.on(`pageerror`, (error) => diagnostics.push(error.message)) + await page.goto(url) + await page.waitForFunction( + () => window.__electricSourcePrefix !== undefined, + ) + await expect + .poll(async () => (await observe(page)).status) + .toBe(`ready`) + await expect + .poll(async () => (await observe(page)).commits) + .toBeGreaterThan(0) + return page + } + const a = await open() + const b = await open() + await b.evaluate(() => window.__electricSourcePrefix!.arm()) + const bCommits = (await observe(b)).commits + // The first change reverses membership; the second restores it. A has + // already applied the first before the second is issued to PostgreSQL. + if (cell.present) + await pg.query(`DELETE FROM public.${table} WHERE id = 'target'`) + else + await pg.query(`INSERT INTO public.${table} VALUES ('target', 'old')`) + await expect + .poll(() => rowIds(a), { timeout: 30_000 }) + .toEqual(cell.present ? [`anchor`] : [`anchor`, `target`]) + // Empty long-poll responses can precede the row response. Deliver those + // without releasing the first actual changed-row response. + await expect + .poll( + async () => { + const state = await observe(b) + if (state.pending[0] === 0) + await b.evaluate(() => + window.__electricSourcePrefix!.releaseOne(), + ) + return (await observe(b)).pending[0] ?? 0 + }, + { timeout: 30_000 }, + ) + .toBeGreaterThan(0) + reach.reach(`held earlier SDK response`) + if (cell.present) + await pg.query(`INSERT INTO public.${table} VALUES ('target', 'new')`) + else await pg.query(`DELETE FROM public.${table} WHERE id = 'target'`) + await expect + .poll(() => rowIds(a), { timeout: 30_000 }) + .toEqual(cell.present ? [`anchor`, `target`] : [`anchor`]) + await b.evaluate(() => window.__electricSourcePrefix!.releaseOne()) + await expect + .poll(async () => (await observe(b)).commits) + .toBeGreaterThan(bCommits) + await expect + .poll(() => rowIds(a)) + .toEqual(cell.present ? [`anchor`] : [`anchor`, `target`]) + reach.reach(`stale commit applied`) + const beforeAdvance = await observe(a) + if (cell.advance === `disjoint`) { + await pg.query( + `UPDATE public.${table} SET label = 'advanced' WHERE id = 'anchor'`, + ) + await expect + .poll( + async () => + (await observe(a)).rows.find(({ id }) => id === `anchor`)?.label, + { timeout: 30_000 }, + ) + .toBe(`advanced`) + } else { + // Ask the real server for a non-live catch-up response on A's next + // SDK request. The current live request may first time out. + await a.evaluate(() => window.__electricSourcePrefix!.refresh()) + await expect + .poll(async () => (await observe(a)).emptyCommits, { + timeout: 70_000, + }) + .toBeGreaterThan(beforeAdvance.emptyCommits) + } + reach.reach(`later advancement applied`) + if (cell.drain) { + await expect + .poll(async () => (await observe(b)).pending.length, { + timeout: 30_000, + }) + .toBeGreaterThan(0) + await b.evaluate(() => window.__electricSourcePrefix!.releaseOne()) + await expect + .poll(() => rowIds(b), { timeout: 30_000 }) + .toEqual(cell.present ? [`anchor`, `target`] : [`anchor`]) + } + await close(b) + if (cell.legacy) + await a.evaluate(() => window.__electricSourcePrefix!.legacy()) + const reopened = await open() + // Source readiness follows an applied SDK receipt. The wrapper may + // still propagate its notice to A, so poll the shared settled judgment. + const source = await pg.query<{ id: string; label: string }>( + `SELECT id, label FROM public.${table} ORDER BY id`, + ) + reach.reach(`reopened source compared`) + await expect + .poll( + async () => { + const states = await Promise.all([observe(a), observe(reopened)]) + return sourceHeadViolation(source.rows, { + durableRows: states[1].durableRows, + tabs: states, + }) + }, + { timeout: 10_000 }, + ) + .toBeUndefined() + console.info( + `real prefix reach`, + cell, + Object.fromEntries(reach.counts()), + ) + reach.assertReached() + } catch (error) { + primary = error + for (const page of pages) { + if (!page.isClosed()) { + try { + diagnostics.push(JSON.stringify(await observe(page))) + } catch (diagnosticError) { + diagnostics.push(String(diagnosticError)) + } + } + } + } finally { + for (const page of [...pages].reverse()) { + try { + await close(page) + } catch (error) { + diagnostics.push(String(error)) + } + } + if (connected) { + try { + await pg.query(`DROP TABLE IF EXISTS public.${table}`) + } catch (error) { + diagnostics.push(String(error)) + } + await pg.end() + } + } + if (primary !== undefined) + throw new Error( + `real prefix comparison failed; diagnostics=${JSON.stringify(diagnostics)}`, + { cause: primary }, + ) + expect(diagnostics).toEqual([]) + }) +} diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.html b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.html new file mode 100644 index 0000000000..93b5b9475b --- /dev/null +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.html @@ -0,0 +1,10 @@ + + + + + Electric source prefix receiving driver + + + + + diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts new file mode 100644 index 0000000000..5f6584b924 --- /dev/null +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts @@ -0,0 +1,178 @@ +/** Real SDK/OPFS receiving driver for the shared source-head convergence law. + * HTTP responses are acquired from Electric before delivery is held. The gate + * never synthesizes messages or positions; it only delays a tab's next response. + */ +import { createCollection } from '@tanstack/db' +import { electricCollectionOptions } from '../../electric-db-collection/src/electric' +import { + BrowserCollectionCoordinator, + createBrowserWASQLitePersistence, + openBrowserWASQLiteOPFSDatabase, + persistedCollectionOptions, +} from '../src/index' + +type Item = { id: string; label: string } +const parameters = new URL(location.href).searchParams +const databaseId = parameters.get(`databaseId`)! +const collectionId = parameters.get(`collectionId`)! +const table = parameters.get(`table`)! +if (!databaseId || !collectionId || !table) + throw new Error(`Missing prefix driver parameters`) +let hold = false +let refresh = false +const pending: Array<{ release: () => void; rows: number }> = [] +const requests: Array<{ offset: string | null; handle: string | null }> = [] +let commits = 0 +let emptyCommits = 0 +let openWrites = 0 +const database = await openBrowserWASQLiteOPFSDatabase({ + databaseName: `${databaseId}.sqlite`, +}) +const coordinator = new BrowserCollectionCoordinator({ dbName: databaseId }) +const persistence = createBrowserWASQLitePersistence({ database, coordinator }) +const source = electricCollectionOptions({ + id: collectionId, + syncMode: `eager`, + getKey: (row) => row.id, + shapeOptions: { + url: `${location.origin}/electric/v1/shape`, + params: { table: `public.${table}` }, + liveSse: false, + fetchClient: async (input, init) => { + const url = new URL( + typeof input === `string` || input instanceof URL ? input : input.url, + ) + requests.push({ + offset: url.searchParams.get(`offset`), + handle: url.searchParams.get(`handle`), + }) + // A non-live catch-up request is answered by Electric with its actual + // up-to-date control, including when no row changed. + if (refresh) { + url.searchParams.delete(`live`) + refresh = false + } + const response = await fetch(url, init) + if (hold && response.ok) { + const messages = (await response.clone().json()) as Array<{ + headers?: { operation?: string } + }> + await new Promise((release) => + pending.push({ + release, + rows: messages.filter( + (message) => message.headers?.operation !== undefined, + ).length, + }), + ) + } + return response + }, + }, +}) +const sourceSync = source.sync +const options = persistedCollectionOptions({ + ...source, + sync: { + ...sourceSync, + sync: (params: Parameters[0]) => + sourceSync.sync({ + ...params, + begin: (...args) => { + openWrites = 0 + return params.begin(...args) + }, + write: (...args) => { + openWrites++ + return params.write(...args) + }, + commit: (...args) => { + const receipt = params.commit(...args) + const writes = openWrites + void Promise.resolve(receipt) + .then(() => { + commits++ + if (writes === 0) emptyCommits++ + }) + .catch(() => undefined) + return receipt + }, + }), + }, + persistence, +}) +const collection = createCollection(options) +const rows = (values: Iterable) => + Array.from(values, ({ id, label }) => ({ id, label })).sort((a, b) => + a.id.localeCompare(b.id), + ) +let cleaned = false +const probe = { + arm: () => { + hold = true + }, + refresh: () => { + refresh = true + }, + legacy: async () => { + // The old format's key ledger followed arrival-order writes even when + // its rows no longer matched the prefix named by the resume position. + await database.execute( + `DELETE FROM collection_metadata WHERE collection_id = ? AND key = ?`, + [collectionId, `electric:source-progress`], + ) + await database.execute( + `UPDATE collection_version SET source_progress_incompatible = 0 WHERE collection_id = ?`, + [collectionId], + ) + }, + releaseOne: () => { + const next = pending.shift() + if (!next) throw new Error(`No held response`) + next.release() + }, + observe: async () => ({ + status: collection.status, + rows: rows(collection.values()), + durableRows: rows( + ( + await options.persistence.adapter.loadResumeSnapshot({ + kind: `eager`, + collectionId, + }) + ).rows.map(({ value }) => value as Item), + ), + commits, + emptyCommits, + pending: pending.map(({ rows: count }) => count), + requests: [...requests], + error: collection.health.error, + }), + cleanup: async () => { + if (cleaned) return [] + cleaned = true + hold = false + const cleanup = collection.cleanup() + for (const entry of pending.splice(0)) entry.release() + const failures: Array = [] + for (const action of [ + () => cleanup, + () => coordinator.dispose(), + () => database.close?.(), + ]) { + try { + await action() + } catch (error) { + failures.push(String(error)) + } + } + return failures + }, +} +declare global { + interface Window { + __electricSourcePrefix?: typeof probe + } +} +window.__electricSourcePrefix = probe +void collection.preload().catch(() => undefined) diff --git a/packages/browser-db-sqlite-persistence/playwright.opfs-electric.config.ts b/packages/browser-db-sqlite-persistence/playwright.opfs-electric.config.ts index 1b3a300d8d..d5ff75a8cb 100644 --- a/packages/browser-db-sqlite-persistence/playwright.opfs-electric.config.ts +++ b/packages/browser-db-sqlite-persistence/playwright.opfs-electric.config.ts @@ -4,6 +4,7 @@ import opfsConfig from './playwright.opfs.config' export default defineConfig({ ...opfsConfig, testMatch: [ + `electric-source-prefix-oracle.opfs.spec.ts`, `electric-resume-two-tab.opfs.spec.ts`, `electric-hydration-straddle.opfs.spec.ts`, `electric-immediate-reload.opfs.spec.ts`, diff --git a/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts b/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts new file mode 100644 index 0000000000..6404f984b0 --- /dev/null +++ b/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts @@ -0,0 +1,49 @@ +/** + * Law 2 of the multi-tab Electric owner: after successful recovery and source + * catch-up, durable rows and every open eager Collection equal the source head. + * The authority is the persisted multi-tab eventual-consistency contract in + * oracle-coverage.md (#2209). Arrival-order regressions before this cut remain + * permitted. This judgment knows no cursor, coordinator, or storage algorithm. + * + * The Node driver supplies its independent server log; the Chromium/real SDK + * driver supplies PostgreSQL rows. Both use this same settled-row judgment. + * It does not judge intermediate publication, on-demand subsets, or optimism. + */ +import { declareOracle } from '../../db/tests/oracle-declaration' + +type SourceRow = { id: string | number; label: string } + +function canonicalRows(rows: ReadonlyArray): Array { + return rows + .map(({ id, label }) => ({ id, label })) + .sort((a, b) => String(a.id).localeCompare(String(b.id))) +} + +export function sourceHeadViolation( + sourceHead: ReadonlyArray, + observation: { + durableRows: ReadonlyArray + tabs: ReadonlyArray<{ status: string; rows: ReadonlyArray }> + }, +): string | undefined { + // Source-head equality is independent of how storage orders or repairs + // interleaved commits. Readiness is supplied by each tab's own source. + const expected = canonicalRows(sourceHead) + if ( + JSON.stringify(canonicalRows(observation.durableRows)) !== + JSON.stringify(expected) + ) { + return `durable rows differ from source head: ${JSON.stringify({ expected, actual: observation.durableRows })}` + } + for (const [index, tab] of observation.tabs.entries()) { + if ( + tab.status !== `ready` || + JSON.stringify(canonicalRows(tab.rows)) !== JSON.stringify(expected) + ) { + return `tab ${index} differs from ready source head: ${JSON.stringify({ expected, actual: tab })}` + } + } + return undefined +} + +declareOracle({ check: [sourceHeadViolation] }) diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index fcdad376c0..c1f6d8b561 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -6,7 +6,7 @@ * runs its own Electric stream, and all of them store through one SQLite * database and one elected leader. Tabs coordinate only through durable * commits and the notices those commits broadcast. This file is the Browser - * host owner for three laws about that arrangement. + * host owner for five laws about that arrangement. * * Law 1, reserved commits survive a refetch storm. Authority: * SyncPersistenceCapabilityV1 `reserveCommitTurn` (packages/db/src/types.ts), @@ -103,6 +103,7 @@ import { declareOracle } from '../../db/tests/oracle-declaration' import { oraclePropertyOptions, oracleReplayRequested, + oracleReplayTest, oracleRuns, } from '../../db/tests/oracle-config' import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' @@ -110,6 +111,7 @@ import { electricCollectionOptions } from '../../electric-db-collection/src/elec import { BrowserCollectionCoordinator } from '../src/browser-coordinator' import { createBrowserWASQLitePersistence } from '../src/browser-persistence' import { createWASQLiteTestDatabase } from './helpers/wa-sqlite-test-db' +import { sourceHeadViolation } from './electric-convergence-oracle' import type { Message, Row, @@ -214,6 +216,19 @@ const update = (row: Item): Message => ({ const nextTask = () => new Promise((resolve) => setTimeout(resolve, 0)) +// Attempt every cleanup, preserving failures after all resources are released. +async function closeAll(actions: Array<() => unknown>) { + const failures: Array = [] + for (const action of actions) { + try { + await action() + } catch (error) { + failures.push(error) + } + } + if (failures.length) throw new AggregateError(failures, `host cleanup failed`) +} + // --------------------------------------------------------------------------- // Browser host: one database file, one coordinator and persistence per tab. // --------------------------------------------------------------------------- @@ -303,9 +318,11 @@ function openHost() { const close = async () => { if (!open) return open = false - await collection.cleanup().catch(() => undefined) - coordinator.dispose() - await database.close?.() + await closeAll([ + () => collection.cleanup(), + () => coordinator.dispose(), + () => database.close?.(), + ]) } closers.push(close) const stream = () => { @@ -386,6 +403,34 @@ function openHost() { openTab, quiesce, commitNotices: () => commitNotices, + // A pre-protocol cache or mismatched provenance record still has rows + // and an Electric cursor. Reopen must not trust that cursor blindly. + rewriteSourceProgress: async (format: `missing` | `foreign`) => { + observer ??= createWASQLiteTestDatabase({ filename, timeout: 1000 }) + // The old format's key ledger followed arrival-order row writes and + // therefore certified these wrong rows. Reconstruct that baseline; + // merely deleting a token from an already incompatible cache would + // not distinguish the startup provenance check. + await observer.execute( + `UPDATE collection_version SET source_progress_incompatible = 0 WHERE collection_id = ?`, + [`items`], + ) + if (format === `missing`) { + await observer.execute( + `DELETE FROM collection_metadata WHERE collection_id = ? AND key = ?`, + [`items`, `electric:source-progress`], + ) + } else { + await observer.execute( + `UPDATE collection_metadata SET value = ? WHERE collection_id = ? AND key = ?`, + [ + JSON.stringify(`foreign-position`), + `items`, + `electric:source-progress`, + ], + ) + } + }, // A peer tab's collection reset, posted on the tabs' coordinator channel // as a peer would post it. No shipped host sends this notice yet, so // the premise is controlled; the tabs, their coordinators, and Electric @@ -426,10 +471,12 @@ function openHost() { .sort((left, right) => left.id - right.id) }, close: async () => { - await observer?.close?.() - for (const close of closers.reverse()) await close() - channel.close() - rmSync(directory, { recursive: true, force: true }) + await closeAll([ + () => observer?.close?.(), + ...closers.reverse(), + () => channel.close(), + () => rmSync(directory, { recursive: true, force: true }), + ]) }, } } @@ -505,11 +552,28 @@ const afterDeploy: Array = [ describe(`multi-tab Electric over one persisted database`, () => { let host: Host | undefined - afterEach(async () => { - await host?.close() - host = undefined - streams.length = 0 - }) + // Shrinking must retain the violated law even when releasing the host fails. + // AggregateError keeps the primary mismatch as cause and cleanup separately. + const finishHistory = async (primary?: { error: unknown }) => { + const failures: Array = [] + if (primary) failures.push(primary.error) + try { + await host?.close() + } catch (error) { + failures.push(error) + } finally { + host = undefined + streams.length = 0 + } + if (primary && failures.length > 1) + throw new AggregateError( + failures, + `${String(primary.error)}; secondary host cleanup failure`, + { cause: primary.error }, + ) + if (failures.length) throw failures[0] + } + afterEach(() => finishHistory()) // Law 1 grammar. Dimensions: two or three tabs; eager or on-demand // Collections; which tab finishes its refetch first; whether every tab's @@ -948,6 +1012,11 @@ describe(`multi-tab Electric over one persisted database`, () => { const runGeneratedHistory = async ( { tabCount, steps }: GeneratedHistory, onKnownGap: () => void, + onRecovery?: ( + divergentBeforeRecovery: boolean, + how: `restart` | `reload`, + ) => void, + progressFormat: `current` | `missing` | `foreign` = `current`, ) => { host = openHost() const activeHost = host @@ -1080,6 +1149,12 @@ describe(`multi-tab Electric over one persisted database`, () => { await activeHost.quiesce(`tab ${tab.tab} delivered`) } const recover = async (index: number, how: `restart` | `reload`) => { + if (progressFormat !== `current`) { + await activeHost.rewriteSourceProgress(progressFormat) + progressFormat = `current` + } + const divergentBeforeRecovery = + JSON.stringify(rowsOf(durable)) !== JSON.stringify(rowsOf(log.rows)) const stopped = tabs[index]! const streamsBefore = stopped.streamCount() let recovered = stopped @@ -1105,6 +1180,7 @@ describe(`multi-tab Electric over one persisted database`, () => { ...requested, }) await activeHost.quiesce(`tab ${recovered.tab} recovered`) + onRecovery?.(divergentBeforeRecovery, how) } const check = async (label: string, head?: Array) => { @@ -1324,9 +1400,138 @@ describe(`multi-tab Electric over one persisted database`, () => { await recover(index, `restart`) } await deliver(index, `all`, true) + // A reset-abandoned transaction fails only when delivery tries to + // commit it. Judge Law 5 at that cut before the restart establishes + // Law 2's successful catch-up premise. + await check(`after final delivery to tab ${tabs[index]!.tab}`) + if (models[index]!.failedByReset) { + await recover(index, `restart`) + await deliver(index, `all`, true) + await check(`after reset recovery of tab ${tabs[index]!.tab}`) + } } await check(`after every tab caught up`, rowsOf(log.rows)) + if (models.every((model) => !model.closed)) { + expect( + sourceHeadViolation(rowsOf(log.rows), { + durableRows: await activeHost.durableRows(), + tabs: tabs.map((tab) => ({ + status: tab.collection.status, + rows: tab.publicRows(), + })), + }), + ).toBeUndefined() + } + } + + // Construct the missing interaction instead of waiting for an unconstrained + // shuffle to draw it. One stream commits an even number of membership + // changes, its peer commits only the first, then the faster stream commits + // either no rows or a disjoint update. Reopening the lagging peer must still + // reach the source head. Swapping tabs, both initial memberships, repeated + // toggles, and draining before recovery distinguish nearby legal histories. + // Recovery 0 stops the source and restarts the same Collection; recovery 1 + // reloads the tab, closing its Collection and opening a new one. The reach + // callback records the driver's actual recovery branch, not the input pick. + // Removing the lagging commit or the later advancement loses the defect; + // draining is the neighboring non-divergent control. A prefix longer than + // the source log is excluded by construction. + const sourcePrefixHistory = fc.record({ + key: fc.constantFrom(1, 3), + pairs: fc.integer({ min: 1, max: 3 }), + fastTab: fc.integer({ min: 0, max: 1 }), + advancingBatch: fc.constantFrom(`empty` as const, `disjoint` as const), + drain: fc.boolean(), + recovery: fc.integer({ min: 0, max: 1 }), + progressFormat: fc.constantFrom( + `current` as const, + `missing` as const, + `foreign` as const, + ), + }) + const runSourcePrefixCampaign = async ( + parameters: ReturnType, + ) => { + const states = [ + `present/empty`, + `present/disjoint`, + `absent/empty`, + `absent/disjoint`, + `drained before recovery`, + `missing progress at recovery`, + `foreign progress at recovery`, + `divergent source restart`, + `divergent source reload`, + ] as const + const reach = createOracleReach(`browser.electric-source-prefix`, states) + await fc.assert( + fc.asyncProperty(sourcePrefixHistory, async (history) => { + reach.beginHistory() + const slowTab = 1 - history.fastTab + const steps: Array = [ + ...Array.from({ length: history.pairs * 2 }, (): HistoryStep => ({ + kind: `server change`, + key: history.key, + pick: 0, + })), + { kind: `catch up`, tab: history.fastTab, pick: 0 }, + { kind: `catch up`, tab: slowTab, pick: 1 }, + ...(history.advancingBatch === `disjoint` + ? [{ kind: `server change` as const, key: 2, pick: 1 }] + : []), + { kind: `catch up`, tab: history.fastTab, pick: 0 }, + ...(history.drain + ? [{ kind: `catch up` as const, tab: slowTab, pick: 0 }] + : []), + ...(history.recovery === 0 + ? [{ kind: `stop` as const, tab: slowTab, pick: 0 }] + : []), + { kind: `recover`, tab: slowTab, pick: history.recovery }, + ] + let primary: { error: unknown } | undefined + try { + await runGeneratedHistory( + { tabCount: 2, steps }, + () => fc.pre(false), + (divergent, how) => { + if (history.drain && !divergent) + reach.reach(`drained before recovery`) + if (!history.drain && divergent) { + reach.reach(`divergent source ${how}`) + reach.reach( + `${history.key === 1 ? `present` : `absent`}/${history.advancingBatch}`, + ) + } + if (history.progressFormat !== `current`) { + reach.reach(`${history.progressFormat} progress at recovery`) + } + }, + history.progressFormat, + ) + } catch (error) { + primary = { error } + } finally { + await finishHistory(primary) + } + }), + parameters, + ) + console.info(`source prefix reach`, Object.fromEntries(reach.counts())) + reach.assertReached() } + it.skipIf(oracleReplayRequested())( + `converges after interleaved source prefixes (fixed seed)`, + () => runSourcePrefixCampaign({ numRuns: oracleRuns(80), seed: 2281 }), + 600_000, + ) + oracleReplayTest(it, `browser.electric-source-prefix`)( + `converges after interleaved source prefixes (random or replayed)`, + () => + runSourcePrefixCampaign( + oraclePropertyOptions(80, `browser.electric-source-prefix`), + ), + 600_000, + ) type GeneratedReach = | `must-refetch delivered` @@ -1345,7 +1550,7 @@ describe(`multi-tab Electric over one persisted database`, () => { // rows. The storm cells and the generated-history arbitrary are the grammar. declareOracle({ check: [runStormHistory, runGeneratedHistory], - grammar: [stormHistories, generatedHistory], + grammar: [stormHistories, generatedHistory, sourcePrefixHistory], }) const createGeneratedReach = () => @@ -1370,13 +1575,14 @@ describe(`multi-tab Electric over one persisted database`, () => { generatedReach = createGeneratedReach() await fc.assert( fc.asyncProperty(generatedHistory, async (history) => { + let primary: { error: unknown } | undefined try { // fc.pre discards a history that reaches the known gap. await runGeneratedHistory(history, () => fc.pre(false)) + } catch (error) { + primary = { error } } finally { - await host?.close() - host = undefined - streams.length = 0 + await finishHistory(primary) } }), parameters, @@ -1396,7 +1602,7 @@ describe(`multi-tab Electric over one persisted database`, () => { }), 600_000, ) - it( + oracleReplayTest(it, GENERATED_PROPERTY)( `keeps reserved commits lawful across generated multi-tab histories (random or replayed)`, () => runGeneratedCampaign( @@ -1430,6 +1636,88 @@ describe(`multi-tab Electric over one persisted database`, () => { 60_000, ) + // The final delivery can be the first attempt to commit a transaction + // abandoned by the reset. The driver must observe its failure, recover, + // and only then judge convergence. This replays the original issue trace. + it( + `recovers a reset-abandoned source that fails during final catch-up`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `server change`, key: 1, pick: 0 }, + { kind: `advance`, tab: 1, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `advance`, tab: 0, pick: 0 }, + { kind: `peer reset` }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `deploy` }, + { kind: `deploy` }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) + + // The later no-deploy reset trace reaches the same final-delivery failure + // from a reloaded tab. Recovery must happen after that failed commit too. + // Seed -1371495859, path 75:0:0:0:0:0:0:1:1:0:0. + it( + `recovers a reloaded source whose final commit was abandoned by a peer reset`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `recover`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `advance`, tab: 0, pick: 0 }, + { kind: `advance`, tab: 1, pick: 0 }, + { kind: `advance`, tab: 0, pick: 0 }, + { kind: `peer reset` }, + { kind: `advance`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) + + // A discontinuous source prefix does not corrupt the cached key set. Keep + // durable rows visible during reload while a fresh source snapshot repairs + // continuation. This is a reduced random history, replayed by the same law. + it( + `keeps cached rows visible while recovering a discontinuous source`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `server change`, key: 1, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `recover`, tab: 0, pick: 0 }, + { kind: `recover`, tab: 0, pick: 0 }, + { kind: `stop`, tab: 0, pick: 0 }, + { kind: `peer reset` }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `advance`, tab: 0, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) + // Known gap #2209, pinned: tab A commits v1 and v2; tab B's lagging stream // then commits only v1, so the durable row falls back to v1, and B closes // for good. Nothing redelivers the head to A, so Law 2's final check fails. @@ -1456,14 +1744,14 @@ describe(`multi-tab Electric over one persisted database`, () => { 60_000, ) - // Known gap, pinned from the random campaign (seed -1660880085, path + // Regression from the random campaign (seed -1660880085, path // 55:0:1:4:4:4:1:1:1:1:3:4:4:4:4): row 1 reaches v2 on the server and both // tabs catch up, but after tab B stops, tab A catches up, and tab A stops, // the durable rows lose row 1. Law 2 expects every open tab and the durable - // rows to show the head. This reproduces on main and is tracked with #2281. + // rows to show the head. The source-progress repair forces a fresh snapshot. // It replays through the same driver and checks with the known-gap detector - // off. Remove `fails` when the durable row survives the stops. - it.fails( + // off. The generated source-prefix campaign extends this same law. + it( `keeps a durable row after both tabs catch up to it and then stop`, () => runGeneratedHistory( @@ -1485,4 +1773,71 @@ describe(`multi-tab Electric over one persisted database`, () => { ), 60_000, ) + + // Issue #2281 reduces to the same six-step history for both membership + // directions. A stores two server changes; B stores only the first. A's + // next up-to-date stores its newer cursor without restoring the second + // change. B then reopens from that cursor and never receives the missing + // change. Law 2 still requires the server head at the final checkpoint. + // These regressions replay the same model and comparison as the generated + // source-prefix campaign above. + for (const { key, behavior } of [ + { key: 1, behavior: `keeps a reinserted row` }, + { key: 3, behavior: `keeps a deleted row absent` }, + ]) { + const stalePrefix: Array = [ + { kind: `server change`, key, pick: 0 }, + { kind: `server change`, key, pick: 0 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + ] + const publishCursor: HistoryStep = { kind: `catch up`, tab: 0, pick: 0 } + const reopenPeer: HistoryStep = { kind: `recover`, tab: 1, pick: 0 } + + // Keeping only the maximum cursor does not repair these rows. + it( + `${behavior} after a cursor-only commit and a lagging tab reload`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [...stalePrefix, publishCursor, reopenPeer], + }, + () => undefined, + ), + 60_000, + ) + + // A peer that drains its backlog supplies the missing change itself. + it( + `${behavior} when the lagging stream drains before reloading`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + ...stalePrefix, + publishCursor, + { kind: `catch up`, tab: 1, pick: 0 }, + reopenPeer, + ], + }, + () => undefined, + ), + 60_000, + ) + + // Without A's cursor-only commit, B resumes from its older position and + // receives the missing change. This distinguishes cursor advancement + // from reload or lag alone as the cause of the permanent mismatch. + it( + `${behavior} when the lagging tab reloads before the cursor advances`, + () => + runGeneratedHistory( + { tabCount: 2, steps: [...stalePrefix, reopenPeer] }, + () => undefined, + ), + 60_000, + ) + } }) diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 666fbf6a1a..666513c54a 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -479,6 +479,7 @@ export type PersistedTx< /** Claim authorizing this transaction's persisted cache generation. */ cacheGenerationClaimId?: string truncate?: boolean + sourceProgress?: { key: string; previous: string | undefined } mutations: Array< | { type: `insert` @@ -516,6 +517,8 @@ export const SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY = Symbol.for( ) export interface PersistenceAdapter { + /** Implements PersistedTx.sourceProgress under the durable writer lock. */ + readonly supportsSourceProgress?: boolean loadSubset: ( target: PersistedStorageTarget, options: LoadSubsetOptions, @@ -544,6 +547,7 @@ export interface PersistenceAdapter { metadata?: unknown }> keySet?: PersistedKeySetEvidence + sourceProgressCompatible?: boolean collectionMetadata: Array<{ key: string; value: unknown }> latestTerm: number latestSeq: number @@ -1304,6 +1308,7 @@ type BufferedSyncTransaction = { lifecycleGeneration: number admittedStorageCollectionId?: string expectedResumeGenerationOwner?: symbol + sourceProgress?: PersistedTx[`sourceProgress`] signal?: AbortSignal prependHydrationRows: ( rows: Array<{ key: TKey; value: T; metadata?: unknown }>, @@ -1668,6 +1673,7 @@ class PersistedCollectionRuntime< private resumeBaselinePromise: Promise | null = null private resumeCertificationPromise: Promise | null = null private persistedKeySetEvidence: PersistedKeySetEvidence | undefined + private sourceProgressCompatible: boolean | undefined private persistedResumeGeneration: PersistedResumeGeneration | undefined private resumeGenerationOwner = Symbol(`persisted resume generation owner`) private lifecycleGeneration = 0 @@ -2631,6 +2637,14 @@ class PersistedCollectionRuntime< return this.persistedKeySetEvidence } + supportsSourceProgress(): boolean { + return this.persistence.adapter.supportsSourceProgress === true + } + + getSourceProgressCompatible(): boolean | undefined { + return this.sourceProgressCompatible + } + hasManagedCacheGeneration(): boolean { return this.cacheGenerationClaim !== undefined } @@ -2913,6 +2927,7 @@ class PersistedCollectionRuntime< if (storageCollectionId !== this.storageCollectionId) return false this.persistedResumeGeneration = this.getResumeSnapshotGeneration(snapshot) this.persistedKeySetEvidence = snapshot.keySet + this.sourceProgressCompatible = snapshot.sourceProgressCompatible this.observeStreamPosition( snapshot.latestTerm, snapshot.latestSeq, @@ -3620,6 +3635,7 @@ class PersistedCollectionRuntime< this.terminalFailure = undefined this.resumeCertificationPromise = null this.persistedKeySetEvidence = undefined + this.sourceProgressCompatible = undefined this.persistedResumeGeneration = undefined this.resumeGenerationOwner = Symbol(`persisted resume generation owner`) } @@ -4705,6 +4721,7 @@ class PersistedCollectionRuntime< rowVersion: streamPosition.rowVersion, cacheGenerationClaimId: this.cacheGenerationClaim?.claimId, truncate: transaction.truncate, + sourceProgress: transaction.sourceProgress, mutations, rowMetadataMutations: toMetadataMutations(transaction.rowMetadataWrites), collectionMetadataMutations: toMetadataMutations( @@ -6729,6 +6746,24 @@ function createWrappedSyncConfig< openTransaction.expectedResumeGenerationOwner = runtime.getResumeGenerationOwner() }, + ...(runtime.supportsSourceProgress() + ? { + expectSourceProgress: ( + progress: NonNullable, + ) => { + if (startupState.cleanedUp) return + const openTransaction = getOpenTransaction() + if (!openTransaction) { + throw new InvalidPersistedCollectionConfigError( + `resumeSnapshot.expectSourceProgress must be called within an open sync transaction`, + ) + } + openTransaction.sourceProgress = progress + }, + getSourceProgressCompatible: () => + runtime.getSourceProgressCompatible(), + } + : {}), }, } @@ -7083,6 +7118,7 @@ function createWrappedSyncConfig< lifecycleGeneration: openTransaction.lifecycleGeneration, expectedResumeGenerationOwner: openTransaction.expectedResumeGenerationOwner, + sourceProgress: openTransaction.sourceProgress, signal, prependHydrationRows: ( rows: Array<{ key: TKey; value: T; metadata?: unknown }>, @@ -7263,6 +7299,22 @@ function createWrappedSyncConfig< if (active()) metadata.persistence.resumeSnapshot.expectCurrentCommit() }, + ...(metadata.persistence.resumeSnapshot.expectSourceProgress + ? { + expectSourceProgress: ( + progress: NonNullable, + ) => { + if (active()) + metadata.persistence.resumeSnapshot.expectSourceProgress?.( + progress, + ) + }, + getSourceProgressCompatible: () => + active() + ? metadata.persistence.resumeSnapshot.getSourceProgressCompatible?.() + : undefined, + } + : {}), }, }, }, diff --git a/packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts b/packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts index 61ef52fcdb..9a624ddd1b 100644 --- a/packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts +++ b/packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts @@ -1653,6 +1653,7 @@ function buildIndexName(collectionId: string, signature: string): string { } export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { + readonly supportsSourceProgress = true private readonly driver: SQLiteDriver private readonly schedulingIdentitySource: SQLiteDriver private scheduler: SharedPersistenceScheduler | undefined @@ -2346,6 +2347,7 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { metadata?: unknown }> keySet: PersistedKeySetEvidence + sourceProgressCompatible: boolean collectionMetadata: Array<{ key: string; value: unknown }> latestTerm: number latestSeq: number @@ -2393,10 +2395,8 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { const rows = includeRows ? await this.loadSubsetInternal(tableMapping, {}, transactionDriver) : [] - const { latestRowVersion, keySet } = await this.readKeySetEvidence( - collectionId, - transactionDriver, - ) + const { latestRowVersion, keySet, sourceProgressCompatible } = + await this.readResumeEvidence(collectionId, transactionDriver) const collectionMetadataRows = await transactionDriver.query<{ key: string value: string @@ -2425,6 +2425,7 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { metadata: row.metadata, })), keySet, + sourceProgressCompatible, collectionMetadata: collectionMetadataRows.map((row) => ({ key: row.key, value: deserializePersistedRowValue(row.value), @@ -2599,6 +2600,33 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { const currentRowVersion = version.latest_row_version const nextRowVersion = Math.max(currentRowVersion + 1, tx.rowVersion) const replacesPersistedBaseline = tx.truncate === true + const collectionMetadataMutations = lastMetadataByKey( + tx.collectionMetadataMutations ?? [], + (mutation) => mutation.key, + ) + // A source may commit from a different prefix than the shared baseline. + // Keep accepting its rows, but never certify the resulting baseline for a + // resume past changes that this composition can have lost. Only a full + // replacement can establish complete evidence again. + let sourceProgressDiscontinuous = false + if (tx.sourceProgress && !replacesPersistedBaseline) { + const progress = await transactionDriver.query<{ value: string }>( + `SELECT value FROM collection_metadata WHERE collection_id = ? AND key = ?`, + [collectionId, tx.sourceProgress.key], + ) + const next = collectionMetadataMutations.find( + (mutation) => mutation.key === tx.sourceProgress!.key, + ) + sourceProgressDiscontinuous = + progress[0]?.value !== + (tx.sourceProgress.previous === undefined + ? undefined + : serializePersistedRowValue(tx.sourceProgress.previous)) && + // Delivering a complete source position twice is idempotent. It + // neither breaks a coherent baseline nor repairs a previous gap. + (next?.type !== `set` || + progress[0]?.value !== serializePersistedRowValue(next.value)) + } const tracksPersistedKeySet = version.key_set_evidence_available === 1 || replacesPersistedBaseline const replayDelta: ReplayableTxDelta | null = replacesPersistedBaseline @@ -2816,10 +2844,7 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { } for (const batch of batches( - lastMetadataByKey( - tx.collectionMetadataMutations ?? [], - (mutation) => mutation.key, - ), + collectionMetadataMutations, this.replacementBatchSize, )) { const deletes = batch.filter((mutation) => mutation.type === `delete`) @@ -2857,12 +2882,19 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { key_set_evidence_incompatible = CASE WHEN ? = 1 THEN 0 ELSE key_set_evidence_incompatible + END, + source_progress_incompatible = CASE + WHEN ? = 1 THEN 0 + WHEN ? = 1 THEN 1 + ELSE source_progress_incompatible END WHERE collection_id = ?`, [ nextRowVersion, replacesPersistedBaseline ? 1 : 0, replacesPersistedBaseline ? 1 : 0, + replacesPersistedBaseline ? 1 : 0, + sourceProgressDiscontinuous ? 1 : 0, collectionId, ], ) @@ -3383,22 +3415,25 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { } } - private async readKeySetEvidence( + private async readResumeEvidence( collectionId: string, driver: SQLiteDriver, ): Promise<{ latestRowVersion: number keySet: PersistedKeySetEvidence + sourceProgressCompatible: boolean }> { const versionRows = await driver.query<{ latest_row_version: number key_set_evidence_available: number key_set_incompatible: number + source_progress_incompatible: number }>( `SELECT latest_row_version, key_set_evidence_available, - key_set_evidence_incompatible AS key_set_incompatible + key_set_evidence_incompatible AS key_set_incompatible, + source_progress_incompatible FROM collection_version WHERE collection_id = ? LIMIT 1`, @@ -3408,6 +3443,7 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { return { latestRowVersion: version?.latest_row_version ?? 0, + sourceProgressCompatible: version?.source_progress_incompatible === 0, keySet: { status: version?.key_set_evidence_available !== 1 @@ -4179,7 +4215,8 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { ON CONFLICT(collection_id) DO UPDATE SET latest_row_version = 0, key_set_evidence_available = 1, - key_set_evidence_incompatible = 0`, + key_set_evidence_incompatible = 0, + source_progress_incompatible = 0`, [collectionId], ) await transactionDriver.run( @@ -4230,7 +4267,8 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { AND EXISTS (SELECT 1 FROM sqlite_master WHERE name = 'collection_version' AND sql LIKE '%key_set_evidence_available%' - AND sql LIKE '%key_set_evidence_incompatible%') + AND sql LIKE '%key_set_evidence_incompatible%' + AND sql LIKE '%source_progress_incompatible%') AS complete`, ) return rows[0]?.complete === 1 @@ -4343,17 +4381,19 @@ export class SQLiteCorePersistenceAdapter implements PersistenceAdapter { collection_id TEXT PRIMARY KEY, latest_row_version INTEGER NOT NULL, key_set_evidence_available INTEGER NOT NULL DEFAULT 0, - key_set_evidence_incompatible INTEGER NOT NULL DEFAULT 0 + key_set_evidence_incompatible INTEGER NOT NULL DEFAULT 0, + source_progress_incompatible INTEGER NOT NULL DEFAULT 0 )`, ) const collectionVersionColumns = await this.driver.query<{ name: string }>( `PRAGMA table_info(collection_version)`, ) - const keyEvidenceColumns = [ + const versionEvidenceColumns = [ `key_set_evidence_available`, `key_set_evidence_incompatible`, + `source_progress_incompatible`, ] as const - for (const columnName of keyEvidenceColumns) { + for (const columnName of versionEvidenceColumns) { if (collectionVersionColumns.some(({ name }) => name === columnName)) { continue } diff --git a/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts index 56884b1d2d..714cb60dd9 100644 --- a/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts @@ -2,8 +2,13 @@ import { DatabaseSync } from 'node:sqlite' import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { createCollection } from '@tanstack/db' -import { oraclePropertyOptions, oracleRuns } from '../../db/tests/oracle-config' +import { + oraclePropertyOptions, + oracleReplayRequested, + oracleReplayTest, oracleRuns +} from '../../db/tests/oracle-config' import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' import { SQLiteCorePersistenceAdapter, createPersistedTableName, @@ -799,14 +804,183 @@ describe(`SQLite resume snapshots`, () => { ) }) + // A resume token may continue only the baseline that established its + // previous token. This is the storage obligation of the optional source + // progress protocol: a discontinuity invalidates resume evidence until an + // authoritative replacement. It does not reject committed rows or order + // opaque tokens. The independent model judges the edges since the last + // replacement; ordinary metadata commits cannot erase a missing edge. + type ProgressStep = { + kind: `advance` | `stale` | `empty` | `snapshot` | `duplicate` + pick: number + } + const progressSteps = fc.array( + fc.record({ + kind: fc.constantFrom( + `advance`, + `stale`, + `empty`, + `snapshot`, + `duplicate`, + ), + pick: fc.nat(3), + }), + { minLength: 3, maxLength: 16 }, + ) + const progressHistory = fc.record({ + legacySchema: fc.boolean(), + steps: progressSteps, + }) + const runProgressCampaign = async ( + parameters: ReturnType, + ) => { + const reach = createOracleReach(`sqlite-resume.source-progress`, [ + `discontinuous commit`, + `matching commit after discontinuity`, + `replacement after discontinuity`, + `duplicate source position`, + `legacy source progress column added`, + ]) + await fc.assert( + fc.asyncProperty( + progressHistory, + async ({ legacySchema, steps: history }) => { + reach.beginHistory() + const database = new DatabaseSync(`:memory:`) + let primary: unknown + try { + let adapter = new SQLiteCorePersistenceAdapter({ + driver: createDriver(database), + }) + const target = { + kind: `eager` as const, + collectionId: `source-progress`, + } + if (legacySchema) { + await adapter.loadResumeSnapshot(target) + database.exec( + `ALTER TABLE collection_version DROP COLUMN source_progress_incompatible`, + ) + adapter = new SQLiteCorePersistenceAdapter({ + driver: createDriver(database), + }) + await adapter.loadResumeSnapshot(target) + reach.reach(`legacy source progress column added`) + } + const tokens = [`initial`] + let current = `initial` + // Model-only path: each edge must begin where its predecessor ended. + let edges: Array<{ from: string; to: string }> = [] + const coherent = () => + edges.every( + (edge, index) => + index === 0 || edge.from === edges[index - 1]!.to, + ) + const steps: Array = [ + { kind: `snapshot`, pick: 0 }, + ...history, + ] + for (const [index, step] of steps.entries()) { + const wasCoherent = coherent() + const previous = + step.kind === `stale` || step.kind === `duplicate` + ? tokens[step.pick % tokens.length]! + : current + const next = + step.kind === `empty` || step.kind === `duplicate` + ? current + : `position-${index}` + if (step.kind === `duplicate`) + reach.reach(`duplicate source position`) + if (step.kind === `snapshot`) { + if (!wasCoherent) reach.reach(`replacement after discontinuity`) + edges = [{ from: previous, to: next }] + } else { + if (previous !== current && next !== current) + reach.reach(`discontinuous commit`) + if (!wasCoherent && previous === current) + reach.reach(`matching commit after discontinuity`) + // Re-delivering the same source position is idempotent; it + // contributes no new edge and cannot repair an earlier gap. + if (next !== current) edges.push({ from: previous, to: next }) + } + // The baseline adapter ignores this proposed optional field. Keeping + // it on a structurally assignable transaction lets the pre-edit RED + // reach the semantic comparison instead of failing type/setup. + const tx = { + txId: `progress-${index}`, + term: 1, + seq: index + 1, + rowVersion: index + 1, + truncate: step.kind === `snapshot`, + sourceProgress: { key: `source-progress`, previous }, + mutations: + step.kind === `empty` + ? [] + : [ + { + type: `insert` as const, + key: `row`, + value: { id: `row`, label: next }, + }, + ], + collectionMetadataMutations: [ + { type: `set` as const, key: `source-progress`, value: next }, + ], + } + await adapter.applyCommittedTx(target, tx) + const observation = await adapter.loadResumeSnapshot(target) + expect( + observation.sourceProgressCompatible, + `source progress at step ${index}: ${JSON.stringify(steps.slice(0, index + 1))}`, + ).toBe(coherent()) + // Physical key integrity and source continuity are separate laws. + // A broken source path must not quarantine an intact cached set. + expect(observation.keySet).toEqual({ status: `consistent` }) + expect( + observation.collectionMetadata.find( + ({ key }) => key === `source-progress`, + )?.value, + ).toBe(next) + current = next + tokens.push(next) + } + } catch (error) { + primary = error + } finally { + closeDatabasePreservingPrimary(database, primary) + } + }, + ), + parameters, + ) + console.info(`source progress reach`, Object.fromEntries(reach.counts())) + reach.assertReached() + } + it.skipIf(oracleReplayRequested())( + `preserves source progress evidence (fixed seed)`, + () => runProgressCampaign({ numRuns: oracleRuns(80), seed: 2281 }), + ) + oracleReplayTest(it, `sqlite-resume.source-progress`)( + `preserves source progress evidence (random or replayed)`, + () => + runProgressCampaign( + oraclePropertyOptions(80, `sqlite-resume.source-progress`), + ), + ) + // The retired-storage and managed-claim checks compare production with independent // models of retained generations and claim intervals. The fixed and generated // cases share those checks. The startup grammar is the generator whose coverage // the fixed startup corpus calibrates. declareOracle({ - check: [assertRetiredStorageHistory, assertManagedClaimReadHistory], + check: [ + assertRetiredStorageHistory, + assertManagedClaimReadHistory, + runProgressCampaign, + ], model: [expectedRetainedGenerations], - grammar: [startupHistoryArbitrary], + grammar: [startupHistoryArbitrary, progressHistory], }) fcTest.prop([startupHistoryArbitrary], { diff --git a/packages/db/src/sync-persistence.ts b/packages/db/src/sync-persistence.ts index 299f47d847..211a19b34a 100644 --- a/packages/db/src/sync-persistence.ts +++ b/packages/db/src/sync-persistence.ts @@ -94,6 +94,18 @@ export function validateSyncPersistenceCapability< `expectCurrentCommit`, `resumeSnapshot.expectCurrentCommit`, ) + if (resumeSnapshot.expectSourceProgress !== undefined) { + requireFunction( + resumeSnapshot, + `expectSourceProgress`, + `resumeSnapshot.expectSourceProgress`, + ) + requireFunction( + resumeSnapshot, + `getSourceProgressCompatible`, + `resumeSnapshot.getSourceProgressCompatible`, + ) + } return value as SyncPersistenceCapabilityV1 } diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index ce37781c99..8d17b2ae8e 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -639,6 +639,21 @@ export type SyncPersistenceCapabilityV1< readonly certify: () => Promise readonly getKeySetEvidence: () => SyncPersistenceKeySetEvidence | undefined readonly expectCurrentCommit: () => void + /** + * Atomically compare the durable source token with this transaction's + * predecessor. The source stages its next opaque string token in collection + * metadata at `key`. A discontinuity invalidates source progress evidence + * until an authoritative truncate; rows and receipts still apply normally. + * Re-delivering the same complete source position must be idempotent; it + * preserves existing evidence, including an earlier incompatibility. + * Only adapters that implement the durable comparison expose this method. + */ + readonly expectSourceProgress?: (progress: { + key: string + previous: string | undefined + }) => void + /** Source progress evidence from the atomic persisted resume snapshot. */ + readonly getSourceProgressCompatible?: () => boolean | undefined } } diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index ca9c1a8927..15e4cb262a 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -160,12 +160,14 @@ const staticOracleProperties = [ `electric.sdk-rotation-release`, `electric-recovery.publication-stream-convergence`, `sqlite-resume.startup-generation`, + `sqlite-resume.source-progress`, `sqlite-resume.managed-claim-read`, `sqlite-resume.retired-storage`, `persisted-demand-coverage.history`, `persisted-demand-coverage.required`, `sqlite-reload.class`, `browser.multi-tab-reserved-history`, + `browser.electric-source-prefix`, `sqlite-reload.class.electron`, `persistence.generation-notice`, `persistence.follower-remote-demand`, diff --git a/packages/db/tests/sync-persistence.test.ts b/packages/db/tests/sync-persistence.test.ts index 5e67c43842..87d56a9b48 100644 --- a/packages/db/tests/sync-persistence.test.ts +++ b/packages/db/tests/sync-persistence.test.ts @@ -97,4 +97,27 @@ describe(`sync persistence capability`, () => { }), ).toThrow(/startupCacheGenerationRotated must be a boolean/) }) + + it(`accepts source progress only when the optional capability is callable`, () => { + const capability = { + ...baseCapability, + resumeSnapshot: { + certify: () => Promise.resolve(), + getKeySetEvidence: () => ({ status: `consistent` as const }), + expectCurrentCommit: () => {}, + expectSourceProgress: () => {}, + getSourceProgressCompatible: () => true, + }, + } + expect(validateSyncPersistenceCapability(capability)).toBe(capability) + expect(() => + validateSyncPersistenceCapability({ + ...capability, + resumeSnapshot: { + ...capability.resumeSnapshot, + expectSourceProgress: true, + }, + }), + ).toThrow(/resumeSnapshot.expectSourceProgress must be a function/) + }) }) diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 14ecda8745..626959a5de 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -1683,6 +1683,8 @@ function createElectricSync>( const certifyResumeSnapshot = resumeSnapshot?.certify const getKeySetEvidence = resumeSnapshot?.getKeySetEvidence const expectCurrentCommit = resumeSnapshot?.expectCurrentCommit + const expectSourceProgress = + syncMode === `eager` ? resumeSnapshot?.expectSourceProgress : undefined const persistedKeySetEvidence = getKeySetEvidence?.() const storedResumeRecord = freshScopedGeneration @@ -1699,15 +1701,26 @@ function createElectricSync>( url: shapeOptions.url, params: shapeOptions.params as Record | undefined, }) + // Handles are opaque. Equality of the complete source position is enough + // to detect a broken continuation; wall-clock timestamps are not progress. + const progressToken = (state: ElectricResumeState | undefined) => + state?.kind === `resume` + ? JSON.stringify([state.shapeId, state.handle, state.offset]) + : undefined + const progressKey = `electric:source-progress` + let sourceProgress = progressToken(persistedResumeState) const hasIncompatiblePersistedResume = persistedResumeState?.kind === `resume` && persistedResumeState.shapeId !== shapeIdentity // A pre-ledger `unknown` baseline cannot justify a non-initial cursor. // One fresh replacement establishes consistent evidence for later resumes. - const lacksCompletePersistedKeySet = + const lacksCompletePersistedResume = persistedResumeState?.kind === `resume` && - getKeySetEvidence !== undefined && - persistedKeySetEvidence?.status !== `consistent` + ((getKeySetEvidence !== undefined && + persistedKeySetEvidence?.status !== `consistent`) || + (expectSourceProgress !== undefined && + (metadata?.collection.get(progressKey) !== sourceProgress || + resumeSnapshot?.getSourceProgressCompatible?.() !== true))) // Claim expiry can leave a partial managed cache without a resume record. // Its incompatible key set alone requires a fresh source snapshot. const hasIncompatibleManagedCache = @@ -1722,7 +1735,7 @@ function createElectricSync>( (malformedPersistedResume || hasIncompatiblePersistedResume || persistedResumeState?.kind === `reset` || - lacksCompletePersistedKeySet || + lacksCompletePersistedResume || (!retainsTagState && persistedResumeState?.kind === `resume` && persistedResumeState.requiresTagState !== false)))) @@ -2048,6 +2061,11 @@ function createElectricSync>( updatedAt: Date.now(), requiresTagState: tagState.hasTags(), } + if (expectSourceProgress) { + expectSourceProgress({ key: progressKey, previous: sourceProgress }) + sourceProgress = progressToken(resumeState) + metadata?.collection.set(progressKey, sourceProgress) + } lifecycle.resumeState = resumeState metadata?.collection.set(`electric:resume`, resumeState) } diff --git a/packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts b/packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts index 925990c2cb..f306d6b511 100644 --- a/packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts +++ b/packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts @@ -198,6 +198,19 @@ async function runRace( { type: `insert`, key: 2, value: { id: 2, name: `two` } }, ], collectionMetadataMutations: [ + // Seed the current eager format so this driver still exercises + // certification of a resumed baseline, not legacy-token migration. + { + type: `set`, + key: `electric:source-progress`, + value: JSON.stringify([ + startupReset === `shape-identity` + ? `{"params":{"table":"other_table"},"url":"http://test-url"}` + : `{"params":{"table":"test_table"},"url":"http://test-url"}`, + `shape-old`, + `10_0`, + ]), + }, { type: `set`, key: `electric:resume`, From 63c3c4a455b70e62cbaebaa04f4e52bf1e2f35e0 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 10:27:28 -0600 Subject: [PATCH 02/11] docs: record committed persistence verification --- .../oracle-reviews/issue-2281-investigation.md | 16 ++++++++++++++++ .../tests/sqlite-resume-snapshot-oracle.test.ts | 3 ++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index ab797cd124..ffd2444f7b 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -907,3 +907,19 @@ final loop restarted both existing Collections. The local `loss-trace.log` records those restarts at lines 101 and 110, followed by the missing-row source-head mismatch at line 119. This supplies the earlier restart defect witness; it does not retroactively supply the focused campaign's missing reach. + + +### Committed-tree verification (2026-10-29) + +Implementation commit `20e2ad9f81c91c60e5d9c219892c6421ae12baca` contains the +reviewed repair and both approved suggestions. Its pre-commit hook removed an +unnecessary test-only non-null assertion and merged duplicate imports in the +SQLite oracle. The merged import then received a formatting correction. These +changes preserve runtime behavior, model rules, histories and comparisons. +The SQLite resume owner passes again (61 reported tests, no type errors), as +does Browser `tsc --noEmit`. Logs: `prep-committed-core-green.log` and +`prep-committed-browser-tsc.log`. All five production files retain their +previously verified content. The 17-file manifest excluding this record is +`e2fda4fe3c0962594f71929acd20c4e8c03b79cdded5e6713594168525a53111`. This identity +includes the accepted driver simplification and the hook's test-only edits. +The earlier full runtime, real-host and mutant evidence retains its scope. diff --git a/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts index 714cb60dd9..61eb42f87c 100644 --- a/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts @@ -5,7 +5,8 @@ import { createCollection } from '@tanstack/db' import { oraclePropertyOptions, oracleReplayRequested, - oracleReplayTest, oracleRuns + oracleReplayTest, + oracleRuns, } from '../../db/tests/oracle-config' import { declareOracle } from '../../db/tests/oracle-declaration' import { createOracleReach } from '../../db/tests/oracle-reach' From e9b2dda669a3071e9cd31d7c2d46626e2a7070bc Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 10:28:33 -0600 Subject: [PATCH 03/11] chore: add Electric persisted recovery changeset --- .changeset/recover-interleaved-electric-source-commits.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 .changeset/recover-interleaved-electric-source-commits.md diff --git a/.changeset/recover-interleaved-electric-source-commits.md b/.changeset/recover-interleaved-electric-source-commits.md new file mode 100644 index 0000000000..567a350ab0 --- /dev/null +++ b/.changeset/recover-interleaved-electric-source-commits.md @@ -0,0 +1,7 @@ +--- +'@tanstack/db': patch +'@tanstack/db-sqlite-persistence-core': patch +'@tanstack/electric-db-collection': patch +--- + +Recover eager Electric collections from a fresh snapshot when interleaved source commits leave persisted rows inconsistent with the resume cursor. Keep cached rows visible while recovery restores the source state. From 297e214fe4b65df03e73566d9201171b5ca37453 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 10:49:06 -0600 Subject: [PATCH 04/11] Fix recovery fixtures and replay registration from CI --- docs/collections/convex-collection.md | 1 + .../issue-2281-investigation.md | 59 +++++++++++++++++++ ...electric-source-prefix-oracle.opfs.spec.ts | 4 +- .../tests/electric-convergence-oracle.ts | 9 +++ packages/db/tests/oracle-replay-manifest.ts | 12 +++- .../tests/electric-recovery-oracle.test.ts | 17 +++++- .../tests/electron-ipc.test.ts | 1 + scripts/doc-snippets-baseline.json | 1 - 8 files changed, 98 insertions(+), 6 deletions(-) diff --git a/docs/collections/convex-collection.md b/docs/collections/convex-collection.md index 36aca89604..da2740d7eb 100644 --- a/docs/collections/convex-collection.md +++ b/docs/collections/convex-collection.md @@ -81,6 +81,7 @@ const tasks = createCollection( }), ) +const taskId: string = selectedTask._id const transaction = tasks.update(taskId, (task) => { task.completed = true }) diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index ffd2444f7b..d452897b27 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -923,3 +923,62 @@ previously verified content. The 17-file manifest excluding this record is `e2fda4fe3c0962594f71929acd20c4e8c03b79cdded5e6713594168525a53111`. This identity includes the accepted driver simplification and the hook's test-only edits. The earlier full runtime, real-host and mutant evidence retains its scope. + +### Published CI and CodeRabbit follow-up (2026-10-29) + +PR #2298 first ran at `e9b2dda669a3071e9cd31d7c2d46626e2a7070bc`. +CI found two missing named replay registrations, four warm-reopen fixture +failures, one Electron snapshot-shape assertion, and documentation-check drift. +The recovery owner reproduced all four failures locally before its fixture +changed. Its warm baseline supplied a resume record but omitted the complete +source-position token. Under the new capability contract, that is a legacy +baseline and correctly requests a full snapshot. The fixture now supplies the +matching token for its existing compatible-reopen premise. Its model, request +and row comparisons remain unchanged. Missing/foreign-token recovery remains +generated in the Browser owner and exercised by the real receiving cells. + +The replay manifest now names both new properties and their exact replay lanes. +The Electron assertion includes the new `sourceProgressCompatible` field. +The Convex example declares its selected task ID as a string, so an ambient +application variable cannot select the array-update overload. The obsolete +passing Preact snippet was removed from the known-failure baseline. No new +failure was added to that baseline. These changes alter no production code. + +Verification commands use `pnpm_config_verify_deps_before_run=false`: + +- Electric recovery owner: 60 reported tests passed, no type errors. The + preceding run reported four failures and 56 passes. +- Electron IPC suite: 116 reported tests passed, no type errors. +- Replay harness: 41 tests passed. Guarded direct replays of + `sqlite-resume.source-progress` and `browser.electric-source-prefix`, seed + `2281`, path `0`, each recorded one completed execution of the exact property. +- Documentation checker: 806 examples; zero new failures, 250 known failures, + and 59 known non-parsing examples. The earlier run had one new failure and + one stale baseline entry. +- Browser `tsc --noEmit` and the focused formatting check pass. +- `pnpm lint:changed` and `git diff --check` pass; no run budget was lowered. +- The real SDK/PostgreSQL/Chromium source-prefix receiver passes all ten cells; + every cell reaches each of its four declared states once. + +CodeRabbit review `5701493057`, inline `4415193533`, and mutable summary +`6502484352` cover the original published head. Their complete raw text and a +seven-item lossless ledger were preserved locally before edits. The startup +polls now explicitly allow 30 seconds. The shared checker now locates its +companion grammar and drivers and explains its independent source-head model. +Its judgment, name, registration, and comparison remain unchanged. + +Two historical phrases receive clarifications here to preserve this record's +append-only chronology: “new generated campaigns” means newly added +generated-history properties; “Browser important generated properties” means +both important generated properties in the Browser package. “Standalone type +examples” retains its existing meaning and spelling. + +The loss audit accounts for seven items: four fixed, one spelling requirement +refuted, one accepted documentation-policy choice, and one duplicate. The +summary's aggregate docstring warning names no missing function obligation or +repository authority for its 80% threshold. ORC-003's proportional prose and +directly named companion modules remain the governing documentation contract. +No behavioral finding, unresolved law decision, or new in-scope counterexample +was identified. CodeRabbit is useful as a supplemental maintenance reviewer; +this review supplies no independent behavioral proof. Earlier evidence and +limits remain unchanged because no production or model semantics changed. diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts index bf05850289..2d1a495c7a 100644 --- a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts @@ -96,10 +96,10 @@ for (const cell of cells) { () => window.__electricSourcePrefix !== undefined, ) await expect - .poll(async () => (await observe(page)).status) + .poll(async () => (await observe(page)).status, { timeout: 30_000 }) .toBe(`ready`) await expect - .poll(async () => (await observe(page)).commits) + .poll(async () => (await observe(page)).commits, { timeout: 30_000 }) .toBeGreaterThan(0) return page } diff --git a/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts b/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts index 6404f984b0..a0498e34b8 100644 --- a/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts +++ b/packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts @@ -8,6 +8,14 @@ * The Node driver supplies its independent server log; the Chromium/real SDK * driver supplies PostgreSQL rows. Both use this same settled-row judgment. * It does not judge intermediate publication, on-demand subsets, or optimism. + * + * This module owns the independent model and refinement check: the source head + * predicts both durable and public rows, ignoring row order but retaining every + * row occurrence and requiring each tab to be ready. The legal history grammar + * and Node production driver live in multi-tab-electric-oracle.test.ts. The + * receiving grammar and Chromium driver live in + * ../e2e/electric-source-prefix-oracle.opfs.spec.ts and its page module. + * Each driver calls this comparison after recovery and complete source delivery. */ import { declareOracle } from '../../db/tests/oracle-declaration' @@ -19,6 +27,7 @@ function canonicalRows(rows: ReadonlyArray): Array { .sort((a, b) => String(a.id).localeCompare(String(b.id))) } +/** Compare durable rows and ready tabs with the independently supplied head. */ export function sourceHeadViolation( sourceHead: ReadonlyArray, observation: { diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 5de3ae76b1..c4e9d54970 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -230,7 +230,7 @@ const ownerGroups: ReadonlyArray< [ `db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts`, `sqlite-resume`, - `startup-generation`, + `startup-generation source-progress`, ], [ `db/tests/query/derived-delete-reconciliation-oracle.test.ts`, @@ -410,7 +410,7 @@ const ownerGroups: ReadonlyArray< [ `browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts`, `browser`, - `multi-tab-reserved-history`, + `multi-tab-reserved-history electric-source-prefix`, ], [ `electron-db-sqlite-persistence/tests/reload-class-oracle.test.ts`, @@ -724,6 +724,14 @@ export type OracleReplayManifestEntry = { // Pagination cells carry their property name in the test title so the guard // can select one cell without running the rest of the matrix. const replayTestNames = new Map([ + [ + `sqlite-resume.source-progress`, + `preserves source progress evidence.*random or replayed`, + ], + [ + `browser.electric-source-prefix`, + `converges after interleaved source prefixes.*random or replayed`, + ], [ `source-health.required-history`, `required histories refine the model.*random`, diff --git a/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts b/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts index 674ea6d7a0..70a7407e63 100644 --- a/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts +++ b/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts @@ -27,6 +27,7 @@ import { withFixtureCleanup, } from './electric-recovery-support' import type { Message } from '@electric-sql/client' +import type { PersistedTx } from '../../db-sqlite-persistence-core/src' import type { ElectricCollectionUtils } from '../src/electric' import type { Item, @@ -186,7 +187,7 @@ async function observePersistedRestart( scenario, modelCanonicalRows, ) - const resumeState = + const resumeState: NonNullable = scenario.resumeKind === `none` ? [] : [ @@ -210,6 +211,20 @@ async function observePersistedRestart( driver, schemaVersion: 1, }) + // A compatible warm baseline includes the source's complete-position token. + // Legacy/missing tokens are a separate full-snapshot recovery regime in the + // Browser source-prefix owner, not a compatible-reopen premise here. + if (scenario.resumeKind === `resume`) { + resumeState.push({ + type: `set`, + key: `electric:source-progress`, + value: JSON.stringify([ + `{"params":{"table":"test_table"},"url":"http://test-url"}`, + `shape-old`, + `10_0`, + ]), + }) + } await originalAdapter.applyCommittedTx( { kind: `eager`, collectionId: collectionId }, { diff --git a/packages/electron-db-sqlite-persistence/tests/electron-ipc.test.ts b/packages/electron-db-sqlite-persistence/tests/electron-ipc.test.ts index 38bc6924cc..ee4077ed8d 100644 --- a/packages/electron-db-sqlite-persistence/tests/electron-ipc.test.ts +++ b/packages/electron-db-sqlite-persistence/tests/electron-ipc.test.ts @@ -491,6 +491,7 @@ describe(`electron sqlite persistence bridge`, () => { }, ], keySet: { status: `consistent` }, + sourceProgressCompatible: true, collectionMetadata: [ { key: `resume:test`, diff --git a/scripts/doc-snippets-baseline.json b/scripts/doc-snippets-baseline.json index 3ecffabb8d..4786a4be77 100644 --- a/scripts/doc-snippets-baseline.json +++ b/scripts/doc-snippets-baseline.json @@ -58,7 +58,6 @@ "a066dfd28d064b54" ], "docs/framework/angular/overview.md": ["55d749380b445d1a"], - "docs/framework/preact/overview.md": ["a3802a251c868f43"], "docs/framework/react/overview.md": ["1c1f2ec0a48394e3", "d89b6c8f8cab169d"], "docs/guides/collection-options-creator.md": [ "305eb0ca15d244e8", From 08d7ededb9d41ae4f8830eff8c7abcc37d9bfd6a Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 11:19:40 -0600 Subject: [PATCH 05/11] Measure final reset recovery in generated histories --- docs/contributing/oracle-coverage.md | 18 ++++++- .../issue-2281-investigation.md | 54 +++++++++++++++++++ .../tests/multi-tab-electric-oracle.test.ts | 5 ++ 3 files changed, 76 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 680bfc3c8c..d140ad3e5b 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4002,6 +4002,10 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. generated histories and pinned with `it.fails`: a resumed tab that lags behind a peer can receive an update for a row the peer already deleted, and Electric's resume check puts it in `error`. + The published Electric resume-safety contract requires rejecting an + unknown partial row. Whether composition must avoid that condition + without failing the source remains a separate contract question; + this repair does not remove the guard or resolve the pinned expectation. The histories also post a peer `collection:reset` on the tabs' coordinator channel; no shipped host sends one yet, so this premise is controlled. The reset abandons every open reserved transaction, and @@ -4062,7 +4066,19 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. no-deploy reset trace (seed `-1371495859`, path `75:0:0:0:0:0:0:1:1:0:0`) pass on unchanged production with that repaired checkpoint. Both remain fixed inputs to the shared driver. A generated - cached-row regression also preserves intermediate observations during + reach separately requires recovery first discovered during that final + delivery, so an earlier reset failure cannot satisfy this premise. + It reached 2 of 80 fixed-seed histories and 6 of 80 random histories. + Removing the final restart makes both reset traces fail the final row + comparison. The broader Node campaign can also log a leadership term + write against a closed test database during teardown (pilot `PILOT/S02`, + fixed seed `952051961`). This is distinct from injected stream errors; + its user-facing effect is unverified. Owners: broadcast coordination and + this host driver. Needed witness: hold the election term write across + coordinator disposal and database close, then observe its settlement + and resource release on controlled and real hosts. Passing convergence + comparisons do not establish clean teardown. + A generated cached-row regression also preserves intermediate observations during source-continuity repair. #2209's permanent-close case and #2267's resumed-update gap remain pinned expected failures under this owner. A claimed leftover-replay/reset failure still needs a concrete schedule; diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index d452897b27..7e3b5e0bbd 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -982,3 +982,57 @@ No behavioral finding, unresolved law decision, or new in-scope counterexample was identified. CodeRabbit is useful as a supplemental maintenance reviewer; this review supplies no independent behavioral proof. Earlier evidence and limits remain unchanged because no production or model semantics changed. + +### Independent cluster report reconciliation (2026-10-29) + +The SQLite/Electric peer pilot investigated frozen baseline `07bfc0eb2`. +Its report, laws, retained oracle patch, raw reset/convergence logs, and claim +ledger were compared with this repair. Its two row/cursor counterexamples +confirm the mechanism already repaired here: an old row commit followed by +another source's newer resume metadata can make restart skip a needed change. +Both exact row traces and both reset traces pass again on `297e214fe`, with +no type errors. The pilot's baseline failures do not describe that newer tree. + +The report contributes one oracle improvement. The broad generated owner now +declares recovery first discovered during final catch-up separately from a +reset failure earlier in the history. An earlier failure can no longer satisfy +the reach requirement for that final branch. The model, history grammar, +comparisons, production code, and run budgets remain unchanged. + +- **Law:** Law 5 requires a reset-abandoned commit to fail; Law 2 compares the + independent source head after successful recovery and complete delivery. + Their authority remains the coverage map and maintainer decisions on #2265. +- **Reach:** `reset-abandoned source recovered during final catch-up` reached + 2 of 80 histories at fixed seed `952051961`, and 6 of 80 random histories. + Both campaigns require at least one. Existing reach requirements also pass. +- **RED / mutant:** Removing only the final restart/delivery block makes both + named reset histories fail at `after every tab caught up`, with wrong rows. + These are assertion failures, not setup failures or timeouts. This calibrates + the driver repair; it is not a new production mutant. Earlier production + RED and hostile source-progress evidence retain their scope. +- **GREEN:** Both normal 80-history campaigns and both positive reset replays + pass (7 reported tests, no type errors). A measurement-only run repeats the + same campaigns (2 runtime tests) and records the counts above. Its temporary + direct stderr recorder was restored afterward because ordinary console + output was suppressed. Logs: `pilot-current-*.log`, + `pilot-final-recovery-ablation.log`, `pilot-final-recovery-green.log`, and + `pilot-final-recovery-measured.log` in the local evidence directory. +- **Limits:** These remain bounded source-head observations. They do not + establish individual receipt fulfillment, clean teardown, generic source + ordering, or the other cluster issues' receiving paths. + +The report also sharpens two qualifications. The published Electric +resume-safety contract requires rejecting a partial update for an unknown row. +The separate #2267 no-error composition expectation needs an authority decision; +this repair preserves the guard and does not resolve that expectation. +The Node campaign's closed-database leadership warning is distinct from +deliberately injected source failures. It also appears in earlier local logs. +Its user-facing effect remains unverified, and the coverage map now records +the teardown owner and needed held-election witness under pilot `PILOT/S02`. +Passing convergence comparisons do not establish clean teardown. + +Follower reacquisition (#2139), seed metadata, real sync-metadata receiving +(#2272), broader leader-close premises (#2085), loaded-subset relay (#2174), and +real unknown-partial-message premises (#2138) remain separate boundaries. +The report supplies no reason to merge them into this source-progress repair. +The complete raw report and 16-item reconciliation ledger remain local. diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index c1f6d8b561..c01aa745f9 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -1408,6 +1408,9 @@ describe(`multi-tab Electric over one persisted database`, () => { await recover(index, `restart`) await deliver(index, `all`, true) await check(`after reset recovery of tab ${tabs[index]!.tab}`) + generatedReach.reach( + `reset-abandoned source recovered during final catch-up`, + ) } } await check(`after every tab caught up`, rowsOf(log.rows)) @@ -1545,6 +1548,7 @@ describe(`multi-tab Electric over one persisted database`, () => { | `peer commit after signal aborted mid-transaction` | `peer reset abandoned an open transaction` | `reset-abandoned commit put the Collection in error` + | `reset-abandoned source recovered during final catch-up` // The storm driver and the generated-history driver are the checks; the // generated driver folds committed batches into its own model of durable // rows. The storm cells and the generated-history arbitrary are the grammar. @@ -1566,6 +1570,7 @@ describe(`multi-tab Electric over one persisted database`, () => { `peer commit after signal aborted mid-transaction`, `peer reset abandoned an open transaction`, `reset-abandoned commit put the Collection in error`, + `reset-abandoned source recovered during final catch-up`, ]) let generatedReach = createGeneratedReach() From 883b565ab6c4fabb4c93e860ad12fc28d0823af1 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 11:29:05 -0600 Subject: [PATCH 06/11] Type the Preact query example source Collection --- .../oracle-reviews/issue-2281-investigation.md | 10 ++++++++++ docs/framework/preact/overview.md | 16 +++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index 7e3b5e0bbd..96ed273e30 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -1036,3 +1036,13 @@ Follower reacquisition (#2139), seed metadata, real sync-metadata receiving real unknown-partial-message premises (#2138) remain separate boundaries. The report supplies no reason to merge them into this source-progress repair. The complete raw report and 16-item reconciliation ledger remain local. + +The next CI run exposed a Preact documentation fragment that inferred `{}` +for todo fields from the checker's synthetic source Collection. The example +now creates its own typed local Collection. A temporary source-resolution probe +against the real Preact hook reproduces both original JSX errors and passes +the typed example; the checker was restored afterward. The full documentation +check passes all 806 examples with zero new failures (250 known, 59 non-parsing). +No baseline entry or production code changed. Logs: +`pilot-preact-source-before.log`, `pilot-preact-source-after.log`, and +`pilot-preact-snippets-full.log`. diff --git a/docs/framework/preact/overview.md b/docs/framework/preact/overview.md index 9520c714d0..ff3963438b 100644 --- a/docs/framework/preact/overview.md +++ b/docs/framework/preact/overview.md @@ -26,7 +26,21 @@ For comprehensive documentation on writing queries (filtering, joins, aggregatio The `useLiveQuery` hook creates a live query that automatically updates your component when data changes: ```tsx -import { eq, useLiveQuery } from '@tanstack/preact-db' +import { createCollection, eq, localOnlyCollectionOptions } from '@tanstack/db' +import { useLiveQuery } from '@tanstack/preact-db' + +type Todo = { + id: string + text: string + completed: boolean +} + +const todoCollection = createCollection( + localOnlyCollectionOptions({ + getKey: (todo) => todo.id, + initialData: [], + }), +) function TodoList() { const { data, isLoading } = useLiveQuery((q) => From 124ce667c61a9e247ea1628991bc72db91d6041e Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 12:07:10 -0600 Subject: [PATCH 07/11] Preserve source-progress recovery with explicit Electric cursors --- docs/collections/electric-collection.md | 5 ++ docs/contributing/oracle-coverage.md | 23 +++++- .../issue-2281-investigation.md | 80 +++++++++++++++++++ ...electric-source-prefix-oracle.opfs.spec.ts | 68 ++++++++++++++-- .../e2e/electric-source-prefix.opfs.ts | 39 +++++---- .../tests/multi-tab-electric-oracle.test.ts | 64 ++++++++++++++- .../electric-db-collection/src/electric.ts | 23 ++++-- 7 files changed, 266 insertions(+), 36 deletions(-) diff --git a/docs/collections/electric-collection.md b/docs/collections/electric-collection.md index a206107d60..fac813a74f 100644 --- a/docs/collections/electric-collection.md +++ b/docs/collections/electric-collection.md @@ -377,6 +377,11 @@ state, or incomplete key-set evidence also invalidates the saved cursor. Untagged shapes with complete resume evidence can still resume from their saved offset. +An eager persisted resume also needs compatible source progress when its adapter +supplies that capability. Missing or incompatible progress requires a fresh +snapshot, even with an explicit offset or handle. Those options do not certify +the cached rows. + Eager and progressive recovery fetch a full snapshot. Cached rows remain visible until that replacement completes; a partial batch or subset completion cannot publish it early. An on-demand collection with the current SQLite persistence diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d140ad3e5b..bb1349f106 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4039,10 +4039,14 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. seed/path replay. It crosses initial membership, one to three toggle pairs, swapped source roles, empty/disjoint advancement, drain/no-drain, tab reload or source failure followed by same-Collection restart, and - current/missing/foreign progress. Declared reach + current/missing/foreign progress. Caller positions vary between omitted, + initial `-1`, handle-only, numeric offset with handle, and `now`. + Declared reach requires all four divergent membership/advancement cells, a draining control, missing/foreign provenance, and divergent recovery through both - actual lifecycle paths. Both Browser random registrations and the SQLite + actual lifecycle paths. It also requires divergent recovery with explicit + numeric and `now` positions, plus initial-offset and handle-only controls. + Both Browser random registrations and the SQLite source-progress registration stand aside during another property's path replay. The SQLite owner separately checks opaque predecessor paths, sticky gaps, duplicate endpoints, @@ -4052,12 +4056,23 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. validation fails the Browser owner. The real installed SDK/HTTP/PostgreSQL/Chromium OPFS receiving driver `e2e/electric-source-prefix-oracle.opfs.spec.ts` shares the source-head - judgment. Ten cells cover both membership directions, empty/disjoint + judgment. The original ten cells cover both membership directions, empty/disjoint advancement and drain/no-drain, plus missing legacy provenance with disjoint advancement. Each declares held earlier response, stale commit, later advancement, and reopened source comparison. All ten pass; gap and legacy-validation mutants fail the matching real disjoint cells at that - comparison. This is bounded same-version eager Browser evidence, not + comparison. Ten more cells cross numeric and `now` caller positions with + both membership directions and current/legacy provenance, plus initial + offset and handle-only controls. Eight wrong-row comparisons were RED on + `883b565ab` before the eager-only repair; all ten explicit cells now pass. + Each explicit cell declares its caller-position reach. A mutant that + requests recovery but preserves the caller position fails the final Node + source-head comparison (seed 2281, path `1:0:0`). Explicit positions cannot + bypass eager source-progress recovery. Other sync modes and explicit + malformed/reset/tag/shape regimes retain their earlier behavior and are + outside this added evidence. Real foreign-token provenance and changing + caller forms within one history still need separate receiving witnesses. + This is bounded same-version eager Browser evidence, not arbitrary scheduling, mixed client versions, native-device, Electron, on-demand, optimistic, or monotonic-live-write coverage. The final checkpoint now observes a reset-abandoned commit's source diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index 96ed273e30..77a549c7a8 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -1046,3 +1046,83 @@ check passes all 806 examples with zero new failures (250 known, 59 non-parsing) No baseline entry or production code changed. Logs: `pilot-preact-source-before.log`, `pilot-preact-source-after.log`, and `pilot-preact-snippets-full.log`. + +### Explicit caller positions cannot bypass source-progress recovery + +CodeRabbit review `5702393171` on `883b565ab` found a missing history dimension. +The supplied cluster report still supports the repair's mechanism. This later +review exposes an incomplete enforcement boundary within that repair. + +The accepted law remains Browser Law 2: after successful recovery and complete +source delivery, durable rows and every open eager Collection equal the source +head. Electric's documented fresh-snapshot contract requires omitted cached rows +to disappear at replacement. A caller's provider position supplies no missing +baseline rows and cannot certify incompatible persisted source progress. + +The primary source-prefix grammar now varies omitted options, initial `-1`, +handle-only, numeric offset with handle, and the SDK's legal `now` offset. +It crosses those choices with existing membership, repeated toggles, roles, +advancement, draining, restart/reopen, and current/missing/foreign provenance. +Numeric noninitial offsets without handles are SDK-invalid and excluded. +The simulated server now interprets `now` as its head, supplying no baseline. +The independent source-head model, comparison, exclusions, and budgets remain. + +The proposed one-clause repair was insufficient. Merely requesting recovery +still let the explicit offset suppress replacement and reach `ShapeStream`. +The final repair separately identifies incompatible eager source progress and +uses the initial effective position for both stream construction and snapshot +replacement. Physical key-set checks retain their prior cursor guard. Other +sync modes and managed caches without scoped recovery retain their explicit +options. An independent reviewer identified that scope distinction before +publication. No generic ordering or new composition contract follows. + +- **Law:** Browser Law 2/#2209 and the Electric fresh-snapshot contract above. +- **Reach:** Both 80-history campaigns require all earlier states plus explicit + numeric/`now` divergent recovery and initial-offset/handle controls. Final + fixed/random counts are numeric 10/8, `now` 7/19, initial 17/12, and handle + 21/20. Fixed seed remains `2281`. Every original required state also passes. + Each of ten explicit real receiving cells reaches its supplied-position + state and all four original states once. +- **RED:** Before production edits, both extended Node campaigns fail the final + `after every tab caught up` row comparison. Numeric replay: seed `2281`, + path `8:0:0:0` in the four-form grammar. Final five-form grammar: seed `2281`, + path `1:0:0`, `now`. Real SDK/HTTP/PostgreSQL/Chromium OPFS checks reject eight + cells: lost/resurrected membership, current/legacy evidence, and numeric/`now` + positions. Initial-offset and handle-only neighbors pass. Failures show + concrete missing or extra durable rows, not setup errors or readiness waits. +- **GREEN:** The final complete Browser owner reports 312 passed and two tracked + expected failures, with no type errors. Electric recovery, resume races, + installed-SDK delivery, and ordinary Electric checks report 385 passed and + no type errors. Browser `tsc --noEmit` passes. The expanded real matrix passes + 20 cells; after narrowing the predicate to eager source progress, all ten + explicit cells pass again. The narrow and earlier predicate are equivalent + for every eager cell. The first Node GREEN had passing runtime comparisons + but failed on a test-driver type import. The final run fixes that import and + succeeds with type checking enabled. +- **Mutant:** Keep the independent recovery decision but restore raw caller + positions, as the one-clause suggested fix would do. The Node replay at + seed `2281`, path `1:0:0`, fails the original final row comparison. Restore + the exact production source and the same replay passes. The repaired source's + SHA-256 is `df8a918e736ca073619a693316c475040db46e2e30fc20fdbcc566934be695e6`. +- **Limits:** No claim covers arbitrary explicit cursor selection over an + otherwise certified baseline, explicit malformed/reset/tag/shape regimes, + changing caller forms within one history, or real foreign-token provenance. + Existing on-demand, optimism, cross-version, permanent-close, receipt and + teardown limits remain. The coverage map retains their separate owners. + +Commands use the earlier owner invocations, with `-t 'converges after interleaved +source prefixes'` for the focused Node campaign, and `--grep 'with explicit'` +for the real explicit cells. Replay sets `TANSTACK_DB_ORACLE_PROPERTY` to +`browser.electric-source-prefix`, seed `2281`, and path `1:0:0`. +`--disableConsoleIntercept` exposes measured reach without a temporary recorder. +Logs are `explicit-cursor-node-red{,-complete}.log`, +`explicit-cursor-{now-,}real-red.log`, `explicit-cursor-browser-final-green.log`, +`explicit-cursor-electric-final-green.log`, `explicit-cursor-real{,-final}-green.log`, +`explicit-cursor-incomplete-fix-mutant.log`, and `explicit-cursor-replay-green.log`. + +The full third-review loss audit includes the outside-diff finding, repeated +risk, aggregate docstring warning, and related-issue prose. The explicit-cursor +finding is fixed with its generated dimension and receiving evidence. The +summary's claim that the reported #2283 trace is outside this repair is stale: +that exact row-loss replay already passes here. The #2267 safety/composition +qualification remains separate. Raw reviews and dispositions remain local. diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts index 2d1a495c7a..05d8515f53 100644 --- a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts @@ -15,7 +15,14 @@ import { declareOracle } from '../../db/tests/oracle-declaration' import type { Page } from '@playwright/test' import type {} from './electric-source-prefix.opfs' -const cells = [true, false] +type Cell = { + present: boolean + advance: `empty` | `disjoint` + drain: boolean + legacy: boolean + callerCursor: `none` | `initial-offset` | `handle` | `resume` | `now` +} +const cells: Array = [true, false] .flatMap((present) => ([`empty`, `disjoint`] as const).flatMap((advance) => [false, true].map((drain) => ({ @@ -23,6 +30,7 @@ const cells = [true, false] advance, drain, legacy: false, + callerCursor: `none` as const, })), ), ) @@ -32,8 +40,32 @@ const cells = [true, false] advance: `disjoint` as const, drain: false, legacy: true, + callerCursor: `none` as const, })), ) +// Repeat the legal explicit resumed position over both membership failures +// and both current/legacy evidence. Initial-offset and handle-only options +// request a complete snapshot and supply the distinguishing neighbors. +cells.push( + ...([`resume`, `now`] as const).flatMap((callerCursor) => + [true, false].flatMap((present) => + [false, true].map((legacy): Cell => ({ + present, + advance: `disjoint`, + drain: false, + legacy, + callerCursor, + })), + ), + ), + ...([`initial-offset`, `handle`] as const).map((callerCursor): Cell => ({ + present: true, + advance: `disjoint`, + drain: false, + legacy: false, + callerCursor, + })), +) const observe = (page: Page) => page.evaluate(() => window.__electricSourcePrefix!.observe()) const rowIds = async (page: Page) => @@ -41,7 +73,7 @@ const rowIds = async (page: Page) => declareOracle({ check: [sourceHeadViolation], grammar: [cells] }) for (const cell of cells) { - test(`recovers ${cell.present ? `lost` : `resurrected`} rows after ${cell.advance} advancement${cell.drain ? ` with peer drain` : ``}${cell.legacy ? ` from legacy progress` : ``}`, async ({ + test(`recovers ${cell.present ? `lost` : `resurrected`} rows after ${cell.advance} advancement${cell.drain ? ` with peer drain` : ``}${cell.legacy ? ` from legacy progress` : ``}${cell.callerCursor !== `none` ? ` with explicit ${cell.callerCursor}` : ``}`, async ({ context, }) => { const reach = createOracleReach( @@ -51,6 +83,9 @@ for (const cell of cells) { `stale commit applied`, `later advancement applied`, `reopened source compared`, + ...(cell.callerCursor === `none` + ? [] + : [`explicit ${cell.callerCursor} supplied at recovery`]), ], { replaying: false }, ) @@ -87,11 +122,11 @@ for (const cell of cells) { await pg.query(`INSERT INTO public.${table} VALUES ('anchor', 'anchor')`) if (cell.present) await pg.query(`INSERT INTO public.${table} VALUES ('target', 'old')`) - const open = async () => { + const open = async (callerPosition: Record = {}) => { const page = await context.newPage() pages.push(page) page.on(`pageerror`, (error) => diagnostics.push(error.message)) - await page.goto(url) + await page.goto(`${url}&${new URLSearchParams(callerPosition)}`) await page.waitForFunction( () => window.__electricSourcePrefix !== undefined, ) @@ -183,7 +218,30 @@ for (const cell of cells) { await close(b) if (cell.legacy) await a.evaluate(() => window.__electricSourcePrefix!.legacy()) - const reopened = await open() + const callerPosition: Record = {} + if (cell.callerCursor !== `none`) { + const { resume } = await observe(a) + if ( + !resume || + typeof resume !== `object` || + !(`offset` in resume) || + typeof resume.offset !== `string` || + !(`handle` in resume) || + typeof resume.handle !== `string` + ) + throw new Error(`No complete real Electric position at recovery`) + if ( + cell.callerCursor !== `initial-offset` && + cell.callerCursor !== `now` + ) + callerPosition.handle = resume.handle + if (cell.callerCursor === `resume`) + callerPosition.offset = resume.offset + if (cell.callerCursor === `initial-offset`) callerPosition.offset = `-1` + if (cell.callerCursor === `now`) callerPosition.offset = `now` + reach.reach(`explicit ${cell.callerCursor} supplied at recovery`) + } + const reopened = await open(callerPosition) // Source readiness follows an applied SDK receipt. The wrapper may // still propagate its notice to A, so poll the shared settled judgment. const source = await pg.query<{ id: string; label: string }>( diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts index 5f6584b924..bbac28588d 100644 --- a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts @@ -10,6 +10,7 @@ import { openBrowserWASQLiteOPFSDatabase, persistedCollectionOptions, } from '../src/index' +import type { Offset } from '../../electric-db-collection/node_modules/@electric-sql/client' type Item = { id: string; label: string } const parameters = new URL(location.href).searchParams @@ -35,6 +36,8 @@ const source = electricCollectionOptions({ syncMode: `eager`, getKey: (row) => row.id, shapeOptions: { + offset: (parameters.get(`offset`) ?? undefined) as Offset | undefined, + handle: parameters.get(`handle`) ?? undefined, url: `${location.origin}/electric/v1/shape`, params: { table: `public.${table}` }, liveSse: false, @@ -131,23 +134,25 @@ const probe = { if (!next) throw new Error(`No held response`) next.release() }, - observe: async () => ({ - status: collection.status, - rows: rows(collection.values()), - durableRows: rows( - ( - await options.persistence.adapter.loadResumeSnapshot({ - kind: `eager`, - collectionId, - }) - ).rows.map(({ value }) => value as Item), - ), - commits, - emptyCommits, - pending: pending.map(({ rows: count }) => count), - requests: [...requests], - error: collection.health.error, - }), + observe: async () => { + const snapshot = await options.persistence.adapter.loadResumeSnapshot({ + kind: `eager`, + collectionId, + }) + return { + status: collection.status, + rows: rows(collection.values()), + durableRows: rows(snapshot.rows.map(({ value }) => value as Item)), + resume: snapshot.collectionMetadata.find( + ({ key }) => key === `electric:resume`, + )?.value, + commits, + emptyCommits, + pending: pending.map(({ rows: count }) => count), + requests: [...requests], + error: collection.health.error, + } + }, cleanup: async () => { if (cleaned) return [] cleaned = true diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index c01aa745f9..03d561ab71 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -119,6 +119,11 @@ import type { import type { ElectricCollectionUtils } from '../../electric-db-collection/src/electric' type Item = Row & { id: number; label: string } +type CallerCursor = `none` | `initial-offset` | `handle` | `resume` | `now` +type ExplicitPosition = { + handle?: string + offset?: `${number}_${number}` | `-1` | `now` +} // --------------------------------------------------------------------------- // Controlled Electric SDK boundary. Each stream records the position Electric @@ -267,7 +272,11 @@ function openHost() { } } - const openTab = (tab: string, syncMode: SyncMode) => { + const openTab = ( + tab: string, + syncMode: SyncMode, + callerPosition: ExplicitPosition = {}, + ) => { const database = createWASQLiteTestDatabase({ filename, timeout: 0 }) const execute = database.execute.bind(database) database.execute = async ( @@ -288,6 +297,7 @@ function openHost() { // The application's signal, which can stop the stream from outside. let external = new AbortController() const shapeOptions = { + ...callerPosition, url: `http://electric.test/v1/shape`, params: { table: `items` }, headers: { 'x-tab': tab }, @@ -368,10 +378,11 @@ function openHost() { // The application's recovery: clean up and start the Collection again. // A signal that already aborted would stop the new stream too, so the // restarted session gets a fresh signal. - restart: async () => { + restart: async (nextPosition: ExplicitPosition = {}) => { await collection.cleanup() external = new AbortController() shapeOptions.signal = external.signal + Object.assign(shapeOptions, nextPosition) void collection.preload().catch(() => undefined) }, publicRows: () => @@ -999,6 +1010,11 @@ describe(`multi-tab Electric over one persisted database`, () => { startedAt: { handle: string | undefined; offset: string }, log: ServerLog, ): Pick => { + // Electric's legal `now` request begins at the current source head and + // supplies no baseline rows, even when no handle was specified. + if (startedAt.offset === `now`) { + return { handle: log.handle, version: log.changes.at(-1)?.version ?? 0 } + } if (startedAt.handle === undefined || startedAt.offset === `-1`) { return { handle: undefined, version: -1 } } @@ -1017,6 +1033,7 @@ describe(`multi-tab Electric over one persisted database`, () => { how: `restart` | `reload`, ) => void, progressFormat: `current` | `missing` | `foreign` = `current`, + callerCursor: CallerCursor = `none`, ) => { host = openHost() const activeHost = host @@ -1157,12 +1174,29 @@ describe(`multi-tab Electric over one persisted database`, () => { JSON.stringify(rowsOf(durable)) !== JSON.stringify(rowsOf(log.rows)) const stopped = tabs[index]! const streamsBefore = stopped.streamCount() + // A caller can repeat the current source position, but that position + // supplies no missing rows. Initial-offset and handle-only requests + // are legal full-snapshot neighbors of the explicit resumed request. + const head = position(log, serverVersion) + const callerPosition: ExplicitPosition = + callerCursor === `resume` + ? { + handle: head.handle, + offset: head.offset as `${number}_${number}`, + } + : callerCursor === `handle` + ? { handle: head.handle } + : callerCursor === `initial-offset` + ? { offset: `-1` } + : callerCursor === `now` + ? { offset: `now` } + : {} let recovered = stopped - if (how === `restart`) await stopped.restart() + if (how === `restart`) await stopped.restart(callerPosition) else { await stopped.close() await activeHost.quiesce(`tab ${stopped.tab} closed`) - recovered = activeHost.openTab(stopped.tab, `eager`) + recovered = activeHost.openTab(stopped.tab, `eager`, callerPosition) tabs[index] = recovered } abandon(models[index]!) @@ -1439,6 +1473,8 @@ describe(`multi-tab Electric over one persisted database`, () => { // Removing the lagging commit or the later advancement loses the defect; // draining is the neighboring non-divergent control. A prefix longer than // the source log is excluded by construction. + // Explicit offset/handle options select a provider position, not a cache + // certificate. The source-head law is unchanged across caller cursor forms. const sourcePrefixHistory = fc.record({ key: fc.constantFrom(1, 3), pairs: fc.integer({ min: 1, max: 3 }), @@ -1451,6 +1487,13 @@ describe(`multi-tab Electric over one persisted database`, () => { `missing` as const, `foreign` as const, ), + callerCursor: fc.constantFrom( + `none`, + `initial-offset`, + `handle`, + `resume`, + `now`, + ), }) const runSourcePrefixCampaign = async ( parameters: ReturnType, @@ -1465,6 +1508,10 @@ describe(`multi-tab Electric over one persisted database`, () => { `foreign progress at recovery`, `divergent source restart`, `divergent source reload`, + `explicit resume at divergent recovery`, + `explicit now at divergent recovery`, + `explicit initial offset at recovery`, + `explicit handle at recovery`, ] as const const reach = createOracleReach(`browser.electric-source-prefix`, states) await fc.assert( @@ -1500,6 +1547,10 @@ describe(`multi-tab Electric over one persisted database`, () => { if (history.drain && !divergent) reach.reach(`drained before recovery`) if (!history.drain && divergent) { + if (history.callerCursor === `resume`) + reach.reach(`explicit resume at divergent recovery`) + if (history.callerCursor === `now`) + reach.reach(`explicit now at divergent recovery`) reach.reach(`divergent source ${how}`) reach.reach( `${history.key === 1 ? `present` : `absent`}/${history.advancingBatch}`, @@ -1508,8 +1559,13 @@ describe(`multi-tab Electric over one persisted database`, () => { if (history.progressFormat !== `current`) { reach.reach(`${history.progressFormat} progress at recovery`) } + if (history.callerCursor === `initial-offset`) + reach.reach(`explicit initial offset at recovery`) + if (history.callerCursor === `handle`) + reach.reach(`explicit handle at recovery`) }, history.progressFormat, + history.callerCursor, ) } catch (error) { primary = { error } diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 626959a5de..885d2e2c4b 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -1712,15 +1712,18 @@ function createElectricSync>( const hasIncompatiblePersistedResume = persistedResumeState?.kind === `resume` && persistedResumeState.shapeId !== shapeIdentity + const hasIncompatibleSourceProgress = + persistedResumeState?.kind === `resume` && + expectSourceProgress !== undefined && + (metadata?.collection.get(progressKey) !== sourceProgress || + resumeSnapshot?.getSourceProgressCompatible?.() !== true) // A pre-ledger `unknown` baseline cannot justify a non-initial cursor. // One fresh replacement establishes consistent evidence for later resumes. const lacksCompletePersistedResume = persistedResumeState?.kind === `resume` && ((getKeySetEvidence !== undefined && persistedKeySetEvidence?.status !== `consistent`) || - (expectSourceProgress !== undefined && - (metadata?.collection.get(progressKey) !== sourceProgress || - resumeSnapshot?.getSourceProgressCompatible?.() !== true))) + hasIncompatibleSourceProgress) // Claim expiry can leave a partial managed cache without a resume record. // Its incompatible key set alone requires a fresh source snapshot. const hasIncompatibleManagedCache = @@ -1729,6 +1732,7 @@ function createElectricSync>( const requiresFreshSourceEvidence = !freshScopedGeneration && (hasIncompatibleManagedCache || + hasIncompatibleSourceProgress || (shapeOptions.offset === undefined && shapeOptions.handle === undefined && (persistedResumeState !== undefined || malformedPersistedResume) && @@ -1744,6 +1748,12 @@ function createElectricSync>( requiresFreshSourceEvidence && persistence?.managedCacheGeneration === true && persistence.startScopedRecovery !== undefined + // A provider position cannot certify incompatible source progress. + // Eager recovery overrides the caller's position and replaces its rows. + // Other sync modes keep their existing explicit-position behavior. + const requestedPosition = hasIncompatibleSourceProgress + ? {} + : { offset: shapeOptions.offset, handle: shapeOptions.handle } if (scopedRecovery) lifecycle.forgetEvidenceAfterScopedRecovery() const usesChangesOnlyLog = syncMode === `on-demand` && @@ -1775,7 +1785,8 @@ function createElectricSync>( // Only a complete adapter ledger can justify a persisted cursor. !requiresFreshSourceEvidence const hasExplicitResumeOffset = - shapeOptions.offset !== undefined && shapeOptions.offset !== `-1` + requestedPosition.offset !== undefined && + requestedPosition.offset !== `-1` if (!canUsePersistedResume && !hasExplicitResumeOffset) { clearTagTrackingState() } @@ -1899,14 +1910,14 @@ function createElectricSync>( // In on-demand mode, we only need the changes from the point of time the collection was created // so we default to `now` when there is no saved offset. offset: - shapeOptions.offset ?? + requestedPosition.offset ?? (canUsePersistedResume ? (persistedResumeState.offset as Offset) : usesChangesOnlyLog ? `now` : undefined), handle: - shapeOptions.handle ?? + requestedPosition.handle ?? (canUsePersistedResume ? persistedResumeState.handle : undefined), signal: abortController.signal, onError: async (errorParams) => { From fd3f32985d15c14238e7289c2dd6178168fe6c0b Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 12:26:32 -0600 Subject: [PATCH 08/11] Construct stopped-source peer boundaries in generated histories --- docs/contributing/oracle-coverage.md | 9 +- .../issue-2281-investigation.md | 91 +++++++++++++++++++ .../tests/multi-tab-electric-oracle.test.ts | 46 +++++++++- 3 files changed, 141 insertions(+), 5 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index bb1349f106..4bdba3e7de 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3988,7 +3988,14 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. (fixed seed 952051961 and a random seed, 80 runs each; the shared `TANSTACK_DB_ORACLE_SEED`, `_PATH`, and `_PROPERTY= browser.multi-tab-reserved-history` replay one). The - model folds committed batches into the expected durable rows. After + grammar constructs a 24-step prefix before its unchanged 8–24 arbitrary + steps. Every stop kind occurs in a generated order: a source opens a + transaction, stops, lets a distinct peer commit, then recovers and catches + up. Source tab, initially absent key, and recovery method vary. Four + membership toggles restore the original rows before the arbitrary tail. + This makes stop/peer reach structural; an earlier CI random campaign + missed the SDK-give-up boundary with the unconstrained shuffle. + The model folds committed batches into the expected durable rows. After every step it judges the durable rows, every Collection status, and the rows of each tab with no open transaction; after every tab catches up it judges them again. A Collection status changes only through its diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index 77a549c7a8..824aca9104 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -1126,3 +1126,94 @@ finding is fixed with its generated dimension and receiving evidence. The summary's claim that the reported #2283 trace is outside this repair is stale: that exact row-loss replay already passes here. The #2267 safety/composition qualification remains separate. Raw reviews and dispositions remain local. + +### Construct stop/peer reach before arbitrary interleavings + +CI on `124ce667c61a9e247ea1628991bc72db91d6041e`, workflow `40927860887`, +SQLite-adapters job `123638080436`, found an instrumentation failure. The random +reserved-history campaign reached `peer commit after SDK gives up +mid-transaction` zero times. It reported no incorrect row comparison. The +unconstrained shuffle made the required four-action ordering too rare. + +Every generated history now begins with four six-action stop blocks in a +generated permutation. Each block toggles an initially absent key, opens a +source transaction, stops the source, commits from a distinct peer, recovers the +source, and catches it up. The source tab, key (3 or 4), recovery method, and +stop order vary. Four toggles restore the original membership before the +unchanged 8–24 arbitrary actions. Histories therefore contain 32–48 actions; +both campaigns still accept 80 histories, with fixed seed `952051961` and the +same replay interface. Every pinned history retains its unchanged direct path. + +This grammar excludes a peer equal to the stopped source and does not recover +that source before its peer commits. Removing partial delivery, the stop, or +the peer commit loses the premise. Removing recovery makes the later blocks +act on a stopped source. The arbitrary tail preserves deploys, resets, +interleavings, and the hostile witness below. The independent model, production +driver, comparison checkpoints, exclusions, and all reach requirements remain. + +- **Law:** Browser Laws 1 and 3 require a stopped source to end its reservation + so peer durable rows can apply. Authority: `reserveCommitTurn` and the + maintainer decision on PR #2265. Laws 2, 4, and 5 retain their earlier bounds. +- **Reach:** Fixed/random counts below include histories that later encounter + the documented #2267 discard: 81/80 executed for 80/80 accepted histories. + The constructed stop premises occur before any such discard. +- **RED:** The CI run above rejects the old grammar's missing SDK-give-up reach. + This is coverage RED, not a production row defect. +- **GREEN:** The focused owner command below passes both campaigns and its + associated type checks: three reported tests pass, no type errors. +- **Mutant:** Removing only Electric's SDK-give-up `abandonOpenTransaction` + rejects both campaigns. The fixed campaign shrinks to seed `952051961`, path + `1:0:0:0:0:0:0:0:1:1:0:0`: at step 31, the stopped tab still shows row 1 + while its peer and SQLite have deleted it. The random campaign also rejects + a stale peer row at final delivery (seed `-1481171989`, path + `15:0:0:1:0:1:1:1:1:1:1:1:2:2:2:2`). These are row assertion failures. + The first one-history diagnostic (seed `952051961`, path `0`) survived the + mutant, despite reaching the give-up callback with an open transaction. + It is not a kill. The retained arbitrary tail supplies the distinguishing + stale-row witnesses. Exact source restoration uses SHA-256 + `df8a918e736ca073619a693316c475040db46e2e30fc20fdbcc566934be695e6`. +- **Limits:** This constructs the controlled SDK stop boundary on the Node + Browser coordinator. It adds no real-provider claim or new product behavior. + Existing partial-update, permanent-close, receipt, and teardown limits remain. + +| Declared state | Fixed/random histories | +| --- | --- | +| must-refetch delivered | 55/43 | +| peer commit during an open reserved transaction | 17/10 | +| source stopped with an open transaction | 81/80 | +| tab recovered | 81/80 | +| durable rows behind the server | 81/80 | +| peer commit while a tab was not ready | 81/80 | +| peer commit after error without retry mid-transaction | 81/80 | +| peer commit after SDK gives up mid-transaction | 81/80 | +| peer commit after signal aborted mid-transaction | 81/80 | +| peer reset abandoned an open transaction | 12/13 | +| reset-abandoned commit put the Collection in error | 6/11 | +| reset-abandoned source recovered during final catch-up | 4/3 | + +```sh +pnpm_config_verify_deps_before_run=false pnpm \ + --filter @tanstack/browser-db-sqlite-persistence exec vitest run \ + tests/multi-tab-electric-oracle.test.ts --maxWorkers=2 \ + --disableConsoleIntercept -t 'keeps reserved commits lawful across generated' +``` + +Replay uses `TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history`, +`TANSTACK_DB_ORACLE_SEED=952051961`, and +`TANSTACK_DB_ORACLE_PATH=1:0:0:0:0:0:0:0:1:1:0:0` with that command. +Logs remain local as `ci-reach-green.log`, `ci-reach-abandon-mutant{,-campaign}.log`, +`ci-reach-abandon-path.log`, and `ci-reach-restored-replay.log`. + +The grammar change retains ORC-001–003's authority, independent judgment, and +literate boundaries. The construction and ablations above address ORC-004; +measured production reach and unchanged public comparisons address ORC-005. +The mutant supplies ORC-006/013 sensitivity, and both normal campaigns and direct +replay retain ORC-007. No model state changes (ORC-008), new abstraction +(ORC-009), or named shared semantic fault requiring another formulation +(ORC-011) arise. Existing cleanup and failure preservation retain ORC-010. +This versioned append supplies ORC-012 evidence; the controlled boundary limits +ORC-014's claim. All declared reaches remain enforced (ORC-015), and the file +continues to run shared judgments without example-specific answers (ORC-016). + +The restored shrunk replay passes with no type errors (two reported tests). +No production file remains changed by this calibration. diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 03d561ab71..39a0576d0d 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -948,10 +948,48 @@ describe(`multi-tab Electric over one persisted database`, () => { }), }, ) - const generatedHistory = fc.record({ - tabCount: fc.integer({ min: 2, max: 3 }), - steps: fc.array(historyStep, { minLength: 8, maxLength: 24 }), - }) + // Construct stop/peer boundaries before the arbitrary interleavings. A + // partial delivery opens the source's reservation; its stop must abandon + // that reservation so the peer's next commit can apply (Laws 1 and 3). + // Every stop kind occurs in a generated order, with a distinct running + // peer. Recovery and catch-up separate the blocks. Four membership toggles + // of an initially absent key restore the original rows before the tail. + // This adds 24 steps to the unchanged 8–24-step arbitrary tail. Removing + // the open delivery, stop, or peer commit loses the law's premise; recovering + // before the peer commit would test a different boundary. Existing pinned + // histories still use the same driver without this prefix. + const generatedHistory = fc + .record({ + tabCount: fc.integer({ min: 2, max: 3 }), + source: fc.nat(2), + key: fc.constantFrom(3, 4), + recovery: fc.nat(1), + stopOrder: fc.shuffledSubarray([0, 1, 2, 3], { + minLength: 4, + maxLength: 4, + }), + steps: fc.array(historyStep, { minLength: 8, maxLength: 24 }), + }) + .map(({ tabCount, source, key, recovery, stopOrder, steps }) => { + const tab = source % tabCount + const peer = (tab + 1) % tabCount + return { + tabCount, + steps: [ + ...stopOrder.flatMap( + (pick): Array => [ + { kind: `server change`, key, pick: 0 }, + { kind: `advance`, tab, pick: 0 }, + { kind: `stop`, tab, pick }, + { kind: `catch up`, tab: peer, pick: 0 }, + { kind: `recover`, tab, pick: recovery }, + { kind: `catch up`, tab, pick: 0 }, + ], + ), + ...steps, + ], + } + }) type GeneratedHistory = { tabCount: number; steps: Array } type ServerLog = { From 623c599555516e6182b2b0d04a48307ed03d49b5 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Thu, 29 Oct 2026 18:28:46 +0000 Subject: [PATCH 09/11] ci: apply automated fixes --- .../tests/multi-tab-electric-oracle.test.ts | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 39a0576d0d..bc8435ce23 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -976,16 +976,14 @@ describe(`multi-tab Electric over one persisted database`, () => { return { tabCount, steps: [ - ...stopOrder.flatMap( - (pick): Array => [ - { kind: `server change`, key, pick: 0 }, - { kind: `advance`, tab, pick: 0 }, - { kind: `stop`, tab, pick }, - { kind: `catch up`, tab: peer, pick: 0 }, - { kind: `recover`, tab, pick: recovery }, - { kind: `catch up`, tab, pick: 0 }, - ], - ), + ...stopOrder.flatMap((pick): Array => [ + { kind: `server change`, key, pick: 0 }, + { kind: `advance`, tab, pick: 0 }, + { kind: `stop`, tab, pick }, + { kind: `catch up`, tab: peer, pick: 0 }, + { kind: `recover`, tab, pick: recovery }, + { kind: `catch up`, tab, pick: 0 }, + ]), ...steps, ], } From 0ec58057de52162da0eb4598616c2fc2eed2129b Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 12:42:10 -0600 Subject: [PATCH 10/11] Honor reset markers during explicit eager recovery Law: An interrupted eager replacement requires a fresh snapshot on the next start. Authority: Electric cleanup/resume safety and Browser Law2/#2209. Reach: 80 paired inputs per fixed/random source-prefix campaign; actual reset before explicit numeric recovery10/6, now7/8. All earlier reaches retained. RED: Extended Node owner fails at final source head on623c59955 (seed2281/path1:0:0). Ten real interrupted receiving cells give four numeric/now wrong-row failures and six passing controls before production edits. GREEN: Browser312 pass/two tracked expected failures; Electric336 and97 pass; Browser types and all30 real SDK/HTTP/PostgreSQL/Chromium OPFS cells pass. Mutant: Keep reset recovery admission but retain caller position. Same replay rejects missing row1; exact production restoration makes it pass. Limits: Bounded same-version eager Browser convergence. Other modes, arbitrary cursor selection, tag/shape/malformed states, native hosts and teardown retain their existing coverage-map limits. No model or comparison relaxation. Versioned evidence: docs/contributing/oracle-reviews/issue-2281-investigation.md. --- docs/collections/electric-collection.md | 2 + docs/contributing/oracle-coverage.md | 20 +- .../issue-2281-investigation.md | 132 +++++++++++++ ...electric-source-prefix-oracle.opfs.spec.ts | 58 +++++- .../e2e/electric-source-prefix.opfs.ts | 8 +- .../tests/multi-tab-electric-oracle.test.ts | 182 ++++++++++++------ .../electric-db-collection/src/electric.ts | 13 +- 7 files changed, 341 insertions(+), 74 deletions(-) diff --git a/docs/collections/electric-collection.md b/docs/collections/electric-collection.md index fac813a74f..a7bc97cc5a 100644 --- a/docs/collections/electric-collection.md +++ b/docs/collections/electric-collection.md @@ -381,6 +381,8 @@ An eager persisted resume also needs compatible source progress when its adapter supplies that capability. Missing or incompatible progress requires a fresh snapshot, even with an explicit offset or handle. Those options do not certify the cached rows. +An interrupted eager recovery keeps its durable reset marker. The next start +also requires a fresh snapshot, even with an explicit offset or handle. Eager and progressive recovery fetch a full snapshot. Cached rows remain visible until that replacement completes; a partial batch or subset completion cannot diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 4bdba3e7de..51d375dcf6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4076,9 +4076,23 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. requests recovery but preserves the caller position fails the final Node source-head comparison (seed 2281, path `1:0:0`). Explicit positions cannot bypass eager source-progress recovery. Other sync modes and explicit - malformed/reset/tag/shape regimes retain their earlier behavior and are - outside this added evidence. Real foreign-token provenance and changing - caller forms within one history still need separate receiving witnesses. + malformed/tag/shape regimes retain their earlier behavior and are + outside this added evidence. Real foreign-token provenance and arbitrary + changes of caller form still need separate receiving witnesses. + Each of the source-prefix property's 80 generated inputs now runs both + its original history and an interrupted-recovery neighbor (160 histories + per campaign). The latter starts recovery without a caller cursor, stops + before any source delivery, and reopens with the generated caller form. + Actual persisted reset markers before divergent numeric/`now` recovery + are required reaches: fixed/random counts are 10/6 and 7/8. Both campaigns + fail on `623c59955` before the eager-only reset fix, and pass afterward. + The real matrix adds ten interrupted histories across both membership + directions and all five caller forms. A held first SDK response prevents + replacement publication; the driver observes the durable reset before + closing that run. Numeric/`now` give four wrong-row RED comparisons; + six neighboring controls pass. All thirty real cells pass after the fix. + The reset marker now requires eager recovery and clears the effective + caller position. No expectation, model rule, or comparison is relaxed. This is bounded same-version eager Browser evidence, not arbitrary scheduling, mixed client versions, native-device, Electron, on-demand, optimistic, or monotonic-live-write coverage. diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index 824aca9104..8da3ed228b 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -1217,3 +1217,135 @@ continues to run shared judgments without example-specific answers (ORC-016). The restored shrunk replay passes with no type errors (two reported tests). No production file remains changed by this calibration. + +### Interrupted eager recovery preserves its reset obligation + +CodeRabbit review `5702891774` on `124ce667c` identified the next missing +transition: recovery can stop after its reset marker commits but before a +replacement snapshot arrives. The next start selected `kind: reset`, which +bypassed the resume-only source-progress predicate. An explicit numeric position +or `now` could then request only incremental changes over the wrong cached rows. + +This violates the existing eager replacement and next-sync reset contracts in +`docs/collections/electric-collection.md`, under “Cleanup and resume safety.” +It requires no new source-ordering law and does not change #2267's safety guard. +The independent review agrees with that authority and identifies the receiving +seams below. No model rule or comparison changes. + +Each source-prefix generated input now runs its original uninterrupted history +and an interrupted neighbor. The neighbor starts recovery without explicit +options, stops before any source delivery, and recovers again with the selected +caller form. Production writes the reset; the driver reads its actual durable +kind before the second start. All original axes remain. Each campaign still +uses 80 inputs, now 160 histories. Original histories contain 6–13 actions; +interrupted histories add two actions. Numeric offsets without handles remain +SDK-invalid. Omitting the first recovery loses the reset premise; completing +its delivery loses the unfinished replacement. The unchanged draining neighbor +keeps the source-head comparison independently sensitive to missing/extra rows. + +The real driver holds the first actual SDK response of the pending recovery +before delivery. It observes the durable reset and closes that run. Caller +positions are captured before the reset removes their fields. The surviving +peer's responses are held too, preventing an empty commit from replacing the +marker. The next page uses the captured numeric position or `now`; PostgreSQL +still supplies the head to the unchanged shared comparison. Ten added cells +cross both membership directions and all five caller forms. They preserve all +20 earlier receiving cells. + +The eager-only reset predicate now participates in both fresh-evidence admission +and effective-position selection. Thus the requested source position and +snapshot replacement agree. Progressive/on-demand explicit-position behavior +and eager resumes with no incompatible/reset evidence retain their prior path. + +- **Law:** After successful eager recovery and complete delivery, durable rows + and every open Collection match the source head. An interrupted replacement's + reset still requires a full snapshot at the next start, including explicit + caller positions. Authority: the existing public recovery contract above and + Browser Law 2/#2209. +- **Reach:** Actual reset before divergent numeric/`now` recovery reaches 10/6 + and 7/8 fixed/random histories. All 13 earlier source-prefix reaches pass; + their final counts appear below. Each added real cell reaches actual durable + reset before interruption and all its original/explicit required states once. +- **RED:** Before production edits, both paired Node campaigns fail the final + `after every tab caught up` comparison with row 1 missing. Fixed seed `2281`, + path `1:0:0`; random seed `-636405297`, path `14:0:0:0:0`. The interrupted + real matrix reports four concrete wrong-row comparisons (numeric/`now` × + lost/resurrected), while six none/initial/handle controls pass. Production + SHA-256 was still `df8a918e736ca073619a693316c475040db46e2e30fc20fdbcc566934be695e6`. + Bot commit `623c59955` only formatted the earlier stop grammar and was + fast-forwarded without dropping any witness or production change. +- **GREEN:** The full Browser owner passes 312 tests and two tracked expected + failures, with no type errors. Browser `tsc --noEmit` passes. Electric ordinary, + recovery, and SDK suites report 336 passes; supplemental recovery, resume + races, and SDK grammar report 97 passes. Both commands report no type errors. + All 30 real receiving cells pass, including all ten interruptions. +- **Mutant:** Retain eager reset recovery admission but remove reset from the + effective-position override. Replay seed `2281`, path `1:0:0` rejects the + missing row at the final source-head comparison. Exact source restoration + uses SHA-256 `46a417f2d65ccc21adcc72f56d80388d6f2f9fc32a172ab6e3fa1e4702508047`. +- **Limits:** This covers bounded eager startup/restart/reopen after interrupted + recovery. It does not decide progressive/on-demand explicit-reset policy, + malformed/tagged/changed-shape explicit resumes, arbitrary cursor selection + over certified rows, real foreign-token receiving, mixed clients, native + hosts, permanent closure, receipt fulfillment, or clean teardown. + +| Source-prefix state | Fixed/random histories | +| --- | --- | +| present/empty | 28/26 | +| present/disjoint | 22/14 | +| absent/empty | 12/34 | +| absent/disjoint | 14/18 | +| drained before recovery | 84/68 | +| missing progress at recovery | 52/64 | +| foreign progress at recovery | 42/56 | +| divergent source restart | 50/52 | +| divergent source reload | 26/40 | +| explicit resume at divergent recovery | 20/12 | +| explicit now at divergent recovery | 14/16 | +| explicit initial offset at recovery | 34/42 | +| explicit handle at recovery | 42/32 | +| interrupted reset before explicit resume recovery | 10/6 | +| interrupted reset before explicit now recovery | 7/8 | + +The broad reserved-history campaign also passes every reach at the original +80-input budgets. Its fixed counts match the earlier constructed-grammar run. +Random counts are refetch 50, open-reservation peer 20, each constructed stop +boundary 80, peer reset 14, reset failure 5, and final reset recovery 2. +The first fixed campaign drives 81 histories including one documented #2267 +discard; its 80 accepted histories remain. No exclusion or reach minimum changes. + +Commands retain the earlier owner invocations. Node RED selects +`-t 'converges after interleaved source prefixes'`; final Browser GREEN omits +that selector. Real RED selects `--grep 'after interrupted recovery'`; final +real GREEN omits it. Replay uses property `browser.electric-source-prefix`, +seed `2281`, and path `1:0:0`. Logs are `reset-cursor-node-red.log`, +`reset-cursor-real-red.log`, `reset-cursor-browser-green.log`, +`reset-cursor-real-green.log`, `reset-cursor-electric{,-neighbors}-green.log`, +`reset-cursor-browser-types.log`, `reset-cursor-incomplete-mutant.log`, and +`reset-cursor-restored-replay.log`. The final production bundle delta is 573 +Gzip bytes across three packages, including 168 for Electric. The typical app +delta remains 45 bytes. The comparison uses merge-base `672684846` of fetched +main `bea84cd4554b7bd49df50cb0c569a34d8b2cd915`. + +The paired histories preserve reconstruction and direct replay (ORC-004/007). +Production-observed reset plus the shared comparison and incomplete-fix mutant +supply ORC-005/006/013/015 evidence. The real held-response receiving path supplies +ORC-014. Authority, independence, literate boundaries, cleanup, and law-only tests +retain ORC-001–003/010/016. No model state changes, new abstraction, or named +shared semantic fault trigger new ORC-008/009/011 work. This versioned append and +its finite limits supply ORC-012 evidence. + +Review maintenance items: the receiver now derives its offset type through +public `ElectricCollectionConfig`, avoiding the sibling `node_modules` path +without approximating the SDK's bigint offset form. The earlier phrase +“test-driver type import” means a type import in the test driver. Historical +validation wording and counts remain intact. The review's repeated reset risk, +aggregate docstring warning, and related-issue prose retain separate local +ledger entries; no review claim or pilot boundary disappears. + +After exact source restoration, the same shrunk replay passes (two reported +tests, no type errors). Lint and whitespace checks pass. The fourth review's +loss audit retains 24 items: eight fixed, four refuted, one accepted design, +and eleven duplicates. No in-scope finding remains open. External review covers +`124ce667c`; its automatic review is paused, so subsequent semantic changes +have the independent local source review and executed evidence above. diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts index 05d8515f53..1fa30afbf0 100644 --- a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts @@ -21,6 +21,7 @@ type Cell = { drain: boolean legacy: boolean callerCursor: `none` | `initial-offset` | `handle` | `resume` | `now` + interrupted?: boolean } const cells: Array = [true, false] .flatMap((present) => @@ -66,6 +67,22 @@ cells.push( callerCursor, })), ) +// Interrupt recovery before the first SDK response can publish its snapshot. +// The real eager adapter durably records reset; numeric/now positions cannot +// turn that marker into a certified baseline. The other forms are controls. +cells.push( + ...([`none`, `initial-offset`, `handle`, `resume`, `now`] as const).flatMap( + (callerCursor) => + [true, false].map((present): Cell => ({ + present, + advance: `disjoint`, + drain: false, + legacy: false, + callerCursor, + interrupted: true, + })), + ), +) const observe = (page: Page) => page.evaluate(() => window.__electricSourcePrefix!.observe()) const rowIds = async (page: Page) => @@ -73,7 +90,7 @@ const rowIds = async (page: Page) => declareOracle({ check: [sourceHeadViolation], grammar: [cells] }) for (const cell of cells) { - test(`recovers ${cell.present ? `lost` : `resurrected`} rows after ${cell.advance} advancement${cell.drain ? ` with peer drain` : ``}${cell.legacy ? ` from legacy progress` : ``}${cell.callerCursor !== `none` ? ` with explicit ${cell.callerCursor}` : ``}`, async ({ + test(`recovers ${cell.present ? `lost` : `resurrected`} rows after ${cell.advance} advancement${cell.drain ? ` with peer drain` : ``}${cell.legacy ? ` from legacy progress` : ``}${cell.callerCursor !== `none` ? ` with explicit ${cell.callerCursor}` : ``}${cell.interrupted ? ` after interrupted recovery` : ``}`, async ({ context, }) => { const reach = createOracleReach( @@ -83,6 +100,7 @@ for (const cell of cells) { `stale commit applied`, `later advancement applied`, `reopened source compared`, + ...(cell.interrupted ? [`interrupted recovery reset persisted`] : []), ...(cell.callerCursor === `none` ? [] : [`explicit ${cell.callerCursor} supplied at recovery`]), @@ -122,14 +140,23 @@ for (const cell of cells) { await pg.query(`INSERT INTO public.${table} VALUES ('anchor', 'anchor')`) if (cell.present) await pg.query(`INSERT INTO public.${table} VALUES ('target', 'old')`) - const open = async (callerPosition: Record = {}) => { + const open = async ( + callerPosition: Record = {}, + holdInitial = false, + ) => { const page = await context.newPage() pages.push(page) page.on(`pageerror`, (error) => diagnostics.push(error.message)) - await page.goto(`${url}&${new URLSearchParams(callerPosition)}`) + await page.goto( + `${url}&${new URLSearchParams({ + ...callerPosition, + ...(holdInitial ? { holdInitial: `true` } : {}), + })}`, + ) await page.waitForFunction( () => window.__electricSourcePrefix !== undefined, ) + if (holdInitial) return page await expect .poll(async () => (await observe(page)).status, { timeout: 30_000 }) .toBe(`ready`) @@ -241,6 +268,31 @@ for (const cell of cells) { if (cell.callerCursor === `now`) callerPosition.offset = `now` reach.reach(`explicit ${cell.callerCursor} supplied at recovery`) } + if (cell.interrupted) { + // Preserve the real position above before recovery replaces its + // resume record with reset. Hold A too, so an empty peer response + // cannot overwrite that marker while this boundary is observed. + await a.evaluate(() => window.__electricSourcePrefix!.arm()) + const pendingRecovery = await open({}, true) + await expect + .poll(async () => (await observe(pendingRecovery)).pending.length, { + timeout: 30_000, + }) + .toBeGreaterThan(0) + await expect + .poll( + async () => { + const { resume } = await observe(pendingRecovery) + return resume && typeof resume === `object` && `kind` in resume + ? resume.kind + : undefined + }, + { timeout: 30_000 }, + ) + .toBe(`reset`) + reach.reach(`interrupted recovery reset persisted`) + await close(pendingRecovery) + } const reopened = await open(callerPosition) // Source readiness follows an applied SDK receipt. The wrapper may // still propagate its notice to A, so poll the shared settled judgment. diff --git a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts index bbac28588d..1912479240 100644 --- a/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts +++ b/packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts @@ -10,7 +10,11 @@ import { openBrowserWASQLiteOPFSDatabase, persistedCollectionOptions, } from '../src/index' -import type { Offset } from '../../electric-db-collection/node_modules/@electric-sql/client' +import type { ElectricCollectionConfig } from '../../electric-db-collection/src/electric' + +type Offset = NonNullable< + ElectricCollectionConfig[`shapeOptions`][`offset`] +> type Item = { id: string; label: string } const parameters = new URL(location.href).searchParams @@ -19,7 +23,7 @@ const collectionId = parameters.get(`collectionId`)! const table = parameters.get(`table`)! if (!databaseId || !collectionId || !table) throw new Error(`Missing prefix driver parameters`) -let hold = false +let hold = parameters.get(`holdInitial`) === `true` let refresh = false const pending: Array<{ release: () => void; rows: number }> = [] const requests: Array<{ offset: string | null; handle: string | null }> = [] diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index bc8435ce23..f109c45c08 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -414,6 +414,20 @@ function openHost() { openTab, quiesce, commitNotices: () => commitNotices, + hasResetMarker: async () => { + observer ??= createWASQLiteTestDatabase({ filename, timeout: 1000 }) + const records = await observer.execute<{ value: string }>( + `SELECT value FROM collection_metadata WHERE collection_id = ? AND key = ?`, + [`items`, `electric:resume`], + ) + const value: unknown = JSON.parse(records[0]?.value ?? `null`) + return ( + value !== null && + typeof value === `object` && + `kind` in value && + value.kind === `reset` + ) + }, // A pre-protocol cache or mismatched provenance record still has rows // and an Electric cursor. Reopen must not trust that cursor blindly. rewriteSourceProgress: async (format: `missing` | `foreign`) => { @@ -895,7 +909,12 @@ describe(`multi-tab Electric over one persisted database`, () => { // only the pinned #2209 replay does (see below). | { kind: `close`; tab: number } | { kind: `stop`; tab: number; pick: number } - | { kind: `recover`; tab: number; pick: number } + | { + kind: `recover` + tab: number + pick: number + callerCursor?: CallerCursor + } const historyStep: fc.Arbitrary = fc.oneof( { weight: 3, @@ -1067,6 +1086,8 @@ describe(`multi-tab Electric over one persisted database`, () => { onRecovery?: ( divergentBeforeRecovery: boolean, how: `restart` | `reload`, + cursor: CallerCursor, + hadResetMarker: boolean, ) => void, progressFormat: `current` | `missing` | `foreign` = `current`, callerCursor: CallerCursor = `none`, @@ -1201,13 +1222,18 @@ describe(`multi-tab Electric over one persisted database`, () => { } await activeHost.quiesce(`tab ${tab.tab} delivered`) } - const recover = async (index: number, how: `restart` | `reload`) => { + const recover = async ( + index: number, + how: `restart` | `reload`, + cursor = callerCursor, + ) => { if (progressFormat !== `current`) { await activeHost.rewriteSourceProgress(progressFormat) progressFormat = `current` } const divergentBeforeRecovery = JSON.stringify(rowsOf(durable)) !== JSON.stringify(rowsOf(log.rows)) + const hadResetMarker = await activeHost.hasResetMarker() const stopped = tabs[index]! const streamsBefore = stopped.streamCount() // A caller can repeat the current source position, but that position @@ -1215,16 +1241,16 @@ describe(`multi-tab Electric over one persisted database`, () => { // are legal full-snapshot neighbors of the explicit resumed request. const head = position(log, serverVersion) const callerPosition: ExplicitPosition = - callerCursor === `resume` + cursor === `resume` ? { handle: head.handle, offset: head.offset as `${number}_${number}`, } - : callerCursor === `handle` + : cursor === `handle` ? { handle: head.handle } - : callerCursor === `initial-offset` + : cursor === `initial-offset` ? { offset: `-1` } - : callerCursor === `now` + : cursor === `now` ? { offset: `now` } : {} let recovered = stopped @@ -1250,7 +1276,7 @@ describe(`multi-tab Electric over one persisted database`, () => { ...requested, }) await activeHost.quiesce(`tab ${recovered.tab} recovered`) - onRecovery?.(divergentBeforeRecovery, how) + onRecovery?.(divergentBeforeRecovery, how, cursor, hadResetMarker) } const check = async (label: string, head?: Array) => { @@ -1400,7 +1426,11 @@ describe(`multi-tab Electric over one persisted database`, () => { (step.kind === `recover` && step.pick === 1) ? `reload` : `restart` - await recover(index, how) + await recover( + index, + how, + step.kind === `recover` ? step.callerCursor : undefined, + ) } else if (step.kind === `stop`) { const how = stops[step.pick % stops.length]! if (model.open !== undefined) { @@ -1548,65 +1578,93 @@ describe(`multi-tab Electric over one persisted database`, () => { `explicit now at divergent recovery`, `explicit initial offset at recovery`, `explicit handle at recovery`, + `interrupted reset before explicit resume recovery`, + `interrupted reset before explicit now recovery`, ] as const const reach = createOracleReach(`browser.electric-source-prefix`, states) await fc.assert( fc.asyncProperty(sourcePrefixHistory, async (history) => { - reach.beginHistory() const slowTab = 1 - history.fastTab - const steps: Array = [ - ...Array.from({ length: history.pairs * 2 }, (): HistoryStep => ({ - kind: `server change`, - key: history.key, - pick: 0, - })), - { kind: `catch up`, tab: history.fastTab, pick: 0 }, - { kind: `catch up`, tab: slowTab, pick: 1 }, - ...(history.advancingBatch === `disjoint` - ? [{ kind: `server change` as const, key: 2, pick: 1 }] - : []), - { kind: `catch up`, tab: history.fastTab, pick: 0 }, - ...(history.drain - ? [{ kind: `catch up` as const, tab: slowTab, pick: 0 }] - : []), - ...(history.recovery === 0 - ? [{ kind: `stop` as const, tab: slowTab, pick: 0 }] - : []), - { kind: `recover`, tab: slowTab, pick: history.recovery }, - ] - let primary: { error: unknown } | undefined - try { - await runGeneratedHistory( - { tabCount: 2, steps }, - () => fc.pre(false), - (divergent, how) => { - if (history.drain && !divergent) - reach.reach(`drained before recovery`) - if (!history.drain && divergent) { - if (history.callerCursor === `resume`) - reach.reach(`explicit resume at divergent recovery`) - if (history.callerCursor === `now`) - reach.reach(`explicit now at divergent recovery`) - reach.reach(`divergent source ${how}`) - reach.reach( - `${history.key === 1 ? `present` : `absent`}/${history.advancingBatch}`, - ) - } - if (history.progressFormat !== `current`) { - reach.reach(`${history.progressFormat} progress at recovery`) - } - if (history.callerCursor === `initial-offset`) - reach.reach(`explicit initial offset at recovery`) - if (history.callerCursor === `handle`) - reach.reach(`explicit handle at recovery`) - }, - history.progressFormat, - history.callerCursor, - ) - } catch (error) { - primary = { error } - } finally { - await finishHistory(primary) + // Keep every original history, then repeat it with a recovery that + // stops before its first source delivery. The first start writes its + // reset marker through production; the second must still recover. + // Pairing both paths preserves the uninterrupted domain and gives + // every generated cursor form an interrupted neighbor. + for (const interrupted of [false, true]) { + reach.beginHistory() + const steps: Array = [ + ...Array.from({ length: history.pairs * 2 }, (): HistoryStep => ({ + kind: `server change`, + key: history.key, + pick: 0, + })), + { kind: `catch up`, tab: history.fastTab, pick: 0 }, + { kind: `catch up`, tab: slowTab, pick: 1 }, + ...(history.advancingBatch === `disjoint` + ? [{ kind: `server change` as const, key: 2, pick: 1 }] + : []), + { kind: `catch up`, tab: history.fastTab, pick: 0 }, + ...(history.drain + ? [{ kind: `catch up` as const, tab: slowTab, pick: 0 }] + : []), + ...(history.recovery === 0 + ? [{ kind: `stop` as const, tab: slowTab, pick: 0 }] + : []), + ...(interrupted + ? [ + { + kind: `recover` as const, + tab: slowTab, + pick: history.recovery, + callerCursor: `none` as const, + }, + { kind: `stop` as const, tab: slowTab, pick: 0 }, + ] + : []), + { kind: `recover`, tab: slowTab, pick: history.recovery }, + ] + let primary: { error: unknown } | undefined + try { + await runGeneratedHistory( + { tabCount: 2, steps }, + () => fc.pre(false), + (divergent, how, cursor, hadResetMarker) => { + if (history.drain && !divergent) + reach.reach(`drained before recovery`) + if (!history.drain && divergent) { + if (cursor === `resume`) + reach.reach(`explicit resume at divergent recovery`) + if (cursor === `now`) + reach.reach(`explicit now at divergent recovery`) + if ( + interrupted && + hadResetMarker && + (cursor === `resume` || cursor === `now`) + ) + reach.reach( + `interrupted reset before explicit ${cursor} recovery`, + ) + reach.reach(`divergent source ${how}`) + reach.reach( + `${history.key === 1 ? `present` : `absent`}/${history.advancingBatch}`, + ) + } + if (history.progressFormat !== `current`) { + reach.reach(`${history.progressFormat} progress at recovery`) + } + if (cursor === `initial-offset`) + reach.reach(`explicit initial offset at recovery`) + if (cursor === `handle`) + reach.reach(`explicit handle at recovery`) + }, + history.progressFormat, + history.callerCursor, + ) + } catch (error) { + primary = { error } + } finally { + await finishHistory(primary) + } } }), parameters, diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 885d2e2c4b..6fb538f0f4 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -1717,6 +1717,8 @@ function createElectricSync>( expectSourceProgress !== undefined && (metadata?.collection.get(progressKey) !== sourceProgress || resumeSnapshot?.getSourceProgressCompatible?.() !== true) + const hasPersistedReset = + syncMode === `eager` && persistedResumeState?.kind === `reset` // A pre-ledger `unknown` baseline cannot justify a non-initial cursor. // One fresh replacement establishes consistent evidence for later resumes. const lacksCompletePersistedResume = @@ -1733,6 +1735,7 @@ function createElectricSync>( !freshScopedGeneration && (hasIncompatibleManagedCache || hasIncompatibleSourceProgress || + hasPersistedReset || (shapeOptions.offset === undefined && shapeOptions.handle === undefined && (persistedResumeState !== undefined || malformedPersistedResume) && @@ -1748,12 +1751,14 @@ function createElectricSync>( requiresFreshSourceEvidence && persistence?.managedCacheGeneration === true && persistence.startScopedRecovery !== undefined - // A provider position cannot certify incompatible source progress. + // A provider position cannot certify incompatible source progress or + // finish an interrupted eager replacement recorded by a reset marker. // Eager recovery overrides the caller's position and replaces its rows. // Other sync modes keep their existing explicit-position behavior. - const requestedPosition = hasIncompatibleSourceProgress - ? {} - : { offset: shapeOptions.offset, handle: shapeOptions.handle } + const requestedPosition = + hasIncompatibleSourceProgress || hasPersistedReset + ? {} + : { offset: shapeOptions.offset, handle: shapeOptions.handle } if (scopedRecovery) lifecycle.forgetEvidenceAfterScopedRecovery() const usesChangesOnlyLog = syncMode === `on-demand` && From 381f41e2034fb5fd3c115a0ab9312e46911f6398 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 14:25:33 -0600 Subject: [PATCH 11/11] docs: condense Electric recovery investigation evidence --- .../issue-2281-investigation.md | 1566 +++-------------- 1 file changed, 227 insertions(+), 1339 deletions(-) diff --git a/docs/contributing/oracle-reviews/issue-2281-investigation.md b/docs/contributing/oracle-reviews/issue-2281-investigation.md index 8da3ed228b..1b0bc305ac 100644 --- a/docs/contributing/oracle-reviews/issue-2281-investigation.md +++ b/docs/contributing/oracle-reviews/issue-2281-investigation.md @@ -1,1351 +1,239 @@ -# Issue #2281 investigation - -Investigated on 2026-10-29 against production commit -`82c5f999b32c9db827dd74e3f172604dfda28013`. The working branch is -`codex/investigate-2281`. -The investigation sections below preserve the original findings. The appended -implementation evidence records the subsequent repair and supersedes their -open-work status. - -This record describes an investigation, not a production repair or a claim -that the bug class is closed. Production and the generated campaign are -unchanged. The test additions retain two reduced counterexamples and four -passing controls through the existing model, driver, and comparison. - -Read the [issue and both comments](https://github.com/TanStack/db/issues/2281). -There are two different accounts of the multi-tab failures: - -1. The original reset/deploy trace reaches a premature convergence checkpoint. - Its final deliveries fail the sources; the driver does not recover them - afterward. Row 1 never leaves durable storage in this trace. -2. Both later multi-tab traces expose a production defect: a lagging source - changes the shared rows, then another source stores a newer resume position - without repairing those rows. A reopened source resumes beyond the missing - change. This is a concrete extension of the ordering problem in - [#2209](https://github.com/TanStack/db/issues/2209). - -The separate sync-metadata trace also reproduces. Its root cause was not -investigated here; no evidence joins it to the source-position defect. - -## Authority and observation - -Law 2 in `multi-tab-electric-oracle.test.ts` promises that durable rows and -every open eager tab equal the server head after the streams catch up. Its -authority is the persisted multi-tab eventual-consistency contract recorded -in the coverage map and #2209. Temporary regressions during interleaving are -allowed. Permanent divergence after recovery and catch-up is a violation. - -Law 5 in the same owner requires a peer reset to abandon older reservations. -Electric's next commit of such a transaction fails its source, and the app -recovers by restarting the tab. That maintainer decision on PR #2265 matters -to the original trace: a failed commit cannot establish catch-up. - -All experiments use the existing production driver: one SQLite file, a -better-sqlite3 connection per tab, the real Browser coordinator with Node -BroadcastChannel/Web Locks, the Browser persistence factory, persisted -Collections, and Electric with a controlled ShapeStream. The existing check -compares durable rows, public rows, and Collection status after each step -and at the final checkpoint. Temporary instrumentation additionally read -`collection_metadata['electric:resume']` through an observer connection and -recorded each recovered stream's requested handle/offset. - -## Original trace: the final delivery creates the failure - -Seed `-1115798760`, path `1:4:3:3:3:3:3:3:3:3:3` reproduces the issue's exact -final mismatch. Both tabs stage row 1's delete without committing, then the -peer reset abandons both reservations. Durable rows remain the initial rows -1 and 2 through every numbered step. There is no durable deletion followed -by a durable resurrection in this execution. - -The final loop checks `failedByReset` before `deliver()`. At that point it is -false: neither abandoned transaction has tried to commit yet. Delivery -encounters must-refetch and eventually attempts the abandoned transaction's -commit. It fails with `AbortError: Sync transaction was aborted before -application`. Both tabs stay ready with source errors, as Law 5 requires. -Only now does the model set `failedByReset = true`. The loop ends and checks -the server head, without restarting either failed source. - -A temporary driver intervention added the following immediately after the -final loop's existing `await deliver(index, 'all', true)`: - -```ts -if (models[index]!.failedByReset) { - await recover(index, `restart`) - await deliver(index, `all`, true) -} -``` - -With unchanged production, the original replay passes. The recorded original -history shows both restarted streams request offset `-1`; both accept the -fresh snapshot, and durable rows and both ready tabs contain only row 2, -with no source error. The intervention was removed after the experiment. -It establishes this trace's missing recovery, not a general driver repair. -A permanent checkpoint repair must retain the comparison of the failed -source state and then establish successful recovery before checking Law 2. - -## Later traces: rows and the resume position describe different prefixes - -The resurrection in the 2026-10-29 comment reproduces with seed `-699129038`, -path `78:2:0:3:9:12:12:13:12:13:13:13:13:14:14:14`: - -| Step | Durable row 3 | Durable offset | +# Electric/SQLite source-progress recovery: issue #2281 + +This record summarizes the repair in [PR #2298](https://github.com/TanStack/db/pull/2298), +verified through implementation commit `0ec58057de52162da0eb4598616c2fc2eed2129b`. +The original investigation used production commit `82c5f999b32c9db827dd74e3f172604dfda28013`. +The [full investigation chronology](https://github.com/TanStack/db/blob/0ec58057de52162da0eb4598616c2fc2eed2129b/docs/contributing/oracle-reviews/issue-2281-investigation.md) +retains superseded results, intermediate manifests, and review corrections. +This summary changes no production code, model, history grammar, comparison, +exclusion, or run budget. + +## Law and failure + +After successful source recovery and complete delivery, durable rows and every +open eager Collection must equal the source head. Authority: Law 2 in the +[Browser multi-tab owner](../../../packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts), +the [coverage map](../oracle-coverage.md), and [#2209](https://github.com/TanStack/db/issues/2209). +Temporary arrival-order regressions remain allowed. The +[Electric recovery contract](../../collections/electric-collection.md#cleanup-and-resume-safety) +requires a fresh baseline after reset. Under the maintainer decision on #2265, +a reset-abandoned commit fails its source; successful catch-up requires restart. + +The row-loss and resurrection traces share this mechanism: + +1. Tab A commits two source changes to one key, reaching position 2. +2. Lagging tab B commits only the first change, restoring an older row state. +3. A advances shared resume metadata to position 2 with an empty batch or a + batch affecting another key. The affected row stays stale. +4. Both tabs restart beyond the missing corrective change, so catch-up cannot + repair the row. + +Atomic rows and metadata within each commit do not establish that commits from +separate streams describe one source history. Keeping the maximum offset still +permits stale row writes. Suppressing empty commits misses disjoint-key batches. +Physical key integrity also cannot certify source progress. + +The two reset traces exposed a separate oracle checkpoint defect. Final delivery +could first fail a reset-abandoned transaction after the initial restart sweep. +The driver now compares that failed state, restarts the newly failed source, +and completes delivery before judging convergence. This preserves the failure +contract; it does not weaken the source-head comparison. + +All four reported traces pass through the shared model, driver, and comparison: + +| Trace | Seed | Shrink path | | --- | --- | --- | -| A commits insert(3), delete(3) | absent | `2_0` | -| B commits only the older insert(3) | `v1` | `1_0` | -| A commits delete(1), insert(1) | still `v1` | `4_0` | -| B reopens, then A reopens | still `v1` | both request `4_0` | -| Both streams catch up with no remaining changes | still `v1` | `4_0` | - -The server deleted row 3 at offset `2_0`. B's late insert reintroduces it. -A's next batch touches only row 1, so it advances the global resume position -without removing row 3. Both new streams skip its delete. Unlike the original -reset trace, this reaches the final comparison with healthy, caught-up -streams and permanently wrong rows. - -The row-loss comment reproduces with seed `-1660880085`, path -`55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`. A commits delete(1) at `1_0`, then insert(1) -at `2_0`. B's older delete removes row 1 and sets the durable offset to `1_0`. -A then receives only up-to-date at its existing `2_0`: its metadata-only -transaction advances the durable offset to `2_0` while leaving row 1 absent. -Both tabs stop and restart from `2_0`, so neither receives the missing insert. -The loss happens at B's stale commit, before either stop. - -## Six-step reductions and controls - -After warming both tabs with rows 1 and 2, use this shared history: - -```ts -[ - { kind: `server change`, key, pick: 0 }, - { kind: `server change`, key, pick: 0 }, - { kind: `catch up`, tab: 0, pick: 0 }, - { kind: `catch up`, tab: 1, pick: 1 }, - { kind: `catch up`, tab: 0, pick: 0 }, - { kind: `recover`, tab: 1, pick: 0 }, -] -``` - -For initially present key 1, the server changes are delete then insert. For -initially absent key 3, they are insert then delete. In each case A commits -through `2_0`, B commits only `1_0`, A stores `2_0` with an empty row batch, -and B reopens from `2_0`. Both cases fail the unchanged final comparison: -key 1 is missing, or key 3 survives with `label: 'v1'`. - -| Continuation after B's older commit | Key 1 | Key 3 | +| Original reset/deploy | `-1115798760` | `1:4:3:3:3:3:3:3:3:3:3` | +| Missing row | `-1660880085` | `55:0:1:4:4:4:1:1:1:1:3:4:4:4:4` | +| Resurrected row | `-699129038` | `78:2:0:3:9:12:12:13:12:13:13:13:13:14:14:14` | +| Reset without deploy | `-1371495859` | `75:0:0:0:0:0:0:1:1:0:0` | + +## Repair + +Each eager Electric commit supplies an opaque source progress token and its +expected predecessor through an optional persistence capability. The token +contains shape identity, handle, and offset. SQLite compares the predecessor +under its existing writer lock, atomically with rows and resume metadata. +A discontinuity makes source progress evidence incompatible until an +authoritative full replacement. A repeated complete endpoint preserves existing +evidence: it neither invents nor clears a gap. Provider handles need no ordering. + +Startup uses existing full-snapshot recovery when progress is missing, foreign, +or incompatible. This recovery overrides explicit caller positions because an +offset cannot supply missing baseline rows. An interrupted eager replacement's +durable reset requires the same recovery on the next start. Both recovery +admission and effective source position honor that obligation. Other sync modes +retain their explicit-position behavior. + +Source progress evidence has its own SQLite column and remains separate from +physical key integrity. A cached-row counterexample rejected combining them: +cached rows must remain visible during recovery. The additive migration also +covers existing databases. Arrival-order writes and their receipts remain. +Custom adapters advertise the optional capability only with an atomic check. + +## Evidence + +Each behavioral extension ran against unchanged production before its repair. +Setup failures and surviving mutants are excluded from the RED evidence. + +| Experiment | RED before its corresponding repair | Repaired GREEN / calibration | | --- | --- | --- | -| A stores its cursor; B reopens | final assertion fails | final assertion fails | -| A stores its cursor; B drains its backlog before reopening | passes | passes | -| B reopens before A stores its cursor | passes | passes | - -These six executions are retained in the owner as two `it.fails` replays and -four passing controls. They contain no independent expected answers. The -existing generated grammar already has the necessary actions; its minimum -length is eight, so the six-step diagnostic reduction is shorter than its -generated domain. The reported longer counterexamples demonstrate generated -reach of the defect. No new generated reach state or sampling guarantee is -claimed by these fixed replays. - -## Production mechanism and repair boundary - -Relevant source locations at the investigated production commit: - -- `packages/electric-db-collection/src/electric.ts:2028`, - `stageResumeMetadata`: writes the current stream's handle and offset. -- `packages/electric-db-collection/src/electric.ts:2531`: an up-to-date with - no open row transaction still opens and commits a metadata transaction. -- `packages/db-sqlite-persistence-core/src/persisted.ts:4529`: forwards each - external source transaction to the coordinator with a local stream position. -- `packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts:2599`: orders - application by durable row version. Row mutations and collection metadata - are applied together, but no Electric server position fences stale copies. - -Atomic rows and metadata within each commit are insufficient: commits from -different stream positions are still composed in local arrival order. The -key-set ledger follows those same row mutations, so agreement with that -ledger does not prove that the rows equal the prefix named by the cursor. - -The accepted convergence law needs a production repair. #2209's proposed -source-position protocol is relevant: the writer must reject or reconcile -older source effects and associate a resumable cursor with the state that -actually establishes its prefix. Its design must also define source identity, -handle transitions, and settlement of a tab's dropped duplicate commits. -This investigation does not choose or implement that protocol. - -Two smaller changes are insufficient by themselves: - -- Keeping only the maximum cursor still allows B's stale row mutation. It - can leave storage already inconsistent with that maximum cursor. -- Suppressing metadata-only commits removes the reduced trigger but does - not repair the reported resurrection: A's ordinary row-1 transaction - advances to `4_0` while row 3 remains wrong. - -The primary oracle's per-step fold deliberately accepts the temporary stale -rows permitted by the contract. Its final server-head comparison correctly -rejects the later traces. Fixed campaigns did not force the combination of a -stale peer write, a later cursor advance, and recovery that skips the missing -change. Existing reach for durable rows behind the server does not establish -that combination. A repair needs a declared reach for that combination in -generated histories, including empty and disjoint-key advancing batches, -both membership directions, and the relevant public receiving paths. - -## Separate metadata comment - -The comment's seed-only command with `_PROPERTY` fails setup on this revision: -the replay parser also requires `_PATH`. That is not evidence of the metadata -law. Instead, the existing pinned replay was run by name. With `it.fails` -temporarily removed, it fails at step 6 (`peer`) with the exact reported law: - -```text -history: begin seed(1,bare) insertOverSeed(1) begin reinsert(1) commit peer(2,reload) -law: source reads its row metadata (metadata.row.get) -expected: {"1":"","2":{"peer":{"s":6}}} -actual: {"1":{"cached":true},"2":{"peer":{"s":6}}} -``` - -The test file was restored. The owning sync-metadata coverage entry already -records this gap. No production root cause or repair is claimed for it. - -## Evidence and replay - -- **Law:** eventual server-head convergence after actual catch-up (#2209 and - the coverage map); reset-abandoned commits fail until restart (PR #2265). -- **Reach:** all three supplied multi-tab traces executed and reached the - reported final mismatch. Six reduced/control histories executed: two - rejected stale final rows and four converged. These are finite witnesses, - not a new normal generated reach measurement. -- **RED:** all three supplied seed/path replays fail on unchanged production; - the two six-step reductions fail through the same comparison. -- **GREEN:** the four neighboring controls pass. The original reset trace - passes after temporary post-failure recovery in the driver. These are - diagnostic greens, not a green production repair. -- **Mutant:** none run; no closure claim. The unchanged defective production - supplies the two reduced wrong-result witnesses. -- **Limits:** eager mode, one Collection, one process, controlled Electric - SDK messages, Node coordinator APIs, and native SQLite. No live Electric - HTTP/SDK, Chromium OPFS scheduling, on-demand rows, optimistic mutations, - source-position protocol, or arbitrary interleaving closure is established. - The original checkpoint still needs repair; both production row defects - remain open, as does the distinct metadata gap. - -Dependencies installed successfully with `pnpm install --frozen-lockfile`; -no override or lockfile change was needed. Diagnostic runs disabled coverage -and typechecking. The final six replay cases also ran with package test -typechecking enabled: four runtime controls pass, two runtime cases are -expected failures, and typechecking reports no errors. - -From the repository root, replay a reported multi-tab case by setting its seed -and path above: - -```sh -TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history \ -TANSTACK_DB_ORACLE_SEED=-699129038 \ -TANSTACK_DB_ORACLE_PATH='78:2:0:3:9:12:12:13:12:13:13:13:13:14:14:14' \ -pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run \ - tests/multi-tab-electric-oracle.test.ts -t 'random or replayed' \ - --coverage.enabled=false --typecheck.enabled=false - -pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run \ - tests/multi-tab-electric-oracle.test.ts \ - -t 'cursor-only commit|lagging stream drains|reloads before the cursor advances' \ - --coverage.enabled=false - -pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest --run \ - tests/sync-metadata-session-persisted.test.ts \ - -t 'a reload after an inner re-insert' --coverage.enabled=false \ - --typecheck.enabled=false -``` - -Raw diagnostic logs and the temporary instrumentation patch are local artifacts -under `notes/issue-2281/` (ignored by Git). The retained tests and this record -contain the essential witnesses independently of those local artifacts. - -## Implementation pre-edit gate (2026-10-29) - -The user authorized implementation after the investigation. No production file -has changed at this gate. The reset checkpoint now checks the failed source -before restarting it and comparing the successful catch-up. Its original -seed/path passes on the unchanged production head. - -The selected repair preserves arrival-order application. An eager Electric -source supplies an opaque progress token and the token its transaction -continues. SQLite compares that predecessor under the same writer transaction -as the rows and resume metadata. A mismatch makes the existing resume key-set -evidence incompatible, sticky until an authoritative truncate. Reopen therefore -uses the existing full-snapshot repair. Tokens contain shape identity, opaque -handle, and offset; timestamps are not progress. No lexical handle ordering, -write suppression, new receipt outcome, or stronger live-ordering law is added. -The optional bridge is exposed only by adapters that implement the atomic check. - -Pre-edit receiving inventory: - -| Path | Observation cut and witness | -| --- | --- | -| Controlled Electric + Browser coordinator + native SQLite, leader/follower | Existing primary owner plus generated `browser.electric-source-prefix`: both membership directions, empty/disjoint advancement, swapped source roles, repeated toggles, reload/restart and draining control; final durable/public source-head equality. | -| Real installed Electric SDK + HTTP + PostgreSQL + Chromium OPFS + actual tab coordinator | New `electric-source-prefix-oracle.opfs.spec.ts`, using real responses held before delivery. It shares `sourceHeadViolation` with the primary owner. Every cell declares held older response, stale commit, later advancement and reopened comparison reach. Empty advancement requests an actual non-live catch-up from Electric. | -| SQLite atomic resume-evidence boundary | `sqlite-resume-snapshot-oracle.test.ts`, generated opaque predecessor edges, duplicate/empty commits and authoritative replacements. The model checks complete paths since the last replacement. Comparisons occur after every durable transaction. | -| Custom persistence adapter | Optional support flag and optional bridge method preserve compatibility; unsupported adapters do not advertise atomic progress checks. Their source-head repair is outside this SQLite claim. | - -The core change uses the same SQLite adapter and transaction semantics on its -other hosts. No native-device, Electron-process, progressive, or on-demand -Electric claim is transferred from these eager Browser witnesses. Those paths -are limits, not additional evidence from Node APIs. - -Pre-edit RED results on `82c5f999b`: - -- `browser.electric-source-prefix`, fixed seed 2281, path `0:0:0:0`: - present key, empty advancement, no drain, final source-head comparison loses - row 1 while both Collections are ready. -- `sqlite-resume.source-progress`, seed 2281, path `0:0:0:0:0:0`: - snapshot → advance → advance → stale predecessor still reports `consistent`. -- Real Chromium disjoint/lost-row cell: both Collections ready, no source error, - actual rows only `anchor:advanced`; PostgreSQL also has `target:new`. - Reopened SDK requests the later real LSN and the shared comparison rejects - the missing row. This is a semantic RED; an earlier recorder setup failure - was repaired before obtaining this result. - -Commands and raw output are in the ignored `notes/issue-2281/` logs. The full -real-host defect/control matrix is running before any production edit. - -The pre-edit real-host matrix completed: **4 semantic REDs and 4 GREEN controls** -(2.4 minutes). Every defect cell reached the final shared source-head comparison; -all draining controls declared each required state once and passed. Production -edits begin only after this completed gate. The matrix covered lost/resurrected -rows × empty/disjoint advancement × drain/no-drain. - -## Implementation evidence (2026-10-29) - -This entry supersedes the investigation-only status above. It preserves the -pre-edit gate and the earlier observations instead of rewriting their outcomes. -The reviewed executable state is identified by the manifest at the end of this -record; its parent is `82c5f999b32c9db827dd74e3f172604dfda28013`. - -### Repair and preserved freedom - -The optional bridge now supplies a predecessor token for each eager Electric -commit. Its next token is collection metadata in that same transaction. -Tokens encode shape identity, handle and offset; timestamp differences do not -create source progress. SQLite compares tokens under its writer transaction. -A matching predecessor or repeated complete endpoint preserves continuity. -A gap remains sticky across later commits until an authoritative truncate. -Startup resumes only when the stored token, resume record and atomic source -continuity evidence agree. Otherwise Electric requests a fresh full snapshot. - -The first implementation reused key-set incompatibility. A subsequent random -history rejected that design at an intermediate public-row checkpoint: cached -rows disappeared during reload even though the physical key set was intact. -Seed `-781606242`, path -`82:0:3:0:1:7:7:9:10:20:23:19:14:14:15:13:12:12:12:12:12:12:12:12:12:12:12:12` -now passes and its reduced history remains in the shared driver. Production now -stores **separate** `source_progress_incompatible` evidence. It does not -quarantine cached rows or reinterpret physical key integrity. The additive -migration checks both the setup fast path and the new column's initialization. -The generated SQLite owner reconstructs the old schema before reopening it. - -Arrival-order regressions, normal applied receipts and rows remain allowed. -No source ordering, write rejection, retry queue or new receipt outcome was -introduced. This repairs unsafe continuation after reopen, while #2209's -permanently closed lagging stream and #2267's resumed-update failure remain -tracked expected failures. The distinct seed-metadata failure also stays open. - -### New comment checked during implementation - -The [new comment](https://github.com/TanStack/db/issues/2281#issuecomment-6500845350) -adds seed `-1371495859`, path `75:0:0:0:0:0:0:1:1:0:0`. Its eight-step trace -reloads A, deletes row 1, advances both sources, posts a peer reset, advances -A again, then reinserts row 1. It has no deploy. - -The original production **and original driver** reproduce its exact CI result: -row 1 is `one` instead of `v2`; A is `error`, B is `ready`, both have source -errors, and A's preload rejects with the reported `AbortError`. Keeping original -production but using the repaired final checkpoint passes. The final delivery -is when the reset-abandoned transactions first fail; they need restart before -catch-up is established. This is an evaluation/checkpoint repair under the -adversarial-review repair-location table. No production expectation was relaxed. -The trace is retained as a fixed replay of the same model, driver and check. -Logs: `new-comment-parent-red.log`, `new-comment-checkpoint-green.log` and -`new-comment-current.log` under the ignored local notes directory. - -### Evidence block - -- **Law:** after successful recovery and actual catch-up, durable rows and each - open eager Collection equal the source head (Law 2, coverage-map contract and - #2209). Reset-abandoned commits fail until restart (PR #2265). The optional - bridge's predecessor continuity and idempotent endpoint contract is stated in - `SyncPersistenceCapabilityV1.resumeSnapshot` and tested independently of SQL. -- **Reach:** `browser.electric-source-prefix` uses 80 fixed-seed and 80 random - histories. It crosses membership, 1–3 toggle pairs, swapped fast/slow roles, - empty/disjoint advancement, drain/no-drain, reload/restart and - current/missing/foreign provenance. `sqlite-resume.source-progress` uses the - same 80/80 budget with 3–16 generated actions plus initial replacement. - Its actions are advance, stale, empty, snapshot and duplicate; half the - generated domain reconstructs an old schema. Measured counts follow below. - The real SDK/OPFS receiver ran 10 cells, each reaching all four declared - states once: held earlier response, stale commit, later advancement, - reopened source comparison. No fabricated messages or source positions enter - that driver. The original 8 cells cross membership × advancement × draining; - 2 more receive missing legacy provenance with disjoint advancement. -- **RED:** before production edits, the generated Browser owner at seed 2281, - path `0:0:0:0`, loses row 1 at its source-head comparison. The SQLite owner - rejects missing gap evidence. The real SDK/OPFS matrix produces 4 semantic - failures and 4 passing drain controls. Setup/recorder failures are not counted - as RED. The exact commands are the owner commands below on the parent - production files with the extended tests retained. -- **GREEN:** current production passes the full Browser owner (312 tests and - 2 tracked expected failures before the final new-comment replay was added), - SQLite resume owner (62 tests), persisted grammar (1188 tests, 49 tracked - expected failures, 2 todo), Electric resume-race suite (49 tests), bridge - validation (8 tests), and Browser TypeScript checking. These runs reported - no type errors. The full real SDK/OPFS run passes 14 tests: the 10 new cells - plus 2 existing recovery histories and 2 diagnostic tests. Final restored - mutant runs and the added replay are recorded below. -- **Mutant:** disabling the durable gap parameter fails the SQLite fixed - campaign at a stale predecessor (`2281`, path - `2:3:2:5:6:5:5:0:0:2:2:2:2`), the Browser fixed campaign at a lost row - (`2281`, `0:0:0:0`), and both real disjoint/no-drain cells. Removing the - duplicate-endpoint exception fails the SQLite comparison (`2281`, - `0:7:1:2:2:2:2:2:2`). Accepting missing/foreign startup tokens fails the - Browser fixed campaign (`2281`, `2:0:0`, missing token), and both real legacy - cells. Each rejection is an assertion failure at the intended evidence or - source-head comparison. The real tests' polling deadlines print the actual - persistent wrong rows; they reached the comparison, rather than timing out - during setup. All mutants were temporary and restored with `finally`. -- **Limits:** bounded, same-version eager Electric sources sharing SQLite. - The generic storage law does not prove provider framing. Real receiving - evidence is Chromium OPFS against the installed SDK and local Electric / - PostgreSQL, not a browser matrix or arbitrary network schedule. No claim is - made for mixed old/new client writes, Electron processes, mobile/Tauri, - multi-process WAL, custom adapters without the optional atomic capability, - on-demand subsets, optimistic mutations, or monotonic live source ordering. - The leftover-replay/reset scheduling handoff remains under the coverage-map - owner. Known #2209, #2267 and metadata counterexamples are not closed here. - -Measured normal-run reach (fixed seed 2281 / random): +| Source continuity | Browser seed `2281`, path `0:0:0:0`, loses row 1 on `82c5f999b`; SQLite accepts missing gap evidence. Real receiver: four wrong-row failures and four passing draining controls. | Full SQLite owner: 61 passes. Browser restart and tab reopen converge. Ignoring durable gaps fails SQLite, Browser, and real disjoint-advancement comparisons. | +| Startup provenance | Missing/foreign tokens cannot justify the cached baseline. | Bypassing validation fails Browser and both real legacy-token cells. Rejecting duplicate endpoints separately fails SQLite at seed `2281`, path `0:7:1:2:2:2:2:2:2`. | +| Explicit caller positions | Extended Node campaigns fail; eight real numeric/`now` cells fail while initial-offset and handle-only controls pass. | Requesting recovery while preserving the raw caller position still fails the shared final comparison. Correcting both decisions passes. | +| Interrupted replacement | Paired Node campaigns fail at seed `2281`, path `1:0:0`, and random seed `-636405297`, path `14:0:0:0:0`. Four real numeric/`now` cells fail; six none/initial/handle controls pass. | All 30 real receiving cells pass. Keeping reset admission but removing its effective-position override fails the same Node replay; exact restoration passes. | +| Final reset checkpoint | Original production and driver reproduce both reported reset mismatches. | Corrected checkpoint passes on original production. Removing only the final restart/delivery block fails both reset replays. | +| Stop/peer coverage | CI reached SDK-give-up followed by peer commit zero times. | Constructed stop blocks establish every required stop boundary. Removing SDK-give-up abandonment fails a public-row assertion at seed `952051961`, path `1:0:0:0:0:0:0:0:1:1:0:0`. | + +Final implementation verification: complete Browser owner, **312 passes and two +tracked expected failures**; all **30 real receiving cells** pass. Electric +checks report 336 and 97 passes across two commands, without type errors. +Browser TypeScript, capability validation, Electron IPC, replay selection, +`pnpm lint:changed`, and whitespace checks pass. Documentation checking reports +zero new failures across 806 examples. Every temporary mutant was restored; +the final Electric source SHA-256 is +`46a417f2d65ccc21adcc72f56d80388d6f2f9fc32a172ab6e3fa1e4702508047`. + +### Independent judgments and exercised paths + +| Owner / driver | Independent judgment | Production path and observation cut | +| --- | --- | --- | +| Browser multi-tab oracle | Server snapshot log, shared source-head comparison | Real Collections, Browser coordinator with Node BroadcastChannel/Web Locks, native SQLite, controlled Electric stream; durable/public rows and health after each step and complete recovery/delivery. | +| SQLite resume-snapshot oracle | Path of opaque predecessor edges since the last authoritative replacement | Runtime capability, SQLite writer transaction, migration and resume snapshot; compatibility evidence after every durable transaction. | +| Real receiver for the Browser law | PostgreSQL source head, same shared comparison | PostgreSQL → Electric HTTP → installed SDK → Chromium Collections → OPFS SQLite; actual earlier responses held before delivery and reopened rows compared after catch-up. | + +The real matrix retains ten original/legacy, ten explicit-position, and ten +interrupted-recovery cells. Every cell requires held earlier response, stale +commit, later advancement, and reopened comparison. Interrupted cells observe +an actual durable reset before stopping replacement. No fabricated SDK messages +or offsets enter this driver. These are finite receiving witnesses. + +The source-prefix property uses 80 fixed-seed inputs and 80 random inputs. Each +input runs uninterrupted and interrupted neighbors: 160 histories per campaign. +Axes include membership, 1–3 toggle pairs, swapped source roles, empty/disjoint +advancement, draining, restart/reopen, token provenance, and five caller forms. +Numeric noninitial offsets without handles are SDK-invalid and excluded. +SQLite uses 80/80 inputs, 3–16 actions plus replacement, with stale/matching/empty +commits, duplicate endpoints, replacements, and old-schema migration. -| Declared state | Browser histories | -| --- | ---: | -| present / empty | 14 / 10 | -| present / disjoint | 11 / 13 | -| absent / empty | 6 / 6 | -| absent / disjoint | 7 / 9 | -| drained before recovery | 42 / 42 | -| missing provenance at recovery | 26 / 31 | -| foreign provenance at recovery | 21 / 30 | +Final measured source-prefix and SQLite reaches: -| Declared state | SQLite histories | +| Declared state | Fixed / random histories | | --- | ---: | -| discontinuous commit | 61 / 57 | -| matching commit after discontinuity | 37 / 41 | -| replacement after discontinuity | 44 / 34 | -| duplicate source position | 64 / 64 | -| legacy source progress column added | 34 / 35 | - -The unchanged broader history campaign also reached every declared state in -both normal runs. Its fixed/random counts were: must-refetch 47/44, peer commit -during reservation 12/13, stopped with open transaction 21/30, recovered 78/80, -durable behind server 64/65, peer commit before readiness 48/52, peer commit -after error/give-up/abort 2/9, 3/2, 3/2, reset abandoned transaction 7/9 and -reset-abandoned commit failed 4/7. Counts include attempted histories discarded -for the already recorded #2267 gap; the focused source-prefix campaign does -not rely on that gap's exclusion. - -### Oracle review (self-review) - -This is a self-review; no independent reviewer was dispatched. - -| Requirement | Outcome and evidence | -| --- | --- | -| ORC-001 | Law 2 and the optional bridge contract state authority and limits. Temporary stale rows remain permitted; only actual catch-up justifies source-head equality. | -| ORC-002 | The primary model is an independent server snapshot log. The core model is a path of opaque predecessor edges. Neither imports SQL continuation decisions. PostgreSQL supplies the real receiver's independent source head. | -| ORC-003 | The owners explain laws, legal actions, models, drivers and cuts beside executable code. The shared source-head checker names its authority and bounded projection. | -| ORC-004 | Both reported membership defects reconstruct as one toggle pair with no drain. Removing the stale commit or later advancement loses the defect; drain and earlier-reopen controls distinguish it. Extra pairs, swapped roles, empty/disjoint batches, recovery type and provenance retain distinct nearby cases. The core grammar reaches gap/duplicate/replacement/migration. Prefixes beyond the source log and a snapshot without authoritative replacement are excluded. | -| ORC-005 | Controlled production path uses real Browser coordination and SQLite, observing durable/public rows and status after each step and after recovery. Real tabs hold actual SDK HTTP responses. Source facts needed for final equality have arrived before each comparison; the new-comment experiment shows why a failed final commit does not meet that premise. | -| ORC-006 | Three hostile designs fail the intended comparisons, including real gap and legacy receivers. The checkpoint's original driver also rejects both reset traces before the repair. No setup failure is credited. | -| ORC-007 | Both new properties register in `oracle-config.ts`, run equal fixed/random budgets of 80, and use direct seed/path replay. The original broad grammar was not changed, preserving all supplied replay paths. Neither uses `fc.commands`. | -| ORC-008 | The core path distinguishes a coherent prefix from one with any missing edge; later matching commits cannot erase that distinction. Replacement resets the path. Duplicate endpoints add no edge. No production queue or recovery controller is copied into the model. | -| ORC-009 | The glossary defines source progress token. Model-only edges and the server snapshot log map to predecessor continuity and source states; they are not durable row versions or timestamps. | -| ORC-010 | Primary histories retain step/checkpoint and expected/actual observations. The real receiver retains its primary error as `cause`, appends observation/cleanup diagnostics and closes pages, coordinators, databases and temporary PostgreSQL tables. Core preserves primary failure during database cleanup. | -| ORC-011 | The concrete shared-fault hypothesis is trusting a wrong cursor as proof of rows. Source-head recomputation independently challenges the storage edge model; PostgreSQL rows challenge the controlled server/SDK premise. No additional duplicate model is needed for that hypothesis. | -| ORC-012 | This record gives each requirement's result, exact parent, executable manifest, witnessed boundaries, mutants and explicit remaining limits. No known in-scope counterexample remains in the bounded reopened-source class. | -| ORC-013 | Matching versus stale predecessor, repeated endpoint versus new position, sticky gap versus authoritative replacement, both memberships, empty/disjoint advancement and draining controls distinguish nearby wrong rules. Missing provenance cannot be rescued by intact key-set evidence. | -| ORC-014 | The real HTTP/SDK/OPFS receiver reproduces delayed old delivery followed by later cursor advancement. It shares `sourceHeadViolation` with the primary owner and reads PostgreSQL for the expected head. Ten finite cells bound the handoff. | -| ORC-015 | Both generated properties use failing `createOracleReach` checks; normal-run counts appear above. Each real receiving cell declares its own four reaches, all once. | -| ORC-016 | Fixed regressions replay `runGeneratedHistory`; the real cells call the shared judgment. Bridge validation examples stay in the ordinary `sync-persistence.test.ts`. No standalone type examples were added to oracle files. | - -The material formulation attack was a premature catch-up cut; the new trace -confirms that evaluation repair. The conceptual/composition attack was treating -source continuity and physical key integrity as one fact; the cached-row -counterexample required separate evidence. The suitability tradeoff is a full -snapshot after a discontinuity, instead of ordering opaque source handles. -That preserves the accepted arrival-order contract and uses existing recovery. -The tests establish correctness for those cuts, not optimal network work. - -Reproduce the main commands from the worktree root: +| Present row / empty advancement | 28 / 26 | +| Present row / disjoint advancement | 22 / 14 | +| Absent row / empty advancement | 12 / 34 | +| Absent row / disjoint advancement | 14 / 18 | +| Drained before recovery | 84 / 68 | +| Missing progress token | 52 / 64 | +| Foreign progress token | 42 / 56 | +| Divergent same-Collection restart | 50 / 52 | +| Divergent tab reopen | 26 / 40 | +| Divergent explicit numeric resume | 20 / 12 | +| Divergent explicit `now` | 14 / 16 | +| Explicit initial offset | 34 / 42 | +| Explicit handle | 42 / 32 | +| Interrupted reset before numeric resume | 10 / 6 | +| Interrupted reset before `now` | 7 / 8 | +| SQLite discontinuity | 61 / 53 | +| SQLite matching commit after discontinuity | 37 / 42 | +| SQLite replacement after discontinuity | 44 / 36 | +| SQLite duplicate endpoint | 64 / 61 | +| SQLite legacy migration | 34 / 39 | + +The broader lifecycle property retains 80 accepted inputs per campaign. Four +constructed stop blocks precede 8–24 arbitrary actions, for 32–48 actions. +All four stop kinds cross a distinct peer commit and recovery. Final fixed/random +stop-boundary reaches are 81/80, including one fixed history later discarded +under the existing #2267 gap. Final-catch-up reset recovery reaches 4/2. +All other declared reaches pass. No comparison, exclusion, minimum reach, or +run budget was relaxed. + +### Review corrections + +Four independent reviewers found an unreachable focused restart branch and +replay registrations that ran unrelated campaigns. Actual-path instrumentation +first measured restart 0/reopen 38 and failed. Adding a stop made restart +reachable; gap and provenance mutants then failed that path. The original +row-loss trace already supplied a pre-production same-Collection restart RED; +the focused campaign's restart repair occurred later. Replay probes in both +Browser directions, unrelated-property selection, and SQLite selection verify +that only the requested campaign runs. + +Later CodeRabbit reviews identified the explicit-position and interrupted-reset +gaps above. The latest completed external review covers `124ce667c`; automatic +review is paused. Subsequent eager reset semantics have independent local +source review plus the generated, real receiving, and mutant evidence above. +The lossless task-local audit retains 30 items: 12 fixed, five refuted, one +accepted design, and 12 duplicates. No known in-scope source-progress finding +remains open. CodeRabbit supplied useful defect and maintenance findings; +its one-clause recovery suggestion was incomplete, as the mutant demonstrates. + +The rubric evidence is concentrated in the contract/model/path distinctions, +measured legal histories, calibrated comparisons, isolated replay, and retained +failure diagnostics. Cleanup attempts every release and preserves the primary +violation alongside cleanup failures; injected combined failures verify this. +Fixed examples use the shared oracle without separate expected answers. + +## Cost and limits + +The production delta is 153 net source lines across five files. Against +merge-base `672684846`, independently compressed package bundles grow by +DB +37, SQLite core +368, and Electric +168 gzip bytes: **573 bytes total**. +The standard application fixture adds 45 bytes but does not use Electric or +SQLite, so it does not measure their application cost. The repair adds one +metadata read per qualifying incremental source-progress transaction, stored +continuity evidence, and token processing. Runtime overhead was not benchmarked. +Incompatible progress can require a full source snapshot. + +The evidence covers bounded, same-version eager sources sharing SQLite. It +does not establish arbitrary schedules, mixed versions, monotonic intermediate +writes, on-demand subsets, optimism, native/Electron receiving, individual +receipt fulfillment, or clean teardown. Explicit changed-shape/malformed/tagged +resumes, arbitrary cursor selection over certified rows, other modes' reset +policy, changing caller forms within one history, and real foreign-token +receiving remain outside the claim. The coverage map names remaining witnesses. + +The permanently closed lagging-source history in #2209 remains a tracked +expected failure. The #2267 partial-update guard preserves the safety contract; +its no-error composition expectation remains a separate contract question. +The first #2281 comment's seed-row-metadata defect, #2272 metadata receiving, +and the pilot's replay/reset scheduling and teardown gaps retain separate owners. +The independent cluster report confirmed this repair's mechanism and led to a +separate reach for reset recovery first discovered during final catch-up. +It did not combine those other boundaries with source-progress recovery. + +CI on implementation `0ec58057d` passed 25 checks after one failed-job rerun. +Its original observer cleanup/restart defect also reproduces on clean current +main `bea84cd45`; the aggregate ordering reach gap is tracked in #2289. The +maintainer chose to keep #2298 focused. Observer evidence was sent to the +“Survey open issues and PRs” chat; both issues remain open. A passing retry +closes neither. CodeRabbit's skipped check reflects its paused review. + +## Reproduction + +Run from the repository root with two workers: ```sh -pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest --run \ - tests/sqlite-resume-snapshot-oracle.test.ts \ - --coverage.enabled=false --disableConsoleIntercept pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run \ - tests/multi-tab-electric-oracle.test.ts \ - --coverage.enabled=false --disableConsoleIntercept + tests/multi-tab-electric-oracle.test.ts --maxWorkers=2 --coverage.enabled=false +pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest --run \ + tests/sqlite-resume-snapshot-oracle.test.ts --maxWorkers=2 --coverage.enabled=false pnpm --filter @tanstack/browser-db-sqlite-persistence exec playwright test \ --config playwright.opfs-electric.config.ts \ - electric-source-prefix-oracle.opfs.spec.ts electric-resume-two-tab.opfs.spec.ts + electric-source-prefix-oracle.opfs.spec.ts electric-resume-two-tab.opfs.spec.ts \ + --workers=2 ``` -Mutation commands used the fixed-seed tests selected by behavior name, with -coverage and typechecking disabled. The real gap mutant selected -`--grep 'after disjoint advancement$'`; the provenance mutant selected -`--grep 'from legacy progress'`. The normal typed owner runs above supply the -separate type evidence. Raw logs and the restoring mutation script remain local -in the ignored `notes/issue-2281/` directory. - -### Restored checks, code weight and cleanup refinement - -After restoring all three mutants, both new generated campaigns pass again. -The SQLite owner passes all 62 tests with no type errors. The Browser selected -run passes 8 reported tests, including the new-comment and cached-row replays, -with no type errors. All 6 real disjoint cells pass, including both legacy -cells that rejected the provenance mutant. The fixed reaches are unchanged; -that restored random run reached Browser states 11/19/11/7/32/36/22 and SQLite -states 56/46/40/64/44, in the table order above. - -Self-review found a separate recorder weakness: the existing Node driver -silently ignored Collection cleanup failures, and campaign `finally` blocks -could replace a primary mismatch. Cleanup now attempts every resource release, -retains errors, and aggregates a secondary cleanup error with the original -violation as `cause`. This changes test machinery only. The broader owner was -rerun after this refinement; its result is appended below. - -`pnpm size:compare --files --packages db,db-sqlite-persistence-core,electric-db-collection` -compares the parent merge-base with production minified bundles: - -| Package/scenario | Gzip change | Brotli change | -| --- | ---: | ---: | -| db | +37 B | -55 B | -| db-sqlite-persistence-core | +368 B | +321 B | -| electric-db-collection | +132 B | +112 B | -| three-package total | +537 B | +378 B | -| typical app scenario | +45 B | -60 B | - -The source delta is 156 inserted and 19 removed production lines across five -files. Test/config additions are separate: 1059 inserted and 23 removed lines -before this final evidence entry; none ships in the package bundles. - -Production grows because safe continuation needs durable provenance and an -atomic predecessor check that did not exist. The repair reuses existing -full-snapshot recovery instead of introducing a second recovery controller. -Collection metadata is deduplicated once and reused for both the source check -and its SQL writes. Reusing the old key-set flag was tried and rejected by the -cached-row counterexample: those two facts cannot be merged. Dropping duplicate -endpoint handling fails its hostile mutant. Dropping provenance validation fails -the legacy receivers. Stronger source ordering would require opaque-handle and -skipped-receipt semantics; it was not substituted for the accepted law. - -`pnpm lint:changed`, Browser `tsc --noEmit`, and `git diff --check` pass. -No oracle budget was lowered or removed. Installation used the lockfile as-is; -there is no dependency override or installation workaround in this diff. - - -### Final verification and reviewed state - -The full primary owner after cleanup refinement reports **314 passed and 2 -expected failures**, with no type errors (`multi-tab-close-final.log`). Every -normal fixed/random reach passed. A combined gap mutant plus a forced host -cleanup failure produced an `AggregateError` retaining the original final -source-head mismatch and the secondary cleanup failure; both source files were -restored. `cleanup-calibration.log` records that assertion-level rejection. - -All four supplied multi-tab seeds were then replayed directly on restored code: -`-1115798760`, `-1660880085`, `-699129038`, and the new `-1371495859`. Each -passed its exact reported shrink path through the unchanged broad grammar. -These are replay results, not substitutes for the normal generated campaigns. -Final lint and diff checks pass. No commit, push or pull request was created. -The local Electric/PostgreSQL test stack remains available for reproduction. - -Final primary-owner normal reaches (fixed / random): - -| Declared state | Histories | -| --- | ---: | -| present/empty | 14 / 6 | -| present/disjoint | 11 / 9 | -| absent/empty | 6 / 15 | -| absent/disjoint | 7 / 6 | -| drained before recovery | 42 / 44 | -| missing progress at recovery | 26 / 22 | -| foreign progress at recovery | 21 / 29 | -| must-refetch delivered | 47 / 49 | -| peer commit during an open reserved transaction | 12 / 14 | -| source stopped with an open transaction | 21 / 21 | -| tab recovered | 78 / 79 | -| durable rows behind the server | 64 / 62 | -| peer commit while a tab was not ready | 48 / 53 | -| peer commit after error without retry mid-transaction | 2 / 3 | -| peer commit after SDK gives up mid-transaction | 3 / 1 | -| peer commit after signal aborted mid-transaction | 3 / 5 | -| peer reset abandoned an open transaction | 7 / 15 | -| reset-abandoned commit put the Collection in error | 4 / 13 | - -This review covers the uncommitted working tree on parent -`82c5f999b32c9db827dd74e3f172604dfda28013`, branch `codex/investigate-2281`. -The SHA-256 identity of the source/test/coverage manifest is -`54d96ca70d3be37242cf060b7c7de6ae1e0eaf59d46059f4aa3bfa4660f4cb1a`. -Compute it from sorted records `path`, NUL, content SHA-256, newline. This -record excludes itself to avoid a circular identity; later edits need a new -append-only verification entry. - -| Reviewed file | Content SHA-256 | -| --- | --- | -| `docs/contributing/glossary.md` | `f8cec460111439ba9ee65b0c1ecf8a60e466b576c4d94d75af222a9ced4ed02d` | -| `docs/contributing/oracle-coverage.md` | `598fa6e4c3874eaafeb48174d3f13afd8626bc56e558d804a422d5bce3c19ba8` | -| `packages/browser-db-sqlite-persistence/e2e/electric-source-prefix-oracle.opfs.spec.ts` | `8fafec444efbcc0a171d8beab0a166b252941e84841e02d90b954ebd80517c81` | -| `packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.html` | `4734264a6227519aeecdf8b3a7eaeb42f917daa8ce784d12baf8a57cb83d4c00` | -| `packages/browser-db-sqlite-persistence/e2e/electric-source-prefix.opfs.ts` | `a71db0fffc376a9b15c2d4669100876d6656df2c448134a17b890fa32f686c2f` | -| `packages/browser-db-sqlite-persistence/playwright.opfs-electric.config.ts` | `d18ef1f5dbbdec19e7db5b5498f039910d60cf43b7da48b617313c6f2ea8c62a` | -| `packages/browser-db-sqlite-persistence/tests/electric-convergence-oracle.ts` | `c3108fc39f4c8d7de91567011de9c829fb8e16fdb2a03d20231aca0a61ec27db` | -| `packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts` | `2868ead1bc73508ecdc685b0590a5a8f487b01db19fc846f6236ec2c0f843cc3` | -| `packages/db-sqlite-persistence-core/src/persisted.ts` | `1a3bc05bbf10646c8d75fb062799c8335a181a8d3a5e91d61a9244c1a4f5027a` | -| `packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts` | `406e4058b85055fc7972c18786f9461856aa46c90932f6346dc6faccd1106a0c` | -| `packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts` | `2ff7303909f08896844b2343f425a26eb87988fc41f1c1137bcf646198c6f779` | -| `packages/db/src/sync-persistence.ts` | `7052e177e444acbe47b57dc5d5bf7f57aae339c7ef6b7308765ea56f81bd4251` | -| `packages/db/src/types.ts` | `152fd447c615a76398e0f8e4f0714957df7879a77cb51f89b4fef40f83bcc367` | -| `packages/db/tests/oracle-config.ts` | `ef56c55cd1e7c759d056ed4d9be85e9cd79d2cdfc9743454170562240c50c908` | -| `packages/db/tests/sync-persistence.test.ts` | `ac531d2397b3a1e39be707c0e9d0656d968fd88cbd863a42f32e6fd0e31d230b` | -| `packages/electric-db-collection/src/electric.ts` | `f8eda904b0c02439196660844d41bafcd754404dabc4d9959caa8659b6c364b8` | -| `packages/electric-db-collection/tests/electric-resume-snapshot-races.test.ts` | `26f430ae9359d989fe221c747fc9f3aeceecd1c9e4137e5cca38e5a8ab257a46` | - -## Independent multi-agent review (2026-10-29) - -Four fresh reviewers examined the unchanged 17-file snapshot identified above: -SQLite model/API, Browser grammar/checkpoints, real SDK/OPFS receiving path, and -policy/evidence scope. They read the oracle-review skill, rubric, glossary and -applicable ownership records. No shared source or test files were changed. -The following findings supersede the earlier self-review's unconditional -conformance statements for the named requirements. These are oracle gaps; -no additional production defect was established. - -### P2: the focused recovery dimension never reaches restart - -`multi-tab-electric-oracle.test.ts:1477` generates a recover step with pick 0 or -1 while the lagging tab is still running. The branch at lines 1323–1328 always -selects reload in that state, so both sampled values close the tab and open a -new Collection. The focused source-prefix campaign does not establish a -same-Collection restart after stale prefix application and later advancement. -Two reviewers independently found this by control-flow inspection. - -The broader generated/storm owners do exercise restart after stops. The real -receiver exercises an actual close/reopen. Those witnesses do not supply this -missing combination. The earlier pre-edit inventory, coverage entry and -ORC-004 self-review overstate the focused reload/restart dimension. Existing -reach counters cannot detect its collapse because none names the actual -recovery path. - -The smallest general repair is to stop the slow source before the restart -branch, retain a running reload branch, and declare reaches from the actual -`how` path for divergent-source restart and reload. The owner remains this -Browser oracle. Until that witness and comparison evidence exist, the focused -restart claim remains open. This finding concerns ORC-004, ORC-005, ORC-012 and -ORC-015, including AGENTS.md's unreachable-claims rule. - -### P2: replay runs unrelated campaigns - -The Browser random registrations at lines 1514 and 1592 use plain `it`. With -the documented `-t 'random or replayed'` selector, both run even when -`TANSTACK_DB_ORACLE_PROPERTY` selects only one. `oraclePropertyOptions` applies -the shrink path to the matching property; it reseeds the other property's -normal 80-history campaign. - -A reviewer executed source-prefix seed 2281, path `0:0:0:0`: the requested -single history passed, then the unrelated broad campaign reported 80 recovered -histories and 49 must-refetch histories. The command reported 2 passed and took -21.74 seconds. This can delay replay or introduce an unrelated failure. - -The SQLite source-progress registration at -`sqlite-resume-snapshot-oracle.test.ts:961` has the same isolation defect, -assessed P3 in that file. Selecting `sqlite-resume.startup-generation`, seed -1659, path `0`, while filtering for source-progress tests still ran its normal -campaign (reach counts 55/32/34/64/40). The intended source-progress replay -itself passed. Existing other unguarded core properties are not evidence that -this new registration is correct. - -Use `oracleReplayTest(it, property)` or `oracleReplaySkips(property)` to stand -aside during another property's replay. Verify both property directions and -an unrelated selector. This is the missing ORC-007 selection check, separate -from successful execution of each property's own seed/path. - -### Other review results and limits - -The SQLite reviewer independently reconstructed stale predecessor, sticky gap, -duplicate endpoint and authoritative replacement behavior. Its fixed/random -campaigns passed with typechecking, as did direct duplicate-endpoint replay. -Browser reviewers passed both focused normal campaigns and new checkpoint -regressions. They found no weakened final source-head judgment or unsupported -removal of the known #2209/#2267 exclusions. - -The receiver reviewer ran both disjoint/no-drain real cells: each passed and -reached all four declared states. An isolated temporary probe held the reopened -page's first actual HTTP response. The Collection stayed loading and `open()` -stayed pending while stale cached rows remained visible. Releasing the response -reached the shared source-head comparison and passed. The commit counter was -already 1 because setup commits reset metadata; the existing status gate, not -the counter alone, prevented premature comparison. No readiness finding was -established. A suspected `pg.end()` cleanup issue was rejected after inspecting -the installed resolve-only completion implementation and finding no concrete -supported failure path. - -The evidence reviewer confirmed the recorded manifest and semantic RED/mutant -results. It found no supported production-first, model-hacking, quiet-exclusion, -pinned-only or missing-mutant shortcut. Historical logs do not hash production -at every execution boundary; they do not contradict the preserved pre-edit -gate. The two verified issues above remain unresolved in this reviewed snapshot. -No production mutants were independently rerun by the reviewers; the receiver -review did not independently repeat the other eight real cells or cleanup fault -injection. The earlier executed evidence retains those distinct claims. - - -## Review repairs and verification (2026-10-29) - -The user authorized both review fixes. The two findings above are resolved in -this state. No production file changed during this pass; all five production -content hashes still match the preceding reviewed manifest. The recorded bundle -measurements and unchanged real SDK/OPFS receiving evidence still apply. - -### Law, reach and failed controls - -The convergence law and its authority remain unchanged: once a source actually -recovers and catches up, durable/public rows equal the source head. The grammar -now distinguishes two legal recovery paths after the forced prefix divergence. -Recovery 0 stops the lagging source, then restarts the same Collection. Recovery -1 closes the tab and opens a new Collection. The callback receives the driver's -actual `how` after stream startup and quiescence. The new reach states count -that branch only when the baseline was divergent, rather than inferring it -from the generated pick. - -Before changing the grammar, those actual-path counters rejected the old -campaign: `divergent source restart: 0`, `divergent source reload: 38`. The fixed -80-history run failed its reach assertion (`review-restart-reach-red.log`). -This is a RED for the oracle's missing reach, not a production row failure. -The conditional stop then made the restart branch reachable. Both fixed/random -80-history campaigns pass at the original source-head comparison. - -The earlier pre-edit inventory's restart claim was inaccurate, as the independent -review records. The new restart evidence was obtained after the original -production repair. This entry does not retroactively attribute it to that gate. -This pass changes test machinery only. The original broad grammar and all four -issue replay paths remain unchanged; the focused recovery recipe now includes -the missing stop. - -### Replay selection - -The Browser source-prefix and broad-history random registrations, and the -SQLite source-progress random registration, now use `oracleReplayTest` with -their property names. Normal campaigns still run. During path replay, these -registrations stand aside unless their property is selected; fixed campaigns -continue to stand aside for every path replay. - -Before the guards, selecting the one-history source-prefix replay also ran the -broad 80-history campaign. Selecting startup-generation while filtering for -source-progress still ran the latter's normal campaign. Those successful test -exits were failed selection controls; their extra campaign output is the -violation (`review-replay-before.log`, `review-core-replay-before.log`). - -After the guards, an integration probe checked five selectors with coverage and -typechecking disabled: - -| Selector | Observed registration behavior | -| --- | --- | -| Browser source-prefix, 2281, `0:0:0:0` | Exactly one requested replay passed; actual restart reach was 1; no broad campaign. | -| Browser broad-history, -1371495859, `75:0:0:0:0:0:0:1:1:0:0` | Exactly one requested replay passed; no source-prefix campaign. | -| An unrelated registered property with both Browser random tests selected | Both registrations skipped; no campaign ran. | -| SQLite startup-generation with source-progress tests selected | Both source-progress registrations skipped. | -| SQLite source-progress duplicate-endpoint replay | Exactly one requested replay passed; fixed campaign skipped. | - -The Browser probe used the documented `-t 'random or replayed'` selector in both -directions. The SQLite probe used `-t 'preserves source progress evidence'`. -These checks target the reviewed registrations; they do not claim to repair -pre-existing replay-selection behavior in unrelated SQLite tests. - -### Mutants and restored GREEN - -A temporary SQL mutant that never retains a source gap failed the new restart -replay at `after every tab caught up`, with a missing source row. The same mutant -failed both existing cursor-only reload replays at that checkpoint. A temporary -startup-provenance bypass failed the missing-token restart replay (seed 2281, -path `2:0:0`) at the same source-head comparison. All were assertion failures -with concrete wrong rows, and the source files were restored in `finally`. -Logs: `review-mutant-gap-restart.log`, `review-mutant-gap-reload.log`, and -`review-mutant-legacy-restart.log`. - -After restoration, both Browser important generated properties ran their -normal fixed/random 80-history campaigns, alongside both cursor-only reload -replays. The selected command passed all 9 reported tests with no type errors -(`review-restored-campaigns-green.log`). The full SQLite owner also passed -(61 reported tests, no type errors, `review-core-green.log`). The changed -registration wrapper changes Vitest's type-test discovery count; runtime -campaigns and run budgets were not removed or lowered. - -Final focused source-prefix reaches (fixed / random): - -| Declared state | Histories | -| --- | ---: | -| present/empty | 14 / 8 | -| present/disjoint | 11 / 17 | -| absent/empty | 6 / 8 | -| absent/disjoint | 7 / 9 | -| drained before recovery | 42 / 38 | -| missing progress at recovery | 26 / 30 | -| foreign progress at recovery | 21 / 27 | -| divergent source restart | 25 / 23 | -| divergent source reload | 13 / 19 | - -Both normal broad-history campaigns also met all existing declared reaches. -The full SQLite source-progress reaches were 61/52 discontinuities, 37/35 later -matching commits, 44/31 replacement repairs, 64/66 duplicates and 34/42 legacy -column migrations. `pnpm lint:changed` and `git diff --check` pass. - -The original Browser reviewer re-reviewed the delta and independently ran both -replay directions. Each reported one passed replay, with the other campaign -absent; its source-prefix replay measured a real same-Collection restart. -The original SQLite reviewer confirmed its guard repair by inspection. Neither -reviewer found a remaining supported issue in these repairs. They did not -independently repeat this pass's normal campaigns or mutants. - -For the applicable rubric changes: ORC-004/005 now have an executable restart -history and its actual-path observation; ORC-007 has isolated selection checks; -ORC-015 has failing reach counters and measured normal counts; ORC-012 is updated -by this entry. Existing laws, independent models, failure handling, shared -receiving judgment and law-only file structure remain unchanged. The earlier -ORC-001–003/006/008–011/013–014/016 evidence still applies. - -The scope remains bounded same-version eager sources. The added in-page restart -interaction is a controlled SDK/Node Browser witness; the real Chromium matrix -continues to supply tab close/reopen evidence. No broader real in-page restart, -mixed-version, on-demand, native-device or monotonic-live-write claim is added. -The separate #2209, #2267 and metadata gaps remain open. No commit or push was -made during these fixes. - -The updated uncommitted 17-file manifest identity is `f326ed14d9464e8ecc7d62c1da21d53589e2a49f31cfa3dec162949f1325e62b`, on the same -parent `82c5f999b32c9db827dd74e3f172604dfda28013`. It uses the sorted path/NUL/ -content-hash/newline algorithm above and excludes this record. Only these -manifest entries differ from the previously reviewed snapshot: - -| Changed file | Content SHA-256 | -| --- | --- | -| `docs/contributing/oracle-coverage.md` | `ef76490d320e037e9362261feb245089f1901a54ce209237946af998cf04a531` | -| `packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts` | `eee52b0972a4920c573b4eb304e84ef256ed32b886583695cd35544415b7b746` | -| `packages/db-sqlite-persistence-core/tests/sqlite-resume-snapshot-oracle.test.ts` | `16781c8eb12ef6698cf412108d705dccb5e345223dc319a53e0528d5e83d521b` | - - -## Prep-pr verification on updated main (2026-10-29) - -The task branch fast-forwarded to `07bfc0eb2584fd83a3efc14ec1fb452954a829ea` -before fresh review. Upstream changed release metadata and unrelated paths; -none overlapped this task's edits. The same 17-file manifest, excluding this -record, remains `f326ed14d9464e8ecc7d62c1da21d53589e2a49f31cfa3dec162949f1325e62b`. -No production, oracle, history grammar, or comparison changed during this pass. -Earlier RED, mutant, scope and ORC audit entries retain their stated provenance. - -Fresh verification against this base: - -- Full Browser multi-tab owner: 312 passed and 2 expected failures; no type errors. -- Full SQLite resume-snapshot owner: 61 reported passed; no type errors. -- Electric resume-snapshot races: 49 passed; no type errors. -- Persistence capability validation: 8 passed. -- Real SDK/PostgreSQL/Chromium OPFS: all 14 passed, including all ten prefix cells. -- Browser `tsc --noEmit`, `pnpm lint:changed`, and `git diff --check`: passed. - -The owner commands above used `--maxWorkers=2`; the real command used -`--workers=2`. Commands set `pnpm_config_verify_deps_before_run=false` after -pnpm tried to reinstall for upstream release-specifier changes. The installed -external dependencies used here did not change; no override or lockfile edit -was made. Local logs are `notes/issue-2281/prep-{browser,core,electric,bridge,real}-green.log`. - -The focused Browser fixed/random reaches, in the earlier table order, were -14/11, 11/10, 6/15, 7/11, 42/33, 26/35, 21/19, 25/24 and 13/23. -SQLite fixed/random reaches were gaps61/53, later matching37/42, -replacement44/36, duplicates64/61 and migration34/39. Every real prefix cell -reached its four declared states once. No budget was lowered. - -Fresh simplification and five independent review angles found no confirmed -behavioral defect. Evaluation also held the eager baseline read after initial -resume selection, then committed a discontinuous predecessor through SQLite. -Both a row-changing and a metadata-only commit advanced the row version while -physical key evidence stayed consistent. After release, the existing generation -fence rejected resume, persisted a reset marker and prevented queued publication. -Both focused probes passed; their temporary fixture copies were removed. -This refutes the suggested startup race at that controlled cut; it does not -establish new monotonic live-write or arbitrary-scheduling guarantees. -The existing finite bounds and separate open gaps remain unchanged. - -No commit, push or PR has been made at this checkpoint. Optional readability -and documentation suggestions remain for the prep-pr user walkthrough. - - -### Accepted test-driver simplification (2026-10-29) - -The user approved flattening the URL-input conversion in the real receiving -driver. Its native string, URL and Request forms retain the same URL result. -Browser `tsc --noEmit` passes. The real lost-row/disjoint-advancement cell -passes and reaches all four declared premises; no production, model, history -grammar or comparison changed. The 17-file manifest excluding this record is -now `dfdcaa048fc45d73c2b5750856ce91e50f812137e41c724235e7e4889637adad`, -on the same base `07bfc0eb2584fd83a3efc14ec1fb452954a829ea`. -Logs: `prep-s1-tsc.log` and `prep-s1-real.log` in the local evidence directory. - -### Restart evidence chronology clarification (2026-10-29) - -The restart reach correction in “Review repairs and verification” concerns the -focused `browser.electric-source-prefix` campaign. Its recipe originally -performed tab reload for both recovery choices; actual restart reach was added -after the production repair. That chronology remains unchanged. - -The earlier [row-loss trace](#later-traces-rows-and-the-resume-position-describe-different-prefixes) -already exercised same-Collection restart on unchanged production. Seed -`-1660880085`, path `55:0:1:4:4:4:1:1:1:1:3:4:4:4:4`, stopped both sources -after the stale-prefix and metadata-only advancement. The original driver's -final loop restarted both existing Collections. The local `loss-trace.log` -records those restarts at lines 101 and 110, followed by the missing-row -source-head mismatch at line 119. This supplies the earlier restart defect -witness; it does not retroactively supply the focused campaign's missing reach. - - -### Committed-tree verification (2026-10-29) - -Implementation commit `20e2ad9f81c91c60e5d9c219892c6421ae12baca` contains the -reviewed repair and both approved suggestions. Its pre-commit hook removed an -unnecessary test-only non-null assertion and merged duplicate imports in the -SQLite oracle. The merged import then received a formatting correction. These -changes preserve runtime behavior, model rules, histories and comparisons. -The SQLite resume owner passes again (61 reported tests, no type errors), as -does Browser `tsc --noEmit`. Logs: `prep-committed-core-green.log` and -`prep-committed-browser-tsc.log`. All five production files retain their -previously verified content. The 17-file manifest excluding this record is -`e2fda4fe3c0962594f71929acd20c4e8c03b79cdded5e6713594168525a53111`. This identity -includes the accepted driver simplification and the hook's test-only edits. -The earlier full runtime, real-host and mutant evidence retains its scope. - -### Published CI and CodeRabbit follow-up (2026-10-29) - -PR #2298 first ran at `e9b2dda669a3071e9cd31d7c2d46626e2a7070bc`. -CI found two missing named replay registrations, four warm-reopen fixture -failures, one Electron snapshot-shape assertion, and documentation-check drift. -The recovery owner reproduced all four failures locally before its fixture -changed. Its warm baseline supplied a resume record but omitted the complete -source-position token. Under the new capability contract, that is a legacy -baseline and correctly requests a full snapshot. The fixture now supplies the -matching token for its existing compatible-reopen premise. Its model, request -and row comparisons remain unchanged. Missing/foreign-token recovery remains -generated in the Browser owner and exercised by the real receiving cells. - -The replay manifest now names both new properties and their exact replay lanes. -The Electron assertion includes the new `sourceProgressCompatible` field. -The Convex example declares its selected task ID as a string, so an ambient -application variable cannot select the array-update overload. The obsolete -passing Preact snippet was removed from the known-failure baseline. No new -failure was added to that baseline. These changes alter no production code. - -Verification commands use `pnpm_config_verify_deps_before_run=false`: - -- Electric recovery owner: 60 reported tests passed, no type errors. The - preceding run reported four failures and 56 passes. -- Electron IPC suite: 116 reported tests passed, no type errors. -- Replay harness: 41 tests passed. Guarded direct replays of - `sqlite-resume.source-progress` and `browser.electric-source-prefix`, seed - `2281`, path `0`, each recorded one completed execution of the exact property. -- Documentation checker: 806 examples; zero new failures, 250 known failures, - and 59 known non-parsing examples. The earlier run had one new failure and - one stale baseline entry. -- Browser `tsc --noEmit` and the focused formatting check pass. -- `pnpm lint:changed` and `git diff --check` pass; no run budget was lowered. -- The real SDK/PostgreSQL/Chromium source-prefix receiver passes all ten cells; - every cell reaches each of its four declared states once. - -CodeRabbit review `5701493057`, inline `4415193533`, and mutable summary -`6502484352` cover the original published head. Their complete raw text and a -seven-item lossless ledger were preserved locally before edits. The startup -polls now explicitly allow 30 seconds. The shared checker now locates its -companion grammar and drivers and explains its independent source-head model. -Its judgment, name, registration, and comparison remain unchanged. - -Two historical phrases receive clarifications here to preserve this record's -append-only chronology: “new generated campaigns” means newly added -generated-history properties; “Browser important generated properties” means -both important generated properties in the Browser package. “Standalone type -examples” retains its existing meaning and spelling. - -The loss audit accounts for seven items: four fixed, one spelling requirement -refuted, one accepted documentation-policy choice, and one duplicate. The -summary's aggregate docstring warning names no missing function obligation or -repository authority for its 80% threshold. ORC-003's proportional prose and -directly named companion modules remain the governing documentation contract. -No behavioral finding, unresolved law decision, or new in-scope counterexample -was identified. CodeRabbit is useful as a supplemental maintenance reviewer; -this review supplies no independent behavioral proof. Earlier evidence and -limits remain unchanged because no production or model semantics changed. - -### Independent cluster report reconciliation (2026-10-29) - -The SQLite/Electric peer pilot investigated frozen baseline `07bfc0eb2`. -Its report, laws, retained oracle patch, raw reset/convergence logs, and claim -ledger were compared with this repair. Its two row/cursor counterexamples -confirm the mechanism already repaired here: an old row commit followed by -another source's newer resume metadata can make restart skip a needed change. -Both exact row traces and both reset traces pass again on `297e214fe`, with -no type errors. The pilot's baseline failures do not describe that newer tree. - -The report contributes one oracle improvement. The broad generated owner now -declares recovery first discovered during final catch-up separately from a -reset failure earlier in the history. An earlier failure can no longer satisfy -the reach requirement for that final branch. The model, history grammar, -comparisons, production code, and run budgets remain unchanged. - -- **Law:** Law 5 requires a reset-abandoned commit to fail; Law 2 compares the - independent source head after successful recovery and complete delivery. - Their authority remains the coverage map and maintainer decisions on #2265. -- **Reach:** `reset-abandoned source recovered during final catch-up` reached - 2 of 80 histories at fixed seed `952051961`, and 6 of 80 random histories. - Both campaigns require at least one. Existing reach requirements also pass. -- **RED / mutant:** Removing only the final restart/delivery block makes both - named reset histories fail at `after every tab caught up`, with wrong rows. - These are assertion failures, not setup failures or timeouts. This calibrates - the driver repair; it is not a new production mutant. Earlier production - RED and hostile source-progress evidence retain their scope. -- **GREEN:** Both normal 80-history campaigns and both positive reset replays - pass (7 reported tests, no type errors). A measurement-only run repeats the - same campaigns (2 runtime tests) and records the counts above. Its temporary - direct stderr recorder was restored afterward because ordinary console - output was suppressed. Logs: `pilot-current-*.log`, - `pilot-final-recovery-ablation.log`, `pilot-final-recovery-green.log`, and - `pilot-final-recovery-measured.log` in the local evidence directory. -- **Limits:** These remain bounded source-head observations. They do not - establish individual receipt fulfillment, clean teardown, generic source - ordering, or the other cluster issues' receiving paths. - -The report also sharpens two qualifications. The published Electric -resume-safety contract requires rejecting a partial update for an unknown row. -The separate #2267 no-error composition expectation needs an authority decision; -this repair preserves the guard and does not resolve that expectation. -The Node campaign's closed-database leadership warning is distinct from -deliberately injected source failures. It also appears in earlier local logs. -Its user-facing effect remains unverified, and the coverage map now records -the teardown owner and needed held-election witness under pilot `PILOT/S02`. -Passing convergence comparisons do not establish clean teardown. - -Follower reacquisition (#2139), seed metadata, real sync-metadata receiving -(#2272), broader leader-close premises (#2085), loaded-subset relay (#2174), and -real unknown-partial-message premises (#2138) remain separate boundaries. -The report supplies no reason to merge them into this source-progress repair. -The complete raw report and 16-item reconciliation ledger remain local. - -The next CI run exposed a Preact documentation fragment that inferred `{}` -for todo fields from the checker's synthetic source Collection. The example -now creates its own typed local Collection. A temporary source-resolution probe -against the real Preact hook reproduces both original JSX errors and passes -the typed example; the checker was restored afterward. The full documentation -check passes all 806 examples with zero new failures (250 known, 59 non-parsing). -No baseline entry or production code changed. Logs: -`pilot-preact-source-before.log`, `pilot-preact-source-after.log`, and -`pilot-preact-snippets-full.log`. - -### Explicit caller positions cannot bypass source-progress recovery - -CodeRabbit review `5702393171` on `883b565ab` found a missing history dimension. -The supplied cluster report still supports the repair's mechanism. This later -review exposes an incomplete enforcement boundary within that repair. - -The accepted law remains Browser Law 2: after successful recovery and complete -source delivery, durable rows and every open eager Collection equal the source -head. Electric's documented fresh-snapshot contract requires omitted cached rows -to disappear at replacement. A caller's provider position supplies no missing -baseline rows and cannot certify incompatible persisted source progress. - -The primary source-prefix grammar now varies omitted options, initial `-1`, -handle-only, numeric offset with handle, and the SDK's legal `now` offset. -It crosses those choices with existing membership, repeated toggles, roles, -advancement, draining, restart/reopen, and current/missing/foreign provenance. -Numeric noninitial offsets without handles are SDK-invalid and excluded. -The simulated server now interprets `now` as its head, supplying no baseline. -The independent source-head model, comparison, exclusions, and budgets remain. - -The proposed one-clause repair was insufficient. Merely requesting recovery -still let the explicit offset suppress replacement and reach `ShapeStream`. -The final repair separately identifies incompatible eager source progress and -uses the initial effective position for both stream construction and snapshot -replacement. Physical key-set checks retain their prior cursor guard. Other -sync modes and managed caches without scoped recovery retain their explicit -options. An independent reviewer identified that scope distinction before -publication. No generic ordering or new composition contract follows. - -- **Law:** Browser Law 2/#2209 and the Electric fresh-snapshot contract above. -- **Reach:** Both 80-history campaigns require all earlier states plus explicit - numeric/`now` divergent recovery and initial-offset/handle controls. Final - fixed/random counts are numeric 10/8, `now` 7/19, initial 17/12, and handle - 21/20. Fixed seed remains `2281`. Every original required state also passes. - Each of ten explicit real receiving cells reaches its supplied-position - state and all four original states once. -- **RED:** Before production edits, both extended Node campaigns fail the final - `after every tab caught up` row comparison. Numeric replay: seed `2281`, - path `8:0:0:0` in the four-form grammar. Final five-form grammar: seed `2281`, - path `1:0:0`, `now`. Real SDK/HTTP/PostgreSQL/Chromium OPFS checks reject eight - cells: lost/resurrected membership, current/legacy evidence, and numeric/`now` - positions. Initial-offset and handle-only neighbors pass. Failures show - concrete missing or extra durable rows, not setup errors or readiness waits. -- **GREEN:** The final complete Browser owner reports 312 passed and two tracked - expected failures, with no type errors. Electric recovery, resume races, - installed-SDK delivery, and ordinary Electric checks report 385 passed and - no type errors. Browser `tsc --noEmit` passes. The expanded real matrix passes - 20 cells; after narrowing the predicate to eager source progress, all ten - explicit cells pass again. The narrow and earlier predicate are equivalent - for every eager cell. The first Node GREEN had passing runtime comparisons - but failed on a test-driver type import. The final run fixes that import and - succeeds with type checking enabled. -- **Mutant:** Keep the independent recovery decision but restore raw caller - positions, as the one-clause suggested fix would do. The Node replay at - seed `2281`, path `1:0:0`, fails the original final row comparison. Restore - the exact production source and the same replay passes. The repaired source's - SHA-256 is `df8a918e736ca073619a693316c475040db46e2e30fc20fdbcc566934be695e6`. -- **Limits:** No claim covers arbitrary explicit cursor selection over an - otherwise certified baseline, explicit malformed/reset/tag/shape regimes, - changing caller forms within one history, or real foreign-token provenance. - Existing on-demand, optimism, cross-version, permanent-close, receipt and - teardown limits remain. The coverage map retains their separate owners. - -Commands use the earlier owner invocations, with `-t 'converges after interleaved -source prefixes'` for the focused Node campaign, and `--grep 'with explicit'` -for the real explicit cells. Replay sets `TANSTACK_DB_ORACLE_PROPERTY` to -`browser.electric-source-prefix`, seed `2281`, and path `1:0:0`. -`--disableConsoleIntercept` exposes measured reach without a temporary recorder. -Logs are `explicit-cursor-node-red{,-complete}.log`, -`explicit-cursor-{now-,}real-red.log`, `explicit-cursor-browser-final-green.log`, -`explicit-cursor-electric-final-green.log`, `explicit-cursor-real{,-final}-green.log`, -`explicit-cursor-incomplete-fix-mutant.log`, and `explicit-cursor-replay-green.log`. - -The full third-review loss audit includes the outside-diff finding, repeated -risk, aggregate docstring warning, and related-issue prose. The explicit-cursor -finding is fixed with its generated dimension and receiving evidence. The -summary's claim that the reported #2283 trace is outside this repair is stale: -that exact row-loss replay already passes here. The #2267 safety/composition -qualification remains separate. Raw reviews and dispositions remain local. - -### Construct stop/peer reach before arbitrary interleavings - -CI on `124ce667c61a9e247ea1628991bc72db91d6041e`, workflow `40927860887`, -SQLite-adapters job `123638080436`, found an instrumentation failure. The random -reserved-history campaign reached `peer commit after SDK gives up -mid-transaction` zero times. It reported no incorrect row comparison. The -unconstrained shuffle made the required four-action ordering too rare. - -Every generated history now begins with four six-action stop blocks in a -generated permutation. Each block toggles an initially absent key, opens a -source transaction, stops the source, commits from a distinct peer, recovers the -source, and catches it up. The source tab, key (3 or 4), recovery method, and -stop order vary. Four toggles restore the original membership before the -unchanged 8–24 arbitrary actions. Histories therefore contain 32–48 actions; -both campaigns still accept 80 histories, with fixed seed `952051961` and the -same replay interface. Every pinned history retains its unchanged direct path. - -This grammar excludes a peer equal to the stopped source and does not recover -that source before its peer commits. Removing partial delivery, the stop, or -the peer commit loses the premise. Removing recovery makes the later blocks -act on a stopped source. The arbitrary tail preserves deploys, resets, -interleavings, and the hostile witness below. The independent model, production -driver, comparison checkpoints, exclusions, and all reach requirements remain. - -- **Law:** Browser Laws 1 and 3 require a stopped source to end its reservation - so peer durable rows can apply. Authority: `reserveCommitTurn` and the - maintainer decision on PR #2265. Laws 2, 4, and 5 retain their earlier bounds. -- **Reach:** Fixed/random counts below include histories that later encounter - the documented #2267 discard: 81/80 executed for 80/80 accepted histories. - The constructed stop premises occur before any such discard. -- **RED:** The CI run above rejects the old grammar's missing SDK-give-up reach. - This is coverage RED, not a production row defect. -- **GREEN:** The focused owner command below passes both campaigns and its - associated type checks: three reported tests pass, no type errors. -- **Mutant:** Removing only Electric's SDK-give-up `abandonOpenTransaction` - rejects both campaigns. The fixed campaign shrinks to seed `952051961`, path - `1:0:0:0:0:0:0:0:1:1:0:0`: at step 31, the stopped tab still shows row 1 - while its peer and SQLite have deleted it. The random campaign also rejects - a stale peer row at final delivery (seed `-1481171989`, path - `15:0:0:1:0:1:1:1:1:1:1:1:2:2:2:2`). These are row assertion failures. - The first one-history diagnostic (seed `952051961`, path `0`) survived the - mutant, despite reaching the give-up callback with an open transaction. - It is not a kill. The retained arbitrary tail supplies the distinguishing - stale-row witnesses. Exact source restoration uses SHA-256 - `df8a918e736ca073619a693316c475040db46e2e30fc20fdbcc566934be695e6`. -- **Limits:** This constructs the controlled SDK stop boundary on the Node - Browser coordinator. It adds no real-provider claim or new product behavior. - Existing partial-update, permanent-close, receipt, and teardown limits remain. - -| Declared state | Fixed/random histories | -| --- | --- | -| must-refetch delivered | 55/43 | -| peer commit during an open reserved transaction | 17/10 | -| source stopped with an open transaction | 81/80 | -| tab recovered | 81/80 | -| durable rows behind the server | 81/80 | -| peer commit while a tab was not ready | 81/80 | -| peer commit after error without retry mid-transaction | 81/80 | -| peer commit after SDK gives up mid-transaction | 81/80 | -| peer commit after signal aborted mid-transaction | 81/80 | -| peer reset abandoned an open transaction | 12/13 | -| reset-abandoned commit put the Collection in error | 6/11 | -| reset-abandoned source recovered during final catch-up | 4/3 | - -```sh -pnpm_config_verify_deps_before_run=false pnpm \ - --filter @tanstack/browser-db-sqlite-persistence exec vitest run \ - tests/multi-tab-electric-oracle.test.ts --maxWorkers=2 \ - --disableConsoleIntercept -t 'keeps reserved commits lawful across generated' -``` - -Replay uses `TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history`, -`TANSTACK_DB_ORACLE_SEED=952051961`, and -`TANSTACK_DB_ORACLE_PATH=1:0:0:0:0:0:0:0:1:1:0:0` with that command. -Logs remain local as `ci-reach-green.log`, `ci-reach-abandon-mutant{,-campaign}.log`, -`ci-reach-abandon-path.log`, and `ci-reach-restored-replay.log`. - -The grammar change retains ORC-001–003's authority, independent judgment, and -literate boundaries. The construction and ablations above address ORC-004; -measured production reach and unchanged public comparisons address ORC-005. -The mutant supplies ORC-006/013 sensitivity, and both normal campaigns and direct -replay retain ORC-007. No model state changes (ORC-008), new abstraction -(ORC-009), or named shared semantic fault requiring another formulation -(ORC-011) arise. Existing cleanup and failure preservation retain ORC-010. -This versioned append supplies ORC-012 evidence; the controlled boundary limits -ORC-014's claim. All declared reaches remain enforced (ORC-015), and the file -continues to run shared judgments without example-specific answers (ORC-016). - -The restored shrunk replay passes with no type errors (two reported tests). -No production file remains changed by this calibration. - -### Interrupted eager recovery preserves its reset obligation - -CodeRabbit review `5702891774` on `124ce667c` identified the next missing -transition: recovery can stop after its reset marker commits but before a -replacement snapshot arrives. The next start selected `kind: reset`, which -bypassed the resume-only source-progress predicate. An explicit numeric position -or `now` could then request only incremental changes over the wrong cached rows. - -This violates the existing eager replacement and next-sync reset contracts in -`docs/collections/electric-collection.md`, under “Cleanup and resume safety.” -It requires no new source-ordering law and does not change #2267's safety guard. -The independent review agrees with that authority and identifies the receiving -seams below. No model rule or comparison changes. - -Each source-prefix generated input now runs its original uninterrupted history -and an interrupted neighbor. The neighbor starts recovery without explicit -options, stops before any source delivery, and recovers again with the selected -caller form. Production writes the reset; the driver reads its actual durable -kind before the second start. All original axes remain. Each campaign still -uses 80 inputs, now 160 histories. Original histories contain 6–13 actions; -interrupted histories add two actions. Numeric offsets without handles remain -SDK-invalid. Omitting the first recovery loses the reset premise; completing -its delivery loses the unfinished replacement. The unchanged draining neighbor -keeps the source-head comparison independently sensitive to missing/extra rows. - -The real driver holds the first actual SDK response of the pending recovery -before delivery. It observes the durable reset and closes that run. Caller -positions are captured before the reset removes their fields. The surviving -peer's responses are held too, preventing an empty commit from replacing the -marker. The next page uses the captured numeric position or `now`; PostgreSQL -still supplies the head to the unchanged shared comparison. Ten added cells -cross both membership directions and all five caller forms. They preserve all -20 earlier receiving cells. - -The eager-only reset predicate now participates in both fresh-evidence admission -and effective-position selection. Thus the requested source position and -snapshot replacement agree. Progressive/on-demand explicit-position behavior -and eager resumes with no incompatible/reset evidence retain their prior path. - -- **Law:** After successful eager recovery and complete delivery, durable rows - and every open Collection match the source head. An interrupted replacement's - reset still requires a full snapshot at the next start, including explicit - caller positions. Authority: the existing public recovery contract above and - Browser Law 2/#2209. -- **Reach:** Actual reset before divergent numeric/`now` recovery reaches 10/6 - and 7/8 fixed/random histories. All 13 earlier source-prefix reaches pass; - their final counts appear below. Each added real cell reaches actual durable - reset before interruption and all its original/explicit required states once. -- **RED:** Before production edits, both paired Node campaigns fail the final - `after every tab caught up` comparison with row 1 missing. Fixed seed `2281`, - path `1:0:0`; random seed `-636405297`, path `14:0:0:0:0`. The interrupted - real matrix reports four concrete wrong-row comparisons (numeric/`now` × - lost/resurrected), while six none/initial/handle controls pass. Production - SHA-256 was still `df8a918e736ca073619a693316c475040db46e2e30fc20fdbcc566934be695e6`. - Bot commit `623c59955` only formatted the earlier stop grammar and was - fast-forwarded without dropping any witness or production change. -- **GREEN:** The full Browser owner passes 312 tests and two tracked expected - failures, with no type errors. Browser `tsc --noEmit` passes. Electric ordinary, - recovery, and SDK suites report 336 passes; supplemental recovery, resume - races, and SDK grammar report 97 passes. Both commands report no type errors. - All 30 real receiving cells pass, including all ten interruptions. -- **Mutant:** Retain eager reset recovery admission but remove reset from the - effective-position override. Replay seed `2281`, path `1:0:0` rejects the - missing row at the final source-head comparison. Exact source restoration - uses SHA-256 `46a417f2d65ccc21adcc72f56d80388d6f2f9fc32a172ab6e3fa1e4702508047`. -- **Limits:** This covers bounded eager startup/restart/reopen after interrupted - recovery. It does not decide progressive/on-demand explicit-reset policy, - malformed/tagged/changed-shape explicit resumes, arbitrary cursor selection - over certified rows, real foreign-token receiving, mixed clients, native - hosts, permanent closure, receipt fulfillment, or clean teardown. - -| Source-prefix state | Fixed/random histories | -| --- | --- | -| present/empty | 28/26 | -| present/disjoint | 22/14 | -| absent/empty | 12/34 | -| absent/disjoint | 14/18 | -| drained before recovery | 84/68 | -| missing progress at recovery | 52/64 | -| foreign progress at recovery | 42/56 | -| divergent source restart | 50/52 | -| divergent source reload | 26/40 | -| explicit resume at divergent recovery | 20/12 | -| explicit now at divergent recovery | 14/16 | -| explicit initial offset at recovery | 34/42 | -| explicit handle at recovery | 42/32 | -| interrupted reset before explicit resume recovery | 10/6 | -| interrupted reset before explicit now recovery | 7/8 | - -The broad reserved-history campaign also passes every reach at the original -80-input budgets. Its fixed counts match the earlier constructed-grammar run. -Random counts are refetch 50, open-reservation peer 20, each constructed stop -boundary 80, peer reset 14, reset failure 5, and final reset recovery 2. -The first fixed campaign drives 81 histories including one documented #2267 -discard; its 80 accepted histories remain. No exclusion or reach minimum changes. - -Commands retain the earlier owner invocations. Node RED selects -`-t 'converges after interleaved source prefixes'`; final Browser GREEN omits -that selector. Real RED selects `--grep 'after interrupted recovery'`; final -real GREEN omits it. Replay uses property `browser.electric-source-prefix`, -seed `2281`, and path `1:0:0`. Logs are `reset-cursor-node-red.log`, -`reset-cursor-real-red.log`, `reset-cursor-browser-green.log`, -`reset-cursor-real-green.log`, `reset-cursor-electric{,-neighbors}-green.log`, -`reset-cursor-browser-types.log`, `reset-cursor-incomplete-mutant.log`, and -`reset-cursor-restored-replay.log`. The final production bundle delta is 573 -Gzip bytes across three packages, including 168 for Electric. The typical app -delta remains 45 bytes. The comparison uses merge-base `672684846` of fetched -main `bea84cd4554b7bd49df50cb0c569a34d8b2cd915`. - -The paired histories preserve reconstruction and direct replay (ORC-004/007). -Production-observed reset plus the shared comparison and incomplete-fix mutant -supply ORC-005/006/013/015 evidence. The real held-response receiving path supplies -ORC-014. Authority, independence, literate boundaries, cleanup, and law-only tests -retain ORC-001–003/010/016. No model state changes, new abstraction, or named -shared semantic fault trigger new ORC-008/009/011 work. This versioned append and -its finite limits supply ORC-012 evidence. - -Review maintenance items: the receiver now derives its offset type through -public `ElectricCollectionConfig`, avoiding the sibling `node_modules` path -without approximating the SDK's bigint offset form. The earlier phrase -“test-driver type import” means a type import in the test driver. Historical -validation wording and counts remain intact. The review's repeated reset risk, -aggregate docstring warning, and related-issue prose retain separate local -ledger entries; no review claim or pilot boundary disappears. - -After exact source restoration, the same shrunk replay passes (two reported -tests, no type errors). Lint and whitespace checks pass. The fourth review's -loss audit retains 24 items: eight fixed, four refuted, one accepted design, -and eleven duplicates. No in-scope finding remains open. External review covers -`124ce667c`; its automatic review is paused, so subsequent semantic changes -have the independent local source review and executed evidence above. +For a reported trace, set `TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history` +and its seed/path from the first table; select `-t 'random or replayed'`. +For the final source-prefix replay, use property `browser.electric-source-prefix`, +seed `2281`, and path `1:0:0` with the same selector. Fixed campaigns stand aside +during path replay. Normal campaigns require no replay variables. + +Final local runs used `pnpm_config_verify_deps_before_run=false` after upstream +release-specifier changes triggered automatic reinstall. Installed external +dependencies were unchanged. No override or lockfile workaround accompanies +this PR. Raw logs and the complete review ledger remain task-local under +`notes/issue-2281/`; essential evidence is summarized here and preserved in the +linked historical record.