diff --git a/.changeset/persisted-reload-keeps-status.md b/.changeset/persisted-reload-keeps-status.md new file mode 100644 index 0000000000..99c3bd75c9 --- /dev/null +++ b/.changeset/persisted-reload-keeps-status.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +--- + +Keep a persisted Collection's status when another tab's commit reloads its rows. Before, a reload from durable rows could mark a tab `ready` while its own source was still loading, or move a tab whose initial sync failed from `error` to `ready`. Now only the Collection's own source changes its status. diff --git a/.changeset/reserved-commit-defers-every-hydration.md b/.changeset/reserved-commit-defers-every-hydration.md new file mode 100644 index 0000000000..24a1f7499a --- /dev/null +++ b/.changeset/reserved-commit-defers-every-hydration.md @@ -0,0 +1,13 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +'@tanstack/db': patch +'@tanstack/electric-db-collection': patch +--- + +Keep a sync transaction that reserved its commit turn from failing when a persisted hydration starts while it is open. Before, only a subset load admitted during the reservation waited for the commit. A subset load admitted earlier, a reload or reset from a peer tab's notice, sequence-gap recovery, or a hydration that a shared scheduler held past its turn advanced the hydration cycle, and the commit then rejected with `a persisted sync transaction cannot cross a hydration cycle`. Electric reserves every stream transaction, so a deploy that made several tabs refetch at once could put their Collections into `error`. Each of these hydrations now waits outside the apply mutex until the reserved transaction commits, then starts. While it waits, readers keep the rows from before it. The `reserveCommitTurn` contract now covers every persisted hydration. + +A peer reset now applies in arrival order with peer commit notices, so a commit from before the reset can no longer publish a row that the reset removed. + +Electric now abandons its open transaction when its stream stops for good, so hydrations that wait for that transaction can run. + +An abandoned reservation now releases the subset loads that waited for it, as a commit does, so they run. Before, a load admitted while the turn was reserved rejected with `SyncTransactionAbortedError` when the source abandoned the transaction. diff --git a/docs/contributing/glossary.md b/docs/contributing/glossary.md index b67a234605..6f484efac5 100644 --- a/docs/contributing/glossary.md +++ b/docs/contributing/glossary.md @@ -34,6 +34,7 @@ production queues, caches, or semantic helpers merely to share their names. | optimistic state | The rows an optimistic transaction overlays on the applied synced rows while its mutation function runs. It drops when the mutation function settles, whether it fulfills or rejects. | Retained snapshot or accepted local snapshot. | | accepted sync transaction | A committed sync transaction that core, or a wrapping sync after its durable step, has taken. It always applies, in commit order. While an optimistic transaction is persisting it is held, and it applies in the publication that drops that transaction's optimistic state. Handler-facing writes and Collection readiness wait for acceptance; commit receipts and subset loads wait for visibility. | Visible or applied sync transaction. | | deferred sync transaction | A committed sync transaction that a wrapping sync has not handed to the Collection yet, because an earlier durable write is still running. Like a queued sync transaction, it holds settled optimistic rows on its keys until it applies. | Queued or accepted sync transaction. | +| reserved sync transaction | A sync transaction whose source called `reserveCommitTurn` and that has not committed or been abandoned. Every persisted hydration that would start while it is open waits for its commit. A source that stops for good abandons it. | Deferred sync transaction or held optimistic row. | | held optimistic row | A successful transaction's optimistic row kept only while a committed sync transaction that has not applied touches its key, so the drop and that transaction publish together. The transaction may be queued in the Collection or deferred behind a wrapping sync's durable step. Its source attribution follows the key-and-timing rules in [VirtualOrigin](../reference/type-aliases/VirtualOrigin.md), not source-client identity. | Retained or accepted snapshot. | | Collection status | One public Collection lifecycle value: `idle`, `loading`, `ready`, `error`, `crashed`, or `cleaned-up`. | Subscription status. | | subscription status | One subscription value: `ready` or `loadingSubset`. | Collection status. | diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b6d649ada2..c34f023efc 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2592,7 +2592,20 @@ consistency between them: while their streams interleave, a lagging tab's copy of an older transaction can be stored and announced after a newer one, so a tab can briefly show an older server state. Every tab delivers the same log in order, so durable state and every tab converge once the streams -catch up. No lane checks this yet. Ordering the copies would need a generic +catch up. No lane checks this yet. +A counterexample bounds that convergence claim +([#2209](https://github.com/TanStack/db/issues/2209)). Two Browser tabs +share one SQLite file through the real coordinator, with a controlled +Electric stream per tab. Tab A's stream applies v1 then v2 at `30_0`. Tab B's +lagging stream applies v1 at `20_0`, and tab B closes before it reaches v2. +Tab A then shows v1, and the durable `electric:resume` offset is back at +`20_0`. Nothing redelivers v2 to tab A while it stays open, because its +stream is already past it. A tab that opens later resumes from `20_0` and +repairs the cache. The convergence above therefore holds only while the +lagging tab keeps running or another tab restarts its stream. The +multi-tab Electric oracle in `browser-db-sqlite-persistence` pins this +history with `it.fails` beside a passing control in which the lagging tab +catches up. Ordering the copies would need a generic source position on sync commits; Electric already stores `{ handle, offset }` with each commit, PowerSync does not use this wrapper, and persisted Query collections have no log position. @@ -3460,6 +3473,182 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. not establish behavior after schema changes or invalidated resume points, or each intermediate publication during a multi-transaction truncate replay. The same owner needs those histories before claiming those stronger laws. + Reserved commits (#2226). The same driver and model also own the + reserved-commit law: a persisted hydration that would start while a + reserved sync transaction is open waits until it ends, and the commit is + accepted. Authority: `reserveCommitTurn` in `packages/db/src/types.ts`, + widened by maintainer decision on #2226 to every persisted hydration. + Further maintainer decisions fix its edges. Readers keep the rows from + before a hydration while it waits; a transaction opened after a + hydration began straddles it. A notice that waited may be retried as + gap recovery. A tab asks the leader only to fill a gap it has not + already filled, and sends no added upstream request. Each reservation + a notice waited for can open one gap, so the oracle checks at most one + more pullSince request per wait (maintainer decision on PR #2265). A source that stops for good abandons its + reservation. + A second registration crosses the routes with the reserving timings, + ending, and follow-up dimensions. The routes are subset, complete + subset, a follower's routed subset, the source's own baseline, + reset, full reload, sequence gap with or without pullSince, sequence + gap whose pullSince deltas reload partway through (a targeted delta, + then one past the targeted-invalidation limit), and a targeted change + that starts no hydration. Gap recovery is a hydration cycle only from + the point where it will change rows (maintainer decision on PR #2265): + after pullSince answers, or at its reload without pullSince. A + reservation opened after that point straddles it; one opened during + pullSince makes it wait, and a transaction that did not reserve may + commit during pullSince. A peer reset abandons a reserved transaction + that began before it (maintainer decision on PR #2265): the commit + rejects and the reset does not wait. The reserving timings are + before a subset load's admission, during gap recovery's pullSince + (with or without a reservation), across the mutex turn, across scope + entry, during the read, in the route's first publication, and across + startup. A load that waits for a reservation runs after it ends, + whether it commits or is abandoned (maintainer decision on PR #2265); + on `851e8d9da` the abandoned admission cells failed with a rejected + load. The ending is commits, + abandoned, cleanup, an aborted demand, or the source awaiting its own + baseline. The follow-up is a reset then a pre-reset commit, or a + reset, a new reservation, and then the pre-reset commit. A reservation + that a reset abandoned cannot reserve again; on `7a6f32fd7` 8 + re-reserve cells failed because a later hydration waited for the + abandoned transaction. An aborted demand still waiting for its turn is + no longer active demand. It receives the reset follow-up, or a peer's + targeted change for a row no live demand covers; nothing reads or asks + the leader for it. On `7a6f32fd7`, 12 of the 24 aborted-demand cells + fail: the reset's reload read for it, and the peer change sent a + pullSince past the bound. Filtering aborted demand only from reloads + still fails the 4 peer-change cells that wait at the mutex turn or + scope entry. + Observations: receipts, final rows, row metadata, cursor, status, + reads during the reservation, upstream requests, pullSince count, + rows while waiting, and every read against the fold of the durable + events before it. Every cell declares reach (ORC-015). + | Evidence | Result | + | --- | --- | + | Row-algebra cross and reservation cross | 677 cells pass. | + | Peer reset against an older reservation, on `2047f4281` | 53 cells fail: the reset waited and the stale commit landed in the new epoch. Abandoning only at the reset's turn fails 6 follow-up cells; abandoning only on arrival fails 40 route cells. | + | Gap recovery's cycle start, on `2047f4281` | 12 pullSince cells fail: an unreserved commit during pullSince rejected, and a reservation opened then straddled. Dropping the reservation check after pullSince fails 8. | + | Gap recovery that reloads partway through, on `851e8d9da` | 28 cells fail: recovery advanced the stream position and applied a targeted delta before it deferred, so its retry skipped the reload. A gate only at recovery start still fails 15 scope-entry cells; reloads that start their own cycle inside recovery fail 61. | + | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | + | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | + A routed load also judges law 7 at its settlement: the owner's row is + public when the load settles. A reservation that ends between the + follower's flush and its wait check is a one-microtask window no + timing reaches; a temporary hook that ended the reservation there + showed the old wait loop settling early in 4 straddled cells and the + current loop keeping the owner's row public. A new reservation that + opens between the loop's wait and its flush defers the owner's + broadcast again. The loop checks for a reservation after each flush; a + temporary hook that opened one there showed the loop on `7a6f32fd7` + settling the routed load in 2 cells before the owner's row was public. + A third registration reaches it without a hook. The owner's reply is + held while its broadcast waits; the source commits 0 to 8 microtasks + after the reply and reserves an empty second transaction 0 to 12 + microtasks later. On `7a6f32fd7` the 12 cells that commit 3 microtasks + after the reply settle the routed load before the owner's row is + public; all 117 cells pass on the current loop. A held write after the + commit does not open the window, because a reservation lasts through + its own durable write. + A second reservation that opens 7 or more microtasks after the commit + holds the broadcast twice, and the notice asks the leader once per + wait, as the bound allows. A peer reset for a + storage generation this Collection no longer uses abandons nothing. + Known limit (maintainer decision on PR #2265): Electric publishes a + transaction's txids as stream evidence before persistence accepts its + commit, so `awaitTxId` can resolve for a commit that a peer reset or a + terminal failure later rejects; the source then fails, and the + Collection's health node records it. + Known limit (maintainer decision on PR #2265): no fairness rule. A + hydration released by a commit waits again for a reservation that + opens before its next mutex turn, so a source that reserves back to + back can delay it; Electric reserves across batches only during an + initial sync or refetch. + Not covered: `forceReloadSubset`, which only a sync-absent Collection + has, and which therefore never meets a reservation; startup hydration, + which source entry follows (an `open across startup` observation); + and row visibility for on-demand Collections without demand beyond the + source's own rows. Cleaned-up cells judge only cleanup. + Real SQLite adapter: + `packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts` + crosses three routes (peer full reload, peer reset, and a subset + admitted earlier) with two timings (a held write of this Collection, + or the shared scheduler holding the scope after its turn) and two + endings (commits, abandoned). It runs on `SQLiteCorePersistenceAdapter`. + It also judges that a subset load that waited settles successfully. + Its peer-reset cells judge that the reset abandons the reservation: + the commit rejects and row-1 keeps its durable value (both committing + peer-reset cells fail on `2047f4281`). + A nesting dimension adds a second reserved transaction inside the + first. A peer reset must abandon both, so both reject with the same + abort; on `5ac5a8711` the inner one rejected with the cross-cycle + error instead. Its 20 cells pass. + Every committing single-transaction cell fails on `5649a07d4`. A + mutant that rejects the waited load after the reservation fails the 4 + subset cells; before the load's outcome was judged, it passed. A peer + full reload behind this Collection's own held write cannot become + durable through the single SQLite writer, so it is excluded here and + left to the grammar's fixture leader. + Browser host: the multi-tab Electric oracle in + `browser-db-sqlite-persistence` crosses two or three tabs, eager or + on-demand Collections, which tab finishes first, a storm or staggered + arrival, the last tab's open reserved transaction (its refetch, or an + ordinary batch after it resumed from the persisted position), how its + source stops for good (an error that `onError` does not retry, the SDK + giving up, a signal abort, cleanup, or resume invalidation), and how + the application recovers it (restart in the page, or reload). It + judges every tab's status once every source has delivered or stopped, + and again after recovery. Eager cells also compare rows at both + checkpoints, except a cleaned-up tab before recovery. It also judges the + full-snapshot request after resume invalidation. A Collection in + `error` keeps showing durable rows (maintainer decision on PR #2265), + so the tab whose source resume invalidation failed must show its + peers' refetch; it stays `ready` with an error on its health node. Its 290 storm cells pass. Every ordinary storm + cell fails on `5649a07d4`. Removing Electric's abandon on the `onError`, + give-up, or signal-abort path fails 12 cells each, and on the resume + invalidation path 16 cells. Cleanup's abandon mutant survives here, + because the persisted wrapper ends its transactions on cleanup; the + Electric descriptor-isolation oracle kills it without persistence. + On-demand cells judge receipts and status only. + Generated histories in the same file interleave server changes, + deploys, partial and complete deliveries, every stop except resume + invalidation, restarts, and reloads across two or three eager tabs + (fixed seed 952051961 and a random seed, 60 runs each; the shared + `TANSTACK_DB_ORACLE_SEED`, `_PATH`, and `_PROPERTY= + browser.multi-tab-reserved-history` replay one). The + model folds committed batches into the expected durable rows. After + every step it judges the durable rows, every Collection status, and + the rows of each tab with no open transaction; after every tab catches + up it judges them again. A Collection status changes only through its + own source (maintainer decision on PR #2265): a peer's commit leaves a + `loading` or failed tab as it is. That law failed on `main` (a peer's + truncate reload marked such a tab ready); the fixed seed fails without + the repair. Removing the reservation gate, or the `onError`, give-up, + or signal-abort abandon, fails both campaigns. Each declared premise, + including a peer commit after each kind of stop mid-transaction, is + reached in every campaign. Known gap #2267 is left out of the + generated histories and pinned with `it.fails`: a resumed tab that lags + behind a peer can receive an update for a row the peer already + deleted, and Electric's resume check puts it in `error`. + The histories also post a peer `collection:reset` on the tabs' + coordinator channel; no shipped host sends one yet, so this premise is + controlled. The reset abandons every open reserved transaction, and + Electric's next commit then fails the source until the tab restarts + (maintainer decision on PR #2265): a ready tab keeps `ready` with an + error on its health node, which the histories judge, and a loading tab + goes to `error` (#2168). A reset that waits and + accepts the stale commit fails both campaigns. + Electric's updates carry only changed columns by default, so an update + for a row the tab does not hold applies nothing; the fold follows that + rule (a CI random campaign found the history where a lagging tab's + update follows a peer's delete). + A delivery may also commit only a prefix of what a stream owes, as a + server answering at an earlier moment does, so lagging commits are + generated. The final checkpoint requires the server head (Law 2). + Known gap #2209 is the one history that breaks it: a lagging tab + commits an older row and closes for good. Generated histories never + close a tab for good, and an `it.fails` replay through the same driver + and checks pins that history. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in @@ -4357,7 +4546,12 @@ public and durable state. A second delivery lands during rotation, before the replacement stream starts; it rejects old callback admission into the new cache. Initial scoped recovery also forgets imported old txid evidence, and retiring an incomplete old stream transaction releases its commit turn so a -new demand can start. These histories killed the one-phase restart, retained +new demand can start. A progressive atomic swap that throws in `getKey` +leaves its reserved transaction open; when the SDK then gives up, Electric +must abandon it, and a subset load that waited runs. On `7a6f32fd7` the swap +did not mark its transaction open, so no stop could abandon it and the load +waited until its checkpoint timed out. The metadata-only commit shares the +same begin and commit helpers but has no fallible step between them. These histories killed the one-phase restart, retained acknowledgement, and open-transaction designs at their named checkpoints. The Electric descriptor owner also holds rotation while aborting one old demand: that demand rejects promptly and a sibling later applies fresh source rows. diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts new file mode 100644 index 0000000000..01aa7342c6 --- /dev/null +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -0,0 +1,1462 @@ +// @vitest-environment node +/** + * # Do Electric tabs that share one persisted database agree with the server? + * + * Several Browser tabs open the same persisted Electric Collection. Each tab + * runs its own Electric stream, and all of them store through one SQLite + * database and one elected leader. Tabs coordinate only through durable + * commits and the notices those commits broadcast. This file is the Browser + * host owner for three laws about that arrangement. + * + * Law 1, reserved commits survive a refetch storm. Authority: + * SyncPersistenceCapabilityV1 `reserveCommitTurn` (packages/db/src/types.ts), + * widened by maintainer decision on issue #2226 to every persisted + * hydration. Electric reserves every stream transaction. After a deploy, each + * tab's stream sends must-refetch; the tab commits a reset resume marker at + * once, then holds a reserved refetch transaction open until up-to-date. A + * peer's marker or refetch commit reaches the other tabs while their + * refetches are open. The law: every tab's refetch commits, no Collection + * enters `error`, and once every stream has delivered the new snapshot every + * tab shows it. The controlled-history owner for the same law is the + * persisted-reads grammar in + * packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts; this + * file is its receiving witness on the real Browser coordinator. + * + * Law 2, an open tab converges to the server head. Authority: the coverage + * map's eventual-consistency note for persisted multi-tab Collections + * (docs/contributing/oracle-coverage.md, issue #2209): a lagging tab may + * briefly store an older transaction, and every tab converges once the + * streams catch up. The generated histories below judge it: after every + * open tab catches up, the durable rows and every open tab show the server + * head, however earlier lagging commits left them. When a lagging tab + * commits an older row and then closes for good, nothing redelivers the head + * to the faster tab, so this law fails today. That history is pinned with + * `it.fails` until #2209 orders source writes by server position; generated + * histories never close a tab for good. + * + * Model: the server is a log of snapshots. A history names which snapshot + * each tab's stream has delivered; the expected public rows of a tab whose + * stream reached the head are the head snapshot. This file states no + * contract for which stream rows an on-demand Collection without subset + * demand exposes, so on-demand histories judge only commit acceptance and + * Collection status (a partial oracle). The model reads nothing from the + * coordinator, adapter, or Collection. + * + * Driver: each tab is a better-sqlite3 connection to one shared file, a + * `BrowserCollectionCoordinator` (BroadcastChannel and Web Locks in Node), and + * `createBrowserWASQLitePersistence`. A recording `ShapeStream` stands in for + * the Electric SDK; the test plays the server and delivers each tab's + * messages. Checkpoints wait for host quiescence: no SQL statement and no + * coordinator message for several tasks. Observations are each tab's + * Collection status and public rows, and the position from which a + * recovered tab's new stream starts. + * + * Law 3, a Collection whose source failed keeps showing durable rows. + * Authority: maintainer decision on PR #2265. A persisted Collection whose + * source failed still applies its peers' durable commits, so its public rows + * match the durable rows. A failure before the first load completes puts it + * in `error`; a later one leaves it `ready` with an error on its health node + * (#2168: after readiness, sync failures never change Collection status). + * This file judges it after resume invalidation, after a failed initial + * sync, and after a reset abandoned the source's transaction; the source + * must end its reserved transaction for the peers' commits to apply. + * + * Law 4, a Collection status changes only through its own source. Authority: + * maintainer decision on PR #2265. A peer's durable commit reloads a tab's + * rows but never moves its status: a tab whose own stream has not reached + * up-to-date stays `loading`, and a tab whose initial sync failed stays in + * `error` until it restarts. + * + * Law 5, a peer reset abandons an older reservation, and Electric fails + * loudly. Authority: maintainer decisions on PR #2265. A peer + * `collection:reset` abandons every reserved transaction that began before + * it. Electric still holds that transaction, so its next commit rejects and + * the source fails: a ready Collection keeps `ready` with an error on its + * health node, one that is still loading goes to `error`. The app recovers by + * restarting the tab. + * No shipped host sends `collection:reset` yet, so the histories post it on + * the tabs' coordinator channel as a peer would (a controlled premise). + * + * The generated histories below judge Laws 1, 3, 4, and 5 over interleaved + * server changes, deploys, deliveries, stops, and recoveries, against a + * model of the durable rows. One known gap is left out of them and pinned: + * a resumed tab that lags behind a peer can receive an update for a row the + * peer already deleted, and Electric's resume check then puts it in `error`. + * + * Electric's own cleanup abandon is redundant behind the persisted wrapper, + * which ends its transactions on cleanup, so that mutant survives here; the + * Electric descriptor-isolation oracle judges it without persistence. + * + * Limits: the stream is controlled, not the installed SDK over HTTP; one + * process hosts every tab; one Collection; no subset demand in on-demand + * Collections; no optimistic mutations; work amplification across tabs is + * not judged. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { fc } from '@fast-check/vitest' +import { createCollection } from '@tanstack/db' +import { createOracleReach } from '../../db/tests/oracle-reach.js' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { + oraclePropertyOptions, + oracleReplayRequested, + oracleRuns, +} from '../../db/tests/oracle-config' +import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' +import { electricCollectionOptions } from '../../electric-db-collection/src/electric' +import { BrowserCollectionCoordinator } from '../src/browser-coordinator' +import { createBrowserWASQLitePersistence } from '../src/browser-persistence' +import { createWASQLiteTestDatabase } from './helpers/wa-sqlite-test-db' +import type { + Message, + Row, +} from '../../electric-db-collection/node_modules/@electric-sql/client' +import type { ElectricCollectionUtils } from '../../electric-db-collection/src/electric' + +type Item = Row & { id: number; label: string } + +// --------------------------------------------------------------------------- +// Controlled Electric SDK boundary. Each stream records the position Electric +// started it at and exposes the subscriber, so the test can play the server. +// --------------------------------------------------------------------------- + +type ControlledStream = { + tab: string + // The position Electric asked the server to start from. + startedAt: { handle: string | undefined; offset: string } + shapeHandle: string | undefined + lastOffset: string + // Set when Electric releases the stream; a stopped stream gets no messages. + stopped: boolean + send: (messages: Array>) => void + // Electric's stream-error handler, which the test calls to fail the stream. + onError: (error: unknown) => unknown + // The subscriber's error callback, which the SDK calls when it stops + // retrying. + giveUp: (error: unknown) => void +} + +const streams: Array = [] + +vi.mock( + `../../electric-db-collection/node_modules/@electric-sql/client`, + async (importOriginal) => { + const actual = await importOriginal>() + return { + ...actual, + ShapeStream: vi.fn( + (options: { + headers?: Record + offset?: string + handle?: string + onError?: (error: unknown) => unknown + }) => { + const stream: ControlledStream = { + tab: options.headers?.[`x-tab`] ?? `unknown`, + startedAt: { + handle: options.handle, + offset: options.offset ?? `-1`, + }, + stopped: false, + shapeHandle: options.handle, + lastOffset: options.offset ?? `-1`, + send: () => { + throw new Error(`stream for tab ${stream.tab} is not subscribed`) + }, + onError: (error) => options.onError?.(error), + giveUp: () => { + throw new Error(`stream for tab ${stream.tab} is not subscribed`) + }, + } + streams.push(stream) + return { + subscribe: ( + send: ControlledStream[`send`], + giveUp: ControlledStream[`giveUp`], + ) => { + stream.send = send + stream.giveUp = giveUp + return () => undefined + }, + unsubscribeAll: () => { + stream.stopped = true + }, + requestSnapshot: vi.fn().mockResolvedValue(undefined), + fetchSnapshot: vi + .fn() + .mockResolvedValue({ metadata: {}, data: [] }), + forceDisconnectAndRefresh: vi.fn().mockResolvedValue(undefined), + isUpToDate: false, + get shapeHandle() { + return stream.shapeHandle + }, + get lastOffset() { + return stream.lastOffset + }, + } + }, + ), + } + }, +) + +const upToDate: Message = { headers: { control: `up-to-date` } } +const mustRefetch: Message = { headers: { control: `must-refetch` } } +const insert = (row: Item): Message => ({ + key: String(row.id), + value: row, + headers: { operation: `insert` }, +}) +const update = (row: Item): Message => ({ + key: String(row.id), + value: row, + headers: { operation: `update` }, +}) + +const nextTask = () => new Promise((resolve) => setTimeout(resolve, 0)) + +// --------------------------------------------------------------------------- +// Browser host: one database file, one coordinator and persistence per tab. +// --------------------------------------------------------------------------- + +type SyncMode = `eager` | `on-demand` +type StreamStop = + | `error without retry` + | `SDK gives up` + | `signal aborted` + | `cleanup` + | `resume invalidated` + +function openHost() { + const directory = mkdtempSync(join(tmpdir(), `tsdb-multi-tab-`)) + const filename = join(directory, `shared.sqlite`) + const dbName = `multi-tab-${Math.random()}` + const closers: Array<() => Promise> = [] + let observer: ReturnType | undefined + // Host activity for the quiescence checkpoint, and the number of commit + // notices, read from the coordinator's own channel. + let activity = 0 + let commitNotices = 0 + let storageCollectionId: string | undefined + let resetEpoch = 0 + const channel = new BroadcastChannel(`tsdb:coord:${dbName}`) + channel.onmessage = (event: MessageEvent) => { + activity++ + const envelope = event.data as { + senderId?: string + collectionId?: string + payload?: { type?: string } + } + if (envelope.payload?.type === `tx:committed` && envelope.senderId) { + commitNotices++ + storageCollectionId ??= envelope.collectionId + } + } + + const openTab = (tab: string, syncMode: SyncMode) => { + const database = createWASQLiteTestDatabase({ filename, timeout: 0 }) + const execute = database.execute.bind(database) + database.execute = async ( + sql: string, + params: ReadonlyArray = [], + ) => { + activity++ + for (;;) { + try { + return await execute(sql, params) + } catch (error) { + if ((error as { code?: string }).code !== `SQLITE_BUSY`) throw error + await nextTask() + } + } + } + const coordinator = new BrowserCollectionCoordinator({ dbName }) + // The application's signal, which can stop the stream from outside. + let external = new AbortController() + const shapeOptions = { + url: `http://electric.test/v1/shape`, + params: { table: `items` }, + headers: { 'x-tab': tab }, + signal: external.signal, + } + const collection = createCollection( + persistedCollectionOptions< + Item, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `items`, + shapeOptions, + syncMode, + getKey: (row) => row.id, + startSync: true, + }), + persistence: createBrowserWASQLitePersistence({ + database, + coordinator, + schemaMismatchPolicy: `reset`, + }), + }), + ) + let open = true + const close = async () => { + if (!open) return + open = false + await collection.cleanup().catch(() => undefined) + coordinator.dispose() + await database.close?.() + } + closers.push(close) + const stream = () => { + const own = streams.filter((candidate) => candidate.tab === tab) + const latest = own[own.length - 1] + if (!latest) throw new Error(`tab ${tab} has no stream`) + return latest + } + return { + tab, + collection, + close, + hasStream: () => streams.some((candidate) => candidate.tab === tab), + streamCount: () => + streams.filter((candidate) => candidate.tab === tab).length, + startedAt: () => stream().startedAt, + deliver: ( + messages: Array>, + position: { handle: string; offset: string }, + ) => { + const current = stream() + if (current.stopped) { + throw new Error(`tab ${tab} delivered to a stopped stream`) + } + current.shapeHandle = position.handle + current.lastOffset = position.offset + current.send(messages) + }, + // Each way a source stops for good: an error that `onError` does not + // retry, the SDK giving up its own retries, the application's signal + // aborting the stream, the application cleaning up the Collection, or + // a resumed stream sending an update for a row it never sent, which + // invalidates the persisted resume position. + stopStream: (how: StreamStop) => { + const error = new Error(`stream failed`) + if (how === `error without retry`) return stream().onError(error) + if (how === `SDK gives up`) return stream().giveUp(error) + if (how === `signal aborted`) return external.abort() + if (how === `cleanup`) return collection.cleanup() + const current = stream() + return current.send([update({ id: 99, label: `never sent` })]) + }, + // The application's recovery: clean up and start the Collection again. + // A signal that already aborted would stop the new stream too, so the + // restarted session gets a fresh signal. + restart: async () => { + await collection.cleanup() + external = new AbortController() + shapeOptions.signal = external.signal + void collection.preload().catch(() => undefined) + }, + publicRows: () => + Array.from(collection.values(), ({ id, label }) => ({ + id, + label, + })).sort((left, right) => left.id - right.id), + } + } + + // The hosts are quiet when five 2 ms tasks pass with no SQL statement and + // no coordinator message, within 400 tasks; the bound turns a livelock into + // a named failure. Quiet is evidence, not proof, that every notice and + // hydration finished, because the public API exposes no completion signal + // for peer notices. Its premises are calibrated instead: declared reach + // shows the peer commit arrived, and the abandon and readiness mutants fail + // only when a hydration ran after that commit. + const quiesce = async (label: string) => { + let quietTasks = 0 + for (let task = 0; task < 400 && quietTasks < 5; task++) { + const before = activity + await new Promise((resolve) => setTimeout(resolve, 2)) + quietTasks = activity === before ? quietTasks + 1 : 0 + } + if (quietTasks < 5) throw new Error(`hosts did not quiesce: ${label}`) + } + + return { + openTab, + quiesce, + commitNotices: () => commitNotices, + // A peer tab's collection reset, posted on the tabs' coordinator channel + // as a peer would post it. No shipped host sends this notice yet, so + // the premise is controlled; the tabs, their coordinators, and Electric + // receive it through production. + broadcastPeerReset: () => { + if (!storageCollectionId) { + throw new Error(`no commit notice named the storage collection yet`) + } + channel.postMessage({ + v: 1, + dbName, + collectionId: storageCollectionId, + senderId: `peer-reset-tab`, + ts: Date.now(), + payload: { + type: `collection:reset`, + schemaVersion: 1, + resetEpoch: ++resetEpoch, + }, + }) + }, + // The durable rows of the shared Collection, read through a connection of + // its own so the read counts as no host activity. + durableRows: async (): Promise> => { + observer ??= createWASQLiteTestDatabase({ filename, timeout: 1000 }) + const registry = await observer.execute<{ table_name: string }>( + `SELECT table_name FROM collection_registry WHERE collection_id = ?`, + [`items`], + ) + const table = registry[0]?.table_name + if (!table) return [] + const rows = await observer.execute<{ value: string }>( + `SELECT value FROM "${table}"`, + ) + return rows + .map((row) => JSON.parse(row.value) as Item) + .map(({ id, label }) => ({ id, label })) + .sort((left, right) => left.id - right.id) + }, + close: async () => { + await observer?.close?.() + for (const close of closers.reverse()) await close() + channel.close() + rmSync(directory, { recursive: true, force: true }) + }, + } +} + +type Host = ReturnType +type Tab = ReturnType + +async function openWarmTabs(host: Host, count: number, syncMode: SyncMode) { + const tabs: Array = [] + for (let index = 0; index < count; index++) { + const tab = host.openTab(String.fromCharCode(65 + index), syncMode) + for (let task = 0; task < 400 && !tab.hasStream(); task++) { + await new Promise((resolve) => setTimeout(resolve, 2)) + } + expect(tab.hasStream()).toBe(true) + tabs.push(tab) + } + return tabs +} + +// Generated-history campaigns (ORC-007): a fixed seed and a random seed +// with the same property and budget. TANSTACK_DB_ORACLE_SEED with +// TANSTACK_DB_ORACLE_PATH and TANSTACK_DB_ORACLE_PROPERTY= +// browser.multi-tab-reserved-history replays one recorded counterexample; +// TANSTACK_DB_ORACLE_RUNS_MULTIPLIER scales the budget. +const GENERATED_PROPERTY = `browser.multi-tab-reserved-history` +const GENERATED_FIXED_SEED = 952051961 +const GENERATED_RUNS = 80 + +async function waitForStream(tab: Tab, count: number) { + for (let task = 0; task < 400 && tab.streamCount() < count; task++) { + await new Promise((resolve) => setTimeout(resolve, 2)) + } + expect(tab.streamCount()).toBe(count) +} + +// The server after the deploy, answering a new stream from where it starts: +// the head handle is up to date, a full-snapshot request gets the head +// snapshot, and the old handle must refetch. +async function serveHead( + host: Host, + tab: Tab, + startedAt: { handle: string | undefined; offset: string }, +) { + const head = { handle: `h2`, offset: `20_0` } + if (startedAt.handle === `h2`) tab.deliver([upToDate], head) + else if (startedAt.handle === undefined || startedAt.offset === `-1`) { + tab.deliver([...afterDeploy.map(insert), upToDate], head) + } else { + tab.deliver([mustRefetch], { + handle: startedAt.handle, + offset: startedAt.offset, + }) + await host.quiesce(`tab ${tab.tab} restart refetch open`) + tab.deliver([...afterDeploy.map(insert), upToDate], head) + } + await host.quiesce(`tab ${tab.tab} restarted`) +} + +// --------------------------------------------------------------------------- +// The server log. Snapshot 1 is the state before a deploy, snapshot 2 after: +// row 1 changes, row 2 is deleted, row 3 is new. +// --------------------------------------------------------------------------- + +const beforeDeploy: Array = [ + { id: 1, label: `one` }, + { id: 2, label: `two` }, +] +const afterDeploy: Array = [ + { id: 1, label: `one, refetched` }, + { id: 3, label: `three` }, +] + +describe(`multi-tab Electric over one persisted database`, () => { + let host: Host | undefined + afterEach(async () => { + await host?.close() + host = undefined + streams.length = 0 + }) + + // Law 1 grammar. Dimensions: two or three tabs; eager or on-demand + // Collections; which tab finishes its refetch first; whether every tab's + // must-refetch arrives before any refetch finishes (a storm) or each tab + // finishes before the next receives must-refetch (staggered, the + // neighbouring control); and whether the last tab's source stops for good + // while it holds an open reserved transaction, through each terminal path + // Electric has. That transaction is either its refetch or an ordinary batch + // of a stream that resumed from the persisted position, with no up-to-date + // yet. Resume invalidation needs the resumed batch: a refetch already + // replaces the snapshot, and on-demand Collections never validate resume. + // + // A source that stops for good abandons its reserved transaction + // (maintainer decision on #2226). So a stopped tab that is still open + // applies its peers' refetch from durable rows, as it did before + // reservations deferred hydrations. Electric then fails no further, except + // that resume invalidation fails the source, as Electric documents: the + // ready tab keeps `ready` with an error on its health node and still shows + // durable rows (Law 3). The + // application's recovery, cleanup and a restart, then brings + // the tab to the head. After resume invalidation the restart must request a + // full snapshot (electric.ts: "a full snapshot is required"). + type StormHistory = { + tabCount: number + syncMode: SyncMode + finisher: `first` | `last` + arrival: `storm` | `staggered` + open: `refetch` | `resumed batch` + stop: `none` | StreamStop + recovery: `none` | `restart` | `reload` + } + const stormHistories: Array = [2, 3].flatMap((tabCount) => + ([`eager`, `on-demand`] as const).flatMap((syncMode) => + ([`first`, `last`] as const).flatMap((finisher) => + ([`storm`, `staggered`] as const).flatMap((arrival) => + ([`refetch`, `resumed batch`] as const).flatMap((open) => + ( + [ + ...(open === `refetch` ? ([`none`] as const) : []), + `error without retry`, + `SDK gives up`, + `signal aborted`, + `cleanup`, + ...(open === `resumed batch` && syncMode === `eager` + ? ([`resume invalidated`] as const) + : []), + ] as const + ).flatMap((stop) => + (stop === `none` + ? ([`none`] as const) + : ([`restart`, `reload`] as const) + ).map((recovery) => ({ + tabCount, + syncMode, + finisher, + arrival, + open, + stop, + recovery, + })), + ), + ), + ), + ), + ), + ) + // The storm driver: it runs one cell on real tabs and checks every tab's + // rows and status against the law at each checkpoint. + const runStormHistory = async ({ + tabCount, + syncMode, + finisher, + arrival, + open, + stop, + recovery, + }: (typeof stormHistories)[number]) => { + host = openHost() + const tabs = await openWarmTabs(host, tabCount, syncMode) + for (const tab of tabs) { + tab.deliver([...beforeDeploy.map(insert), upToDate], { + handle: `h1`, + offset: `10_0`, + }) + await host.quiesce(`tab ${tab.tab} warm`) + } + type RefetchReach = + | `peer commit announced during an open refetch` + | `stopped tab resumed from the persisted position` + | `peer commit announced during the stopped tab's open transaction` + | `a source stopped while its transaction was open` + | `restarted tab opened a new stream` + const reach = createOracleReach( + `browser.multi-tab-reserved-refetch`, + [ + ...(arrival === `storm` + ? ([`peer commit announced during an open refetch`] as const) + : []), + ...(open === `resumed batch` + ? ([`stopped tab resumed from the persisted position`] as const) + : []), + ...(stop === `none` + ? [] + : ([ + `peer commit announced during the stopped tab's open transaction`, + `a source stopped while its transaction was open`, + `restarted tab opened a new stream`, + ] as const)), + ], + ) + reach.beginHistory() + + // The last tab to finish is the one whose source may stop. + const finishOrder = finisher === `first` ? tabs : [...tabs].reverse() + const stoppedIndex = + stop === `none` ? undefined : tabs.indexOf(finishOrder.at(-1)!) + if (open === `resumed batch`) { + // A page reload: the tab closes and reopens over the same database, + // resuming from its persisted position. Its first batch after the + // resume stays open, as a batch with no up-to-date does. + const closing = tabs[stoppedIndex!]! + const streamsBefore = closing.streamCount() + await closing.close() + await host.quiesce(`stopped tab closed`) + const reopened = host.openTab(closing.tab, syncMode) + await waitForStream(reopened, streamsBefore + 1) + tabs[stoppedIndex!] = reopened + if (reopened.startedAt().handle === `h1`) { + reach.reach(`stopped tab resumed from the persisted position`) + } + reopened.deliver([upToDate], { handle: `h1`, offset: `10_0` }) + await host.quiesce(`stopped tab resumed`) + reopened.deliver([update({ id: 1, label: `one, unfinished` })], { + handle: `h1`, + offset: `11_0`, + }) + await host.quiesce(`resumed batch open`) + } + const stoppedTab = + stoppedIndex === undefined ? undefined : tabs[stoppedIndex] + const refetchingTabs = ( + finisher === `first` ? tabs : [...tabs].reverse() + ).filter((tab) => open === `refetch` || tab !== stoppedTab) + const noticesBefore = host.commitNotices() + const stopSource = async () => { + if (!stoppedTab || stop === `none`) return + if (host!.commitNotices() > noticesBefore) { + reach.reach( + `peer commit announced during the stopped tab's open transaction`, + ) + } + void Promise.resolve(stoppedTab.stopStream(stop)).catch(() => undefined) + await host!.quiesce(`tab ${stoppedTab.tab} source stopped`) + reach.reach(`a source stopped while its transaction was open`) + } + const refetch = (tab: Tab) => + tab.deliver([...afterDeploy.map(insert), upToDate], { + handle: `h2`, + offset: `20_0`, + }) + if (arrival === `storm`) { + for (const tab of refetchingTabs) + tab.deliver([mustRefetch], { handle: `h1`, offset: `10_0` }) + const noticesBeforeRefetch = host.commitNotices() + await host.quiesce(`every refetch open`) + // Each tab's reset marker is a peer notice for every other tab while + // its own reserved refetch is open. + if (host.commitNotices() > noticesBeforeRefetch) { + reach.reach(`peer commit announced during an open refetch`) + } + for (const tab of refetchingTabs) { + if (tab === stoppedTab) continue + refetch(tab) + await host.quiesce(`tab ${tab.tab} refetched`) + } + await stopSource() + } else { + for (const tab of refetchingTabs) { + tab.deliver([mustRefetch], { handle: `h1`, offset: `10_0` }) + await host.quiesce(`tab ${tab.tab} refetch open`) + if (tab === stoppedTab) continue + refetch(tab) + await host.quiesce(`tab ${tab.tab} refetched`) + } + await stopSource() + } + + // Checkpoint 1, every source has delivered or stopped. A rejected + // reserved commit would fail its source. A stopped tab that is still + // open shows its peers' refetch, including one whose source failed on + // resume invalidation (Law 3); a cleaned-up tab has no rows. Eager rows + // must equal the head snapshot; on-demand rows are not judged (above). + // Resume invalidation comes after the tab was ready, so the tab stays + // `ready` and its health node records the source's error (#2168: after + // readiness, sync failures never change Collection status). + const stoppedStatus = stop === `cleanup` ? `cleaned-up` : `ready` + const judgesRowsBeforeRestart = (tab: Tab) => + syncMode === `eager` && (tab !== stoppedTab || stop !== `cleanup`) + const judgesSourceError = (tab: Tab) => + tab === stoppedTab && stop === `resume invalidated` + const observe = (judgesRows: (tab: Tab) => boolean) => + tabs.map((tab) => ({ + tab: tab.tab, + status: tab.collection.status, + ...(judgesRows(tab) ? { rows: tab.publicRows() } : {}), + ...(judgesSourceError(tab) + ? { sourceError: tab.collection.health.error !== null } + : {}), + })) + expect(observe(judgesRowsBeforeRestart)).toEqual( + tabs.map((tab) => ({ + tab: tab.tab, + status: tab === stoppedTab ? stoppedStatus : `ready`, + ...(judgesRowsBeforeRestart(tab) ? { rows: afterDeploy } : {}), + ...(judgesSourceError(tab) ? { sourceError: true } : {}), + })), + ) + + // Checkpoint 2, after the application recovers the stopped tab and the + // server answers its new stream from where it starts. A restart cleans + // up and starts the same Collection again; a reload closes the tab and + // opens it again over the same database, so it reads only durable + // state. + if (stoppedTab) { + const streamsBefore = stoppedTab.streamCount() + let recovered = stoppedTab + if (recovery === `restart`) await stoppedTab.restart() + else { + await stoppedTab.close() + await host.quiesce(`tab ${stoppedTab.tab} closed`) + recovered = host.openTab(stoppedTab.tab, syncMode) + tabs[stoppedIndex!] = recovered + } + await waitForStream(recovered, streamsBefore + 1) + reach.reach(`restarted tab opened a new stream`) + const restartedAt = recovered.startedAt() + await serveHead(host, recovered, restartedAt) + if (stop === `resume invalidated`) { + expect(restartedAt).toEqual({ handle: undefined, offset: `-1` }) + } + // The new sync run starts without the retired run's problems. + expect(observe(() => syncMode === `eager`)).toEqual( + tabs.map((tab) => ({ + tab: tab.tab, + status: `ready`, + ...(syncMode === `eager` ? { rows: afterDeploy } : {}), + ...(judgesSourceError(tab) ? { sourceError: false } : {}), + })), + ) + } + reach.assertReached() + } + it.each(stormHistories)( + `commits every reserved refetch across tabs ($tabCount tabs, $syncMode, $finisher finishes first, $arrival, open $open, stop: $stop, recovery: $recovery)`, + runStormHistory, + 30_000, + ) + + // Law 1 and Law 3, generated histories. The enumerated grammar above fixes + // one refetch storm and one stopped tab. These histories interleave, in any + // legal order: + // + // - server changes (insert, update, or delete of one of four rows); + // - deploys, after which every stream must refetch; + // - a peer reset, which abandons every open reserved transaction (Law 5); + // - deliveries to one tab: part of what its stream still owes, with no + // up-to-date, which leaves a reserved transaction open, or everything it + // owes and up-to-date, which commits it; + // - a tab's source stopping for good (each terminal path except resume + // invalidation, which the enumerated grammar owns); + // - recovery of a stopped tab, and a reload of a running tab. + // + // Model. The server is a log per shape handle. A tab's stream owes the log + // after its position, or must-refetch then the snapshot when its handle is + // stale. A tab's open transaction holds the messages delivered since it + // began; a commit applies them to the durable rows in commit order, after a + // truncate when the transaction is a refetch or a fresh snapshot. An + // abandoned transaction applies nothing. So the expected durable rows are a + // fold of committed batches. A lagging tab can commit an older value after + // a peer committed a newer one; that is the eventual-consistency note in + // the coverage map (#2209), so the fold keeps it rather than the server + // head. + // + // Checkpoint after every step, at host quiescence: + // - every status is the one the model owes: `error` only for a failed + // first load, `cleaned-up` only for a tab stopped by cleanup; a + // reset-abandoned commit fails the source, which keeps a ready tab + // `ready` with an error on its health node (Law 5); + // - the durable rows equal the fold; + // - every open tab with no open transaction shows the durable rows. That + // includes a tab whose source stopped: it abandoned its transaction, so + // its peers' commits apply to it (Law 1, Law 3). + // After the history, every tab recovers and catches up. Then every tab must + // show the durable rows, and the durable rows must be the server head when + // the fold says so. + // + // The model reads one production fact: the position a new stream asks the + // server to start from. The simulated server answers that request; the + // model does not judge it, except that a full-snapshot request must get a + // full snapshot. + type HistoryStep = + | { kind: `server change`; key: number; pick: number } + | { kind: `deploy` } + | { kind: `peer reset` } + | { kind: `advance`; tab: number; pick: number } + | { kind: `catch up`; tab: number; pick: number } + // A tab that closes for good. Generated histories never close a tab; + // only the pinned #2209 replay does (see below). + | { kind: `close`; tab: number } + | { kind: `stop`; tab: number; pick: number } + | { kind: `recover`; tab: number; pick: number } + const historyStep: fc.Arbitrary = fc.oneof( + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`server change` as const), + key: fc.integer({ min: 1, max: 4 }), + pick: fc.nat(5), + }), + }, + { + weight: 1, + arbitrary: fc.record({ kind: fc.constant(`deploy` as const) }), + }, + { + weight: 1, + arbitrary: fc.record({ kind: fc.constant(`peer reset` as const) }), + }, + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`advance` as const), + tab: fc.nat(2), + pick: fc.nat(5), + }), + }, + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`catch up` as const), + tab: fc.nat(2), + // 0 delivers everything owed; n commits only the next n changes, as + // a server answering at an earlier moment does. + pick: fc.nat(3), + }), + }, + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`stop` as const), + tab: fc.nat(2), + pick: fc.nat(3), + }), + }, + { + weight: 1, + arbitrary: fc.record({ + kind: fc.constant(`recover` as const), + tab: fc.nat(2), + pick: fc.nat(1), + }), + }, + ) + const generatedHistory = fc.record({ + tabCount: fc.integer({ min: 2, max: 3 }), + steps: fc.array(historyStep, { minLength: 8, maxLength: 24 }), + }) + type GeneratedHistory = { tabCount: number; steps: Array } + + type ServerLog = { + handle: number + // Versions under the current handle; version 0 is the snapshot. + snapshot: Map + changes: Array<{ version: number; message: Message }> + rows: Map + } + type TabModel = { + stopped: StreamStop | undefined + // The position the stream has delivered: a handle and a version. An + // undefined handle means the stream owes a full snapshot. + handle: number | undefined + version: number + // Messages of the open transaction, and whether it truncates first. + open: Array> | undefined + truncates: boolean + // Collection readiness: a new sync run is `loading` until its own + // stream's first up-to-date. A stop before then is the final initial + // failure, which puts the Collection in `error` (Electric's initial-sync + // law). A Collection status changes only through its own source: a + // peer's durable commit reloads the rows but leaves the status alone + // (maintainer decision on PR #2265, Law 4). + ready: boolean + // A sync run that resumed from a persisted position, so Electric checks + // that each update names a row it holds. + resumed: boolean + failedInitialSync: boolean + // The stop that ended an open transaction, until the tab recovers. A + // peer commit in that interval is the premise of the abandon law. + stoppedMidTransaction: StreamStop | undefined + closed: boolean + // A peer reset abandoned this tab's open transaction. Electric still + // holds it, so its next commit rejects and the source fails (maintainer + // decision on PR #2265, Law 5); the app then restarts it. + abandonedByReset: boolean + failedByReset: boolean + } + const stops: ReadonlyArray = [ + `error without retry`, + `SDK gives up`, + `signal aborted`, + `cleanup`, + ] + const rowsOf = (rows: Map): Array => + [...rows] + .map(([id, label]) => ({ id, label })) + .sort((left, right) => left.id - right.id) + const position = (log: ServerLog, version: number) => ({ + handle: `h${log.handle}`, + offset: `${version}_0`, + }) + // The request a new stream made, read as a model position. + const requestedPosition = ( + startedAt: { handle: string | undefined; offset: string }, + log: ServerLog, + ): Pick => { + if (startedAt.handle === undefined || startedAt.offset === `-1`) { + return { handle: undefined, version: -1 } + } + const handle = Number(startedAt.handle.slice(1)) + const version = Number(startedAt.offset.split(`_`)[0]) + return handle === log.handle + ? { handle, version } + : { handle, version: Number.NaN } + } + + const runGeneratedHistory = async ( + { tabCount, steps }: GeneratedHistory, + onKnownGap: () => void, + ) => { + host = openHost() + const activeHost = host + const tabs = await openWarmTabs(activeHost, tabCount, `eager`) + const log: ServerLog = { + handle: 1, + snapshot: new Map(beforeDeploy.map((row) => [row.id, row.label])), + changes: [], + rows: new Map(beforeDeploy.map((row) => [row.id, row.label])), + } + const durable = new Map() + const models: Array = tabs.map(() => ({ + stopped: undefined, + handle: undefined, + version: -1, + open: undefined, + truncates: false, + ready: false, + failedInitialSync: false, + resumed: false, + stoppedMidTransaction: undefined, + closed: false, + abandonedByReset: false, + failedByReset: false, + })) + let serverVersion = 0 + generatedReach.beginHistory() + + const commit = (model: TabModel) => { + if (model.truncates) durable.clear() + for (const message of model.open ?? []) { + if (!(`key` in message)) continue + const row = message.value + if (message.headers.operation === `delete`) durable.delete(row.id) + else durable.set(row.id, row.label) + } + model.open = undefined + model.truncates = false + } + const abandon = (model: TabModel) => { + model.open = undefined + model.truncates = false + } + // What the server sends one tab next: must-refetch for a stale handle, + // the snapshot for a full-snapshot request, or the changes it owes. + const deliver = async ( + index: number, + upTo: number | `all`, + commits: boolean, + ) => { + const tab = tabs[index]! + const model = models[index]! + if (model.handle !== undefined && model.handle !== log.handle) { + tab.deliver([mustRefetch], { + handle: `h${model.handle}`, + offset: `${Math.max(model.version, 0)}_0`, + }) + generatedReach.reach(`must-refetch delivered`) + abandon(model) + model.open = [] + model.truncates = true + model.handle = undefined + model.version = -1 + await activeHost.quiesce(`tab ${tab.tab} must-refetch`) + if (!commits) return + } + const messages: Array> = [] + if (model.handle === undefined) { + model.open ??= [] + model.truncates = true + messages.push( + ...rowsOf(log.snapshot).map(insert), + ...log.changes.map((change) => change.message), + ) + model.handle = log.handle + model.version = serverVersion + } else { + const owed = log.changes.filter( + (change) => change.version > model.version, + ) + const count = upTo === `all` ? owed.length : Math.min(upTo, owed.length) + messages.push(...owed.slice(0, count).map((change) => change.message)) + model.version = count > 0 ? owed[count - 1]!.version : model.version + if (messages.length > 0) model.open ??= [] + } + // Electric's updates carry only changed columns (the shape's default + // replica), so an update cannot create a row the tab does not hold, + // and Electric drops it. The tab holds the durable rows, or, inside a + // refetch, the snapshot it has received. Known gap: a resumed tab + // instead reports an unseen row and fails (issue #2267). Generated + // campaigns leave that history out; the pinned replay below keeps it. + const held = new Set(model.truncates ? [] : durable.keys()) + const track = (message: Message) => { + if (!(`key` in message)) return + if (message.headers.operation === `delete`) + held.delete(message.value.id) + else held.add(message.value.id) + } + for (const message of model.open ?? []) track(message) + const applied = messages.filter((message) => { + if ( + `key` in message && + message.headers.operation === `update` && + !held.has(message.value.id) + ) { + if (model.resumed && !model.truncates) onKnownGap() + return false + } + track(message) + return true + }) + model.open?.push(...applied) + const complete = commits + if (messages.length === 0 && !complete) return + tab.deliver( + complete ? [...messages, upToDate] : messages, + position(log, model.version), + ) + if (complete && model.abandonedByReset) { + generatedReach.reach( + `reset-abandoned commit put the Collection in error`, + ) + abandon(model) + model.abandonedByReset = false + model.failedByReset = true + } else if (complete) { + commit(model) + model.ready = true + } + await activeHost.quiesce(`tab ${tab.tab} delivered`) + } + const recover = async (index: number, how: `restart` | `reload`) => { + const stopped = tabs[index]! + const streamsBefore = stopped.streamCount() + let recovered = stopped + if (how === `restart`) await stopped.restart() + else { + await stopped.close() + await activeHost.quiesce(`tab ${stopped.tab} closed`) + recovered = activeHost.openTab(stopped.tab, `eager`) + tabs[index] = recovered + } + abandon(models[index]!) + await waitForStream(recovered, streamsBefore + 1) + generatedReach.reach(`tab recovered`) + const requested = requestedPosition(recovered.startedAt(), log) + Object.assign(models[index]!, { + stopped: undefined, + ready: false, + failedInitialSync: false, + resumed: requested.handle !== undefined, + stoppedMidTransaction: undefined, + abandonedByReset: false, + failedByReset: false, + ...requested, + }) + await activeHost.quiesce(`tab ${recovered.tab} recovered`) + } + + const check = async (label: string, head?: Array) => { + const expectedDurable = head ?? rowsOf(durable) + // A tab closed for good is a cleaned-up Collection with no rows. + const judgesRows = (index: number) => + models[index]!.open === undefined && + models[index]!.stopped !== `cleanup` && + !models[index]!.closed + const expectedStatus = (index: number) => + models[index]!.stopped === `cleanup` || models[index]!.closed + ? `cleaned-up` + : models[index]!.failedInitialSync || + (models[index]!.failedByReset && !models[index]!.ready) + ? `error` + : models[index]!.ready + ? `ready` + : `loading` + const expected = { + durableRows: expectedDurable, + tabs: tabs.map((tab, index) => ({ + tab: tab.tab, + status: expectedStatus(index), + ...(judgesRows(index) ? { rows: expectedDurable } : {}), + // The reset-abandoned source failed (Law 5). + ...(models[index]!.failedByReset ? { sourceError: true } : {}), + })), + } + const actual = { + durableRows: await activeHost.durableRows(), + tabs: tabs.map((tab, index) => ({ + tab: tab.tab, + status: tab.collection.status, + ...(judgesRows(index) ? { rows: tab.publicRows() } : {}), + ...(models[index]!.failedByReset + ? { sourceError: tab.collection.health.error !== null } + : {}), + })), + } + // fast-check reports only the message, so it carries both sides, plus + // each errored Collection's reason, which `preload()` rejects with. + const errors = await Promise.all( + tabs.map((tab) => + tab.collection.status === `error` + ? tab.collection.preload().then( + () => undefined, + (error: unknown) => `${tab.tab}: ${String(error)}`, + ) + : undefined, + ), + ) + expect( + actual, + `${label}\nactual ${JSON.stringify(actual)}\nexpected ${JSON.stringify(expected)}\nerrors ${JSON.stringify(errors.filter(Boolean))}`, + ).toEqual(expected) + } + + // Warm start: every tab delivers the snapshot and commits it. + for (const index of tabs.keys()) await deliver(index, `all`, true) + await check(`warm`) + + for (const [stepIndex, step] of steps.entries()) { + const openBefore = models.some((model) => model.open !== undefined) + const unreadyBefore = models.some( + (model) => + model.stopped !== `cleanup` && + (!model.ready || model.failedInitialSync), + ) + const noticesBefore = activeHost.commitNotices() + if (step.kind === `server change`) { + serverVersion++ + const present = log.rows.has(step.key) + const operation = !present + ? `insert` + : step.pick % 3 === 0 + ? `delete` + : `update` + const row = { id: step.key, label: `v${serverVersion}` } + if (operation === `delete`) log.rows.delete(step.key) + else log.rows.set(step.key, row.label) + log.changes.push({ + version: serverVersion, + message: { + key: String(row.id), + value: row, + headers: { operation }, + }, + }) + } else if (step.kind === `close`) { + const index = step.tab % tabs.length + await tabs[index]!.close() + abandon(models[index]!) + models[index]!.closed = true + await activeHost.quiesce(`tab ${tabs[index]!.tab} closed for good`) + } else if (step.kind === `peer reset`) { + // A peer reset abandons every open reserved transaction that began + // before it (maintainer decision on PR #2265). It changes no durable + // row, so every tab reloads the same durable rows. + if (models.some((model) => model.open !== undefined)) { + generatedReach.reach(`peer reset abandoned an open transaction`) + } + for (const model of models) { + if (model.open !== undefined && model.stopped === undefined) { + model.abandonedByReset = true + } + abandon(model) + } + activeHost.broadcastPeerReset() + await activeHost.quiesce(`peer reset applied`) + } else if (step.kind === `deploy`) { + log.handle++ + serverVersion = 0 + log.snapshot = new Map(log.rows) + log.changes = [] + } else { + // A stop goes to a tab with an open transaction when there is one, + // so the generator reaches the state the abandon law is about. + const openTabs = models.flatMap((model, index) => + model.open !== undefined && model.stopped === undefined + ? [index] + : [], + ) + // Only `recover` recovers a stopped tab. Another step aimed at a + // stopped tab goes to a running one, so a stopped tab stays stopped + // while its peers commit. + const runningTabs = models.flatMap((model, index) => + model.stopped === undefined && !model.failedByReset ? [index] : [], + ) + const index = + step.kind === `stop` && openTabs.length > 0 + ? openTabs[step.tab % openTabs.length]! + : step.kind !== `recover` && + (models[step.tab % tabs.length]!.stopped !== undefined || + models[step.tab % tabs.length]!.failedByReset) && + runningTabs.length > 0 + ? runningTabs[step.tab % runningTabs.length]! + : step.tab % tabs.length + const model = models[index]! + if ( + model.stopped !== undefined || + model.failedByReset || + step.kind === `recover` + ) { + // A running tab can only reload; a stopped tab restarts or reloads. + const how = + (model.stopped === undefined && !model.failedByReset) || + (step.kind === `recover` && step.pick === 1) + ? `reload` + : `restart` + await recover(index, how) + } else if (step.kind === `stop`) { + const how = stops[step.pick % stops.length]! + if (model.open !== undefined) { + generatedReach.reach(`source stopped with an open transaction`) + model.stoppedMidTransaction = how + } + void Promise.resolve(tabs[index]!.stopStream(how)).catch( + () => undefined, + ) + model.stopped = how + // Electric abandons the dead transaction itself, so nothing errors. + model.abandonedByReset = false + abandon(model) + model.failedInitialSync = how !== `cleanup` && !model.ready + await activeHost.quiesce(`tab ${tabs[index]!.tab} stopped`) + } else { + await deliver( + index, + step.kind === `catch up` + ? step.pick === 0 + ? `all` + : step.pick + : step.pick + 1, + step.kind === `catch up`, + ) + } + } + if (activeHost.commitNotices() > noticesBefore) { + for (const model of models) { + const stop = model.stoppedMidTransaction + if ( + stop === `error without retry` || + stop === `SDK gives up` || + stop === `signal aborted` + ) { + generatedReach.reach(`peer commit after ${stop} mid-transaction`) + } + } + } + if (unreadyBefore && activeHost.commitNotices() > noticesBefore) { + generatedReach.reach(`peer commit while a tab was not ready`) + } + if (openBefore && activeHost.commitNotices() > noticesBefore) { + const stillOpen = models.some((model) => model.open !== undefined) + if (stillOpen) { + generatedReach.reach( + `peer commit during an open reserved transaction`, + ) + } + } + if (rowsOf(durable).some(({ id, label }) => log.rows.get(id) !== label)) { + generatedReach.reach(`durable rows behind the server`) + } + await check(`after step ${stepIndex}: ${JSON.stringify(step)}`) + } + + // Law 2: every open tab recovers and catches up, and then the durable + // rows and every open tab show the server head. Each tab's last batch + // rewrites every row that changed after its position, so no lagging + // commit survives the catch-up. + for (const index of tabs.keys()) { + if (models[index]!.closed) continue + if ( + models[index]!.stopped !== undefined || + models[index]!.failedByReset + ) { + await recover(index, `restart`) + } + await deliver(index, `all`, true) + } + await check(`after every tab caught up`, rowsOf(log.rows)) + } + + type GeneratedReach = + | `must-refetch delivered` + | `peer commit during an open reserved transaction` + | `source stopped with an open transaction` + | `tab recovered` + | `durable rows behind the server` + | `peer commit while a tab was not ready` + | `peer commit after error without retry mid-transaction` + | `peer commit after SDK gives up mid-transaction` + | `peer commit after signal aborted mid-transaction` + | `peer reset abandoned an open transaction` + | `reset-abandoned commit put the Collection in error` + // The storm driver and the generated-history driver are the checks; the + // generated driver folds committed batches into its own model of durable + // rows. The storm cells and the generated-history arbitrary are the grammar. + declareOracle({ + check: [runStormHistory, runGeneratedHistory], + grammar: [stormHistories, generatedHistory], + }) + + const createGeneratedReach = () => + createOracleReach(`browser.multi-tab-reserved-history`, [ + `must-refetch delivered`, + `peer commit during an open reserved transaction`, + `source stopped with an open transaction`, + `tab recovered`, + `durable rows behind the server`, + `peer commit while a tab was not ready`, + `peer commit after error without retry mid-transaction`, + `peer commit after SDK gives up mid-transaction`, + `peer commit after signal aborted mid-transaction`, + `peer reset abandoned an open transaction`, + `reset-abandoned commit put the Collection in error`, + ]) + let generatedReach = createGeneratedReach() + + const runGeneratedCampaign = async ( + parameters: ReturnType, + ) => { + generatedReach = createGeneratedReach() + await fc.assert( + fc.asyncProperty(generatedHistory, async (history) => { + try { + // fc.pre discards a history that reaches the known gap. + await runGeneratedHistory(history, () => fc.pre(false)) + } finally { + await host?.close() + host = undefined + streams.length = 0 + } + }), + parameters, + ) + console.info( + `multi-tab generated reach`, + Object.fromEntries(generatedReach.counts()), + ) + generatedReach.assertReached() + } + it.skipIf(oracleReplayRequested())( + `keeps reserved commits lawful across generated multi-tab histories (fixed seed)`, + () => + runGeneratedCampaign({ + numRuns: oracleRuns(GENERATED_RUNS), + seed: GENERATED_FIXED_SEED, + }), + 600_000, + ) + it( + `keeps reserved commits lawful across generated multi-tab histories (random or replayed)`, + () => + runGeneratedCampaign( + oraclePropertyOptions(GENERATED_RUNS, GENERATED_PROPERTY), + ), + 600_000, + ) + + // Known gap #2267, pinned: the shrunk generated history in which a resumed tab + // that lags behind its peer receives an update for a row the peer already + // deleted. It replays through the same driver and checks with the known-gap + // detector off. Remove `fails` when Electric's resume check stops treating + // rows from peers' commits as this stream's evidence. + it.fails( + `keeps a resumed lagging tab out of error when a peer deleted the row it updates`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `recover`, tab: 1, pick: 0 }, + { kind: `server change`, key: 2, pick: 1 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `server change`, key: 2, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `advance`, tab: 0, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) + + // Known gap #2209, pinned: tab A commits v1 and v2; tab B's lagging stream + // then commits only v1, so the durable row falls back to v1, and B closes + // for good. Nothing redelivers the head to A, so Law 2's final check fails. + // Generated histories never close a tab for good, which leaves this + // history out of them; it replays through the same driver and checks. + // Remove `fails` when source writes are ordered by server position. + it.fails( + `converges an open tab to the head after a lagging tab commits an older row and closes`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `server change`, key: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + { kind: `close`, tab: 1 }, + ], + }, + () => undefined, + ), + 60_000, + ) +}) diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 38e9035841..3859ea985d 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -962,7 +962,9 @@ type SyncControlFns = { ) => void) | null commit: ((signal?: AbortSignal) => SyncAppliedReceipt) | null - truncate: ((options?: { markReady?: boolean }) => void) | null + // A runtime truncate replaces rows from durable state. It never changes + // Collection status: only the Collection's own source does. + truncate: (() => void) | null metadata: SyncMetadataApi | null fail: ((error: unknown) => void) | null } @@ -1295,13 +1297,13 @@ type OpenSyncTransaction = Omit< applicationReceipt?: SyncAppliedReceipt operationKeys: Set queuedBecauseHydrating: boolean - // Set by `reserveCommitTurn`: subset hydration waits for this commit. + // Set by `reserveCommitTurn`: every hydration waits for this commit. reservesCommitTurn?: boolean + // Settles once the reservation ends; every wait shares it. + reservationEnded?: Promise hasDependentSuccessor: boolean - publicationAdmissionWaiters?: Set<{ - resolve: () => void - reject: (error: unknown) => void - }> + // Every outcome of the transaction releases its admission waiters. + publicationAdmissionWaiters?: Set<() => void> terminalFailure?: { error: unknown } // Set just before the transaction is handed to the Collection; from then // on core holds its keys itself. @@ -1316,6 +1318,24 @@ type SyncWriteNormalization = { operation: NormalizedSyncOperation } +/** + * Returns undefined when no reserved sync transaction (`reserveCommitTurn`) + * is open. Otherwise returns a promise that settles when it commits or ends. + * No hydration may start while one is open: a new hydration cycle would make + * its commit fail. + */ +type WaitForReservedCommit = () => Promise | undefined + +/** + * Thrown where a hydration would start while a reserved sync transaction is + * open, before the hydration changes any state. Its apply-mutex turn ends, + * waits for the commit outside the mutex, and retries: the commit's own + * durable write needs the mutex. + */ +class ReservedCommitPending extends Error {} +// One instance signals every deferral; callers compare it by identity. +const reservedCommitPending = new ReservedCommitPending() + class ApplyMutex { private queue: Promise = Promise.resolve() private pending = 0 @@ -1549,6 +1569,9 @@ class PersistedCollectionRuntime< private readonly activeSubsets = new Map() private activeHydrationContext: { suppliedRowKeys: Set } | undefined private hydrationSequence = 0 + private waitForReservedCommit: WaitForReservedCommit | undefined + private abandonReservedCommit: (() => void) | undefined + private reservedCommitFlush: Promise | undefined // An unscheduled startup read may finish after a coordinator reset publishes. private resetSequence = 0 private readonly pendingRemoteSubsetEnsures = new Map< @@ -1569,10 +1592,13 @@ class PersistedCollectionRuntime< private readonly queuedHydrationTransactions: Array< BufferedSyncTransaction > = [] + // Peer commits and resets, in arrival order. private readonly queuedTxCommitted: Array<{ - txCommitted: TxCommitted + notice: TxCommitted | CollectionReset lifecycleGeneration: number storageCollectionId: string + // Set when the notice met an open reservation; it retries after that ends. + waitsForReservedCommit?: boolean }> = [] private readonly requestIds = new WeakMap() private readonly hydratedDemands = new Set() @@ -1903,6 +1929,60 @@ class PersistedCollectionRuntime< this.restartSourceAfterScopedRecovery = restart ?? null } + setReservedCommitControls( + wait: WaitForReservedCommit, + abandon: () => void, + ): void { + this.waitForReservedCommit = wait + this.abandonReservedCommit = abandon + } + + // Every hydration start calls this before it changes state. The check runs + // inside the hydration's own scope, because a shared scheduler can hold the + // scope long after its apply-mutex turn began. + // A hydration cycle begins: it waits for an open reservation, then moves + // the sequence that open sync transactions are bound to. + private beginHydrationCycle(): void { + this.throwIfReservedCommitOpen() + this.hydrationSequence++ + } + + private throwIfReservedCommitOpen(): void { + if (this.waitForReservedCommit?.()) throw reservedCommitPending + } + + /** Runs a hydration turn, retrying it after a reserved commit it met. */ + private async runHydrationTurn( + turn: () => Promise, + ): Promise { + for (;;) { + try { + return await this.applyMutex.run(turn) + } catch (error) { + if (error !== reservedCommitPending) throw error + await this.waitForReservedCommit?.() + } + } + } + + // A deferred notice stays at the head of the queue, so later notices still + // apply in arrival order once the reserved transaction commits. + private flushAfterReservedCommit(): void { + const lifecycleGeneration = this.lifecycleGeneration + const ended = this.waitForReservedCommit?.() + // One deferred flush per reservation drains every notice behind it. + if (ended && ended === this.reservedCommitFlush) return + this.reservedCommitFlush = ended + void Promise.resolve(ended) + .then(() => { + if (lifecycleGeneration !== this.lifecycleGeneration) return + return this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + }) + .catch((error) => { + this.markTerminalFailure(error, lifecycleGeneration) + }) + } + clearSourceRestartAfterScopedRecovery(): void { this.restartSourceAfterScopedRecovery = undefined } @@ -2027,6 +2107,24 @@ class PersistedCollectionRuntime< await this.ensureCacheClaimActive() } await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + // A broadcast whose reload waits for a reserved commit has not applied + // yet, so the load waits with it, including when the reservation ends + // before this check. Notices behind a head that does not wait arrived + // after this flush and are not the owner's. + for (;;) { + const head = this.queuedTxCommitted[0] + if (!head?.waitsForReservedCommit) break + const reservation = this.waitForReservedCommit?.() + await reservation + await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + // With no reservation left, a head still queued is held by other work, + // such as a claim recovery, which flushes it. + if ( + this.queuedTxCommitted[0] === head && + !this.waitForReservedCommit?.() + ) + break + } // A broadcast that failed to apply made the runtime terminal; its rows // are not visible, so the load must not settle successfully. this.throwIfTerminal() @@ -2372,7 +2470,7 @@ class PersistedCollectionRuntime< if (this.syncMode !== `on-demand`) return if (this.scopedRecovery) return - const appliedCursor = await this.applyMutex.run(async () => { + const appliedCursor = await this.runHydrationTurn(async () => { const result = await this.runInHydrationScope((adapter) => this.hydrateBaseline(lifecycleGeneration, adapter), ) @@ -2550,7 +2648,7 @@ class PersistedCollectionRuntime< if (this.syncControls.begin && this.syncControls.commit) { const applied = this.withInternalApply(() => { this.syncControls.begin?.() - this.syncControls.truncate?.({ markReady: false }) + this.syncControls.truncate?.() if (resetMetadata) { this.syncControls.metadata?.collection.set( resetMetadata.key, @@ -2807,17 +2905,36 @@ class PersistedCollectionRuntime< // Wire admission failures are permanent input errors, not transport retries. if (routeRemoteDemandDuringHydration) validateRemoteSubsetOptions(options) this.activeSubsets.set(subsetKey, options) + // An aborted demand is not active, even while it waits for its turn. + const retireAbortedDemand = () => { + if (this.activeSubsets.get(subsetKey) === options) { + this.activeSubsets.delete(subsetKey) + } + } + options.signal?.addEventListener(`abort`, retireAbortedDemand, { + once: true, + }) const appliedCursor = this.appliedReceiptSequence try { - const hydrated = await this.applyMutex.run( + const hydrated = await this.runHydrationTurn( async (): Promise => { - if (options.signal?.aborted) return `aborted` + // The turn may start long after admission, behind other work or a + // reserved commit; demand retired or cleaned up meanwhile reads + // nothing. + if ( + options.signal?.aborted || + (isDemandRetained !== undefined && !isDemandRetained()) + ) { + return `aborted` + } try { return await this.runInHydrationScope( (adapter) => { if (options.signal?.aborted) { return Promise.resolve(`aborted`) } + // Until the reserved commit lands, the load can still abort. + this.throwIfReservedCommitOpen() onHydrationStarted?.() return this.hydrateSubsetUnsafe( options, @@ -2878,6 +2995,8 @@ class PersistedCollectionRuntime< this.activeSubsets.delete(subsetKey) } throw error + } finally { + options.signal?.removeEventListener(`abort`, retireAbortedDemand) } if ( @@ -3034,6 +3153,7 @@ class PersistedCollectionRuntime< await this.ensureCacheClaimActive() const lifecycleGeneration = this.lifecycleGeneration // A one-shot refresh does not acquire an enduring subscription lease. + // Only a sync-absent Collection exposes it, so no reservation can be open. const hydrated = await this.applyMutex.run(async () => { const result = await this.runInHydrationScope((adapter) => this.hydrateSubsetUnsafe( @@ -3374,11 +3494,11 @@ class PersistedCollectionRuntime< }, adapter: HydrationPersistenceAdapter, ): Promise { + this.beginHydrationCycle() let rowsLoaded = false let boundRowVersion: number | undefined let replayFailure: { reason: unknown } | undefined const resetSequence = this.resetSequence - this.hydrationSequence++ const hydrationContext = { suppliedRowKeys: new Set() } this.activeHydrationContext = hydrationContext try { @@ -4661,7 +4781,7 @@ class PersistedCollectionRuntime< // a renewed generation still owes its rows. A hydration flushes the // queue when it ends. this.queuedTxCommitted.push({ - txCommitted: payload, + notice: payload, lifecycleGeneration, storageCollectionId: message.collectionId, }) @@ -4683,32 +4803,44 @@ class PersistedCollectionRuntime< } if (isCollectionResetPayload(payload)) { - void this.applyMutex - .run(async () => { - if (message.collectionId !== this.storageCollectionId) return - await this.runInHydrationScope(async (adapter) => { - await this.truncateAndReloadUnsafe(adapter, lifecycleGeneration) - await this.rebaseDurableStreamFromSnapshotUnsafe( - adapter, - lifecycleGeneration, - ) - if (lifecycleGeneration === this.lifecycleGeneration) { - // Reloaded rows and the later position-only read are separate - // cuts. Zero is the conservative public version in the new epoch. - this.publicRowVersion = 0 - this.resetTermFence = this.latestTerm - } + // A reservation built before the reset cannot commit into it + // (maintainer decision on PR #2265). + this.abandonReservedCommit?.() + // A reset queues with peer commits, so a commit that arrived after it + // applies after it, behind its term fence. + this.queuedTxCommitted.push({ + notice: payload, + lifecycleGeneration, + storageCollectionId: message.collectionId, + }) + if (!this.isHydratingNow()) { + void this.applyMutex + .run(() => this.flushQueuedTxCommittedUnsafe()) + .catch((error) => { + this.markTerminalFailure(error, lifecycleGeneration) }) - if (lifecycleGeneration === this.lifecycleGeneration) { - await this.flushQueuedTxCommittedUnsafe() - } - }) - .catch((error) => { - this.markTerminalFailure(error, lifecycleGeneration) - }) + } } } + private async applyCollectionResetUnsafe( + lifecycleGeneration: number, + ): Promise { + await this.runInHydrationScope(async (adapter) => { + await this.truncateAndReloadUnsafe(adapter, lifecycleGeneration) + await this.rebaseDurableStreamFromSnapshotUnsafe( + adapter, + lifecycleGeneration, + ) + if (lifecycleGeneration === this.lifecycleGeneration) { + // Reloaded rows and the later position-only read are separate + // cuts. Zero is the conservative public version in the new epoch. + this.publicRowVersion = 0 + this.resetTermFence = this.latestTerm + } + }) + } + private async flushQueuedTxCommittedUnsafe(): Promise { while (this.queuedTxCommitted.length > 0) { // Claim recovery flushes the queue once the generation is decided. @@ -4716,19 +4848,39 @@ class PersistedCollectionRuntime< this.scheduleCacheClaimRecovery(this.lifecycleGeneration) return } - const queued = this.queuedTxCommitted.shift() - if (!queued) { - continue + // A notice that waited retries only after the reservation ends, so + // notices that arrive meanwhile neither retry it nor overtake it. + if ( + this.queuedTxCommitted[0]!.waitsForReservedCommit && + this.waitForReservedCommit?.() + ) { + this.flushAfterReservedCommit() + return } + const queued = this.queuedTxCommitted.shift()! if (queued.storageCollectionId !== this.storageCollectionId) continue - // A queued commit has left the queue, so a failure must be terminal, - // as it is for a commit applied when it arrives. try { - await this.processCommittedTxUnsafe( - queued.txCommitted, - queued.lifecycleGeneration, - ) + if (isCollectionResetPayload(queued.notice)) { + await this.applyCollectionResetUnsafe(queued.lifecycleGeneration) + } else { + await this.processCommittedTxUnsafe( + queued.notice, + queued.lifecycleGeneration, + ) + } } catch (error) { + if (error === reservedCommitPending) { + // Only a notice that reaches a hydration waits; later notices stay + // queued behind it. The reserved commit lands first and takes the + // next stream position, so the retry may run as gap recovery, which + // rereads durable rows (maintainer decision on #2226). + queued.waitsForReservedCommit = true + this.queuedTxCommitted.unshift(queued) + this.flushAfterReservedCommit() + return + } + // A queued commit has left the queue, so a failure must be terminal, + // as it is for a commit applied when it arrives. throw this.markTerminalFailure(error, queued.lifecycleGeneration) } } @@ -4837,6 +4989,21 @@ class PersistedCollectionRuntime< lifecycleGeneration: number, ): Promise { if (lifecycleGeneration !== this.lifecycleGeneration) return false + // Gap recovery waits for a reserved commit before it asks the leader. It + // is a hydration cycle only from the point where it will change rows + // (maintainer decision on PR #2265): after pullSince answers, and before + // it moves the stream position. A sync transaction that begins later + // straddles it; one that ends before then never crosses it. + this.throwIfReservedCommitOpen() + // Every reload in this recovery continues its hydration cycle. + const reloadInCycle = (adapter: HydrationPersistenceAdapter) => + this.reloadActiveSubsetsUnsafe( + adapter, + /* skipUnchangedReplace */ false, + /* restoreResetBaseline */ false, + /* replayDemand */ false, + /* continuesHydrationCycle */ true, + ) if (this.persistence.coordinator.pullSince && this.latestRowVersion >= 0) { let pullResponse: PullSinceResponse | undefined try { @@ -4866,15 +5033,14 @@ class PersistedCollectionRuntime< if (lifecycleGeneration !== this.lifecycleGeneration) return false if (pullResponse.ok) { + this.beginHydrationCycle() this.observeStreamPosition( pullResponse.latestTerm, pullResponse.latestSeq, pullResponse.latestRowVersion, ) if (pullResponse.requiresFullReload) { - await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe(adapter), - ) + await this.runInHydrationScope((adapter) => reloadInCycle(adapter)) this.publicRowVersion = Math.max( this.publicRowVersion, pullResponse.latestRowVersion, @@ -4883,9 +5049,7 @@ class PersistedCollectionRuntime< } const deltas = pullResponse.deltas if (!deltas) { - await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe(adapter), - ) + await this.runInHydrationScope((adapter) => reloadInCycle(adapter)) this.publicRowVersion = Math.max( this.publicRowVersion, pullResponse.latestRowVersion, @@ -4911,6 +5075,7 @@ class PersistedCollectionRuntime< delta.collectionMetadataMutations, }, adapter, + true, ) if (lifecycleGeneration !== this.lifecycleGeneration) return } @@ -4989,6 +5154,30 @@ class PersistedCollectionRuntime< this.refreshScopedSubsets() return } + // The reset is one hydration cycle from here, before its eager truncate + // publishes. It abandons a reservation that opened before it, instead of + // waiting for it; a sync transaction that begins during the reset + // straddles it, so readers never wait on a truncated Collection. + this.abandonReservedCommit?.() + this.hydrationSequence++ + this.hydratingGeneration = lifecycleGeneration + try { + await this.truncateThenReloadUnsafe(adapter, lifecycleGeneration) + } finally { + // The reload ends the cycle; this covers an early return before it. + if ( + this.hydratingGeneration === lifecycleGeneration && + this.activeHydrationContext === undefined + ) { + this.hydratingGeneration = null + } + } + } + + private async truncateThenReloadUnsafe( + adapter: HydrationPersistenceAdapter, + lifecycleGeneration: number, + ): Promise { // A subscriber may reacquire reentrantly from the truncate commit. this.hydratedDemands.clear() this.resetSequence++ @@ -5013,15 +5202,17 @@ class PersistedCollectionRuntime< // follower requests its demand again. await this.reloadActiveSubsetsUnsafe( adapter, - false, - this.syncMode === `eager`, - true, + /* skipUnchangedReplace */ false, + /* restoreResetBaseline */ this.syncMode === `eager`, + /* replayDemand */ true, + /* continuesHydrationCycle */ true, ) } private async invalidateFromCommittedTxUnsafe( txCommitted: TxCommitted, adapter: HydrationPersistenceAdapter, + continuesHydrationCycle = false, ): Promise { if (this.scopedRecovery) { this.refreshScopedSubsets() @@ -5030,7 +5221,13 @@ class PersistedCollectionRuntime< const reloadActiveSubsets = (skipUnchangedReplace = false) => this.runInHydrationScope( (scopedAdapter) => - this.reloadActiveSubsetsUnsafe(scopedAdapter, skipUnchangedReplace), + this.reloadActiveSubsetsUnsafe( + scopedAdapter, + skipUnchangedReplace, + /* restoreResetBaseline */ false, + /* replayDemand */ false, + continuesHydrationCycle, + ), adapter, ) @@ -5165,11 +5362,14 @@ class PersistedCollectionRuntime< skipUnchangedReplace = false, restoreResetBaseline = false, replayDemand = false, + // A reset's reload continues the cycle its truncate began. + continuesHydrationCycle = false, ): Promise { if (this.scopedRecovery) { this.refreshScopedSubsets() return } + if (!continuesHydrationCycle) this.beginHydrationCycle() const lifecycleGeneration = this.lifecycleGeneration const truncateGeneration = this.sourceTruncateGeneration const activeSubsetOptions = Array.from(this.activeSubsets.values()) @@ -5198,7 +5398,6 @@ class PersistedCollectionRuntime< .map(([key]) => key) this.hydratedDemands.clear() - this.hydrationSequence++ const hydrationContext = { suppliedRowKeys: new Set() } this.activeHydrationContext = hydrationContext this.hydratingGeneration = lifecycleGeneration @@ -5616,15 +5815,10 @@ function createWrappedSyncConfig< transactionStack[transactionStack.length - 1] const settlePublicationAdmissionWaiters = ( transaction: OpenSyncTransaction, - error?: unknown, ) => { const waiters = transaction.publicationAdmissionWaiters transaction.publicationAdmissionWaiters = undefined - if (!waiters) return - for (const waiter of waiters) { - if (error === undefined) waiter.resolve() - else waiter.reject(error) - } + waiters?.forEach((release) => release()) } const bindToCurrentHydration = ( transaction: OpenSyncTransaction, @@ -5740,7 +5934,7 @@ function createWrappedSyncConfig< error: unknown, ) => { removePendingPublicationTransaction(transaction) - settlePublicationAdmissionWaiters(transaction, error) + settlePublicationAdmissionWaiters(transaction) return createHandledRejection(error) } const updateMetadataWrite = ( @@ -5776,42 +5970,29 @@ function createWrappedSyncConfig< cancelAdmissionWait?: () => void } const acquisitions = new Map() + // Resolves when the reservation ends (committed, abandoned, or + // discarded by cleanup; maintainer decision on PR #2265) or when the + // load's own demand aborts. The load then runs, or rejects through its + // own signal. const waitForPublicationAdmission = ( transaction: OpenSyncTransaction, options: LoadSubsetOptions, acquisition: SubsetAcquisition, - ): Promise => { - if (options.signal?.aborted) { - return Promise.reject( - options.signal.reason ?? new SyncTransactionAbortedError(), - ) - } - return new Promise((resolve, reject) => { - const waiters = - transaction.publicationAdmissionWaiters ?? - (transaction.publicationAdmissionWaiters = new Set()) - let settled = false - const settle = (action: () => void) => { - if (settled) return - settled = true - waiters.delete(waiter) - options.signal?.removeEventListener(`abort`, abort) + ): Promise => + new Promise((resolve) => { + const waiters = (transaction.publicationAdmissionWaiters ??= + new Set()) + const settle = () => { + waiters.delete(settle) + options.signal?.removeEventListener(`abort`, settle) acquisition.cancelAdmissionWait = undefined - action() - } - const waiter = { - resolve: () => settle(resolve), - reject: (error: unknown) => settle(() => reject(error)), + resolve() } - const abort = () => - waiter.reject( - options.signal?.reason ?? new SyncTransactionAbortedError(), - ) - waiters.add(waiter) - options.signal?.addEventListener(`abort`, abort, { once: true }) - acquisition.cancelAdmissionWait = waiter.resolve + if (options.signal?.aborted) return settle() + waiters.add(settle) + options.signal?.addEventListener(`abort`, settle, { once: true }) + acquisition.cancelAdmissionWait = settle }) - } const getTerminalFailure = () => runtime.getCurrentTerminalFailure() // Network-first initial rendering: once the deadline passes, or the // remote fails, a completed restore that installed rows opens the @@ -5917,12 +6098,44 @@ function createWrappedSyncConfig< begin: params.begin, write: params.write as SyncControlFns[`write`], commit: params.commit, - truncate: params.truncate, + truncate: () => params.truncate({ markReady: false }), metadata: params.metadata ?? null, fail: params.fail, }, ) + const findReservedTransaction = () => + transactionStack.find( + (transaction) => transaction.reservesCommitTurn === true, + ) + runtime.setReservedCommitControls( + () => { + const transaction = findReservedTransaction() + if (!transaction) return undefined + // Admission waiters settle when the transaction commits, aborts, or + // cleanup discards it. Every outcome closes the reservation. + return (transaction.reservationEnded ??= new Promise( + (resolve) => { + ;(transaction.publicationAdmissionWaiters ??= new Set()).add( + resolve, + ) + }, + )) + }, + // A peer reset abandons every reservation built before it: later writes + // are dropped, the commit rejects, and its waiters run. + () => { + for (const transaction of transactionStack) { + if (!transaction.reservesCommitTurn) continue + transaction.reservesCommitTurn = false + transaction.terminalFailure ??= { + error: new SyncTransactionAbortedError(), + } + settlePublicationAdmissionWaiters(transaction) + } + }, + ) + const persistenceCapability: SyncPersistenceCapabilityV1 = { protocol: SYNC_PERSISTENCE_PROTOCOL, version: SYNC_PERSISTENCE_VERSION, @@ -5934,11 +6147,23 @@ function createWrappedSyncConfig< }, reserveCommitTurn: () => { const openTransaction = getOpenTransaction() - if (openTransaction && !openTransaction.internal) + // A transaction a reset abandoned cannot reserve again. + if ( + openTransaction && + !openTransaction.internal && + !openTransaction.terminalFailure + ) openTransaction.reservesCommitTurn = true }, hydrateBaseline: async () => { if (startupState.cleanedUp) return + // The source holds this reservation itself, so waiting for its + // commit here would wait for the caller. + if (findReservedTransaction()) { + throw new InvalidPersistedCollectionConfigError( + `hydrateBaseline cannot run while this source's reserved sync transaction is open`, + ) + } try { await runtime.ensureResumeBaselineHydrated() } catch (error) { @@ -6285,7 +6510,7 @@ function createWrappedSyncConfig< bindToCurrentHydration(openTransaction) assertHydrationSequenceCurrent(openTransaction) } catch (error) { - settlePublicationAdmissionWaiters(openTransaction, error) + settlePublicationAdmissionWaiters(openTransaction) throw error } const transaction = { @@ -6363,11 +6588,9 @@ function createWrappedSyncConfig< let applied: Promise try { applied = runtime.applyHydrationBufferedTransaction(transaction) - } catch (error) { - settlePublicationAdmissionWaiters(openTransaction, error) - throw error + } finally { + settlePublicationAdmissionWaiters(openTransaction) } - settlePublicationAdmissionWaiters(openTransaction) settleRuntimeTransaction(openTransaction, applied) return visibleAfterDurable(openTransaction, applied) }, @@ -6542,9 +6765,8 @@ function createWrappedSyncConfig< // open old-source transaction is abandoned before the new cache binds. if (!sourceCleanupPromise) { sourceSession++ - const abortError = new SyncTransactionAbortedError() for (const transaction of transactionStack) { - settlePublicationAdmissionWaiters(transaction, abortError) + settlePublicationAdmissionWaiters(transaction) } transactionStack.length = 0 const retiredResult = sourceResult @@ -6620,7 +6842,6 @@ function createWrappedSyncConfig< // reenters cleanup before returning, wait for that exact synchronous // entry to publish its cleanup callback before sampling the result. const sourceSyncEntry = activeSourceSyncEntry - const cleanupError = new SyncTransactionAbortedError() for (const acquisition of acquisitions.values()) { acquisition.cancelAdmissionWait?.() } @@ -6628,7 +6849,7 @@ function createWrappedSyncConfig< acquisitions.clear() pendingPublicationTransactions.length = 0 for (const transaction of transactionStack) { - settlePublicationAdmissionWaiters(transaction, cleanupError) + settlePublicationAdmissionWaiters(transaction) } transactionStack.length = 0 let sourceCleanup: Promise @@ -6693,9 +6914,8 @@ function createWrappedSyncConfig< } } - const openTransaction = getOpenTransaction() const needsPublicationAdmission = - openTransaction?.reservesCommitTurn === true + findReservedTransaction() !== undefined let hydrationStarted = false const raceRetainedLoad = ( work: Promise, @@ -6756,10 +6976,14 @@ function createWrappedSyncConfig< ) { return } - const currentTransaction = getOpenTransaction() - // A source transaction that reserved its FIFO turn commits before + const currentTransaction = findReservedTransaction() + // A sync transaction that reserved its FIFO turn commits before // subset hydration advances the generation it was built against. - if (currentTransaction?.reservesCommitTurn) { + // A load admitted during the reservation waits here, before it + // takes an apply-mutex turn, so it runs after the commit's durable + // write. The runtime's deferral covers hydrations whose turn + // began earlier; it cannot keep that FIFO order. + if (currentTransaction) { await waitForPublicationAdmission( currentTransaction, options, @@ -6813,7 +7037,7 @@ function createLoopbackSyncConfig< begin: params.begin, write: params.write as SyncControlFns[`write`], commit: params.commit, - truncate: params.truncate, + truncate: () => params.truncate({ markReady: false }), metadata: params.metadata ?? null, fail: params.fail, }, diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 74f3ddbf7d..2953a91c78 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -1,3 +1,4 @@ +import { isDeepStrictEqual } from 'node:util' import { describe, expect, it, vi } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' import { @@ -24,6 +25,7 @@ import { oracleRuns, } from '../../db/tests/oracle-config.js' import { declareOracle } from '../../db/tests/oracle-declaration.js' +import { createOracleReach } from '../../db/tests/oracle-reach.js' import { assertGenerationNoticeHistory, atPersistedOracleCheckpoint, @@ -32,6 +34,7 @@ import { createCoordinatorHarness, createDeferred, createEventGate, + createFailStopCoordinatorHarness, createFollowerTab, createRecordingAdapter, describeUnlessAnyOracleReplay, @@ -2227,7 +2230,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // insert/update, and absent/present dependent successors. The two-read // unreserved case // reconstructs the report; one and three distinguish a misplaced threshold. - // Open transactions crossing hydration are excluded here and rejected below. + // Open transactions crossing hydration are excluded here. Unreserved ones are + // rejected below; reserved ones are covered by the persisted-read precedence + // grammar's open commit timings. // This driver supplies adapter scheduling, not live Electric or OPFS delivery. it.each( ([`insert`, `update`] as const).flatMap((type) => @@ -4987,6 +4992,1684 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } }, ) + + // # Do persisted reads preserve sync history, and do reserved commits survive them? + // + // Law 1 (persisted-baseline precedence, maintainer decision on #2036): + // persisted rows are an older baseline for newer sync-adapter changes, and + // an internal read never revokes an accepted sync transaction. Every + // history must reach the independent final rows, row metadata, and cursor + // below, and a persistence read must observe the durable rows after exactly + // the source steps whose durable writes completed before it. + // + // Law 2 (reserved commits): a persisted hydration that would start while a + // reserved sync transaction is open waits until that transaction ends, and + // the commit is accepted. Authority: SyncPersistenceCapabilityV1 + // `reserveCommitTurn` (packages/db/src/types.ts), widened by maintainer + // decision on #2226 to every persisted hydration. Maintainer decisions on + // #2226 also fix its edges: + // - While a hydration waits, readers keep the rows from before it began. + // A hydration that already began when the reservation opened continues; + // the transaction straddles it. + // - A notice that waited may be retried as gap recovery, because the + // reserved commit lands first and takes the next stream position. A tab + // asks the leader only to fill a gap it has not already filled, and + // sends no added upstream request. Each reservation a notice waited for + // can open one gap, so the oracle allows one more pullSince request per + // wait (maintainer decision on PR #2265). + // - A source that stops for good abandons its reserved transaction; waiting + // hydrations then proceed and the abandoned writes never apply. + // - A peer change that starts no hydration is not delayed. A load aborted + // while it waits settles before the commit. Cleanup while a hydration + // waits settles and reads nothing more. A pre-reset commit never publishes + // its own rows. A source that awaits its own baseline hydration inside its + // reservation is rejected at once rather than waiting on itself. + // + // History grammar. A held earlier durable write ("gate") queues the route, + // the hydration that a cell exercises: + // - `subset`, `complete subset`: on-demand demand; + // - `routed subset`: a follower's demand, which the owner serves by + // committing a row, broadcasting it, then replying; + // - `source baseline`: the source's own `hydrateBaseline` request; + // - `collection-reset`, `full-reload`, `seq-gap`, `seq-gap pull`, + // `seq-gap deltas`: peer notices (`seq-gap pull` recovers through a + // pullSince full reload; `seq-gap deltas` through pullSince deltas, a + // targeted row change and then a delta too large to apply in place, + // which reloads partway through the recovery); + // - `targeted`: a peer change for a key that is not public, which starts + // no hydration. + // The commit timing places the first sync transaction against the route: + // - `after read`: it begins after the read is published; + // - `queued`: it begins and commits while the read waits; + // - `open before admission`: it begins and reserves before the route's + // subset load is admitted, so the load waits in the wrapper's admission + // path. Whether the reservation commits or is abandoned, the load runs + // after it ends (maintainer decision on PR #2265); + // - `unreserved across pullSince`: it begins without reserving before a + // gap notice and commits while recovery waits for pullSince. Recovery + // is a hydration cycle only from the point where it will change rows, so + // the commit is accepted (maintainer decision on PR #2265); + // - `open during pullSince`: it begins and reserves while recovery waits + // for pullSince, before recovery's cycle begins, so recovery waits for + // its commit and asks the leader again; + // - `open across mutex turn`: it begins and reserves while the read waits + // behind the gate, before the read's turn; + // - `open across scope entry`: it reserves after the read's turn began and + // its hydration scope was entered, as a shared scheduler can hold it; + // - `open during the read`: it reserves while the route's first persisted + // read is in progress, so the transaction straddles the hydration (for an + // eager reset, after its truncate published); + // - `open in the first publication`: it reserves from the route's first + // public change, through a microtask, as a source reacting to it would; + // for an eager reset that lands between its truncate and its reload; + // - `open across startup`: the source begins and reserves inside its sync + // function, before startup has settled. + // The ending says how the reserved transaction ends: it `commits`, is + // `abandoned` (committed with an aborted signal), meets `cleanup` of the + // Collection, or the route's demand is aborted while it waits (`demand + // aborted`); with `source awaits own baseline` the source also awaits + // `hydrateBaseline` inside its reservation. A follow-up delivers, while + // the route waits, a peer reset and then a pre-reset peer commit whose row + // the reset superseded. After an aborted demand, the follow-up + // `peer change after the abort` delivers a peer's targeted change while the aborted + // load still waits for its turn. No live demand covers that row, so it + // stays out of the Collection and nothing reads for it. For a follower's + // routed subset, `owner replies, then the source reserves again` releases + // the owner's held reply while the owner's broadcast waits for the + // reservation. The source commits a few microtasks later and reserves an + // empty second transaction a few microtasks after that, as a source that + // reacts through promise continuations does. The follower can then find no + // reservation open while the broadcast still waits, and the second + // reservation defers the broadcast again. Law 7 still requires the load to + // settle after the owner's row is public. The delays range over 0 to 8 and + // 0 to 12 microtasks. A late second reservation holds the broadcast twice, + // so it may ask the leader twice. + // + // Three registrations share this one driver and model: the row-algebra + // cross (every operation and truncate replay; committed endings; the + // original routes and timings), the reservation cross (every route, timing, + // ending, and follow-up, with one insert and no truncate replay), and the + // follower window (the second-reservation delays above). The + // controlled coordinator does not prove host delivery; the Browser-host + // receiving witness is the multi-tab Electric oracle. + type PersistedReadRoute = + | `subset` + | `complete subset` + | `routed subset` + | `source baseline` + | `collection-reset` + | `full-reload` + | `seq-gap` + | `seq-gap pull` + | `seq-gap deltas` + | `targeted` + type PersistedReadTiming = + | `after read` + | `queued` + | `open before admission` + | `unreserved across pullSince` + | `open during pullSince` + | `open across mutex turn` + | `open across scope entry` + | `open during the read` + | `open in the first publication` + | `open across startup` + type PersistedReadEnding = + | `commits` + | `abandoned` + | `cleanup` + | `demand aborted` + | `source awaits own baseline` + type PersistedReadHistory = { + syncMode: `eager` | `on-demand` + route: PersistedReadRoute + operation: `insert` | `update` | `delete` + truncateReplay: `none` | `before` | `after` + commitTiming: PersistedReadTiming + ending: PersistedReadEnding + // For `owner replies, then the source reserves again`: the microtasks + // the source takes after the owner replies to commit its reservation, + // then to reserve a second transaction. + reReserveDelays?: readonly [number, number] + followUp: + | `none` + | `reset then pre-reset commit` + | `reset, re-reserve, then pre-reset commit` + | `retired-storage reset` + | `peer change after the abort` + | `owner replies, then the source reserves again` + } + type PersistedReadReach = + | `route performed a persisted read` + | `reservation opened while the read was pending` + | `reservation opened before the route was admitted` + | `unreserved transaction committed while recovery waited for pullSince` + | `reservation opened while recovery waited for pullSince` + | `reservation opened while a read was in progress` + | `reservation opened in the route's first publication` + | `reservation opened before startup settled` + | `peer change announced while reserved` + | `reset and pre-reset commit announced while waiting` + | `retired-storage reset announced while reserved` + | `peer change announced after the demand aborted` + | `source reserved again after the owner replied` + + // The rows of the persisted-reads histories. Each history gets fresh + // objects, so no history can observe another's. + const createPersistedReadRows = () => { + const storedRow = { id: `shared`, title: `persisted`, detail: `old` } + const storedOnly = { id: `stored-only`, title: `removed by truncate` } + const gateRow = { id: `gate`, title: `earlier sync transaction` } + const networkRow = { id: `shared`, title: `network`, detail: `new` } + const replacementRow = { id: `replacement`, title: `truncate replay` } + const peerRow = { id: `peer`, title: `peer change` } + const remoteRow = { id: `remote`, title: `owner` } + const supersededRow = { id: `superseded`, title: `before the reset` } + return { + storedRow, + storedOnly, + gateRow, + networkRow, + replacementRow, + peerRow, + remoteRow, + supersededRow, + } + } + type PersistedReadRows = ReturnType + + // Independent model. It reads only the history and its input rows, + // never an observation from production, so it can be reviewed and + // calibrated apart from the driver. + const expectedPersistedReadOutcome = ( + history: PersistedReadHistory, + fixtureRows: PersistedReadRows, + ) => { + const { + syncMode, + route, + operation, + truncateReplay, + commitTiming, + ending, + followUp, + } = history + const { + storedRow, + storedOnly, + gateRow, + networkRow, + replacementRow, + peerRow, + remoteRow, + } = fixtureRows + const waits = + commitTiming === `open before admission` || + commitTiming === `open during pullSince` || + commitTiming === `open across mutex turn` || + commitTiming === `open across scope entry` + // The transaction straddles a hydration that already began. + const straddles = + commitTiming === `open during the read` || + commitTiming === `open in the first publication` + const reserves = + waits || straddles || commitTiming === `open across startup` + // Gap recovery is a hydration cycle only from the point where it will + // change rows (maintainer decision on PR #2265). With pullSince that is + // after the leader answers, before recovery enters its hydration scope, + // so a reservation that opens at scope entry straddles the recovery. + // Without pullSince it is the reload inside the scope, so such a + // reservation comes first and the recovery waits. Every other route + // begins its hydration inside the scope. + const recoveryStraddles = + commitTiming === `open across scope entry` && + (route === `seq-gap pull` || route === `seq-gap deltas`) + // A peer reset abandons a reserved transaction that opened before the + // reset's hydration began: the route reset when the reservation opened + // before its turn, or a follow-up reset delivered while the reservation + // is open (maintainer decision on PR #2265). Its commit rejects, its + // first step applies nothing, and the reset runs without waiting. + const resetAbandons = + waits && + (route === `collection-reset` || + followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit`) + const hydrationWaits = waits && !recoveryStraddles && !resetAbandons + const hydrationStraddles = straddles || recoveryStraddles || resetAbandons + const noticeWaited = waits && !recoveryStraddles + const sourceSteps = + truncateReplay === `none` + ? [`network`] + : truncateReplay === `before` + ? [`replacement`, `network`] + : [`network`, `replacement`] + const demandRoute = + route === `subset` || + route === `complete subset` || + route === `routed subset` + const noDemandRead = + syncMode === `on-demand` && !demandRoute && route !== `source baseline` + // Rows the route adds durably. The owner's row for a routed subset lands + // after the follower's own read, so reads never see it. + const routeRows = + route === `targeted` || + route === `seq-gap deltas` || + followUp === `peer change after the abort` + ? [peerRow] + : route === `routed subset` + ? [remoteRow] + : [] + const lateRowIds = new Set(route === `routed subset` ? [remoteRow.id] : []) + const readBaselineRows = [ + storedRow, + storedOnly, + gateRow, + ...routeRows, + ].filter((row) => !lateRowIds.has(row.id)) + // Steps that apply: an abandoned reservation applies none of its writes. + const firstStepAbandoned = ending === `abandoned` || resetAbandons + const appliedSteps = firstStepAbandoned ? sourceSteps.slice(1) : sourceSteps + const abandonedStep = firstStepAbandoned ? sourceSteps[0] : undefined + + // Independent snapshot algebra over the source steps that apply, in + // order. A replacement discards every earlier row and leaves only its + // own; a network step writes or deletes the shared key. An abandoned step + // applies nothing. Complete row values and explicit metadata avoid the + // separate partial-update contract. Source steps are input history, never + // observations from production. + const baselineRows = [storedRow, storedOnly, gateRow, ...routeRows] + const foldSteps = (initialRows: Array) => { + const rows = new Map( + initialRows.map((row) => [row.id, row]), + ) + for (const step of appliedSteps) { + if (step === `replacement`) { + rows.clear() + rows.set(replacementRow.id, replacementRow) + } else if (operation === `delete`) rows.delete(`shared`) + else rows.set(networkRow.id, networkRow) + } + return [...rows.values()] + } + let durableSharedMetadata: unknown = { owner: `persisted` } + let expectedCursor = `persisted` as string + for (const step of appliedSteps) { + durableSharedMetadata = + step === `replacement` || operation === `delete` + ? undefined + : { owner: `network` } + expectedCursor = step + } + const expectedRows = foldSteps(baselineRows) + // On-demand without demand exposes the source's own rows, folded from its + // earlier row; an aborted demand loads nothing. Every other route exposes + // the durable rows. Uncertified on-demand cache rows stay quarantined + // after a baseline hydration (electric.ts: "loaded only through fresh + // subset snapshots"). + const exposesSourceRowsOnly = + noDemandRead || route === `source baseline` || ending === `demand aborted` + const expectedPublicRows = exposesSourceRowsOnly + ? foldSteps([gateRow]) + : expectedRows + // The persisted shared metadata is public only where the durable rows are. + const sharedMetadata = { + public: + exposesSourceRowsOnly && appliedSteps.length === 0 + ? undefined + : durableSharedMetadata, + durable: durableSharedMetadata, + } + const replacementMetadata = appliedSteps.includes(`replacement`) + ? { owner: `replacement` } + : undefined + const readsSomething = + !noDemandRead && + route !== `targeted` && + route !== `source baseline` && + ending !== `demand aborted` + // A routed subset reads its local rows, then rereads to admit the + // owner's row; a reset follow-up reloads, and so does the fenced + // pre-reset notice. Those reloads read only active demand: an eager + // Collection's baseline, or a subset whose demand was not aborted. An + // aborted demand loads no rows. + const expectedReadCount = + (!readsSomething ? 0 : route === `routed subset` ? 2 : 1) + + ((followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit`) && + (syncMode === `eager` || readsSomething) + ? 2 + : 0) + const applyDurableEvent = (rows: Map, event: string) => { + if (event === `replacement`) { + rows.clear() + rows.set(replacementRow.id, replacementRow) + } else if (event === `network`) { + if (operation === `delete`) rows.delete(`shared`) + else rows.set(networkRow.id, networkRow) + } else if (event === `owner commit`) rows.set(remoteRow.id, remoteRow) + } + // The first step committed before the route's reads inside startup. A + // commit made while gap recovery waits for pullSince is accepted, but its + // durable write waits behind recovery's apply-mutex turn, so recovery's + // reads still come first. + const startupStepCommitted = + commitTiming === `open across startup` && abandonedStep === undefined + const noWaitPullSinceCalls = + route === `seq-gap pull` || route === `seq-gap deltas` ? 1 : 0 + // A route notice waits in the waiting timings; a follower's owner notice + // arrives during a straddled reservation and waits there. A tab asks the + // leader only to fill a new gap, and each reservation a notice waited + // for can open one, so the bound is one request per wait; nothing else + // opens a gap. A second reservation can hold the owner's notice again. + const waitedNotices = + (noticeWaited && + (route === `full-reload` || + route === `seq-gap pull` || + route === `seq-gap deltas`)) || + (straddles && route === `routed subset`) + ? followUp === `owner replies, then the source reserves again` + ? 2 + : 1 + : 0 + const expectedSourceLoads = + route === `subset` || + route === `complete subset` || + route === `routed subset` + ? ending === `demand aborted` + ? 0 + : 1 + : route === `seq-gap` && syncMode === `eager` + ? 1 + : 0 + const metadataKeys = [`shared`, `stored-only`, `gate`, `replacement`] + const expectedMetadata = (side: `public` | `durable`) => + metadataKeys.map((key) => + key === `shared` + ? sharedMetadata[side] + : key === `replacement` + ? replacementMetadata + : undefined, + ) + return { + waits, + straddles, + reserves, + hydrationWaits, + resetAbandons, + hydrationStraddles, + sourceSteps, + demandRoute, + readBaselineRows, + appliedSteps, + expectedRows, + exposesSourceRowsOnly, + expectedPublicRows, + expectedCursor, + expectedReadCount, + applyDurableEvent, + startupStepCommitted, + noWaitPullSinceCalls, + waitedNotices, + expectedSourceLoads, + metadataKeys, + expectedMetadata, + } + } + + const runPersistedReadHistory = async (history: PersistedReadHistory) => { + const { syncMode, route, operation, commitTiming, ending, followUp } = + history + const fixtureRows = createPersistedReadRows() + const { + storedRow, + storedOnly, + gateRow, + networkRow, + replacementRow, + peerRow, + remoteRow, + supersededRow, + } = fixtureRows + const { + waits, + reserves, + hydrationWaits, + resetAbandons, + hydrationStraddles, + sourceSteps, + demandRoute, + readBaselineRows, + appliedSteps, + expectedRows, + exposesSourceRowsOnly, + expectedPublicRows, + expectedCursor, + expectedReadCount, + applyDurableEvent, + startupStepCommitted, + noWaitPullSinceCalls, + waitedNotices, + expectedSourceLoads, + metadataKeys, + expectedMetadata, + } = expectedPersistedReadOutcome(history, fixtureRows) + const adapter = createRecordingAdapter([ + { ...storedRow }, + { ...storedOnly }, + ]) + adapter.rowMetadata.set(`shared`, { owner: `persisted` }) + adapter.collectionMetadata.set(`cursor`, `persisted`) + const coordinator = createFailStopCoordinatorHarness( + `persisted-read-precedence`, + ) + const persistenceEntered = createEventGate() + const releasePersistence = createEventGate() + const readApplied = createEventGate() + const trace: Array = [] + // Every adapter call advances this counter. The quiescence checkpoint + // reads it; it is driver evidence, not part of the model. + let adapterCalls = 0 + const applyCommittedTx = adapter.applyCommittedTx.bind(adapter) + adapter.applyCommittedTx = async (...args) => { + adapterCalls++ + if (args[1].mutations.some(({ key }) => key === `gate`)) { + persistenceEntered.resolve() + await releasePersistence.promise + } + await applyCommittedTx(...args) + // Record durable completion, including a delete or truncate. A rejected + // adapter write must never appear as a completed source obligation. + if (args[1].truncate) trace.push(`replacement`) + else if (args[1].mutations.some(({ key }) => key === `shared`)) { + trace.push(`network`) + } + } + // The fixture leader allocates each commit the next durable position, + // after every commit already stored, including the peer commits that + // notices announce. A Browser leader does the same + // (`nextPersistedTxIfLeader`), so two commits never share a position. + const leaderPosition = { seq: 0, rowVersion: 0 } + const advanceLeaderPosition = (seq: number, rowVersion: number) => { + leaderPosition.seq = Math.max(leaderPosition.seq, seq) + leaderPosition.rowVersion = Math.max( + leaderPosition.rowVersion, + rowVersion, + ) + } + const nextPeerPosition = () => { + advanceLeaderPosition( + leaderPosition.seq + 1, + leaderPosition.rowVersion + 1, + ) + return { seq: leaderPosition.seq, rowVersion: leaderPosition.rowVersion } + } + // A peer commit stored durably at the next position, then announced. + const peerCommit = async (row: Todo, txId: string) => { + const position = nextPeerPosition() + await applyCommittedTx( + { kind: `eager`, collectionId: `persisted-read-precedence` }, + { + txId, + term: 1, + seq: position.seq, + rowVersion: position.rowVersion, + mutations: [{ type: `insert`, key: row.id, value: row }], + } as never, + ) + if (row.id === remoteRow.id) trace.push(`owner commit`) + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId, + latestRowVersion: position.rowVersion, + requiresFullReload: false, + changedRows: [{ key: row.id, value: row }], + deletedKeys: [], + }) + } + // Every route but `seq-gap` has a leader that answers pullSince with a + // full reload. + let pullSinceCalls = 0 + // Holds the next pullSince request open for the pullSince timings. + let pullSinceHold: + | { + entered: ReturnType + release: ReturnType + } + | undefined = + commitTiming === `unreserved across pullSince` || + commitTiming === `open during pullSince` + ? { entered: createEventGate(), release: createEventGate() } + : undefined + const pullHold = pullSinceHold + coordinator.pullSince = + route === `seq-gap` + ? undefined + : async () => { + pullSinceCalls++ + const hold = pullSinceHold + pullSinceHold = undefined + if (hold) { + hold.entered.resolve() + await hold.release.promise + } + if (route === `seq-gap deltas`) { + // A targeted delta for the row the peer stored at the skipped + // position, then a delta past the targeted-invalidation limit. + const bulkKeys = Array.from( + { length: 129 }, + (_, index) => `absent-${index}`, + ) + return Promise.resolve({ + type: `rpc:pullSince:res`, + rpcId: `gap-deltas`, + ok: true, + latestTerm: 1, + latestSeq: leaderPosition.seq, + latestRowVersion: leaderPosition.rowVersion, + requiresFullReload: false, + changedKeys: [peerRow.id], + deletedKeys: bulkKeys, + deltas: [ + { + txId: `gap-delta-row`, + latestRowVersion: leaderPosition.rowVersion - 1, + changedRows: [{ key: peerRow.id, value: peerRow }], + deletedKeys: [], + }, + { + txId: `gap-delta-bulk`, + latestRowVersion: leaderPosition.rowVersion, + changedRows: [], + deletedKeys: bulkKeys, + }, + ], + } as never) + } + return Promise.resolve({ + type: `rpc:pullSince:res`, + rpcId: `gap-pull`, + ok: true, + latestTerm: 1, + latestSeq: leaderPosition.seq, + latestRowVersion: leaderPosition.rowVersion, + requiresFullReload: true, + }) + } + // Upstream requests: direct source loads and remote-subset ensures, + // which owned demand, a follower's routed demand, and recovery's refetch + // go through. A follower's owner commits its row, broadcasts, then + // replies, as law 7 assumes. + let sourceLoads = 0 + if (route === `routed subset`) coordinator.isLeader = () => false + const requestEnsureRemoteSubset = + coordinator.requestEnsureRemoteSubset.bind(coordinator) + let ownerCommitted = false + const ownerReply = createEventGate() + if (followUp !== `owner replies, then the source reserves again`) { + ownerReply.resolve() + } + coordinator.requestEnsureRemoteSubset = async (...args) => { + sourceLoads++ + if (route === `routed subset` && !ownerCommitted) { + ownerCommitted = true + await peerCommit(remoteRow, `owner-commit`) + await ownerReply.promise + return + } + return requestEnsureRemoteSubset(...args) + } + coordinator.requestApplyCommittedTx = async (id, tx) => { + const positioned = { + ...tx, + seq: Math.max(tx.seq, leaderPosition.seq + 1), + rowVersion: Math.max(tx.rowVersion, leaderPosition.rowVersion + 1), + } + advanceLeaderPosition(positioned.seq, positioned.rowVersion) + await adapter.applyCommittedTx( + { kind: `eager`, collectionId: id }, + positioned, + ) + return { + type: `rpc:applyCommittedTx:res`, + rpcId: tx.txId, + ok: true, + term: positioned.term, + seq: positioned.seq, + latestRowVersion: positioned.rowVersion, + } + } + // Reads. A reserved sync transaction is open between its reservation and + // its end. A read that starts in that window, outside a straddled + // hydration, belongs to a hydration that did not wait. Every adapter read + // method is recorded. When armed, the first read after the route was + // admitted is held, so a reservation can open while it is in progress. + let reservationOpen = false + let routeAdmitted = false + let cleanupStarted = false + const routeReads: Array = [] + const readsDuringReservation: Array = [] + const readsAfterCleanup: Array = [] + const startupOrder: Array = [] + let heldRead: + | { + entered: ReturnType + release: ReturnType + } + | undefined + const recordRead = async (method: string) => { + adapterCalls++ + if (!routeAdmitted) startupOrder.push(`startup read`) + if (routeAdmitted) routeReads.push(method) + if (reservationOpen) readsDuringReservation.push(method) + if (cleanupStarted) readsAfterCleanup.push(method) + const hold = routeAdmitted ? heldRead : undefined + if (hold) { + heldRead = undefined + hold.entered.resolve() + await hold.release.promise + } + } + const loadSubset = adapter.loadSubset.bind(adapter) + const readRows: Array>> = [] + adapter.loadSubset = async (...args) => { + await recordRead(`loadSubset`) + const rows = await loadSubset(...args) + readRows.push(rows.map(({ value }) => ({ ...value }))) + trace.push(`persisted read`) + return rows + } + const loadResumeSnapshot = adapter.loadResumeSnapshot.bind(adapter) + adapter.loadResumeSnapshot = async (...args) => { + await recordRead(`loadResumeSnapshot`) + return loadResumeSnapshot(...args) + } + // The recording adapter defines every optional read method. + const loadCollectionMetadata = adapter.loadCollectionMetadata!.bind(adapter) + adapter.loadCollectionMetadata = async (...args) => { + await recordRead(`loadCollectionMetadata`) + return loadCollectionMetadata(...args) + } + const scanRows = adapter.scanRows!.bind(adapter) + adapter.scanRows = async (...args) => { + await recordRead(`scanRows`) + return scanRows(...args) + } + let sourceParams!: TodoSyncParams + const writeSourceStep = (step: (typeof sourceSteps)[number]) => { + if (step === `replacement`) { + sourceParams.truncate() + sourceParams.write({ type: `insert`, value: { ...replacementRow } }) + sourceParams.metadata!.row.set(`replacement`, { owner: `replacement` }) + } else if (operation === `delete`) { + sourceParams.write({ type: `delete`, key: `shared` }) + } else { + sourceParams.write({ type: operation, value: { ...networkRow } }) + sourceParams.metadata!.row.set(`shared`, { owner: `network` }) + } + sourceParams.metadata!.collection.set(`cursor`, step) + } + const reach = createOracleReach( + `persistence.reserved-commit-read-precedence`, + [ + ...(ending === `cleanup` || + ((route === `targeted` || ending === `demand aborted`) && + followUp !== `reset then pre-reset commit` && + followUp !== `reset, re-reserve, then pre-reset commit`) + ? [] + : ([`route performed a persisted read`] as const)), + ...(waits && + commitTiming !== `open before admission` && + commitTiming !== `open during pullSince` + ? ([`reservation opened while the read was pending`] as const) + : []), + ...(commitTiming === `open before admission` + ? ([`reservation opened before the route was admitted`] as const) + : []), + ...(commitTiming === `unreserved across pullSince` + ? ([ + `unreserved transaction committed while recovery waited for pullSince`, + ] as const) + : []), + ...(commitTiming === `open during pullSince` + ? ([ + `reservation opened while recovery waited for pullSince`, + ] as const) + : []), + ...(commitTiming === `open during the read` + ? ([`reservation opened while a read was in progress`] as const) + : []), + ...(commitTiming === `open in the first publication` + ? ([`reservation opened in the route's first publication`] as const) + : []), + ...(commitTiming === `open across startup` + ? ([`reservation opened before startup settled`] as const) + : []), + ...(route === `targeted` && waits + ? ([`peer change announced while reserved`] as const) + : []), + ...((followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit`) && + waits + ? ([`reset and pre-reset commit announced while waiting`] as const) + : []), + ...(followUp === `retired-storage reset` + ? ([`retired-storage reset announced while reserved`] as const) + : []), + ...(followUp === `peer change after the abort` + ? ([`peer change announced after the demand aborted`] as const) + : []), + ...(followUp === `owner replies, then the source reserves again` + ? ([`source reserved again after the owner replied`] as const) + : []), + ], + ) + reach.beginHistory() + const beginReservedSourceStep = () => { + sourceParams.begin() + sourceParams.metadata!.persistence!.reserveCommitTurn() + writeSourceStep(sourceSteps[0]!) + reservationOpen = true + } + const collection = createCollection( + persistedCollectionOptions({ + id: `persisted-read-precedence`, + getKey: (row) => row.id, + syncMode, + sync: { + sync: (params) => { + sourceParams = params + if (commitTiming === `open across startup`) { + startupOrder.push(`source entered`) + beginReservedSourceStep() + reach.reach(`reservation opened before startup settled`) + } + params.markReady() + return { + loadSubset: () => { + sourceLoads++ + return true + }, + } + }, + }, + persistence: { adapter, coordinator }, + }), + ) + const abortDemand = new AbortController() + const options = + route === `complete subset` + ? { signal: abortDemand.signal } + : { limit: 100, signal: abortDemand.signal } + // Pending operations and the outcome each one owes. + const pendingOperations: Array> = [] + const expectedOutcomes: Array<`fulfilled` | `rejected`> = [] + const retainOperation = ( + receipt: unknown, + expected: `fulfilled` | `rejected` = `fulfilled`, + ) => { + const pending = Promise.resolve(receipt) + void pending.catch(() => undefined) + pendingOperations.push(pending) + expectedOutcomes.push(expected) + } + const commitSourceTransaction = ( + signal?: AbortSignal, + expected: `fulfilled` | `rejected` = signal ? `rejected` : `fulfilled`, + ) => { + try { + retainOperation(sourceParams.commit(signal), expected) + } catch (error) { + retainOperation(Promise.reject(error), expected) + } + } + const endReservation = () => { + reservationOpen = false + if (ending === `abandoned`) { + const cancelled = new AbortController() + cancelled.abort() + commitSourceTransaction(cancelled.signal) + } else { + // A reset that abandoned the reservation rejects its commit. + commitSourceTransaction( + undefined, + resetAbandons ? `rejected` : `fulfilled`, + ) + } + } + const published = new Set() + let hasPrimaryFailure = false + try { + await atPersistedOracleCheckpoint( + collection.stateWhenReady(), + `precedence collection ready`, + ) + const startupObservation = + commitTiming === `open across startup` + ? { + readsBeforeSource: startupOrder.indexOf(`source entered`) > 0, + sourceEnteredLast: + startupOrder[startupOrder.length - 1] === `source entered`, + } + : undefined + if (commitTiming === `open across startup`) endReservation() + const subscription = collection.subscribeChanges( + () => { + for (const key of collection.keys()) published.add(String(key)) + }, + { includeInitialState: false }, + ) + // The scope callback witnesses publication, not just completion of the + // adapter's load promise. When armed, it holds one scope after entry + // and before its task, as a shared logical scheduler can. + let heldScope: + | { + entered: ReturnType + release: ReturnType + } + | undefined + // Set inside the scope hook, so TypeScript cannot narrow it here. + let readPublished = false as boolean + adapter.runInHydrationScope = async (task) => { + const hold = heldScope + heldScope = undefined + if (hold) { + hold.entered.resolve() + await hold.release.promise + } + const result = await task(adapter) + if (routeAdmitted) { + readPublished = true + readApplied.resolve() + } + return result + } + // Progress checkpoint: wait until three consecutive tasks pass with no + // adapter call; the bound turns a livelock into a named failure. + const persistenceQuiesced = async (label: string) => { + let quietTasks = 0 + for (let task = 0; task < 200 && quietTasks < 3; task++) { + const callsBefore = adapterCalls + await flushAsyncWork() + quietTasks = adapterCalls === callsBefore ? quietTasks + 1 : 0 + } + if (quietTasks < 3) { + throw new Error( + `Persisted oracle checkpoint did not quiesce: ${label}`, + ) + } + } + sourceParams.begin() + sourceParams.write({ type: `insert`, value: { ...gateRow } }) + retainOperation(sourceParams.commit()) + await atPersistedOracleCheckpoint( + persistenceEntered.promise, + `predecessor held`, + ) + const rowsAtAdmission = new Map( + Array.from(collection.values(), (row) => [ + row.id, + stripVirtualProps(row), + ]), + ) + if (commitTiming === `unreserved across pullSince`) { + sourceParams.begin() + writeSourceStep(sourceSteps[0]!) + } + if (commitTiming === `open before admission`) { + beginReservedSourceStep() + reach.reach(`reservation opened before the route was admitted`) + } + routeAdmitted = true + let ownBaselineRejection: string | undefined + let ownerVisibleAtSettlement: boolean | undefined + if ( + route === `subset` || + route === `complete subset` || + route === `routed subset` + ) { + const load = collection._sync.loadSubset(options) + retainOperation( + load, + ending === `demand aborted` ? `rejected` : `fulfilled`, + ) + // Law 7 at the load's settlement: a routed load settles only after + // the owner's broadcast rows are public. + if (route === `routed subset`) { + void Promise.resolve(load).then( + () => { + ownerVisibleAtSettlement = collection.has(remoteRow.id) + }, + () => undefined, + ) + } + } else if (route === `source baseline`) { + retainOperation(sourceParams.metadata!.persistence!.hydrateBaseline()) + } else if (route === `collection-reset`) { + coordinator.emit({ + type: `collection:reset`, + schemaVersion: 1, + resetEpoch: 1, + }) + } else if (route === `full-reload`) { + const position = nextPeerPosition() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId: `remote-reload`, + latestRowVersion: position.rowVersion, + requiresFullReload: true, + }) + } else if (route === `targeted`) { + // Waiting timings deliver the change after the reservation opens. + if (!waits && commitTiming !== `open in the first publication`) { + await peerCommit(peerRow, `peer-change`) + } + } else { + // Sequence gap, with or without pullSince. A peer commit this tab has + // not observed arrives past a sequence gap and carries no rows, as a + // peer's must-refetch reset commit does in a multi-tab refetch storm. + // Recovery rereads durable rows. For `seq-gap deltas` the skipped + // position holds a peer row that only pullSince reports. + const skipped = nextPeerPosition() + if (route === `seq-gap deltas`) { + await applyCommittedTx( + { kind: `eager`, collectionId: `persisted-read-precedence` }, + { + txId: `gap-row`, + term: 1, + seq: skipped.seq, + rowVersion: skipped.rowVersion, + mutations: [{ type: `insert`, key: peerRow.id, value: peerRow }], + } as never, + ) + } + const position = nextPeerPosition() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId: `remote-gap`, + latestRowVersion: position.rowVersion, + requiresFullReload: false, + changedRows: [], + deletedKeys: [], + }) + } + await flushAsyncWork() + expect(readRows).toEqual([]) + // A peer reset, then a pre-reset peer commit whose row the reset + // superseded. The reset removed that commit's row from durable storage. + const deliverFollowUp = () => { + if (followUp === `none`) return + // A reset for a storage generation this Collection no longer uses + // does not apply here, so it abandons nothing. + if (followUp === `retired-storage reset`) { + coordinator.emit( + { type: `collection:reset`, schemaVersion: 1, resetEpoch: 1 }, + undefined, + `retired-storage`, + ) + return + } + coordinator.emit({ + type: `collection:reset`, + schemaVersion: 1, + resetEpoch: 1, + }) + // The source reserves its turn again on the transaction the reset + // abandoned; a dead transaction cannot reserve. + if ( + followUp === `reset, re-reserve, then pre-reset commit` && + reservationOpen + ) { + sourceParams.metadata!.persistence!.reserveCommitTurn() + } + const position = nextPeerPosition() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId: `pre-reset-commit`, + latestRowVersion: position.rowVersion, + requiresFullReload: false, + changedRows: [{ key: supersededRow.id, value: supersededRow }], + deletedKeys: [], + }) + } + // With `owner replies, then the source reserves again`, the owner's + // reply is released first; the source then ends its reservation and + // reserves an empty second transaction after the history's delays. + const endStraddledReservation = async () => { + if (followUp !== `owner replies, then the source reserves again`) { + return endReservation() + } + const [commitDelay, reserveDelay] = history.reReserveDelays! + ownerReply.resolve() + for (let i = 0; i < commitDelay; i++) await Promise.resolve() + endReservation() + for (let i = 0; i < reserveDelay; i++) await Promise.resolve() + sourceParams.begin() + sourceParams.metadata!.persistence!.reserveCommitTurn() + reservationOpen = true + reach.reach(`source reserved again after the owner replied`) + await persistenceQuiesced(`owner's broadcast waits again`) + reservationOpen = false + commitSourceTransaction() + } + let remainingSteps: ReadonlyArray<(typeof sourceSteps)[number]> = + commitTiming === `open across startup` + ? sourceSteps.slice(1) + : sourceSteps + let rowsWhileReserved: Map | undefined + let peerWhileReserved: unknown + let abortedBeforeEnd: string | undefined + const observeWhileReserved = () => { + rowsWhileReserved = new Map( + Array.from(collection.values(), (row) => [ + row.id, + stripVirtualProps(row), + ]), + ) + } + if ( + commitTiming === `after read` || + commitTiming === `open across startup` + ) { + releasePersistence.resolve() + if (route !== `targeted`) { + await atPersistedOracleCheckpoint( + readApplied.promise, + `read published`, + ) + } else await persistenceQuiesced(`targeted change applied`) + deliverFollowUp() + await persistenceQuiesced(`follow-up applied`) + } else if (commitTiming === `unreserved across pullSince`) { + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pullHold!.entered.promise, + `recovery waits for pullSince`, + ) + commitSourceTransaction() + reach.reach( + `unreserved transaction committed while recovery waited for pullSince`, + ) + pullHold!.release.resolve() + await persistenceQuiesced(`recovery finished`) + remainingSteps = sourceSteps.slice(1) + } else if (commitTiming === `queued`) { + // The sync transaction commits while the read waits (below). + } else if (commitTiming === `open in the first publication`) { + const reserved = createEventGate() + let armed = true + const firstPublication = collection.subscribeChanges( + () => { + if (!armed) return + armed = false + queueMicrotask(() => { + beginReservedSourceStep() + reach.reach(`reservation opened in the route's first publication`) + reserved.resolve() + }) + }, + { includeInitialState: false }, + ) + releasePersistence.resolve() + if (route === `targeted`) await peerCommit(peerRow, `peer-change`) + await atPersistedOracleCheckpoint(reserved.promise, `route published`) + firstPublication.unsubscribe() + await persistenceQuiesced(`straddled hydration finished`) + observeWhileReserved() + await endStraddledReservation() + remainingSteps = sourceSteps.slice(1) + } else if (commitTiming === `open during the read`) { + heldRead = { entered: createEventGate(), release: createEventGate() } + const { entered, release } = heldRead + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + entered.promise, + `route read in progress`, + ) + beginReservedSourceStep() + reach.reach(`reservation opened while a read was in progress`) + release.resolve() + await persistenceQuiesced(`straddled read finished`) + observeWhileReserved() + await endStraddledReservation() + remainingSteps = sourceSteps.slice(1) + } else { + if (commitTiming === `open during pullSince`) { + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pullHold!.entered.promise, + `recovery waits for pullSince`, + ) + beginReservedSourceStep() + reach.reach(`reservation opened while recovery waited for pullSince`) + pullHold!.release.resolve() + } else if (commitTiming === `open before admission`) { + // The reservation opened before the route was admitted. + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pendingOperations[0]!, + `predecessor applied while the reservation is open`, + ) + } else if (commitTiming === `open across mutex turn`) { + // The read was admitted before this reservation. + beginReservedSourceStep() + if (!readPublished) { + reach.reach(`reservation opened while the read was pending`) + } + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pendingOperations[0]!, + `predecessor applied while the reservation is open`, + ) + } else { + // The read already holds its mutex turn and hydration scope. + heldScope = { entered: createEventGate(), release: createEventGate() } + const { entered, release } = heldScope + releasePersistence.resolve() + if (route === `targeted`) { + await persistenceQuiesced(`targeted change pending`) + heldScope = undefined + release.resolve() + } else { + await atPersistedOracleCheckpoint( + entered.promise, + `read entered its scope`, + ) + } + beginReservedSourceStep() + if (!readPublished) { + reach.reach(`reservation opened while the read was pending`) + } + release.resolve() + } + if (route === `targeted`) { + await peerCommit(peerRow, `peer-change`) + reach.reach(`peer change announced while reserved`) + } + const announceFollowUp = () => { + deliverFollowUp() + reach.reach( + followUp === `retired-storage reset` + ? `retired-storage reset announced while reserved` + : `reset and pre-reset commit announced while waiting`, + ) + } + // An aborted demand meets its follow-up after the abort (below). + if (followUp !== `none` && ending !== `demand aborted`) { + announceFollowUp() + } + await persistenceQuiesced(`reservation open, route waiting`) + observeWhileReserved() + peerWhileReserved = stripVirtualProps(collection.get(peerRow.id)) + if (ending === `cleanup`) { + cleanupStarted = true + reservationOpen = false + await atPersistedOracleCheckpoint( + collection.cleanup(), + `cleanup settles while the route waits`, + ) + } else { + if (ending === `demand aborted`) { + abortDemand.abort() + abortedBeforeEnd = await atPersistedOracleCheckpoint( + pendingOperations[1]!.then( + () => `fulfilled`, + () => `rejected`, + ), + `aborted demand settles before the reservation ends`, + ) + // A reset after the abort reloads only demand that is still + // active, so the aborted demand loads nothing. + if (followUp === `peer change after the abort`) { + await peerCommit(peerRow, `peer-change`) + reach.reach(`peer change announced after the demand aborted`) + await persistenceQuiesced(`peer change after the abort applied`) + } else if (followUp !== `none`) { + announceFollowUp() + await persistenceQuiesced(`follow-up after the abort applied`) + } + } + if (ending === `source awaits own baseline`) { + ownBaselineRejection = await atPersistedOracleCheckpoint( + sourceParams.metadata!.persistence!.hydrateBaseline().then( + () => `fulfilled`, + (error: unknown) => (error as Error).message, + ), + `own baseline settles inside the reservation`, + ) + } + endReservation() + } + remainingSteps = sourceSteps.slice(1) + } + if (ending !== `cleanup`) { + for (const step of remainingSteps) { + sourceParams.begin() + writeSourceStep(step) + commitSourceTransaction() + } + } + releasePersistence.resolve() + const outcomes = await atPersistedOracleCheckpoint( + Promise.allSettled(pendingOperations), + `sync history applied and durable`, + ) + if (ending === `cleanup`) { + await flushAsyncWork() + subscription.unsubscribe() + reach.assertReached() + expect({ status: collection.status, readsAfterCleanup }).toEqual({ + status: `cleaned-up`, + readsAfterCleanup: [], + }) + return + } + // A read that waited runs after the reservation ends. + if (reserves && route !== `targeted` && ending !== `demand aborted`) { + await atPersistedOracleCheckpoint( + readApplied.promise, + `route read published`, + ) + } + if (followUp !== `none`) await persistenceQuiesced(`follow-up settled`) + subscription.unsubscribe() + + // Observations and the values the model owes. The trace records durable + // completions (source steps and the owner's commit) beside persisted + // reads. Each read must observe the fold of the durable events before + // it, which proves the baseline it saw. Without waiting, every read + // precedes every source step, so the older baseline runs first; after a + // startup reservation commits, its step precedes the reads; while a read + // waits for a reserved commit, reads and later steps may interleave. + // Rows are compared by key: the adapter's row order is not part of the law. + const durableBefore = (index: number) => { + const rows = new Map(readBaselineRows.map((row) => [row.id, row])) + for (const event of trace.slice(0, index)) + applyDurableEvent(rows, event) + return rows + } + const sameRows = ( + left: Map, + right: Map, + ) => + left.size === right.size && + [...left].every(([key, value]) => + isDeepStrictEqual(value, right.get(key)), + ) + const byKey = (rows: Array) => + new Map(rows.map((row) => [row.id, row])) + const readIndexes = trace.flatMap((event, index) => + event === `persisted read` ? [index] : [], + ) + const stepIndexes = trace.flatMap((event, index) => + event === `network` || event === `replacement` ? [index] : [], + ) + const observedReadOrder = hydrationWaits + ? `may interleave` + : hydrationStraddles + ? readIndexes.length === 0 || + stepIndexes.every((step) => readIndexes[0]! < step) + ? `straddled read before every step` + : `other` + : startupStepCommitted + ? readIndexes.every( + (read) => + read > stepIndexes[0]! && + stepIndexes.slice(1).every((step) => read < step), + ) + ? `after the startup step` + : `other` + : readIndexes.every((read) => + stepIndexes.every((step) => read < step), + ) + ? `before every step` + : `other` + const observed = { + outcomes: outcomes.map(({ status }) => status), + readsDuringReservation: hydrationStraddles + ? [] + : readsDuringReservation, + stepOrder: trace.filter( + (event) => event === `network` || event === `replacement`, + ), + ownerCommits: trace.filter((event) => event === `owner commit`).length, + readCount: readIndexes.length, + readsObserveDurable: readIndexes.map((index, read) => + sameRows(byKey(readRows[read] as Array), durableBefore(index)), + ), + readOrder: observedReadOrder, + finalDurable: durableBefore(trace.length), + pullSinceWithinBound: + pullSinceCalls >= noWaitPullSinceCalls && + pullSinceCalls <= noWaitPullSinceCalls + waitedNotices, + sourceLoads, + status: collection.status, + publicRows: new Map( + Array.from(collection.values(), (row) => [ + row.id, + stripVirtualProps(row), + ]), + ), + durableRows: adapter.rows, + publicMetadata: metadataKeys.map((key) => + sourceParams.metadata!.row.get(key), + ), + durableMetadata: metadataKeys.map((key) => + adapter.rowMetadata.get(key), + ), + publicCursor: sourceParams.metadata!.collection.get(`cursor`), + durableCursor: adapter.collectionMetadata.get(`cursor`), + ...(hydrationWaits && route !== `targeted` + ? { rowsWhileReserved } + : {}), + ...(hydrationStraddles ? { rowsWhileReserved } : {}), + ...(route === `targeted` && waits ? { peerWhileReserved } : {}), + ...(followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit` + ? { publishedSuperseded: published.has(supersededRow.id) } + : {}), + ...(route === `routed subset` ? { ownerVisibleAtSettlement } : {}), + ...(ending === `demand aborted` ? { abortedBeforeEnd } : {}), + ...(ending === `source awaits own baseline` + ? { ownBaselineRejection } + : {}), + ...(startupObservation ? { startupObservation } : {}), + } + const expected = { + outcomes: expectedOutcomes, + readsDuringReservation: [], + stepOrder: appliedSteps, + ownerCommits: route === `routed subset` ? 1 : 0, + readCount: expectedReadCount, + readsObserveDurable: Array.from( + { length: expectedReadCount }, + () => true, + ), + readOrder: hydrationWaits + ? `may interleave` + : hydrationStraddles + ? `straddled read before every step` + : startupStepCommitted + ? `after the startup step` + : `before every step`, + // The fold over durable events must agree with the closed form. + finalDurable: byKey(expectedRows), + pullSinceWithinBound: true, + sourceLoads: expectedSourceLoads, + status: `ready`, + publicRows: byKey(expectedPublicRows), + durableRows: byKey(expectedRows), + publicMetadata: expectedMetadata(`public`), + durableMetadata: expectedMetadata(`durable`), + publicCursor: expectedCursor, + durableCursor: expectedCursor, + // While a hydration waits, readers keep the rows from before it; a + // straddled read has published the pre-source durable rows. + ...(hydrationWaits && route !== `targeted` + ? { rowsWhileReserved: rowsAtAdmission } + : {}), + ...(hydrationStraddles + ? { + rowsWhileReserved: byKey( + exposesSourceRowsOnly ? [gateRow] : readBaselineRows, + ), + } + : {}), + ...(route === `targeted` && waits + ? { peerWhileReserved: peerRow } + : {}), + ...(followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit` + ? { publishedSuperseded: false } + : {}), + ...(route === `routed subset` + ? { ownerVisibleAtSettlement: true } + : {}), + ...(ending === `demand aborted` + ? { abortedBeforeEnd: `rejected` } + : {}), + ...(ending === `source awaits own baseline` + ? { + ownBaselineRejection: `hydrateBaseline cannot run while this source's reserved sync transaction is open`, + } + : {}), + ...(startupObservation + ? { + startupObservation: { + readsBeforeSource: true, + sourceEnteredLast: true, + }, + } + : {}), + } + expect(observed).toEqual(expected) + // Reach is judged after the comparison, so an unexpected read fails on + // the read count rather than as an undeclared reach state. + if (routeReads.length > 0) reach.reach(`route performed a persisted read`) + reach.assertReached() + } catch (error) { + hasPrimaryFailure = true + throw error + } finally { + releasePersistence.resolve() + await cleanupPersistedOracle( + [ + ...(demandRoute + ? [() => collection._sync.unloadSubset(options)] + : []), + () => Promise.allSettled(pendingOperations), + () => collection.cleanup(), + ], + hasPrimaryFailure, + ) + } + } + + // Row-algebra cross: the original routes and timings, every operation and + // truncate replay, with a reservation that commits. + it.each( + ( + [ + { syncMode: `on-demand`, route: `subset` }, + { syncMode: `on-demand`, route: `complete subset` }, + { syncMode: `on-demand`, route: `collection-reset` }, + { syncMode: `on-demand`, route: `full-reload` }, + { syncMode: `on-demand`, route: `seq-gap` }, + { syncMode: `on-demand`, route: `seq-gap pull` }, + { syncMode: `on-demand`, route: `seq-gap deltas` }, + { syncMode: `eager`, route: `collection-reset` }, + { syncMode: `eager`, route: `full-reload` }, + { syncMode: `eager`, route: `seq-gap` }, + { syncMode: `eager`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `seq-gap deltas` }, + ] as const + ).flatMap(({ syncMode, route }) => + ([`insert`, `update`, `delete`] as const).flatMap((operation) => + ([`none`, `before`, `after`] as const).flatMap((truncateReplay) => + ( + [ + `after read`, + `queued`, + `open across mutex turn`, + `open across scope entry`, + ] as const + ).map((commitTiming): PersistedReadHistory => ({ + syncMode, + route, + operation, + truncateReplay, + commitTiming, + ending: `commits`, + followUp: `none`, + })), + ), + ), + ), + )( + `preserves sync history over persisted reads ($syncMode, $route, $operation, truncate=$truncateReplay, timing=$commitTiming)`, + runPersistedReadHistory, + ) + + // Reservation cross: every route, reserving timing, ending, and follow-up, + // with one insert and no truncate replay. `after read` is each route's + // control. An ending or follow-up appears only where it can happen: cleanup, + // aborted demand, and the source's own baseline need a waiting route; the + // follow-up's reset fence is judged on eager Collections, whose reset + // reloads every durable row. + const reservationRoutes = [ + { syncMode: `on-demand`, route: `subset` }, + { syncMode: `on-demand`, route: `complete subset` }, + { syncMode: `on-demand`, route: `routed subset` }, + { syncMode: `on-demand`, route: `source baseline` }, + { syncMode: `on-demand`, route: `collection-reset` }, + { syncMode: `on-demand`, route: `full-reload` }, + { syncMode: `on-demand`, route: `seq-gap` }, + { syncMode: `on-demand`, route: `seq-gap pull` }, + { syncMode: `on-demand`, route: `seq-gap deltas` }, + { syncMode: `eager`, route: `collection-reset` }, + { syncMode: `eager`, route: `full-reload` }, + { syncMode: `eager`, route: `seq-gap` }, + { syncMode: `eager`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `seq-gap deltas` }, + { syncMode: `eager`, route: `targeted` }, + ] as const + const reservationHistories = reservationRoutes.flatMap( + ({ syncMode, route }) => + ( + [ + `after read`, + `open before admission`, + `unreserved across pullSince`, + `open during pullSince`, + `open across mutex turn`, + `open across scope entry`, + `open during the read`, + `open in the first publication`, + `open across startup`, + ] as const + ).flatMap((commitTiming) => { + // Only a gap recovery that asks the leader waits for pullSince. + if ( + (commitTiming === `unreserved across pullSince` || + commitTiming === `open during pullSince`) && + route !== `seq-gap pull` && + route !== `seq-gap deltas` + ) { + return [] + } + // Only a subset load is admitted through the wrapper's admission. + if ( + commitTiming === `open before admission` && + route !== `subset` && + route !== `complete subset` + ) { + return [] + } + if (commitTiming === `open during the read` && route === `targeted`) + return [] + // Only a route whose hydration publishes rows reaches a publication. + if ( + commitTiming === `open in the first publication` && + syncMode === `on-demand` && + route !== `subset` && + route !== `complete subset` && + route !== `routed subset` + ) { + return [] + } + const waits = + commitTiming === `open before admission` || + commitTiming === `open during pullSince` || + commitTiming === `open across mutex turn` || + commitTiming === `open across scope entry` + const endings: Array = + commitTiming === `after read` || + commitTiming === `unreserved across pullSince` + ? [`commits`] + : waits + ? [ + `commits`, + `abandoned`, + `cleanup`, + ...(route === `subset` || route === `complete subset` + ? ([`demand aborted`] as const) + : []), + ...(route === `source baseline` + ? ([`source awaits own baseline`] as const) + : []), + ] + : [`commits`, `abandoned`] + return endings.flatMap((ending) => + ( + [ + `none`, + ...((syncMode === `eager` && + (commitTiming === `after read` || waits) && + (ending === `commits` || ending === `abandoned`)) || + ending === `demand aborted` + ? ([ + `reset then pre-reset commit`, + `reset, re-reserve, then pre-reset commit`, + ] as const) + : []), + ...(ending === `demand aborted` + ? ([`peer change after the abort`] as const) + : []), + ...(syncMode === `eager` && + waits && + (ending === `commits` || ending === `abandoned`) + ? ([`retired-storage reset`] as const) + : []), + ] as const + ).map((followUp): PersistedReadHistory => ({ + syncMode, + route, + operation: `insert`, + truncateReplay: `none`, + commitTiming, + ending, + followUp, + })), + ) + }), + ) + // The follower window: a second reservation after the owner replied. + const reReserveHistories = Array.from({ length: 9 }, (_, commitDelay) => + Array.from({ length: 13 }, (_, reserveDelay): PersistedReadHistory => ({ + syncMode: `on-demand`, + route: `routed subset`, + operation: `insert`, + truncateReplay: `none`, + commitTiming: `open during the read`, + ending: `commits`, + followUp: `owner replies, then the source reserves again`, + reReserveDelays: [commitDelay, reserveDelay], + })), + ).flat() + it.each(reReserveHistories)( + `keeps a follower's load behind a second reservation (delays=$reReserveDelays)`, + runPersistedReadHistory, + ) + it.each(reservationHistories)( + `keeps reserved commits and their neighbours lawful ($syncMode, $route, timing=$commitTiming, ending=$ending, follow-up=$followUp)`, + runPersistedReadHistory, + ) + + // The model predicts every observation from the history and its input + // rows; the driver runs the real Collection and compares; the history + // lists are the grammar the registrations below draw from. + declareOracle({ + check: [runPersistedReadHistory], + model: [expectedPersistedReadOutcome], + grammar: [reservationHistories, reReserveHistories], + }) }) /** diff --git a/packages/db-sqlite-persistence-core/tests/persisted-reload-metadata.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-reload-metadata.test.ts index c1b8964ea1..311b058a1d 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-reload-metadata.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-reload-metadata.test.ts @@ -1934,292 +1934,6 @@ describe(`persistedCollectionOptions`, () => { }, ) - // This bounded grammar crosses insert/update/delete with no truncate replay, - // replay before the change, and replay after it. Each history runs with the - // persistence read already finished and with accepted commits waiting behind - // that read. Both schedules must satisfy the same independent result below. - // Eager and on-demand Collections receive reset/full-reload notifications; - // on-demand Collections acquire bounded or complete subsets. A full reload - // with no demand validates already public rows without publishing unrelated - // stored rows. The controlled coordinator does not prove host delivery. - it.each( - ( - [ - { syncMode: `on-demand`, route: `subset` }, - { syncMode: `on-demand`, route: `complete subset` }, - { syncMode: `on-demand`, route: `collection-reset` }, - { syncMode: `on-demand`, route: `full-reload` }, - { syncMode: `eager`, route: `collection-reset` }, - { syncMode: `eager`, route: `full-reload` }, - ] as const - ).flatMap(({ syncMode, route }) => - ([`insert`, `update`, `delete`] as const).flatMap((operation) => - ([`none`, `before`, `after`] as const).flatMap((truncateReplay) => - [false, true].map((commitBeforeRead) => ({ - syncMode, - route, - operation, - truncateReplay, - commitBeforeRead, - })), - ), - ), - ), - )( - `preserves sync history over persisted reads ($syncMode, $route, $operation, truncate=$truncateReplay, queued=$commitBeforeRead)`, - async ({ - syncMode, - route, - operation, - truncateReplay, - commitBeforeRead, - }) => { - const storedRow = { id: `shared`, title: `persisted`, detail: `old` } - const storedOnly = { id: `stored-only`, title: `removed by truncate` } - const gateRow = { id: `gate`, title: `earlier sync transaction` } - const networkRow = { id: `shared`, title: `network`, detail: `new` } - const replacementRow = { id: `replacement`, title: `truncate replay` } - const adapter = createRecordingAdapter([ - { ...storedRow }, - { ...storedOnly }, - ]) - adapter.rowMetadata.set(`shared`, { owner: `persisted` }) - adapter.collectionMetadata.set(`cursor`, `persisted`) - const coordinator = createFailStopCoordinatorHarness( - `persisted-read-precedence`, - ) - const persistenceEntered = createEventGate() - const releasePersistence = createEventGate() - const readApplied = createEventGate() - const trace: Array = [] - const applyCommittedTx = adapter.applyCommittedTx.bind(adapter) - adapter.applyCommittedTx = async (...args) => { - if (args[1].mutations.some(({ key }) => key === `gate`)) { - persistenceEntered.resolve() - await releasePersistence.promise - } - await applyCommittedTx(...args) - // Record durable completion, including a delete or truncate. A rejected - // adapter write must never appear as a completed source obligation. - if (args[1].truncate) trace.push(`replacement`) - else if (args[1].mutations.some(({ key }) => key === `shared`)) { - trace.push(`network`) - } - } - coordinator.requestApplyCommittedTx = async (id, tx) => { - await adapter.applyCommittedTx({ kind: `eager`, collectionId: id }, tx) - return { - type: `rpc:applyCommittedTx:res`, - rpcId: tx.txId, - ok: true, - term: tx.term, - seq: tx.seq, - latestRowVersion: tx.rowVersion, - } - } - const loadSubset = adapter.loadSubset.bind(adapter) - const readRows: Array>> = [] - adapter.loadSubset = async (...args) => { - const rows = await loadSubset(...args) - readRows.push(rows.map(({ value }) => ({ ...value }))) - trace.push(`persisted read`) - return rows - } - let sourceParams!: TodoSyncParams - const collection = createCollection( - persistedCollectionOptions({ - id: `persisted-read-precedence`, - getKey: (row) => row.id, - syncMode, - sync: { - sync: (params) => { - sourceParams = params - params.markReady() - return { loadSubset: () => true } - }, - }, - persistence: { adapter, coordinator }, - }), - ) - const options = route === `complete subset` ? {} : { limit: 100 } - const sourceSteps = - truncateReplay === `none` - ? [`network`] - : truncateReplay === `before` - ? [`replacement`, `network`] - : [`network`, `replacement`] - const noDemandRead = - syncMode === `on-demand` && - route !== `subset` && - route !== `complete subset` - - // Independent snapshot algebra: a later replacement discards everything - // before it. Otherwise the baseline survives only when there is no - // replacement, and the newer change decides the shared key. Complete row - // values and explicit metadata avoid the separate partial-update contract. - // Source steps are input history, never observations from production. - const expectedRows = - truncateReplay === `after` - ? [replacementRow] - : [ - ...(truncateReplay === `before` - ? [replacementRow] - : [storedOnly, gateRow]), - ...(operation === `delete` ? [] : [networkRow]), - ] - const expectedPublicRows = - noDemandRead && truncateReplay === `none` - ? [gateRow, ...(operation === `delete` ? [] : [networkRow])] - : expectedRows - const expectedMetadata = new Map([ - ...(truncateReplay === `none` - ? [] - : [[`replacement`, { owner: `replacement` }] as const]), - ...(truncateReplay !== `after` && operation !== `delete` - ? [[`shared`, { owner: `network` }] as const] - : []), - ]) - const expectedCursor = - truncateReplay === `after` ? `replacement` : `network` - // Await applied receipts and the optional subset-load promise. - const pendingOperations: Array> = [] - const retainOperation = (receipt: true | void | Promise) => { - const pending = Promise.resolve(receipt) - void pending.catch(() => undefined) - pendingOperations.push(pending) - } - let hasPrimaryFailure = false - try { - await atPersistedOracleCheckpoint( - collection.stateWhenReady(), - `precedence collection ready`, - ) - // Hold an earlier durable write so the real runtime queues its read and - // ordinary sync transactions. The scope callback witnesses publication, - // not just completion of the adapter's load promise. - adapter.runInHydrationScope = async (task) => { - const result = await task(adapter) - readApplied.resolve() - return result - } - sourceParams.begin() - sourceParams.write({ type: `insert`, value: { ...gateRow } }) - retainOperation(sourceParams.commit()) - await atPersistedOracleCheckpoint( - persistenceEntered.promise, - `precedence predecessor held`, - ) - if (route === `subset` || route === `complete subset`) { - retainOperation(collection._sync.loadSubset(options)) - } else if (route === `collection-reset`) { - coordinator.emit({ - type: `collection:reset`, - schemaVersion: 1, - resetEpoch: 1, - }) - } else { - coordinator.emit({ - type: `tx:committed`, - term: 1, - seq: 2, - txId: `remote-reload`, - latestRowVersion: 2, - requiresFullReload: true, - }) - } - await flushAsyncWork() - expect(readRows).toEqual([]) - expect(collection.get(`shared`)?.title).toBe( - syncMode === `eager` ? `persisted` : undefined, - ) - if (!commitBeforeRead) { - releasePersistence.resolve() - await atPersistedOracleCheckpoint( - readApplied.promise, - `persistence read published before commit`, - ) - } - for (const step of sourceSteps) { - sourceParams.begin() - if (step === `replacement`) { - sourceParams.truncate() - sourceParams.write({ type: `insert`, value: { ...replacementRow } }) - sourceParams.metadata!.row.set(`replacement`, { - owner: `replacement`, - }) - } else if (operation === `delete`) { - sourceParams.write({ type: `delete`, key: `shared` }) - } else { - sourceParams.write({ type: operation, value: { ...networkRow } }) - sourceParams.metadata!.row.set(`shared`, { owner: `network` }) - } - sourceParams.metadata!.collection.set(`cursor`, step) - retainOperation(sourceParams.commit()) - } - releasePersistence.resolve() - const outcomes = await atPersistedOracleCheckpoint( - Promise.allSettled(pendingOperations), - `sync history applied and durable`, - ) - // Compare after applied receipts and the subset load settle, before cleanup. Exact - // keys detect resurrected deletes and rows wrongly retained by truncate. - // Read contents and completion order prove that the older baseline ran - // first in both schedules. No intermediate-publication claim is made. - const metadataKeys = [`shared`, `stored-only`, `gate`, `replacement`] - expect({ - outcomes, - trace, - readRows, - status: collection.status, - publicRows: new Map( - Array.from(collection.values(), (row) => [ - row.id, - stripVirtualProps(row), - ]), - ), - durableRows: adapter.rows, - publicMetadata: metadataKeys.map((key) => - sourceParams.metadata!.row.get(key), - ), - durableMetadata: metadataKeys.map((key) => - adapter.rowMetadata.get(key), - ), - publicCursor: sourceParams.metadata!.collection.get(`cursor`), - durableCursor: adapter.collectionMetadata.get(`cursor`), - }).toEqual({ - outcomes: pendingOperations.map(() => ({ - status: `fulfilled`, - value: undefined, - })), - trace: [...(noDemandRead ? [] : [`persisted read`]), ...sourceSteps], - readRows: noDemandRead ? [] : [[storedRow, storedOnly, gateRow]], - status: `ready`, - publicRows: new Map(expectedPublicRows.map((row) => [row.id, row])), - durableRows: new Map(expectedRows.map((row) => [row.id, row])), - publicMetadata: metadataKeys.map((key) => expectedMetadata.get(key)), - durableMetadata: metadataKeys.map((key) => expectedMetadata.get(key)), - publicCursor: expectedCursor, - durableCursor: expectedCursor, - }) - } catch (error) { - hasPrimaryFailure = true - throw error - } finally { - releasePersistence.resolve() - await cleanupPersistedOracle( - [ - ...(route === `subset` || route === `complete subset` - ? [() => collection._sync.unloadSubset(options)] - : []), - () => Promise.allSettled(pendingOperations), - () => collection.cleanup(), - ], - hasPrimaryFailure, - ) - } - }, - ) - it(`rejects a source transaction that crosses a hydration cycle`, async () => { const adapter = createRecordingAdapter() const firstEntered = createEventGate() diff --git a/packages/db-sqlite-persistence-core/tests/persisted-support.ts b/packages/db-sqlite-persistence-core/tests/persisted-support.ts index e3ed2e64de..9113feb50f 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-support.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-support.ts @@ -371,7 +371,7 @@ export function createCoordinatorHarness(): CoordinatorHarness { } export type FailStopCoordinatorHarness = PersistedCollectionCoordinator & { - emit: (payload: unknown, senderId?: string) => void + emit: (payload: unknown, senderId?: string, storageId?: string) => void publishCalls: Array> remoteEnsureCalls: Array remoteReleaseCalls: Array @@ -417,11 +417,11 @@ export function createFailStopCoordinatorHarness( seq: tx.seq, latestRowVersion: tx.rowVersion, }), - emit: (payload, senderId = `remote-fail-stop-peer`) => { + emit: (payload, senderId = `remote-fail-stop-peer`, storageId) => { subscriber?.({ v: 1, dbName: `test-db`, - collectionId, + collectionId: storageId ?? collectionId, senderId, ts: Date.now(), payload, diff --git a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts new file mode 100644 index 0000000000..24ab31cfc2 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts @@ -0,0 +1,494 @@ +// @vitest-environment node +/** + * # Do reserved commits survive hydrations on the real SQLite adapter? + * + * Law: a persisted hydration that would start while a reserved sync + * transaction is open waits until it ends; a commit is then accepted, and an + * abandoned transaction applies nothing (`reserveCommitTurn`, + * packages/db/src/types.ts, widened by maintainer decision on #2226). A peer + * reset is the exception: it abandons a reserved transaction that began + * before it, so that commit rejects and the reset does not wait (maintainer + * decision on PR #2265). The + * primary owner is the persisted-reads grammar in persisted-oracle.test.ts, + * which uses a recording adapter and simulates a held hydration scope. This + * oracle drives the same law through `SQLiteCorePersistenceAdapter` over + * `node:sqlite`, including the shared logical scheduler, which can hold a + * hydration after its apply-mutex turn began. Its histories were reduced from + * the report in https://github.com/TanStack/db/issues/2226. + * + * Grammar: the hydration route (a peer full-reload notice, a peer reset, or a + * subset load admitted before the reservation) crossed with the timing (the + * route waits behind this Collection's held write, or the shared scheduler + * holds its scope behind another Collection's write after its turn began), + * the nesting (one reserved transaction, or a second one nested inside it), + * and the ending (the reserved transactions commit, or are abandoned). A + * peer reset abandons every reserved transaction that began before it, so a + * nested one rejects with the same abort as its outer one. + * + * Model: row-1's durable value before the reservation follows from the + * history alone: the initial row, a peer edit, or the held write's value. A + * commit makes row-1 `refetched`; an abandoned transaction, or one a peer + * reset abandoned, leaves the durable value. Each cell reaches its premise: the hydration entered its scope while + * the reservation was open. + * + * Limits: the simulated peer writes straight to the adapter and announces + * through a test coordinator, outside a leader that allocates positions, so + * only row-1, the commit outcome, and Collection status are judged. The + * Browser-host owner (the multi-tab Electric oracle) judges peer rows across + * real tabs. + */ + +import { DatabaseSync } from 'node:sqlite' +import { createCollection } from '@tanstack/db' +import { afterEach, describe, expect, it } from 'vitest' +import { createOracleReach } from '../../db/tests/oracle-reach.js' +import { declareOracle } from '../../db/tests/oracle-declaration.js' +import { + SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, + SQLiteCorePersistenceAdapter, + SingleProcessCoordinator, + eagerStorageTarget, + persistedCollectionOptions, +} from '../src' +import type { ProtocolEnvelope, SQLiteDriver, TxCommitted } from '../src' +import type { SyncConfig } from '@tanstack/db' +import type { SQLInputValue } from 'node:sqlite' + +interface ShapeRow extends Record { + readonly id: string + readonly value: string +} + +/** + * A real in-memory SQLite driver whose next write transaction can be held to keep the persistence queue busy. + * `sharedScheduling` marks it as one physical database, as the browser driver does, so every adapter on it + * shares one write/hydrate scheduler. + */ +const openHoldableDatabase = ({ + sharedScheduling = false, +}: { sharedScheduling?: boolean } = {}) => { + const database = new DatabaseSync(':memory:') + let hold: { reached: () => void; released: Promise } | undefined + let transactionQueue: Promise = Promise.resolve() + const driver: SQLiteDriver = { + ...(sharedScheduling + ? { [SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY]: {} } + : {}), + exec: async (sql) => { + database.exec(sql) + }, + query: async (sql: string, params: ReadonlyArray = []) => + // SQLiteDriver's generic contract erases SQL parameter and result types. + database + .prepare(sql) + .all(...(params as Array)) as Array, + run: async (sql, params = []) => { + database.prepare(sql).run(...(params as Array)) + }, + transaction: (work) => { + const transaction = transactionQueue.then(async () => { + const current = hold + hold = undefined + if (current) { + current.reached() + await current.released + } + database.exec('BEGIN IMMEDIATE') + try { + const result = await work(driver) + database.exec('COMMIT') + return result + } catch (error) { + database.exec('ROLLBACK') + throw error + } + }) + transactionQueue = transaction.catch(() => undefined) + return transaction + }, + } + const holdNextWrite = () => { + let release!: () => void + let reached!: () => void + const wasReached = new Promise((resolve) => { + reached = resolve + }) + hold = { + reached, + released: new Promise((resolve) => (release = resolve)), + } + return { release, wasReached } + } + return { database, driver, holdNextWrite } +} + +const waitFor = async ( + assertion: () => void, + timeoutMs: number = 2_000, +): Promise => { + const deadline = Date.now() + timeoutMs + for (;;) { + try { + assertion() + return + } catch (error) { + if (Date.now() >= deadline) throw error + await new Promise((resolve) => setTimeout(resolve, 5)) + } + } +} + +/** A coordinator for one tab that lets the test deliver another tab's broadcast, as BroadcastChannel does. */ +class PeerTabCoordinator extends SingleProcessCoordinator { + private readonly listeners = new Map< + string, + (message: ProtocolEnvelope) => void + >() + + override subscribe( + collectionId?: string, + onMessage?: (message: ProtocolEnvelope) => void, + ): () => void { + if (collectionId === undefined || onMessage === undefined) + return () => undefined + this.listeners.set(collectionId, onMessage) + return () => this.listeners.delete(collectionId) + } + + deliverFromPeer(collectionId: string, payload: unknown): void { + const listener = this.listeners.get(collectionId) + if (!listener) throw new Error(`No runtime subscribed to ${collectionId}.`) + listener({ + collectionId, + dbName: 'repro', + payload, + senderId: 'peer-tab', + ts: Date.now(), + v: 1, + }) + } +} + +const COLLECTION_ID = 'items' + +type SourceSyncParams = Parameters['sync']>[0] + +/** + * A persisted collection whose source the test drives by hand, with another tab committing to the same SQLite + * database. `reserve()` opens a source transaction that reserves its commit turn, as Electric's refetch does. + */ +const openPeerTabHarness = async ({ + sharedScheduling = false, + onDemand = false, +}: { sharedScheduling?: boolean; onDemand?: boolean } = {}) => { + const { database, driver, holdNextWrite } = openHoldableDatabase({ + sharedScheduling, + }) + // The concrete class exposes the stream-position and hydration-scope methods the factory's interface marks optional. + const adapter = new SQLiteCorePersistenceAdapter({ driver }) + // Reach checkpoint: resolves when a hydration scope starts its task, after + // any scheduler delay. A case commits only after the hydration it tests has + // tried to start, so the history overlaps the reservation however slow the + // runner is. + const scopeEntryWaiters: Array<() => void> = [] + const runScope = adapter.runInHydrationScope.bind(adapter) + adapter.runInHydrationScope = (task) => + runScope((scoped) => { + for (const resolve of scopeEntryWaiters.splice(0)) resolve() + return task(scoped) + }) + const nextScopeEntry = () => + new Promise((resolve) => scopeEntryWaiters.push(resolve)) + const coordinator = new PeerTabCoordinator() + let source: SourceSyncParams | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: COLLECTION_ID, + persistence: { adapter, coordinator }, + schemaVersion: 1, + getKey: (row) => row.id, + ...(onDemand ? { syncMode: 'on-demand' as const } : {}), + sync: { + sync: (params) => { + source = params + params.begin() + params.write({ + type: 'insert', + value: { id: 'row-1', value: 'initial' }, + }) + params.commit() + params.markReady() + return onDemand ? { loadSubset: () => true } : {} + }, + }, + }), + ) + const requireSource = (): SourceSyncParams => { + if (!source) throw new Error('The collection has not started its source.') + return source + } + const start = async () => { + await collection.preload() + await waitFor(() => + expect(collection.get('row-1')).toMatchObject({ value: 'initial' }), + ) + } + /** Commits a row change from the peer tab into the shared database and returns its broadcast notice. */ + const commitFromPeer = async (value: string): Promise => { + const position = await adapter.getStreamPosition( + eagerStorageTarget(COLLECTION_ID), + ) + const tx = { + mutations: [ + { + key: 'row-1', + type: 'update' as const, + value: { id: 'row-1', value }, + }, + ], + rowVersion: position.latestRowVersion + 1, + seq: position.latestSeq + 1, + term: position.latestTerm, + txId: value, + } + await adapter.applyCommittedTx(eagerStorageTarget(COLLECTION_ID), tx) + return { + latestRowVersion: tx.rowVersion, + requiresFullReload: true, + seq: tx.seq, + term: tx.term, + txId: tx.txId, + type: 'tx:committed', + } + } + const reserve = (value?: string) => { + const params = requireSource() + params.begin() + if (value) params.write({ type: 'update', value: { id: 'row-1', value } }) + params.metadata?.persistence?.reserveCommitTurn() + } + // A second reserved transaction nested inside the open one. + const reserveNested = () => { + const params = requireSource() + params.begin() + params.write({ type: 'insert', value: { id: 'row-2', value: 'nested' } }) + params.metadata?.persistence?.reserveCommitTurn() + } + const commit = () => requireSource().commit() + // Committing with an aborted signal abandons the transaction. + const abandon = () => { + const cancelled = new AbortController() + cancelled.abort() + return requireSource().commit(cancelled.signal) + } + await start() + return { + reserveNested, + abandon, + adapter, + coordinator, + collection, + commit, + commitFromPeer, + database, + holdNextWrite, + nextScopeEntry, + reserve, + } +} + +type AdapterHistory = { + route: 'peer full reload' | 'peer reset' | 'subset admitted earlier' + timing: 'mutex turn' | 'scheduler hold' + nesting: 'single' | 'nested' + ending: 'commits' | 'abandoned' +} + +const adapterHistories: Array = ( + ['peer full reload', 'peer reset', 'subset admitted earlier'] as const +).flatMap((route) => + (['mutex turn', 'scheduler hold'] as const).flatMap((timing) => + // A peer commit cannot become durable while this Collection's held write + // occupies the only SQLite writer, so that history needs a separate + // leader; the persisted-reads grammar's fixture leader covers it. + route === 'peer full reload' && timing === 'mutex turn' + ? [] + : (['single', 'nested'] as const).flatMap((nesting) => + (['commits', 'abandoned'] as const).map((ending) => ({ + route, + timing, + nesting, + ending, + })), + ), + ), +) + +/** A premise that must be reached; a miss fails by name instead of hanging. */ +const reached = (promise: Promise, label: string): Promise => + Promise.race([ + promise, + new Promise((_resolve, reject) => + setTimeout( + () => reject(new Error(`premise not reached: ${label}`)), + 2_000, + ), + ), + ]) + +// The model: a peer reset abandons the reservation, which began before it, +// so only a committing transaction on another route is accepted. An accepted +// commit shows its rows; a waited subset load runs after the reservation and +// succeeds either way. +const expectedAdapterOutcome = ({ route, nesting, ending }: AdapterHistory) => { + const commitAccepted = ending === 'commits' && route !== 'peer reset' + return { + commitAccepted, + observation: { + outcomes: Array.from({ length: nesting === 'nested' ? 2 : 1 }, () => + commitAccepted ? 'fulfilled' : 'rejected (AbortError)', + ), + nestedRow: nesting === 'nested' && commitAccepted ? 'nested' : undefined, + status: 'ready', + loadOutcome: + route === 'subset admitted earlier' ? 'fulfilled' : undefined, + }, + } +} + +describe('reserved commits across real-adapter hydrations', () => { + const cleanups: Array<() => Promise | void> = [] + afterEach(async () => { + for (const cleanup of cleanups.splice(0).reverse()) await cleanup() + }) + + // The driver runs one history on the real adapter and compares it with the + // model. + const runAdapterHistory = async (history: AdapterHistory) => { + const { route, timing, nesting, ending } = history + const harness = await openPeerTabHarness({ + sharedScheduling: timing === 'scheduler hold', + onDemand: route === 'subset admitted earlier', + }) + cleanups.push(() => harness.database.close()) + cleanups.push(() => harness.collection.cleanup()) + const reach = createOracleReach('sqlite.reserved-commit-hydration', [ + 'hydration entered its scope while reserved', + ]) + reach.beginHistory() + // row-1's durable value just before the reservation, from the history. + let durableBefore = 'initial' + const notice = + route === 'peer full reload' + ? await harness.commitFromPeer('peer edit') + : undefined + if (notice) durableBefore = 'peer edit' + // The subset load's settlement, judged after the reservation ends. + let load: Promise<'fulfilled' | 'rejected'> | undefined + const admitRoute = () => { + if (notice) harness.coordinator.deliverFromPeer(COLLECTION_ID, notice) + else if (route === 'peer reset') { + harness.coordinator.deliverFromPeer(COLLECTION_ID, { + type: 'collection:reset', + schemaVersion: 1, + resetEpoch: 1, + }) + } else { + load = Promise.resolve( + harness.collection._sync.loadSubset({ limit: 1 }), + ).then( + () => 'fulfilled' as const, + () => 'rejected' as const, + ) + } + } + if (timing === 'mutex turn') { + // This Collection's held write is ahead of the route on the mutex. + const write = harness.holdNextWrite() + harness.reserve('durable predecessor') + durableBefore = 'durable predecessor' + const blocker = Promise.resolve(harness.commit()).catch(() => undefined) + await write.wasReached + admitRoute() + // One task lets the route take its mutex turn before the reservation. + await new Promise((resolve) => setTimeout(resolve, 0)) + harness.reserve('refetched') + if (nesting === 'nested') harness.reserveNested() + const entered = harness.nextScopeEntry() + write.release() + await blocker + await reached(entered, 'hydration entered its scope') + } else { + // Another Collection's held write holds the shared scheduler, so the + // route's scope waits after its mutex turn began. + await harness.adapter.loadSubset( + eagerStorageTarget('other_collection'), + {}, + ) + const otherWrite = harness.holdNextWrite() + const blocker = harness.adapter.applyCommittedTx( + eagerStorageTarget('other_collection'), + { mutations: [], rowVersion: 1, seq: 1, term: 1, txId: 'blocker' }, + ) + await otherWrite.wasReached + let turnBegan!: () => void + const began = new Promise((resolve) => { + turnBegan = resolve + }) + const runInHydrationScope = harness.adapter.runInHydrationScope.bind( + harness.adapter, + ) + harness.adapter.runInHydrationScope = (task) => { + turnBegan() + return runInHydrationScope(task) + } + admitRoute() + await reached(began, 'route turn began') + harness.reserve('refetched') + if (nesting === 'nested') harness.reserveNested() + const entered = harness.nextScopeEntry() + otherWrite.release() + await blocker + await reached(entered, 'hydration entered its scope') + } + reach.reach('hydration entered its scope while reserved') + // Commits the innermost transaction first; a rejection keeps its name, + // so an abort and a cross-cycle failure stay distinct. + const settle = async () => { + try { + await (ending === 'commits' ? harness.commit() : harness.abandon()) + return 'fulfilled' + } catch (error) { + return `rejected (${(error as Error).name})` + } + } + const outcomes: Array = [] + if (nesting === 'nested') outcomes.push(await settle()) + outcomes.push(await settle()) + const loadOutcome = await load + reach.assertReached() + const expected = expectedAdapterOutcome(history) + const expectedValue = expected.commitAccepted ? 'refetched' : durableBefore + await waitFor(() => + expect(harness.collection.get('row-1')).toMatchObject({ + value: expectedValue, + }), + ) + // A subset load that waited runs after the reservation and succeeds. + expect({ + outcomes, + nestedRow: harness.collection.get('row-2')?.value, + status: harness.collection.status, + loadOutcome, + }).toEqual(expected.observation) + } + declareOracle({ + check: [runAdapterHistory], + model: [expectedAdapterOutcome], + grammar: [adapterHistories], + }) + it.each(adapterHistories)( + 'keeps a reserved commit lawful across a real-adapter hydration ($route, $timing, $nesting, $ending)', + runAdapterHistory, + ) +}) diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 0e3e03affc..2f23228eb5 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -614,8 +614,21 @@ export type SyncPersistenceCapabilityV1< value: unknown }) => Promise /** - * Reserve the open sync transaction's FIFO turn, so a subset hydration - * that starts before it commits waits for that commit. + * Reserve the open sync transaction's FIFO turn. Every persisted hydration + * that would start before it commits waits for that commit: a subset load, + * a reload or reset from a peer notice, and sequence-gap recovery. A subset + * load admitted while the turn is reserved also runs after the commit's + * durable write. The source must end the transaction: commit it, or abandon + * it by committing with an aborted signal. An abandon releases the waiting + * hydrations as a commit does, so they run. A peer reset abandons a + * reserved transaction that began before the reset: its commit rejects + * with `SyncTransactionAbortedError`, and the reset does not wait. + * Sequence-gap recovery counts as a hydration only from the point where it + * changes rows. A source that stops for good + * abandons it, or every later hydration keeps waiting. The source must not + * await a `hydrateBaseline` that it started before reserving while the + * reservation is open: that baseline waits for the commit. Calling + * `hydrateBaseline` while the reservation is open rejects. */ readonly reserveCommitTurn: () => void readonly scanPersistedRows: ( diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index f950fea895..907038e6e5 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -157,6 +157,7 @@ const staticOracleProperties = [ `sqlite-resume.managed-claim-read`, `sqlite-resume.retired-storage`, `sqlite-reload.class`, + `browser.multi-tab-reserved-history`, `sqlite-reload.class.electron`, `persistence.generation-notice`, `persistence.follower-remote-demand`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 13e1bd2322..873a1092ef 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -397,6 +397,11 @@ const ownerGroups: ReadonlyArray< `sqlite-reload`, `class`, ], + [ + `browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts`, + `browser`, + `multi-tab-reserved-history`, + ], [ `electron-db-sqlite-persistence/tests/reload-class-oracle.test.ts`, `sqlite-reload`, diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 54ee092413..9605b30a19 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -888,7 +888,11 @@ function createLoadSubsetDedupe>({ } /** - * Type for the awaitTxId utility function + * Type for the awaitTxId utility function. It resolves when the txid + * appears in the Electric stream (the server's confirmation), before this + * tab's persistence accepts the transaction. If that commit later rejects, + * the source fails instead, and the Collection's health node records it + * (maintainer decision on PR #2265). */ export type AwaitTxIdFn = (txId: Txid, timeout?: number) => Promise @@ -1939,6 +1943,17 @@ function createElectricSync>( // Abort controller for the stream - wraps the signal if provided const abortController = new AbortController() let cleanupAbortedStream = false + let transactionStarted = false + // Commit with an aborted signal: the wrapper discards the transaction and + // its reserved commit turn ends. + const abandonOpenTransaction = () => { + if (!transactionStarted) return + transactionStarted = false + const cancellation = new AbortController() + cancellation.abort() + const abandoned = commit(cancellation.signal) + if (abandoned !== true) void abandoned.catch(() => undefined) + } // A final initial failure decides its waiting loads. A cleanup that a // status listener runs inside that failure must not relabel it. let initialSyncFailedFinally = false @@ -1970,6 +1985,9 @@ function createElectricSync>( const fails = isActiveLifecycle() && !ready abortController.abort() if (fails) failInitialSync(new StreamAbortedError(collectionId)) + // A source that stops must end its reserved transaction. The initial + // failure decides first: abandoning commits. + abandonOpenTransaction() } if (shapeOptions.signal) { @@ -2015,6 +2033,9 @@ function createElectricSync>( const failWaits = (cause: unknown) => { rejectFullSnapshot(cause) if (!ready) failInitialSync(cause) + // No retry follows, so the open transaction can never commit. A + // source that stops must end its reserved transaction. + abandonOpenTransaction() } if (shapeOptions.onError) { @@ -2043,7 +2064,6 @@ function createElectricSync>( return }, }) - let transactionStarted = false const newTxids = new Set() const newSnapshots: Array = [] // Track if we've completed initial sync in progressive mode. A persisted @@ -2092,8 +2112,10 @@ function createElectricSync>( usesBaseline = false } + // The commit consumes the transaction even if it throws. const commitSourceTransaction = (): SyncAppliedReceipt => { const committedPresenceRevision = pendingPresenceRevision + transactionStarted = false const applied = commit() if (metadata?.persistence) { const retireAppliedPresence = () => { @@ -2109,8 +2131,10 @@ function createElectricSync>( } // Subset hydration must not split a stream transaction it overlaps. + // Every reserved transaction is marked open, so a stop can abandon it. const beginSourceTransaction = () => { begin() + transactionStarted = true metadata?.persistence?.reserveCommitTurn() } @@ -2328,12 +2352,7 @@ function createElectricSync>( const invalidateResume = () => { resumeInvalid = true - if (transactionStarted) { - const cancellation = new AbortController() - cancellation.abort() - commit(cancellation.signal) - transactionStarted = false - } + abandonOpenTransaction() syncedKeys.clear() newTxids.clear() newSnapshots.length = 0 @@ -2356,7 +2375,6 @@ function createElectricSync>( if (freshSnapshotPending) { freshSnapshotPending = false beginSourceTransaction() - transactionStarted = true truncate() recordPendingTruncate() syncedKeys.clear() @@ -2461,7 +2479,6 @@ function createElectricSync>( // Normal processing: write changes immediately if (!transactionStarted) { beginSourceTransaction() - transactionStarted = true } processChangeMessage(message) @@ -2520,7 +2537,6 @@ function createElectricSync>( // Start a transaction and truncate the collection if (!transactionStarted) { beginSourceTransaction() - transactionStarted = true } truncate() @@ -2603,8 +2619,7 @@ function createElectricSync>( bufferedMsg.headers.patterns, begin, write, - // The swap already opened a transaction, even though the - // normal-stream transactionStarted flag is still false. + // The swap already opened a transaction. true, (rowId) => { recordPendingPresence(rowId, false) @@ -2637,7 +2652,6 @@ function createElectricSync>( stageResumeMetadata() } applied = commitSourceTransaction() - transactionStarted = false } else if (commitPoint === `up-to-date` && metadata) { beginSourceTransaction() stageResumeMetadata() @@ -2735,6 +2749,9 @@ function createElectricSync>( if (!ready) failInitialSync(error) else fail(error) } + // A source that stops must end its reserved transaction. The initial + // failure decides first: abandoning commits. + abandonOpenTransaction() }, ) @@ -2774,13 +2791,7 @@ function createElectricSync>( shapeOptions.signal?.removeEventListener(`abort`, forwardExternalAbort) stream.unsubscribeAll() abortController.abort() - if (transactionStarted) { - transactionStarted = false - const cancellation = new AbortController() - cancellation.abort() - const abandoned = commit(cancellation.signal) - if (abandoned !== true) void abandoned.catch(() => undefined) - } + abandonOpenTransaction() pendingResumeBatches.length = 0 // Reset deduplication tracking so collection can load fresh data if restarted loadSubsetDedupe?.reset() diff --git a/packages/electric-db-collection/tests/electric-descriptor-harness.ts b/packages/electric-db-collection/tests/electric-descriptor-harness.ts index a31fe81e80..7d28ebc69d 100644 --- a/packages/electric-db-collection/tests/electric-descriptor-harness.ts +++ b/packages/electric-db-collection/tests/electric-descriptor-harness.ts @@ -31,6 +31,7 @@ export function expectWholeTagRecovery( export type StreamHarness = { send: (messages: Array>) => void + giveUp: (error: Error) => void unsubscribe: ReturnType holdSnapshots: boolean ignoreAbort: boolean @@ -71,11 +72,15 @@ export async function createMockedElectricClient() { ) let harness: StreamHarness | undefined return { - subscribe: (send: StreamHarness[`send`]) => { + subscribe: ( + send: StreamHarness[`send`], + giveUp: StreamHarness[`giveUp`], + ) => { harness = { // Deliberately retain the raw callback for late-delivery tests. // Release calls are observed separately from these injections. send, + giveUp, unsubscribe, holdSnapshots: harnessState.holdNewSnapshots, ignoreAbort: false, diff --git a/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts b/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts index d6002c82d8..19c805a20a 100644 --- a/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts +++ b/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts @@ -1010,6 +1010,54 @@ it(`starts a new subset after retiring an incomplete old stream transaction`, as } }) +// A progressive Collection opens its atomic swap as a reserved transaction. +// A throw inside the swap leaves that transaction open. When the SDK then +// gives up, the source has stopped for good and must abandon the transaction, +// as for any other reserved transaction. Otherwise a subset load that waits +// for the reservation never runs. +it(`releases a waiting subset when the SDK gives up after a failed atomic swap`, async () => { + const { adapter } = tagPersistence() + let poisoned = false + const collection = createCollection( + persistedCollectionOptions< + TestRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...descriptor(`once-spread`, false, `progressive`), + id: `failed-atomic-swap`, + getKey: (row) => { + if (poisoned) throw new Error(`getKey failed`) + return row.id + }, + persistence: { adapter }, + }), + ) + let demand: Promise | undefined + try { + collection.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(1)) + const stream = streams[0]! + stream.send([insert(1, `buffered`)]) + poisoned = true + expect(() => stream.send([upToDate])).toThrow(`getKey failed`) + demand = Promise.resolve(collection._sync.loadSubset({ limit: 1 })).then( + () => `ran`, + () => `rejected`, + ) + stream.giveUp(new Error(`SDK gave up`)) + // An abandon releases a waiting load as a commit does, so it runs. + expect( + await atCheckpoint(demand, `subset after the failed atomic swap`), + ).toBe(`ran`) + } finally { + poisoned = false + await atCheckpoint(collection.cleanup(), `failed atomic swap cleanup`) + await demand + } +}) + it(`rejects torn or reverted tag-recovery histories even when their last snapshot is correct`, () => { const cached = [{ id: 1, name: `cached`, stable: `stable-1` }] const replacement = [{ id: 2, name: `replacement`, stable: `stable-2` }] diff --git a/packages/electric-db-collection/tests/electric-oracle.property.test.ts b/packages/electric-db-collection/tests/electric-oracle.property.test.ts index 53e0f2ae92..1062739bdd 100644 --- a/packages/electric-db-collection/tests/electric-oracle.property.test.ts +++ b/packages/electric-db-collection/tests/electric-oracle.property.test.ts @@ -3690,13 +3690,15 @@ describe(`Electric adapter laws`, () => { const heldAcceptance = createDeferred() const holdAcceptance = (params: TParams): TParams => { const syncParams = params as unknown as { - commit: () => true | Promise + commit: (signal?: AbortSignal) => true | Promise } const commit = syncParams.commit return { ...params, - commit: () => { - const receipt = commit() + // A wrapper passes the source's signal through, so an abandoned + // transaction stays abandoned. + commit: (signal?: AbortSignal) => { + const receipt = commit(signal) if (!armAcceptance) return receipt return withAcceptedReceipt( receipt === true ? Promise.resolve() : receipt,