From 526d9bcfe415e2c313441cf4d32e7f85a24151a3 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 14:20:50 -0600 Subject: [PATCH 1/2] Preserve accepted metadata on implicit durable re-inserts Snapshot accepted row metadata for durable inserts that omit explicit metadata writes. Keep explicit metadata writes authoritative and avoid attributing the preserved metadata to the source. Share completed-reach classification between the controlled and native SQLite receiving drivers. Law: Sync metadata session decisions4,5,6,8,9,14,15,18 in docs/contributing/oracle-reviews/2026-10-22-sync-metadata-session.md, the implicit row-metadata rules, and glossary hydration-seed contract. An inner equal re-insert preserves accepted metadata through persistence and peer reload. Reach: Each controlled/native host campaign completes40 histories in each of three lanes. Fixed seed527962096 reaches2 completed bare-key1 implicit keeps and2 completed non-bare keeps per lane, with21 named/19 fullreload notices. RED: M01 controlled extended owner fails unchanged parent production at seed2281001/path7:0; M06/M07 native SQLite receiving replays/property fail the same metadata law, generated path7:0:0. GREEN: M08/M09 paired40-run campaigns and receiving replays pass. H02 repeats both hosts with the shared helper; H01b proves final-helper equivalence over 2304 histories/102912 comparisons. Final types/lint/format pass. Mutant: Original durable-metadata map and always-delete rival both fail the shared metadata comparison in controlled/native generated histories. An all-explicit-inner grammar ablation fails the completed implicit-keep reaches. Limits: One process/native SQLite with two production coordinators. No OPFS, Electron IPC, transport-loss/leadership-turnover or broader2272 claim. Random scarce reach is measured; fixed guards enforce both scarce branches. Host runs precede only an explicit terminal undefined return, linked by exact equivalence. --- docs/contributing/oracle-coverage.md | 52 +- .../src/persisted.ts | 14 +- .../sync-metadata-session-persisted.test.ts | 211 +++++--- .../sync-metadata-session-sqlite.test.ts | 462 ++++++++++++++++++ packages/db/tests/oracle-config.ts | 2 + packages/db/tests/oracle-replay-manifest.ts | 7 +- .../db/tests/sync-metadata-session-oracle.ts | 111 ++++- 7 files changed, 778 insertions(+), 81 deletions(-) create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index afa469b5b0..e63484720e 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4422,24 +4422,50 @@ must equal the model's, which a restart would read. `persisted-peers-full` repeats the lane under full row updates, where an update replaces its row in storage too; `persisted-peers-on-demand` repeats it on demand with a demand for every row. A reset over metadata of a missing -row is too rare in generated histories to declare; a replay pins it. A -random CI campaign (seed `654774679`) found a `persisted-peers-full` history -that reproduces on `main`: an outer transaction inserts over a bare seed, an -inner one re-inserts the row and commits, and a peer's full reload leaves the -row with the cached seed's metadata, where the model expects none (law: -the source reads its row metadata). The maintainer chose to leave it open, -tracked with #2281; an `it.fails` replay through the same model, driver, and -comparison pins its shrunk prefix, and the random campaign can still draw it. -Owner: this oracle. A reset +row is too rare in generated histories to declare; a replay pins it. Durable source equal re-inserts preserve accepted row metadata, including absence +or metadata supplied by a later DbClient hydration seed, without making the keep +a source metadata write (decisions 9, 14, and 18). The constructive +`seedReinsertReloadHistoryArbitrary` and ordinary original replay use the shared +independent model and per-action source metadata comparison. Controlled +partial/full/on-demand lanes and native SQLite receiving lanes each pass fixed +40 and random 40 generated histories, with actual named/full-reload notices in +every native lane. Final fixed seed `527962096` independently reaches 40 inner +commits, 21 named notices, 19 full reloads, 2 completed bare-key1 implicit keeps, +and 2 completed non-bare implicit keeps per lane on both hosts. Every normal run +enforces those completed fixed-branch guards; random branch counts are measured +without a guarantee of reach. Controlled final random seed `-704889444` measures +0 bare/4 non-bare; native `1784284659` measures 2 bare/1 non-bare. Older +commit/notice guards still apply to both campaigns. Original RED and direct +replay seed `2281001` remain preserved. Original-defect and +always-delete-kept-metadata mutants both fail the shared product comparison +through both drivers. An all-explicit-inner grammar ablation passes 40 product +comparisons per host but fails the new completed-branch guards, with all hashes +restored. Native on-demand RED can occur at inner commit before the explicit peer +notice because actual coordination reads storage sooner; eager partial/full RED +reaches explicit peer step 6. + +The completed-branch guards were added in independent review after the first +production edit. M16 revealed zero bare-key1 reach at fixed seed `2281001`, then +measured seed `527962096` supplied both premises. The bare conjunction has +probability 1/24, so the reproducible fixed campaign supplies its durable +assertion while random campaigns measure it. No prior guard, comparison, +minimum, or run budget was removed or lowered. The model, grammar, and comparison +are unchanged; only the new properties' fixed seeds were calibrated. Broader +#2272 transport and host gaps remain open: this receiver covers two real Browser +coordinators over Node BroadcastChannel/Web Locks and native SQLite in one +process. It does not establish Chromium OPFS, Electron, transport gaps, +leadership turnover, every partial-notice shape, or every shared session grammar. +The constructive receiver uses core holds; broad controlled histories retain +store/read holds. Owner: this sync metadata session oracle. A reset may rewrite its rows, which its reload then applies as source writes. `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end (decision 21); a seed that arrived after it stays judged against it (decision 22). Dependent suffixes of an abandoned transaction belong to the -abort-graph oracle. The peer is the driver, not a second -Collection, so the claim stops at the controlled coordinator; a second -production driver with two real coordinators is the open receiving -witness (issue #2272). Missed notices and notices over the targeted key limit are not +abort-graph oracle. The broad peer grammar uses a controlled coordinator. The new constructive +seed/re-insert/reload receiver uses two real Collections and coordinators for +the bounded metadata law above; it does not close the broader host and transport +receiving scope of issue #2272. Missed notices and notices over the targeted key limit are not generated; the persisted oracle's grids cover them. Two held-cleanup Query retention witnesses race on CI; the maintainer deferred the race until a failure reports the timer trace these witnesses now print, and the review diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 666fbf6a1a..0d93565e26 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -4698,6 +4698,18 @@ class PersistedCollectionRuntime< value: operation.value as Record, } }) + // An equal re-insert keeps accepted metadata without writing it in this + // sync run. Its durable replacement must keep that same value, including + // a DbClient seed that never reached storage (decisions 9 and 14). + const rowMetadataWrites = new Map(transaction.rowMetadataWrites) + for (const { key } of transaction.insertOperations) { + if (rowMetadataWrites.has(key)) continue + const value = this.syncControls.metadata?.row.get(key) + rowMetadataWrites.set( + key, + value === undefined ? { type: `delete` } : { type: `set`, value }, + ) + } return { txId: safeRandomUUID(), term: streamPosition.term, @@ -4706,7 +4718,7 @@ class PersistedCollectionRuntime< cacheGenerationClaimId: this.cacheGenerationClaim?.claimId, truncate: transaction.truncate, mutations, - rowMetadataMutations: toMetadataMutations(transaction.rowMetadataWrites), + rowMetadataMutations: toMetadataMutations(rowMetadataWrites), collectionMetadataMutations: toMetadataMutations( transaction.collectionMetadataWrites, ), diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index 7e50570609..b483f91cd2 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -5,6 +5,7 @@ import { absentMarker, cachedMetadata, cachedRow, + completedSeedReinsertReloadRegime, describeMismatch, historyRegimes, persistedCachedLane, @@ -19,13 +20,17 @@ import { persistedReadSeedLane, persistedSeedLane, runSessionHistory, + seedReinsertReloadHistoryArbitrary, + seedReinsertReloadReplay, sessionCollectionKeys, sessionHistoryArbitrary, } from '../../db/tests/sync-metadata-session-oracle' import { oraclePropertyOptions, + oracleReplayTest, oracleRuns, readOracleRunConfig, + recordOracleReplay, } from '../../db/tests/oracle-config' import { createOracleReach } from '../../db/tests/oracle-reach' import { SingleProcessCoordinator, persistedCollectionOptions } from '../src' @@ -1209,29 +1214,13 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla it.each(peerFullReplays)(`replays under full rows: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerFullLane), ) - // Known gap, pinned from the persisted-peers-full random campaign (seed - // 654774679, shrunk to this prefix): an outer transaction inserts over a - // bare seed, an inner one re-inserts the row and commits, and a peer's full - // reload then leaves row 1 with the cached seed's metadata, where the model - // expects none. This reproduces on main and is tracked with #2281. It - // replays through the same model, driver, and comparison. Remove `fails` - // when the reload stops restoring that metadata. - it.fails( - `replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, - () => - assertPersistedHistory( - [ - { type: `begin` }, - { type: `seed`, key: 1, narrow: false, bare: true }, - { type: `insertOverSeed`, key: 1, metadata: false }, - { type: `begin` }, - { type: `reinsert`, key: 1, metadata: false }, - { type: `commit` }, - { type: `peer`, key: 2, notice: `reload` }, - ], - persistedPeerFullLane, - ), - ) + // Pinned from the persisted-peers-full random campaign (seed 654774679): + // an outer transaction inserts over a bare seed and an inner equal re-insert + // commits first. Its durable replacement must keep the accepted metadata + // (absent here), so a peer's full reload cannot restore the older cache + // metadata. This uses the campaigns' model, driver, and comparison. + it(`replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, () => + assertPersistedHistory(seedReinsertReloadReplay, persistedPeerFullLane)) it.each(peerOnDemandReplays)(`replays on demand: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerOnDemandLane), ) @@ -1241,48 +1230,140 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla ) }) -describe(`sync metadata session oracle through the persisted wrapper: nesting and seeds`, () => { - for (const { - lane, - property: lanePropertyName, - seed, - runs, - reach: states, - } of extraLanes) { - const runLane = (history: Array) => - assertPersistedHistory(history, lane) - if (requestedReplayProperty === undefined) { - const reach = createOracleReach(`${lanePropertyName} (fixed)`, states, { - minimum: 5, - }) - fcTest.prop([sessionHistoryArbitrary(lane)], { - numRuns: oracleRuns(runs), - seed, - })( - `keeps the session law on ${lane.name} histories (fixed)`, - (history) => { +// The broad peer campaigns can already find this defect. This constructive +// grammar measures its scarce overlap and adjacent legal histories directly; +// it uses the existing independent model, driver, and per-action comparison. +const seedReloadProperty = `sync-metadata-session.persisted-seed-reinsert-reload` +for (const campaign of [`fixed`, `random`] as const) { + if ( + requestedReplayProperty !== undefined && + (requestedReplayProperty !== seedReloadProperty || campaign === `fixed`) + ) + continue + it(`keeps inserted metadata across peer reloads (${campaign})`, async () => { + const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ] + const states = lanes.flatMap((lane) => [ + `${lane.name}: inner commit`, + `${lane.name}: row`, + `${lane.name}: reload`, + ]) + const completedStates = lanes.flatMap((lane) => [ + `${lane.name}: completed kept bare seed reload`, + `${lane.name}: completed kept non-bare seed reload`, + ]) + const reach = createOracleReach(seedReloadProperty, states) + const completedReach = createOracleReach( + seedReloadProperty, + completedStates, + ) + const result = await fc.check( + fc.asyncProperty(seedReinsertReloadHistoryArbitrary, async (history) => { + const failures: Array = [] + for (const lane of lanes) { reach.beginHistory() - for (const regime of historyRegimes(lane, history)) - if (states.includes(regime)) reach.reach(regime) - return runLane(history) - }, - 240_000, - ) - it(`reaches the declared states in the ${lane.name} fixed campaign`, () => - reach.assertReached()) - fcTest.prop( - [sessionHistoryArbitrary(lane)], - oraclePropertyOptions(50, lanePropertyName), - )( - `keeps the session law on ${lane.name} histories (random)`, - runLane, - 120_000, + completedReach.beginHistory() + const result = await runSessionHistory(lane, history, () => + createPersistedDriver( + true, + true, + false, + true, + lane.fullRows === true, + lane.onDemand === true, + ), + ) + const last = history.at(-1)! + const innerCommit = history.findIndex( + (action) => action.type === `commit`, + ) + if (result.steps > innerCommit) + reach.reach(`${lane.name}: inner commit`) + if (last.type === `peer` && result.steps === history.length) + reach.reach(`${lane.name}: ${last.notice}`) + const completed = completedSeedReinsertReloadRegime(history, result) + if (completed !== undefined) + completedReach.reach(`${lane.name}: ${completed}`) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }), + campaign === `fixed` + ? { numRuns: oracleRuns(40), seed: 527962096 } + : oraclePropertyOptions(40, seedReloadProperty), + ) + recordOracleReplay(seedReloadProperty, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property: seedReloadProperty, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, ) - } else if (requestedReplayProperty === lanePropertyName) { - fcTest.prop( - [sessionHistoryArbitrary(lane)], - oraclePropertyOptions(50, lanePropertyName), - )(`replays ${lane.name} session histories`, runLane, 120_000) + reach.assertReached() + // The bare-key-1 conjunction has probability 1/24: random40 would miss it + // about 18% of the time. Every normal run enforces the fixed40 branch + // guards; the identical random40 campaign measures them without a promise + // of reach. Existing inner/row/reload guards apply to both campaigns. + if (campaign === `fixed`) completedReach.assertReached() + }, 120_000) +} + +const describeNested = oracleReplayTest( + describe, + ...extraLanes.map(({ property }) => property), +) +describeNested( + `sync metadata session oracle through the persisted wrapper: nesting and seeds`, + () => { + for (const { + lane, + property: lanePropertyName, + seed, + runs, + reach: states, + } of extraLanes) { + const runLane = (history: Array) => + assertPersistedHistory(history, lane) + if (requestedReplayProperty === undefined) { + const reach = createOracleReach(`${lanePropertyName} (fixed)`, states, { + minimum: 5, + }) + fcTest.prop([sessionHistoryArbitrary(lane)], { + numRuns: oracleRuns(runs), + seed, + })( + `keeps the session law on ${lane.name} histories (fixed)`, + (history) => { + reach.beginHistory() + for (const regime of historyRegimes(lane, history)) + if (states.includes(regime)) reach.reach(regime) + return runLane(history) + }, + 240_000, + ) + it(`reaches the declared states in the ${lane.name} fixed campaign`, () => + reach.assertReached()) + fcTest.prop( + [sessionHistoryArbitrary(lane)], + oraclePropertyOptions(50, lanePropertyName), + )( + `keeps the session law on ${lane.name} histories (random)`, + runLane, + 120_000, + ) + } else if (requestedReplayProperty === lanePropertyName) { + fcTest.prop( + [sessionHistoryArbitrary(lane)], + oraclePropertyOptions(50, lanePropertyName), + )(`replays ${lane.name} session histories`, runLane, 120_000) + } } - } -}) + }, +) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts new file mode 100644 index 0000000000..cb4edf4e61 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts @@ -0,0 +1,462 @@ +// @vitest-environment node +/** + * The sync metadata session oracle's seed/re-insert/reload law crosses real + * SQLite and coordinator boundaries here. The law, model, legal histories, + * and comparison are imported from sync-metadata-session-oracle.ts. This + * driver supplies two persisted Collections, two production coordinators over + * Node's BroadcastChannel/Web Locks, and one node:sqlite database. Each + * Collection has its own adapter; their shared physical database supplies the + * shared scheduler premise. + * + * A peer's named notice comes from its real source commit. For a full reload, + * that source takes a durable snapshot, changes the requested row, and commits + * a snapshot transaction. This preserves every other durable row and causes + * production to announce requiresFullReload. The receiver records the actual + * notice and waits for SQL/message quiescence before the shared comparison. + * Neither this driver nor the peer invents an expected metadata value. + * + * Limits: one process, one SQLite database, two Collections of one ID, no + * transport loss or leadership turnover, and core holds only. This is a real + * SQLite/coordinator receiving witness, not a Chromium OPFS or Electron claim. + */ +import { DatabaseSync } from 'node:sqlite' +import { fc } from '@fast-check/vitest' +import { it } from 'vitest' +import { DbClient, collectionOptions } from '@tanstack/db' +import { BrowserCollectionCoordinator } from '../../browser-db-sqlite-persistence/src/browser-coordinator' +import { + absentMarker, + cachedMetadata, + cachedRow, + completedSeedReinsertReloadRegime, + describeMismatch, + persistedPeerFullLane, + persistedPeerLane, + persistedPeerOnDemandLane, + runSessionHistory, + seedReinsertReloadHistoryArbitrary, + seedReinsertReloadReplay, + sessionCollectionKeys, +} from '../../db/tests/sync-metadata-session-oracle' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, + readOracleRunConfig, + recordOracleReplay, +} from '../../db/tests/oracle-config' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, + SQLiteCorePersistenceAdapter, + eagerStorageTarget, + managedStorageTarget, + persistedCollectionOptions, +} from '../src' +import type { SQLInputValue } from 'node:sqlite' +import type { SyncConfig } from '@tanstack/db' +import type { + ReceiptState, + SessionAction, + SessionDriver, + SessionLane, +} from '../../db/tests/sync-metadata-session-oracle' +import type { SQLiteDriver } from '../src' + +type Row = { id: number; v: unknown } +type Actions = Parameters[`sync`]>[0] +let run = 0 + +async function createSQLiteDriver(lane: SessionLane): Promise { + const database = new DatabaseSync(`:memory:`) + let activity = 0 + let queue: Promise = Promise.resolve() + const driver: SQLiteDriver = { + exec: async (sql) => { + activity++ + database.exec(sql) + }, + query: async (sql: string, params: ReadonlyArray = []) => { + activity++ + return database + .prepare(sql) + .all(...(params as Array)) as Array + }, + run: async (sql, params = []) => { + activity++ + database.prepare(sql).run(...(params as Array)) + }, + transaction: (work) => { + const next = queue.then(async () => { + database.exec(`BEGIN IMMEDIATE`) + try { + const result = await work(driver) + database.exec(`COMMIT`) + return result + } catch (error) { + database.exec(`ROLLBACK`) + throw error + } + }) + queue = next.catch(() => undefined) + return next + }, + } + Object.defineProperty(driver, SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, { + value: {}, + }) + const adapters = [ + new SQLiteCorePersistenceAdapter({ driver }), + new SQLiteCorePersistenceAdapter({ driver }), + ] + const id = `metadata-items` + const seedClaim = lane.onDemand + ? await adapters[0]!.claimCacheGeneration(id) + : undefined + const target = seedClaim + ? managedStorageTarget(seedClaim.storageCollectionId, seedClaim.claimId) + : eagerStorageTarget(id) + const storageId = seedClaim?.storageCollectionId ?? id + await adapters[0]!.applyCommittedTx(target, { + txId: `cached`, + term: 1, + seq: 1, + rowVersion: 1, + cacheGenerationClaimId: seedClaim?.claimId, + mutations: [{ type: `insert`, key: 1, value: cachedRow }], + rowMetadataMutations: [{ type: `set`, key: 1, value: cachedMetadata }], + }) + const dbName = `sync-metadata-receiving-${process.pid}-${run++}` + const coordinators = adapters.map( + (adapter) => new BrowserCollectionCoordinator({ dbName, adapter }), + ) + const notices: Array<{ reload: boolean }> = [] + let durableRows: ReadonlyArray<{ key: string | number; value: unknown }> = [] + const stopNotices = coordinators[0]!.subscribe(storageId, (message) => { + activity++ + const payload = message.payload as { + type: string + requiresFullReload?: boolean + } + if (payload.type === `tx:committed`) + notices.push({ reload: payload.requiresFullReload === true }) + }) + const quiesce = async () => { + let quiet = 0 + for (let turn = 0; turn < 400 && quiet < 5; turn++) { + const before = activity + await new Promise((resolve) => setTimeout(resolve, 2)) + quiet = before === activity ? quiet + 1 : 0 + } + if (quiet < 5) throw new Error(`SQLite metadata hosts did not quiesce`) + durableRows = await adapters[0]!.loadSubset(target, {}) + } + const actions: Array = [] + let resolveMutation: (() => void) | undefined + const clients = [new DbClient(), new DbClient()] + const collections = clients.map((client, index) => + client.collection( + collectionOptions( + persistedCollectionOptions({ + id, + getKey: (row) => row.id, + ...(lane.onDemand ? { syncMode: `on-demand` as const } : {}), + sync: { + rowUpdateMode: lane.fullRows ? `full` : `partial`, + sync: (sync) => { + actions[index] = sync + sync.begin() + sync.commit() + sync.markReady() + return lane.onDemand ? { loadSubset: () => true } : undefined + }, + }, + onInsert: () => + new Promise((resolve) => { + resolveMutation = resolve + }), + persistence: { + adapter: adapters[index]!, + coordinator: coordinators[index]!, + }, + }), + ), + ), + ) + // These direct source histories have no live query to start on-demand sync. + // Start both sources before a follower can route its first demand to its peer. + for (const current of collections) current.startSyncImmediate() + if (!lane.onDemand) + await Promise.all(collections.map((collection) => collection.preload())) + const demands = collections.map(() => ({})) + if (lane.onDemand) + await Promise.all( + collections.map((collection, index) => + collection._sync.loadSubset(demands[index]!), + ), + ) + await quiesce() + const sync = actions[0]! + const metadata = sync.metadata! + const collection = collections[0]! + const receipts: Array> = [] + const receiptStates: Array = [] + const rejections: Array = [] + let depth = 0 + let holdCount = 0 + let blocker: { when: (state: 'settled') => Promise } | undefined + const release = async () => { + resolveMutation?.() + resolveMutation = undefined + await blocker?.when(`settled`) + blocker = undefined + } + const abort = () => { + const controller = new AbortController() + controller.abort() + const receipt = sync.commit(controller.signal) + depth-- + if (receipt !== true) + receipts.push( + receipt.catch((error: unknown) => { + if (!(error instanceof Error) || error.name !== `AbortError`) + rejections.push(String(error)) + }), + ) + } + const read = (value: unknown) => (value === undefined ? absentMarker : value) + const sorted = (entries: ReadonlyArray<{ key: string; value: unknown }>) => + entries + .map(({ key, value }): [string, unknown] => [key, value]) + .sort(([a], [b]) => a.localeCompare(b)) + const rowMetadata = (value: unknown) => + value === undefined ? {} : { metadata: value as Record } + return { + consumerCheckpoint: `final`, + begin: () => { + sync.begin() + depth++ + }, + commit: () => { + const index = receiptStates.length + const receipt = sync.commit() + depth-- + receiptStates.push(receipt === true ? `applied` : `pending`) + if (receipt !== true) + receipts.push( + receipt.then( + () => { + receiptStates[index] = `applied` + }, + () => { + receiptStates[index] = `rejected` + }, + ), + ) + }, + cancel: abort, + rowSet: (key, value) => metadata.row.set(key, value), + rowDelete: (key) => metadata.row.delete(key), + insert: (_key, value, meta) => + sync.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }), + update: (_key, value, meta) => + sync.write({ type: `update`, value: value as Row, ...rowMetadata(meta) }), + delete: (key) => sync.write({ type: `delete`, key }), + truncate: () => sync.truncate(), + collectionSet: (key, value) => metadata.collection.set(key, value), + collectionDelete: (key) => metadata.collection.delete(key), + hold: (store, readHold) => { + if (store || readHold) + throw new Error(`the SQLite seed/re-insert grammar uses core holds`) + blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) + }, + release, + seed: (key, value, meta) => + clients[0]!.hydrate({ + collections: [ + { + collectionId: id, + rows: [ + { + key, + value: value as Row, + metadata: meta as Record, + }, + ], + }, + ], + }), + peer: async (key, notice, value, meta) => { + await quiesce() + const before = notices.length + const peer = actions[1]! + if (notice === `reload`) { + const rows = await adapters[1]!.loadSubset(target, {}) + peer.begin({ snapshot: true }) + for (const row of rows) + if (row.key !== key) + peer.write({ + type: `insert`, + value: row.value as Row, + ...rowMetadata(row.metadata), + }) + } else if (notice === `row`) peer.begin() + else + throw new Error( + `unexpected peer notice in the seed/re-insert grammar: ${notice}`, + ) + peer.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }) + await peer.commit() + await quiesce() + if ( + !notices + .slice(before) + .some((entry) => entry.reload === (notice === `reload`)) + ) + throw new Error( + `the real coordinator did not deliver the ${notice} notice`, + ) + console.info( + `SQLITE_METADATA_NOTICE ${JSON.stringify({ lane: lane.name, notice, received: notices.length - before })}`, + ) + }, + observe: () => ({ + rowMetadata: { + 1: read(metadata.row.get(1)), + 2: read(metadata.row.get(2)), + }, + collection: Object.fromEntries( + sessionCollectionKeys.map((key) => [ + key, + read(metadata.collection.get(key)), + ]), + ), + list: sorted(metadata.collection.list()), + prefixList: sorted(metadata.collection.list(`gc:`)), + base: { + 1: read(collection.base.get(1)), + 2: read(collection.base.get(2)), + }, + receipts: [...receiptStates], + }), + storedRows: () => + Object.fromEntries( + [1, 2].map((key) => [ + key, + read(durableRows.find((row) => row.key === key)?.value), + ]), + ), + flush: quiesce, + settle: async () => { + await release() + while (depth) abort() + await quiesce() + await Promise.all(receipts) + await quiesce() + return rejections + }, + cleanup: async () => { + await release() + while (depth) abort() + await Promise.all(receipts) + if (lane.onDemand) + for (const [index, current] of collections.entries()) + current._sync.unloadSubset(demands[index]!) + await quiesce() + await Promise.all(collections.map((current) => current.cleanup())) + stopNotices() + for (const coordinator of coordinators) coordinator.dispose() + if (seedClaim) + await adapters[0]!.releaseCacheGenerationClaim(seedClaim.claimId) + await queue + database.close() + }, + } +} + +const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, +] +async function check(history: ReadonlyArray, lane: SessionLane) { + return runSessionHistory(lane, history, () => createSQLiteDriver(lane)) +} +oracleReplayTest(it).each(lanes)( + `receives the seed/re-insert/reload replay through SQLite ($name)`, + async (lane) => { + const result = await check(seedReinsertReloadReplay, lane) + if (result.mismatch) + throw new Error( + describeMismatch(seedReinsertReloadReplay, result.mismatch), + ) + }, +) + +const property = `sync-metadata-session.sqlite-seed-reinsert-reload` +const requested = readOracleRunConfig().replayProperty +for (const campaign of [`fixed`, `random`] as const) { + if ( + requested !== undefined && + (requested !== property || campaign === `fixed`) + ) + continue + it(`receives generated seed/re-insert/reload histories through SQLite (${campaign})`, async () => { + const states = lanes.flatMap((lane) => [ + `${lane.name}: inner commit`, + `${lane.name}: row`, + `${lane.name}: reload`, + ]) + const completedStates = lanes.flatMap((lane) => [ + `${lane.name}: completed kept bare seed reload`, + `${lane.name}: completed kept non-bare seed reload`, + ]) + const reach = createOracleReach(property, states) + const completedReach = createOracleReach(property, completedStates) + const result = await fc.check( + fc.asyncProperty(seedReinsertReloadHistoryArbitrary, async (history) => { + const failures: Array = [] + for (const lane of lanes) { + reach.beginHistory() + completedReach.beginHistory() + const result = await check(history, lane) + const last = history.at(-1)! + const innerCommit = history.findIndex( + (action) => action.type === `commit`, + ) + if (result.steps > innerCommit) + reach.reach(`${lane.name}: inner commit`) + if (last.type === `peer` && result.steps === history.length) + reach.reach(`${lane.name}: ${last.notice}`) + // The peer action has asserted actual requiresFullReload delivery + // before the shared comparison passes. + const completed = completedSeedReinsertReloadRegime(history, result) + if (completed !== undefined) + completedReach.reach(`${lane.name}: ${completed}`) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }), + campaign === `fixed` + ? { numRuns: oracleRuns(40), seed: 527962096 } + : oraclePropertyOptions(40, property), + ) + recordOracleReplay(property, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() + // The fixed40 campaign enforces this scarce bare-key-1 premise on every + // normal run. Its probability is 1/24, so random40 would miss it about + // 18% of the time; that identical campaign reports measured branch reach. + // Existing inner/row/reload guards remain enforced in both campaigns. + if (campaign === `fixed`) completedReach.assertReached() + }, 120_000) +} diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index ca9c1a8927..952349fa99 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -238,6 +238,8 @@ const staticOracleProperties = [ `sync-metadata-session.persisted-peers`, `sync-metadata-session.persisted-peers-full`, `sync-metadata-session.persisted-peers-on-demand`, + `sync-metadata-session.persisted-seed-reinsert-reload`, + `sync-metadata-session.sqlite-seed-reinsert-reload`, `sync-metadata-session.nested-conflicts.hold-invariance`, `sync-metadata-session.nested-conflicts.key-independence`, `d2-source.exact-retractions`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 5de3ae76b1..fc31946410 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -636,7 +636,12 @@ const ownerGroups: ReadonlyArray< [ `db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts`, `sync-metadata-session`, - `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand`, + `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand persisted-seed-reinsert-reload`, + ], + [ + `db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts`, + `sync-metadata-session`, + `sqlite-seed-reinsert-reload`, ], ] diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index 6e7772325a..a942262634 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -582,6 +582,109 @@ export function sessionHistoryArbitrary( }) } +/** + * A source insert can be visible to an inner transaction before it commits. + * The inner equal re-insert keeps metadata; its durable replacement must keep + * the metadata of the accepted row, including a later seed, rather than the + * older cache's metadata. A peer notice then reads that durable replacement. + * + * This constructive subgrammar rebuilds the #2281 metadata prefix and varies + * both keys, seed shape/metadata and arrival, explicit metadata, repeated + * re-inserts, a core hold, the enclosing transaction's ending, and named + * versus full-reload notices. Every action is legal in the peer lanes. Store + * and read holds are excluded because their pending work blocks peer notices. + * The general session grammar continues to cover their independent laws. + */ +export const seedReinsertReloadHistoryArbitrary: fc.Arbitrary< + Array +> = fc + .record({ + key: fc.constantFrom(...sessionRowKeys), + bare: fc.boolean(), + narrow: fc.boolean(), + seedInside: fc.boolean(), + outerMetadata: fc.boolean(), + innerMetadata: fc.boolean(), + repeats: fc.integer({ min: 1, max: 3 }), + hold: fc.boolean(), + ending: fc.constantFrom(`open`, `commit`, `cancel`), + notice: fc.constantFrom(`row` as const, `reload` as const), + }) + .map((input): Array => { + const seed: SessionAction = { + type: `seed`, + key: input.key, + bare: input.bare, + narrow: input.narrow, + } + return [ + ...(input.hold ? [{ type: `hold` as const }] : []), + ...(input.seedInside + ? [{ type: `begin` as const }, seed] + : [seed, { type: `begin` as const }]), + { type: `insertOverSeed`, key: input.key, metadata: input.outerMetadata }, + { type: `begin` }, + ...Array.from({ length: input.repeats }, (): SessionAction => ({ + type: `reinsert`, + key: input.key, + metadata: input.innerMetadata, + })), + { type: `commit` }, + ...(input.ending === `open` + ? [] + : [{ type: input.ending } as SessionAction]), + { type: `peer`, key: input.key === 1 ? 2 : 1, notice: input.notice }, + ] + }) + +/** The source report is input to the same model and comparison as generation. */ +export const seedReinsertReloadReplay: Array = [ + { type: `begin` }, + { type: `seed`, key: 1, narrow: false, bare: true }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + { type: `reinsert`, key: 1, metadata: false }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `reload` }, +] + +/** + * A completed implicit re-insert must keep the later seed across a full reload + * while the enclosing write remains unaccepted. This reach classifier judges + * only the grammar and completed comparison, never production metadata values. + * Only key 1 starts with cached metadata, so only its bare seed distinguishes + * a stale cache value from absence. Both receiving drivers use these premises. + */ +export function completedSeedReinsertReloadRegime( + history: ReadonlyArray, + result: { mismatch: SessionMismatch | undefined; steps: number }, +): + | `completed kept bare seed reload` + | `completed kept non-bare seed reload` + | undefined { + const last = history.at(-1)! + const seed = history.find((action) => action.type === `seed`) + const keptSeedReload = + last.type === `peer` && + last.notice === `reload` && + history.some((action) => action.type === `reinsert`) && + history.every( + (action) => action.type !== `reinsert` || action.metadata === false, + ) && + history.filter((action) => action.type === `commit`).length === 1 + if ( + !result.mismatch && + result.steps === history.length && + keptSeedReload && + seed?.type === `seed` + ) { + if (seed.bare === true && seed.key === 1) + return `completed kept bare seed reload` + if (seed.bare === false) return `completed kept non-bare seed reload` + } + return undefined +} + /** * The regimes that a legal history reaches, read from the model as it * replays. The campaign reach test requires each law's premise. @@ -2525,5 +2628,11 @@ declareOracle({ runSeedOrderInvariance, ], model: [SessionModel, createSessionModel, modelCheckpoints], - grammar: [sessionActionArbitrary, sessionHistoryArbitrary, historyRegimes], + grammar: [ + sessionActionArbitrary, + sessionHistoryArbitrary, + historyRegimes, + seedReinsertReloadHistoryArbitrary, + completedSeedReinsertReloadRegime, + ], }) From ea769ec8d3b4c1bffc286e1d88b1b0c553c5cc5f Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 14:25:30 -0600 Subject: [PATCH 2/2] Document implicit re-insert metadata repair release --- .changeset/preserve-implicit-reinsert-metadata.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changeset/preserve-implicit-reinsert-metadata.md diff --git a/.changeset/preserve-implicit-reinsert-metadata.md b/.changeset/preserve-implicit-reinsert-metadata.md new file mode 100644 index 0000000000..bf6bf1192b --- /dev/null +++ b/.changeset/preserve-implicit-reinsert-metadata.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +--- + +Preserve accepted row metadata when an equal source re-insert omits an explicit metadata write. Prevent peer reloads from restoring older cached metadata after that insert.