diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 1db29afe24..310e16a9b5 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3787,6 +3787,66 @@ does not generate a hook input without `cwd`, a worktree outside the launch checkout, or a worktree branch that has no copy of the script. In that last case the launch checkout's rule lints the worktree. +`packages/db/tests/declared-reach-oracle.property.test.ts` +(`oracle.declared-reach`; [review record](oracle-reviews/pr-2290-declared-reach.md)) +owns what `createOracleReach` (ORC-015) counts, when +it checks reach, and what fixed examples prove. It also owns the run +parameters that `oracleRandomParameters` and `oraclePropertyOptions` return, +and the campaigns that `runOracleCampaign` in +`packages/db/tests/oracle-campaign.ts` runs. + +- A history counts once per declared state it enters. A `reach()` call for an + undeclared state, or before the first history, throws and counts nothing. + Options that cannot describe a reach (a minimum below one, a fractional or + negative number of fixed histories, a state declared twice) throw. +- The check runs in an ordinary run and in a run with only + `TANSTACK_DB_ORACLE_SEED`, which reseeds the whole campaign at its full + budget. A path replay, which runs one recorded history, skips it in every + reach of its process: a reach need not carry its property's name, so it + cannot tell whose replay runs. An explicit `replaying` option overrides the + environment. +- A property can run fixed examples first in every run, through + `oraclePropertyOptions(…, { examples })`, on top of the base run budget. It + declares their number as `fixedHistories` and begins each example's history + with `beginHistory({ fixed: true })`, which it can do because fast-check + passes an example's input by reference. The check then also fails for each + declared state that no fixed history entered. Fixedness belongs to the + history, so another test may share the reach. Shrinking can run an example + again, so an extra fixed history counts as fixed. When fewer fixed histories + ran than were declared, the check reports only that. +- A replay keeps its seed, and a path replay of the property keeps its path + and runs one history. `runOracleCampaign` runs a fixed-seed and a random + campaign without a prefix seed, one replay campaign with one, passes each + its parameters, and reports each passing campaign on one `ORACLE_REACH` + line. An invalid prefix environment throws. + +The `source-health.on-demand` and `source-health.lazy-demand` properties run +each reach prefix, and each driver option that a declared state needs, as +examples, so their reach does not depend on the seed (#2287). +`source-health.history` gets the same guarantee from +`source-health.required-history`, which runs every reach prefix in each +campaign and shares its reach without fixed histories. The oracle generates +ledgers over two to four states, up to six histories with fixed marks, +minimums up to three, up to four declared fixed histories, invalid options, +undeclared and early reaches, an optional `replaying` option, a fixed campaign +seed, and invalid campaign environments, under four run kinds: ordinary, +seed-only, and path replays of the property and of another property. + +Limits. A declared state of another property that only random histories +reach can still miss by chance at the normal budget. Those properties have not +been measured, about twenty set fast-check's `examples` inside their base runs +instead of through the option, and a `runOracleCampaign` reach cannot declare +fixed histories. Destination: #2289. Needed witness: per property, the declared +states that no fixed example enters, and how often a normal-budget run misses +each one. A path replay also skips reach in a property that the replay does not +name and that still runs its seeded budget. When a property fails after its +examples, the check still reports the states that the stopped run missed. The +oracle does not judge reporter keys, which only a replay witness channel adds, +the rejection of an unregistered property, or the compile-time rule that a call +without examples returns parameters without an `examples` key; type-aware +lint in every package that infers from these parameters is that rule's only +witness. + `packages/db/tests/oracle-replay.test.ts` checks every replay manifest owner: from its cwd, the command's files exist, its target is the owner, and a Vitest config applies. diff --git a/docs/contributing/oracle-reviews/pr-2290-declared-reach.md b/docs/contributing/oracle-reviews/pr-2290-declared-reach.md new file mode 100644 index 0000000000..d0c568f74d --- /dev/null +++ b/docs/contributing/oracle-reviews/pr-2290-declared-reach.md @@ -0,0 +1,139 @@ +# Declared reach machinery (PR #2290) + +This record applies to the executable oracle at commit `590588a65`. It is +append-only: a later change adds an entry with its own commit and does not +edit an earlier one. It records the evidence behind the +`oracle.declared-reach` owner in the coverage map's "Oracle tooling" section. + +## Entry 1: commit `590588a65` + +### Scope and laws + +The oracle owns these laws of `createOracleReach`, `runOracleCampaign`, +`oracleRandomParameters`, and `oraclePropertyOptions`. Authority is ORC-015, +ORC-007, issue #2287, and maintainer decisions in the PR #2290 session. + +| Law | Authority | Generated witness | +| --- | --- | --- | +| A history counts once per declared state it enters | ORC-015 | repeats within a history | +| A reach call for an undeclared state, or before the first history, throws and counts nothing | maintainer decision | undeclared index, early reaches | +| Invalid options throw at creation | maintainer decision | minimums 0, -1, and 1.5. Fixed counts -1 and 0.5. A state declared twice | +| The check fails below the minimum | ORC-015 | minimums 1 to 3 | +| A fixed history is one begun with `fixed: true`. The check fails for a state that no fixed history entered. An extra fixed history counts as fixed. Too few fixed histories report only the stop | #2287, maintainer decision | fixed marks per history, an unfixed history first, extra and missing fixed histories | +| A seed-only run checks reach, any path replay in the process skips it, an explicit `replaying` overrides the environment | maintainer decisions | four run kinds, optional override | +| Examples come on top of the base runs, a replay keeps its seed, a path replay of the property keeps its path and runs once | #2287, ORC-007 | run kinds, base runs 1 to 200, up to five examples, the process multiplier | +| `runOracleCampaign` runs fixed-seed and random campaigns, or one replay campaign, passes each its parameters, reports each passing campaign, and rejects an invalid environment | ORC-007, ORC-015 | run kinds, a fixed seed, three invalid environments | + +The law discovery behind this table, with the receiving-path inventory and +tension scan, found that the earlier oracle stated only the fixed-example law. +Reviews then found the other laws one round at a time. + +### Experiments + +RED results. Each was run on the parent of its production change with the +extended oracle: + +- Option validation: the oracle failed at fixed ledger 9 (`minimum: 0`), where + the reach did not throw. +- Fixed marks: with validation in place, the parent's reach failed at fixed + ledger 12, where the model of that draft expected a throw. With the final + model, a mutant that restores the parent's positional counting fails at + fixed ledger 13, an unfixed history before the fixed one. +- An extra fixed history: an intermediate draft threw for it. The oracle + failed with that throw, and a mutant hunt showed that the throw replaced the + original assertion failure while fast-check shrank an example. The model + now credits the extra history, and production no longer throws. +- Early reach, the stopped-run report, and seed-only checks in + `createOracleReach` and `runOracleCampaign` failed on their parents in the + same way, in earlier commits of this PR. + +Limit on order: the first commit of this PR changed the seed-only rule and +added `fixedHistories` before this oracle existed. Their RED results were run +afterward against that commit's parent. They are not a pre-change gate. + +GREEN result at `590588a65`, from `packages/db`: + +```sh +pnpm exec vitest run --maxWorkers=2 --coverage.enabled=false \ + tests/declared-reach-oracle.property.test.ts \ + tests/source-health-oracle.test.ts tests/oracle-reach.test.ts \ + tests/oracle-campaign.test.ts tests/mutation-handler-types-oracle.test.ts +``` + +All 556 tests pass with no type errors. The declared-reach property runs 200 +generated ledgers and 14 fixed ledgers. The 14 fixed ledgers enter its 14 +declared reach states in every run. + +Mutants. Each changed one decision in the machinery, ran the declared-reach +oracle, and was restored. All 23 failed at an assertion of the intended +comparison (`reach verdict`, `campaign observation`, `run parameters`, or +`property options under this process`): + +- Fixed counting by position, extra fixed histories not credited. +- Accepting minimum 0, a fractional fixed count, or a duplicate state. +- An ignored campaign fixed seed, a replay campaign without its seed, a path + replay campaign without `numRuns: 1`, a wrong `ORACLE_REACH` minimum, + accepting a non-numeric campaign path. +- An ignored `replaying` override, accepting an undeclared state, counting an + early reach. +- Skipping reach only for the named replay property, skipping reach in a + seed-only run, in `createOracleReach` or in `runOracleCampaign`, checking + reach in a campaign path replay. +- Dropping `examples` in `oraclePropertyOptions`, dropping the replay seed and + path with examples, letting examples displace random runs. +- Naming states after a stopped run, removing the fixed-history check, + counting a state twice in one history. + +A first draft of the extra-fixed mutant capped a counter that is only compared +with the declared count. It survived, and it is equivalent by that invariant. +A separate run with `TANSTACK_DB_ORACLE_RUNS_MULTIPLIER=2` rejected a mutant +that ignores the multiplier. + +Receiving paths. With the new rules, the complete suites of `db` (385 files, +24,216 tests), `db-sqlite-persistence-core`, `offline-transactions`, +`electric-db-collection`, `svelte-db`, `convex-db-collection`, `preact-db`, +and `db-ivm` pass. One `db` file, `mutation-handler-types-oracle.test.ts`, +declared each state six times on purpose and threw. It now declares each +once, and its counts are unchanged. `react-db` passes every test, with type +errors from unbuilt packages in this checkout. Two `query-db-collection` files +and `browser-db-sqlite-persistence` need native SQLite bindings that this +checkout lacks, so only CI runs them. + +### Rubric outcomes + +- ORC-001, ORC-003: the oracle's opening prose states each law, its authority, + and its limits. The model, grammar, drivers, and checks have their own + comments. +- ORC-002: the model recomputes counts and failures from the ledger and reads + the environment on its own. It imports no production helper. +- ORC-004: the grammar reconstructs every known witness above as a fixed + ledger. Its ranges and invalid values are listed in the law table. +- ORC-005, ORC-006: the drivers call the real entry points and compare whole + observations. The mutants above classify as assertion failures. +- ORC-007: the property runs through `oraclePropertyOptions` with fixed + ledgers, and replays through `oracle.declared-reach` in the replay manifest. +- ORC-008: the model keeps a fixed mark per history. The unfixed-first ledger + shows that position alone cannot replace it. +- ORC-009: "ledger" and "run kind" are model-only terms. The oracle maps them + to histories and replay environments. +- ORC-010: the extra-fixed rule exists so that a shrinking rerun cannot + replace the original failure. +- ORC-011: no shared-fault hypothesis needs a second formulation. +- ORC-013: each conditional law has a premise case and a neighbor among the + fixed ledgers (for example, a fixed miss that a later unfixed history enters). +- ORC-014: not applicable. No provider premise. +- ORC-015: 14 declared states, each entered in every run. +- ORC-016: the oracle file has one generated property and no unit tests. + +### Open and limits + +- Other reach properties are not measured for chance misses, about twenty set + examples inside their base runs, and a campaign reach cannot declare fixed + histories. Owner: #2289. +- A path replay also skips reach in a property that the replay does not name. +- After a late property failure, the check still reports missed states. +- Not judged: reporter keys, the rejection of an unregistered property, and + the compile-time rule that a call without examples returns no `examples` + key. Type-aware lint is that rule's only witness. +- Before the merge with main, one local batch failed 1 of 512 tests and its log + was lost. It did not recur in 12 reruns. Its cause is unknown. diff --git a/packages/db/tests/declared-reach-oracle.property.test.ts b/packages/db/tests/declared-reach-oracle.property.test.ts new file mode 100644 index 0000000000..939195b898 --- /dev/null +++ b/packages/db/tests/declared-reach-oracle.property.test.ts @@ -0,0 +1,745 @@ +/** + * Declared reach (ORC-015) cannot miss by chance where a property fixes it. + * + * A generated property declares the states its laws depend on, and + * `createOracleReach` fails a run in which too few histories entered one. + * These rules decide what a reach counts, when its check runs, and what the + * check proves (#2287): + * + * - A history counts once per declared state it enters. A `reach()` call for + * an undeclared state, or before the first `beginHistory()`, is a driver + * error: it throws and counts nothing. Options that cannot describe a reach + * (a minimum below one, a fractional or negative number of fixed + * histories, a state declared twice) throw when the reach is created. + * - The check fails for each state that fewer than `minimum` histories + * entered. It runs in an ordinary run and in a run with only + * `TANSTACK_DB_ORACLE_SEED`: a seed alone reseeds the whole campaign at its + * full budget. A path replay, which runs one recorded history, skips it in + * every reach of that process, also in a property that the replay does not + * name: a reach need not carry its property's name, so it cannot tell whose + * replay runs. An explicit `replaying` option overrides the environment. + * - A property can run fixed examples first in every run. It declares their + * number as `fixedHistories: n` and begins each example's history with + * `beginHistory({ fixed: true })`. The check then also fails for each state + * that no fixed history entered: such a state's reach would depend on the + * seed. Fixedness belongs to the history, not to its position, so another + * test may share the reach. Shrinking can run an example again, so a fixed + * history beyond the n declared ones counts as a fixed history too. When + * fewer than n fixed histories ran, a failure stopped the run inside the + * examples, and the check reports only that. + * - The examples come on top of the base run budget, so they never displace + * a random history. A replay keeps its seed, and a path replay of the + * property keeps its path and still runs one history. + * + * `runOracleCampaign` runs a fixed-seed and a random campaign, each with its + * own reach and a minimum of one. A seed-only run of its replay prefix runs one + * campaign with that seed and checks its reach; a seed and path run one + * recorded history and check none. Each campaign that passes reports its + * counts on one `ORACLE_REACH` line. An invalid prefix environment throws. + * + * The model below states these rules over a generated ledger: a run kind, an + * optional `replaying` override, the options, the states reached before the + * first history, each history with its fixed mark and the states it enters, + * the campaign runner's fixed seed, and an optional invalid campaign + * environment. The drivers run `createOracleReach`, `runOracleCampaign`, and + * `oracleRandomParameters` under that run kind's environment, and + * `oraclePropertyOptions` under the process's own. Each check compares one + * whole observation with the model. + */ +import { fc, test as fcTest } from '@fast-check/vitest' +import { afterAll, expect, vi } from 'vitest' +import { createOracleReach } from './oracle-reach.js' +import { runOracleCampaign } from './oracle-campaign.js' +import { + oraclePropertyOptions, + oracleRandomParameters, +} from './oracle-config.js' +import { declareOracle } from './oracle-declaration.js' + +const property = `oracle.declared-reach` + +// `other-path` is a path replay of another property, in the same process. +type RunKind = `ordinary` | `seed` | `path` | `other-path` +const otherProperty = `lint.part-reach` +type CampaignFault = `bad-seed` | `path-without-seed` | `bad-path` +type LedgerHistory = { + fixed: boolean + // The indices of the states it enters, with repeats. The index equal to + // the number of declared states names an undeclared one. + states: ReadonlyArray +} +type Ledger = { + run: RunKind + replaying: boolean | undefined + states: ReadonlyArray + minimum: number + fixedHistories: number + // The indices of the states reached before the first history. + early: ReadonlyArray + histories: ReadonlyArray + fixedSeed: number + campaignFault: CampaignFault | undefined +} +// The states a failing reach check names, and an absent failure when the check +// passes or does not run. +type Failure = { + belowMinimum: ReadonlyMap + unfixed: ReadonlyArray + // How many fixed histories ran, when fewer than the declared ones did. + stoppedAt?: number +} +type Rejection = readonly [string, `undeclared` | `early`] +type Verdict = { + // Whether creating the reach threw. + invalid: boolean + // The reach calls that threw, in call order. + rejected: ReadonlyArray + counts: ReadonlyMap + failure: Failure | undefined +} + +const stateName = (index: number) => `state ${index}` + +// The grammar: two to four states, and up to six histories over them. About +// one ledger in twenty has each kind of invalid option, a quarter reach a +// state or two before any history, and about one reach in ten names an +// undeclared state. +const ledgerArbitrary: fc.Arbitrary = fc + .integer({ min: 2, max: 4 }) + .chain((stateCount) => { + const names = Array.from({ length: stateCount }, (_, index) => + stateName(index), + ) + const reachIndex = fc.oneof( + { arbitrary: fc.integer({ min: 0, max: stateCount - 1 }), weight: 9 }, + { arbitrary: fc.constant(stateCount), weight: 1 }, + ) + const mostly = (valid: fc.Arbitrary, invalid: fc.Arbitrary) => + fc.oneof( + { arbitrary: valid, weight: 19 }, + { arbitrary: invalid, weight: 1 }, + ) + return fc.record({ + run: fc.constantFrom(`ordinary`, `seed`, `path`, `other-path`), + replaying: fc.oneof( + { arbitrary: fc.constant(undefined), weight: 3 }, + { arbitrary: fc.boolean(), weight: 1 }, + ), + states: mostly(fc.constant(names), fc.constant([...names, names[0]!])), + minimum: mostly( + fc.integer({ min: 1, max: 3 }), + fc.constantFrom(0, -1, 1.5), + ), + fixedHistories: mostly( + fc.integer({ min: 0, max: 4 }), + fc.constantFrom(-1, 0.5), + ), + early: fc.oneof( + { arbitrary: fc.constant([]), weight: 3 }, + { + arbitrary: fc.array(reachIndex, { minLength: 1, maxLength: 2 }), + weight: 1, + }, + ), + histories: fc.array( + fc.record({ + fixed: fc.boolean(), + states: fc.array(reachIndex, { maxLength: 4 }), + }), + { maxLength: 6 }, + ), + fixedSeed: fc.integer(), + campaignFault: mostly( + fc.constant(undefined), + fc.constantFrom( + `bad-seed`, + `path-without-seed`, + `bad-path`, + ), + ), + }) + }) + +// The model. Invalid options throw before any history. A reach call for an +// undeclared state, or before the first history, throws and counts nothing. +// Every fixed history counts as fixed, also one beyond the declared number, +// which shrinking runs again. A history counts once per declared state it +// enters. +// A path replay of any property checks nothing, unless an explicit +// `replaying: false` says otherwise. Otherwise a state fails below the +// minimum, and, when fixed histories are declared, a state that no fixed +// history entered fails too. When fewer fixed histories ran than were +// declared, a failure stopped the run, and the check names how many ran and +// no state. +function expectedVerdict(ledger: Ledger): Verdict { + const valid = + Number.isInteger(ledger.minimum) && + ledger.minimum >= 1 && + Number.isInteger(ledger.fixedHistories) && + ledger.fixedHistories >= 0 && + new Set(ledger.states).size === ledger.states.length + if (!valid) { + return { + invalid: true, + rejected: [], + counts: new Map(), + failure: undefined, + } + } + const declared = new Set(ledger.states) + const rejected: Array = ledger.early.map((index) => [ + stateName(index), + declared.has(stateName(index)) ? `early` : `undeclared`, + ]) + let fixedBegun = 0 + const begun: Array<{ fixed: boolean; entered: Set }> = [] + for (const history of ledger.histories) { + if (history.fixed) fixedBegun += 1 + const entered = new Set() + for (const index of history.states) { + if (declared.has(stateName(index))) entered.add(stateName(index)) + else rejected.push([stateName(index), `undeclared`]) + } + begun.push({ fixed: history.fixed, entered }) + } + const counts = new Map( + ledger.states.map((state) => [ + state, + begun.filter(({ entered }) => entered.has(state)).length, + ]), + ) + const stopped = fixedBegun < ledger.fixedHistories + const belowMinimum = new Map( + [...counts].filter(([, count]) => count < ledger.minimum), + ) + const unfixed = + ledger.fixedHistories === 0 + ? [] + : ledger.states.filter((state) => + begun.every(({ fixed, entered }) => !fixed || !entered.has(state)), + ) + const failure: Failure | undefined = stopped + ? { belowMinimum: new Map(), unfixed: [], stoppedAt: fixedBegun } + : belowMinimum.size > 0 || unfixed.length > 0 + ? { belowMinimum, unfixed } + : undefined + const checked = + ledger.replaying === undefined + ? ledger.run === `ordinary` || ledger.run === `seed` + : !ledger.replaying + return { + invalid: false, + rejected, + counts, + failure: checked ? failure : undefined, + } +} + +// The environment each run kind sets. +function runEnvironment(run: RunKind): Record { + if (run === `ordinary`) return {} + if (run === `seed`) return { TANSTACK_DB_ORACLE_SEED: `1` } + return { + TANSTACK_DB_ORACLE_SEED: `1`, + TANSTACK_DB_ORACLE_PATH: `0`, + TANSTACK_DB_ORACLE_PROPERTY: run === `path` ? property : otherProperty, + } +} + +// Runs `run` with the given variables set and the other named ones unset, and +// restores the environment afterward. +async function withEnvironment( + names: ReadonlyArray, + variables: Record, + run: () => T | Promise, +): Promise { + for (const name of names) vi.stubEnv(name, variables[name]) + try { + return await run() + } finally { + vi.unstubAllEnvs() + } +} +const replayNames = [ + `TANSTACK_DB_ORACLE_SEED`, + `TANSTACK_DB_ORACLE_PATH`, + `TANSTACK_DB_ORACLE_PROPERTY`, +] + +// The driver: one ledger through createOracleReach. +async function observeLedger(ledger: Ledger): Promise { + let reach: ReturnType> + try { + reach = await withEnvironment(replayNames, runEnvironment(ledger.run), () => + createOracleReach(`ledger`, ledger.states, { + minimum: ledger.minimum, + fixedHistories: ledger.fixedHistories, + ...(ledger.replaying === undefined + ? {} + : { replaying: ledger.replaying }), + }), + ) + } catch { + return { + invalid: true, + rejected: [], + counts: new Map(), + failure: undefined, + } + } + const rejected: Array = [] + const attempt = (state: string) => { + try { + reach.reach(state) + } catch (error) { + const message = (error as Error).message + rejected.push([ + state, + /undeclared reach state/.test(message) + ? `undeclared` + : /before the first beginHistory/.test(message) + ? `early` + : (message as `early`), + ]) + } + } + for (const index of ledger.early) attempt(stateName(index)) + for (const history of ledger.histories) { + reach.beginHistory({ fixed: history.fixed }) + for (const index of history.states) attempt(stateName(index)) + } + let failure: string | undefined + try { + reach.assertReached() + } catch (error) { + failure = (error as Error).message + } + return { + invalid: false, + rejected, + counts: new Map(reach.counts()), + failure: namedStates(failure), + } +} + +// The states a failure names: `name (count)` below the minimum, and a list of +// names for the fixed histories, or how many fixed histories ran. +function namedStates(failure: string | undefined): Failure | undefined { + if (failure === undefined) return undefined + const below = /not enforced there: (.*)$/m.exec(failure)?.[1] + const unfixed = /reach depends on the seed: (.*)$/m.exec(failure)?.[1] + const stoppedAt = /only (\d+) of its \d+ fixed histories ran/.exec( + failure, + )?.[1] + return { + ...(stoppedAt === undefined ? {} : { stoppedAt: Number(stoppedAt) }), + belowMinimum: new Map( + (below?.split(`, `) ?? []).map((entry) => { + const match = /^(.*) \((\d+)\)$/.exec(entry)! + return [match[1]!, Number(match[2])] as const + }), + ), + unfixed: unfixed?.split(`, `) ?? [], + } +} + +// The check: the observed ledger refines the model. +async function checkLedger(ledger: Ledger) { + const expected = expectedVerdict(ledger) + expect(await observeLedger(ledger), `reach verdict`).toEqual(expected) + return expected +} + +// The campaign runner's replay prefix, and the environment each run kind or +// invalid environment sets for it. The runner reads only its own prefix, so a +// replay of another property is an ordinary run for it. +const campaignPrefix = `TANSTACK_DB_REACH_LEDGER` +function campaignRun(run: RunKind): RunKind { + return run === `other-path` ? `ordinary` : run +} +function campaignEnvironment(ledger: Ledger): Record { + const seed = `${campaignPrefix}_SEED` + const path = `${campaignPrefix}_PATH` + if (ledger.campaignFault === `bad-seed`) return { [seed]: `x` } + if (ledger.campaignFault === `path-without-seed`) return { [path]: `0` } + if (ledger.campaignFault === `bad-path`) return { [seed]: `1`, [path]: `a` } + const run = campaignRun(ledger.run) + if (run === `ordinary`) return {} + if (run === `seed`) return { [seed]: `1` } + return { [seed]: `1`, [path]: `0` } +} + +type CampaignLine = { + property: string + campaign: string + minimum: number | null + counts: Record +} +type CampaignObservation = { + // The parameters the runner passed to each campaign, in order. + parameters: ReadonlyArray<{ seed?: number; path?: string; numRuns?: number }> + lines: ReadonlyArray + failure: Failure | `invalid environment` | undefined +} + +// The campaign driver replays the ledger's declared reaches, without fixed +// marks or early reaches, and with each state declared once: those rules +// belong to the ledger check above. +function campaignLedger(ledger: Ledger): Ledger { + const states = [...new Set(ledger.states)] + return { + ...ledger, + states, + early: [], + histories: ledger.histories.map((history) => ({ + fixed: false, + states: history.states.filter((index) => index < states.length), + })), + } +} + +// The campaign model: two campaigns without a prefix seed, one replay +// campaign with one. Each campaign has a minimum of one and no fixed +// histories, checks reach unless a path replays it, and reports its counts +// when it passes. The first failing campaign ends the run. +function expectedCampaign(ledger: Ledger): CampaignObservation { + if (ledger.campaignFault !== undefined) { + return { parameters: [], lines: [], failure: `invalid environment` } + } + const run = campaignRun(ledger.run) + const campaigns: ReadonlyArray< + readonly [string, { seed?: number; path?: string; numRuns?: number }] + > = + run === `ordinary` + ? [ + [`fixed`, { seed: ledger.fixedSeed }], + [`random`, {}], + ] + : [ + [ + `replay`, + run === `seed` ? { seed: 1 } : { seed: 1, path: `0`, numRuns: 1 }, + ], + ] + const replayed = campaignLedger(ledger) + const parameters: Array<{ seed?: number; path?: string; numRuns?: number }> = + [] + const lines: Array = [] + for (const [campaign, campaignParameters] of campaigns) { + parameters.push(campaignParameters) + const { counts, failure } = expectedVerdict({ + ...replayed, + minimum: 1, + fixedHistories: 0, + replaying: run === `path`, + }) + if (failure !== undefined) return { parameters, lines, failure } + lines.push({ + property: campaignPrefix, + campaign, + minimum: run === `path` ? null : 1, + counts: Object.fromEntries(counts), + }) + } + return { parameters, lines, failure: undefined } +} + +// The driver: one ledger through each campaign that runOracleCampaign runs. +function observeCampaign(ledger: Ledger): Promise { + const replayed = campaignLedger(ledger) + return withEnvironment( + [`${campaignPrefix}_SEED`, `${campaignPrefix}_PATH`], + campaignEnvironment(ledger), + async () => { + const parameters: Array<{ + seed?: number + path?: string + numRuns?: number + }> = [] + // The runner reports each campaign's counts on the console. + const info = vi.spyOn(console, `info`).mockImplementation(() => {}) + const lines = () => + info.mock.calls.map( + ([line]) => + JSON.parse( + String(line).replace(/^ORACLE_REACH /, ``), + ) as CampaignLine, + ) + try { + await runOracleCampaign( + campaignPrefix, + replayed.states, + (reach, campaignParameters) => { + parameters.push(campaignParameters) + for (const history of replayed.histories) { + reach.beginHistory() + for (const index of history.states) { + reach.reach(stateName(index)) + } + } + }, + { fixedSeed: ledger.fixedSeed }, + ) + return { parameters, lines: lines(), failure: undefined } + } catch (error) { + const message = (error as Error).message + return { + parameters, + lines: lines(), + failure: /_SEED must be an integer|_PATH requires a seed/.test( + message, + ) + ? (`invalid environment` as const) + : namedStates(message), + } + } finally { + info.mockRestore() + } + }, + ) +} + +// The check: the observed campaigns refine the campaign model. +async function checkCampaign(ledger: Ledger) { + expect(await observeCampaign(ledger), `campaign observation`).toEqual( + expectedCampaign(ledger), + ) +} + +// The run parameters: the examples come on top of the base runs, and the +// replay selection stays. A seed replay keeps its seed, as does a path replay +// of another property. A path replay of this property keeps its seed and path, +// and runs one history. A replay witness channel adds reporter keys; no run +// kind here opens one, so the check leaves them out. +type Selection = { run: RunKind; seed: number; path: string } +function expectedRunParameters( + { run, seed, path }: Selection, + baseRuns: number, + examples: ReadonlyArray, +) { + const numRuns = run === `path` ? 1 : baseRuns + examples.length + if (run === `ordinary`) return { numRuns, examples } + if (run === `path`) return { numRuns, seed, path, examples } + return { numRuns, seed, examples } +} + +// oraclePropertyOptions reads the process's replay selection and multiplier +// once, at load. The model reads the same variables on its own. +const processEnvironment = process.env +const processSelection: Selection = { + run: + processEnvironment.TANSTACK_DB_ORACLE_PATH !== undefined + ? processEnvironment.TANSTACK_DB_ORACLE_PROPERTY === property + ? `path` + : `other-path` + : processEnvironment.TANSTACK_DB_ORACLE_SEED === undefined + ? `ordinary` + : `seed`, + seed: Number(processEnvironment.TANSTACK_DB_ORACLE_SEED), + path: processEnvironment.TANSTACK_DB_ORACLE_PATH ?? ``, +} +const processMultiplier = Number( + processEnvironment.TANSTACK_DB_ORACLE_RUNS_MULTIPLIER ?? `1`, +) + +function checkRunParameters( + run: RunKind, + baseRuns: number, + examples: ReadonlyArray, +) { + const project = (parameters: { + numRuns: number + seed?: number + path?: string + examples?: ReadonlyArray + }) => ({ + numRuns: parameters.numRuns, + seed: parameters.seed, + path: parameters.path, + examples: parameters.examples, + }) + const replayed = run === `path` ? property : otherProperty + // toEqual treats an undefined key as absent. + expect( + project( + oracleRandomParameters( + baseRuns, + run === `ordinary` + ? undefined + : run === `seed` + ? { + replaySeed: 1, + replayPath: undefined, + replayProperty: undefined, + } + : { replaySeed: 1, replayPath: `0`, replayProperty: replayed }, + property, + examples, + ), + ), + `run parameters`, + ).toEqual( + expectedRunParameters({ run, seed: 1, path: `0` }, baseRuns, examples), + ) + expect( + project(oraclePropertyOptions(baseRuns, property, { examples })), + `property options under this process`, + ).toEqual( + expectedRunParameters( + processSelection, + baseRuns * processMultiplier, + examples, + ), + ) +} + +declareOracle({ + check: [checkLedger, checkCampaign, checkRunParameters], + model: [expectedVerdict, expectedCampaign, expectedRunParameters], + grammar: [ledgerArbitrary], +}) + +const reachStates = [ + `a seed-only run fails below the minimum`, + `a path replay with a state below the minimum`, + `a fixed history miss that a later history enters`, + `a state below a minimum above one`, + `a ledger that passes`, + `a reach before the first history`, + `a path replay of another property with a state below the minimum`, + `a run that stopped inside the fixed histories`, + `invalid reach options`, + `a reach of an undeclared state`, + `a replaying option that overrides the environment`, + `a fixed history beyond the declared ones`, + `an unfixed history before the first fixed one`, + `an invalid campaign environment`, +] as const + +// One fixed ledger enters each reach state, first in every run, so this +// property's own reach cannot miss by chance. +const fixedLedger = ( + overrides: Partial, +): [Ledger, number, Array] => [ + { + run: `ordinary`, + replaying: undefined, + states: [stateName(0), stateName(1)], + minimum: 1, + fixedHistories: 0, + early: [], + histories: [], + fixedSeed: 7, + campaignFault: undefined, + ...overrides, + }, + 60, + [1], +] +const history = (fixed: boolean, ...states: Array) => ({ + fixed, + states, +}) +const ledgerExamples = [ + fixedLedger({ run: `seed` }), + fixedLedger({ run: `path` }), + fixedLedger({ + fixedHistories: 1, + histories: [history(true, 0), history(false, 1)], + }), + fixedLedger({ minimum: 2, histories: [history(false, 0, 1)] }), + fixedLedger({ histories: [history(false, 0, 1)] }), + fixedLedger({ early: [0], histories: [history(false, 0, 1)] }), + fixedLedger({ run: `other-path` }), + fixedLedger({ fixedHistories: 3, histories: [history(true, 0, 1)] }), + fixedLedger({ minimum: 0 }), + fixedLedger({ histories: [history(false, 0, 1, 2)] }), + fixedLedger({ replaying: true }), + fixedLedger({ + fixedHistories: 1, + histories: [history(true, 0, 1), history(true, 0)], + }), + fixedLedger({ + fixedHistories: 1, + histories: [history(false, 0), history(true, 0, 1)], + }), + fixedLedger({ campaignFault: `bad-seed`, histories: [history(false, 0, 1)] }), +] +const ledgerExampleInputs = new Set(ledgerExamples.map(([ledger]) => ledger)) +const ledgerReach = createOracleReach(`oracle.declared-reach`, reachStates, { + fixedHistories: ledgerExamples.length, +}) +let ledgersRan = false +afterAll(() => { + if (ledgersRan) ledgerReach.assertReached() +}) + +fcTest.prop( + [ + ledgerArbitrary, + fc.integer({ min: 1, max: 200 }), + fc.array(fc.integer(), { maxLength: 5 }), + ], + oraclePropertyOptions(200, property, { examples: ledgerExamples }), +)( + `reach ledgers refine the declared reach model`, + async (ledger, baseRuns, examples) => { + ledgersRan = true + ledgerReach.beginHistory({ fixed: ledgerExampleInputs.has(ledger) }) + const verdict = await checkLedger(ledger) + await checkCampaign(ledger) + checkRunParameters(ledger.run, baseRuns, examples) + const { counts, failure } = verdict + const ordinary = expectedVerdict({ + ...ledger, + run: `ordinary`, + replaying: undefined, + }).failure + const environmentChecks = ledger.run === `ordinary` || ledger.run === `seed` + if (verdict.invalid) ledgerReach.reach(`invalid reach options`) + if (ledger.campaignFault !== undefined) + ledgerReach.reach(`an invalid campaign environment`) + if (verdict.invalid) return + if ( + ledger.replaying === undefined && + ledger.run === `path` && + (ordinary?.belowMinimum.size ?? 0) > 0 + ) + ledgerReach.reach(`a path replay with a state below the minimum`) + if ( + ledger.replaying === undefined && + ledger.run === `other-path` && + (ordinary?.belowMinimum.size ?? 0) > 0 + ) + ledgerReach.reach( + `a path replay of another property with a state below the minimum`, + ) + if ( + ledger.replaying !== undefined && + ledger.replaying === environmentChecks + ) + ledgerReach.reach(`a replaying option that overrides the environment`) + if (failure?.stoppedAt !== undefined) + ledgerReach.reach(`a run that stopped inside the fixed histories`) + if (ledger.run === `seed` && (failure?.belowMinimum.size ?? 0) > 0) + ledgerReach.reach(`a seed-only run fails below the minimum`) + if (failure?.unfixed.some((state) => (counts.get(state) ?? 0) > 0)) + ledgerReach.reach(`a fixed history miss that a later history enters`) + if (ledger.minimum > 1 && (failure?.belowMinimum.size ?? 0) > 0) + ledgerReach.reach(`a state below a minimum above one`) + if (environmentChecks && ledger.replaying === undefined && !failure) + ledgerReach.reach(`a ledger that passes`) + if (verdict.rejected.some(([, reason]) => reason === `early`)) + ledgerReach.reach(`a reach before the first history`) + if (verdict.rejected.some(([, reason]) => reason === `undeclared`)) + ledgerReach.reach(`a reach of an undeclared state`) + if ( + ledger.histories.filter(({ fixed }) => fixed).length > + ledger.fixedHistories + ) + ledgerReach.reach(`a fixed history beyond the declared ones`) + const firstFixed = ledger.histories.findIndex(({ fixed }) => fixed) + if (ledger.fixedHistories > 0 && firstFixed > 0) + ledgerReach.reach(`an unfixed history before the first fixed one`) + }, +) diff --git a/packages/db/tests/mutation-handler-types-oracle.test.ts b/packages/db/tests/mutation-handler-types-oracle.test.ts index 6a37d856d1..84b18b6558 100644 --- a/packages/db/tests/mutation-handler-types-oracle.test.ts +++ b/packages/db/tests/mutation-handler-types-oracle.test.ts @@ -306,16 +306,22 @@ function checkHandlerAdmission( ) } -const states = typeShapes.flatMap((shape) => - surfaces.flatMap((surface) => - handlers.flatMap((handler) => - results.map( - (result) => - `${shape.adapter}/${shape.schema ? `schema` : `plain`}/${shape.data}/${surface}/${handler}/${result.name}`, +// A state name leaves out the key and scalar types, so the six shapes that +// differ only there share one state, and `minimum: 6` asks for all of them. +const states = [ + ...new Set( + typeShapes.flatMap((shape) => + surfaces.flatMap((surface) => + handlers.flatMap((handler) => + results.map( + (result) => + `${shape.adapter}/${shape.schema ? `schema` : `plain`}/${shape.data}/${surface}/${handler}/${result.name}`, + ), + ), ), ), ), -) +] const reach = createOracleReach(`adapter.handler-types`, states, { minimum: 6 }) describe(`public handler result admission`, () => { afterAll(() => { diff --git a/packages/db/tests/oracle-campaign.ts b/packages/db/tests/oracle-campaign.ts index 754b33f854..d99f4fb480 100644 --- a/packages/db/tests/oracle-campaign.ts +++ b/packages/db/tests/oracle-campaign.ts @@ -5,7 +5,9 @@ import { createOracleReach } from './oracle-reach' * owns its property, grammar, recorder, comparison, and run budget; only seed * selection changes between the two calls. Counts are histories, not events: * call beginHistory once and reach only after observing the named checkpoint. - * A direct prefix SEED/PATH replay runs immediately without campaign minima. + * A prefix SEED replay runs one campaign at the callback's full budget, and + * checks its reach. A SEED/PATH replay runs one recorded history and checks + * no reach. `declared-reach-oracle.property.test.ts` owns this rule. * This is runner support, not an independent product judgment. */ export async function runOracleCampaign( @@ -50,7 +52,7 @@ export async function runOracleCampaign( ] for (const campaign of campaigns) { const reach = createOracleReach(prefix, states, { - replaying: seed !== undefined, + replaying: path !== undefined, }) await run(reach, campaign.parameters) reach.assertReached() @@ -58,7 +60,7 @@ export async function runOracleCampaign( `ORACLE_REACH ${JSON.stringify({ property: prefix, campaign: campaign.name, - minimum: seed === undefined ? 1 : null, + minimum: path === undefined ? 1 : null, counts: Object.fromEntries(reach.counts()), })}`, ) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 15e4cb262a..4c2f38468a 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -6,6 +6,7 @@ type OracleEnvironment = Record const staticOracleProperties = [ `lint.part-reach`, + `oracle.declared-reach`, `scripts.lint-scope`, `scripts.run-budget`, `scripts.release-version`, @@ -653,13 +654,41 @@ export function readOracleRunConfig( return { multiplier, replaySeed, replayPath, replayProperty } } +type OracleRunParameters = { + numRuns: number + seed?: number + path?: string +} & ReturnType + +// fast-check infers a property's argument types from `examples` too, so only +// a call with examples returns them. `ReturnType` reads the last overload, +// the one without examples. +export function oracleRandomParameters( + numRuns: number, + replay: OracleReplayConfig | number | undefined, + property: string | undefined, + examples: ReadonlyArray, +): OracleRunParameters & { examples: Array } export function oracleRandomParameters( numRuns: number, replay: OracleReplayConfig | number | undefined, property?: string, -): { numRuns: number; seed?: number; path?: string } & ReturnType< - typeof oracleReplayReporter -> { +): OracleRunParameters +export function oracleRandomParameters( + numRuns: number, + replay: OracleReplayConfig | number | undefined, + property?: string, + examples?: ReadonlyArray, +): OracleRunParameters & { examples?: Array } { + return randomParameters(numRuns, replay, property, examples) +} + +function randomParameters( + numRuns: number, + replay: OracleReplayConfig | number | undefined, + property: string | undefined, + examples: ReadonlyArray | undefined, +): OracleRunParameters & { examples?: Array } { if (property !== undefined) assertRegisteredOracleProperty(property) const { replaySeed, replayPath, replayProperty } = typeof replay === `object` @@ -669,22 +698,36 @@ export function oracleRandomParameters( replayPath: undefined, replayProperty: undefined, } - if (replaySeed === undefined) return { numRuns } - return { - numRuns, - seed: replaySeed, - // A path replays one recorded counterexample, so the replay runs once. - // From a path with one segment, fast-check would otherwise run `numRuns` - // histories that start at that index. - ...(property !== undefined && - replayPath !== undefined && - replayProperty === property - ? { - numRuns: 1, - path: replayPath, - ...oracleReplayReporter(property, replaySeed, replayPath), + const parameters: OracleRunParameters = + replaySeed === undefined + ? { numRuns } + : { + numRuns, + seed: replaySeed, + // A path replays one recorded counterexample, so the replay runs + // once. From a path with one segment, fast-check would otherwise + // run `numRuns` histories that start at that index. + ...(property !== undefined && + replayPath !== undefined && + replayProperty === property + ? { + numRuns: 1, + path: replayPath, + ...oracleReplayReporter(property, replaySeed, replayPath), + } + : {}), } - : {}), + if (examples === undefined) return parameters + // fast-check runs the examples first and counts them in numRuns, so they + // come on top of the random runs: the examples never displace one. A path + // replay runs one recorded history, which can be an example. + return { + ...parameters, + examples: [...examples], + numRuns: + parameters.path === undefined + ? parameters.numRuns + examples.length + : parameters.numRuns, } } @@ -767,10 +810,25 @@ export function oracleRuns(baseRuns: number): number { return baseRuns * multiplier } -/** Replays broad randomized properties when a campaign seed is supplied. */ +/** + * Replays broad randomized properties when a campaign seed is supplied. + * `examples` run first in every run, on top of the base runs. A site that + * spreads these options and sets fast-check's `examples` itself runs them + * inside the base runs instead, so they displace random histories. + */ +export function oraclePropertyOptions( + baseRuns: number, + property: string | undefined, + options: { examples: ReadonlyArray }, +): OracleRunParameters & { examples: Array } export function oraclePropertyOptions( baseRuns: number, property?: string, -): ReturnType { - return oracleRandomParameters(oracleRuns(baseRuns), replay, property) +): OracleRunParameters +export function oraclePropertyOptions( + baseRuns: number, + property?: string, + { examples }: { examples?: ReadonlyArray } = {}, +): OracleRunParameters & { examples?: Array } { + return randomParameters(oracleRuns(baseRuns), replay, property, examples) } diff --git a/packages/db/tests/oracle-reach.test.ts b/packages/db/tests/oracle-reach.test.ts index 51f6f08a35..5836dc705e 100644 --- a/packages/db/tests/oracle-reach.test.ts +++ b/packages/db/tests/oracle-reach.test.ts @@ -1,9 +1,13 @@ import { describe, expect, it } from 'vitest' import { createOracleReach } from './oracle-reach.js' +// A replay selection in the environment would switch the reach check off, so +// these examples choose an ordinary run. describe(`createOracleReach`, () => { it(`fails for a declared state that no history entered`, () => { - const reach = createOracleReach(`example`, [`seen`, `unseen`] as const) + const reach = createOracleReach(`example`, [`seen`, `unseen`] as const, { + replaying: false, + }) reach.beginHistory() reach.reach(`seen`) expect(() => reach.assertReached()).toThrow(/unseen \(0\)/) @@ -12,6 +16,7 @@ describe(`createOracleReach`, () => { it(`counts a history once per state`, () => { const reach = createOracleReach(`example`, [`state`] as const, { minimum: 2, + replaying: false, }) reach.beginHistory() reach.reach(`state`) diff --git a/packages/db/tests/oracle-reach.ts b/packages/db/tests/oracle-reach.ts index b6ec08295a..ecf6e43b01 100644 --- a/packages/db/tests/oracle-reach.ts +++ b/packages/db/tests/oracle-reach.ts @@ -9,42 +9,77 @@ import { readOracleRunConfig } from './oracle-config.js' * produces that state. Counting passing runs does not show reach. * * A property declares the states its laws depend on. Its driver calls - * `reach(state)` when a history enters one. After the property ran, - * `assertReached()` fails for every declared state that fewer than `minimum` - * histories entered. A history counts once per state, however often it - * enters it. + * `reach(state)` when a history enters one. A call for an undeclared state, or + * before the first `beginHistory()`, throws: it is a driver error, not a + * history. Options that cannot describe a reach throw at creation. After the + * property ran, `assertReached()` fails for every declared state that fewer + * than `minimum` histories entered. A history counts once per state, however + * often it enters it. * - * A replay of one seed or path checks one history, not reach, so the check - * does nothing then. + * A property that runs fixed examples first, through `oraclePropertyOptions`' + * `examples`, passes their number as `fixedHistories` and begins each + * example's history with `beginHistory({ fixed: true })`. The check then also + * fails for every declared state that no fixed history entered: that state's + * reach would depend on the seed (#2287). Fixedness belongs to the history, so + * other tests may share the reach. Shrinking can run an example again, so a + * fixed history beyond the declared number counts as fixed too. When fewer + * fixed histories ran than were declared, a failure stopped the run inside the + * examples, and the check reports only that, without naming states. + * + * A path replay checks one recorded history, not reach, so the check does + * nothing then, in every reach of that process: a reach need not carry its + * property's name, so it cannot tell whose replay runs. A seed alone reseeds + * the whole campaign, which runs its full budget, so its reach is checked like + * an ordinary run's. + * `packages/db/tests/declared-reach-oracle.property.test.ts` owns these rules. */ export function createOracleReach( property: string, states: ReadonlyArray, { minimum = 1, + fixedHistories = 0, replaying: replayOverride, - }: { minimum?: number; replaying?: boolean } = {}, + }: { minimum?: number; fixedHistories?: number; replaying?: boolean } = {}, ) { + if (!Number.isInteger(minimum) || minimum < 1) { + throw new Error(`${property}: minimum must be an integer of at least 1`) + } + if (!Number.isInteger(fixedHistories) || fixedHistories < 0) { + throw new Error(`${property}: fixedHistories must be a nonnegative integer`) + } + if (new Set(states).size !== states.length) { + throw new Error(`${property}: a reach state is declared twice`) + } const counts = new Map(states.map((state) => [state, 0])) + const enteredByFixed = new Set() + let histories = 0 + let fixedBegun = 0 + let currentFixed = false let current = new Set() // Other packages use their own replay prefixes. An explicit campaign choice // must not inherit an unrelated DB replay selector from the environment. const replaying = - replayOverride ?? - (() => { - const { replaySeed, replayPath } = readOracleRunConfig() - return replaySeed !== undefined || replayPath !== undefined - })() + replayOverride ?? readOracleRunConfig().replayPath !== undefined return { - /** Starts one generated history. */ - beginHistory(): void { + /** Starts one generated history, a fixed example's when `fixed`. */ + beginHistory({ fixed = false }: { fixed?: boolean } = {}): void { + if (fixed) fixedBegun += 1 + currentFixed = fixed current = new Set() + histories += 1 }, /** The current history entered a declared state. */ reach(state: TState): void { if (!counts.has(state)) { throw new Error(`${property}: undeclared reach state ${state}`) } + if (histories === 0) { + throw new Error( + `${property}: reach(${state}) came before the first beginHistory()`, + ) + } + if (currentFixed) enteredByFixed.add(state) if (current.has(state)) return current.add(state) counts.set(state, counts.get(state)! + 1) @@ -56,16 +91,37 @@ export function createOracleReach( /** Fails for each declared state that too few histories entered. */ assertReached(): void { if (replaying) return - const missed = [...counts] - .filter(([, count]) => count < minimum) - .map(([state, count]) => `${state} (${count})`) - if (missed.length > 0) { + if (fixedBegun < fixedHistories) { throw new Error( - `${property}: generated histories reached these states fewer than ` + - `${minimum} times, so their laws were not enforced there: ` + - missed.join(`, `), + `${property}: only ${fixedBegun} of its ${fixedHistories} fixed ` + + `histories ran, so a failure stopped the run inside the fixed ` + + `examples`, ) } + const missed = [...counts] + .filter(([, count]) => count < minimum) + .map(([state, count]) => `${state} (${count})`) + const unfixed = + fixedHistories === 0 + ? [] + : states.filter((state) => !enteredByFixed.has(state)) + const failures = [ + ...(missed.length > 0 + ? [ + `${property}: generated histories reached these states fewer ` + + `than ${minimum} times, so their laws were not enforced ` + + `there: ${missed.join(`, `)}`, + ] + : []), + ...(unfixed.length > 0 + ? [ + `${property}: its ${fixedHistories} fixed histories, the ` + + `examples, did not enter these states, so their reach ` + + `depends on the seed: ${unfixed.join(`, `)}`, + ] + : []), + ] + if (failures.length > 0) throw new Error(failures.join(`\n`)) }, } } diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index c4e9d54970..222631c23a 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -20,6 +20,11 @@ const ownerGroups: ReadonlyArray< ], [`../scripts/run-budget-oracle.property.test.js`, `scripts`, `run-budget`], [`../scripts/lint-scope-oracle.property.test.js`, `scripts`, `lint-scope`], + [ + `db/tests/declared-reach-oracle.property.test.ts`, + `oracle`, + `declared-reach`, + ], [ `../scripts/release-history-oracle.property.test.js`, `scripts`, diff --git a/packages/db/tests/source-health-oracle.test.ts b/packages/db/tests/source-health-oracle.test.ts index 4ba8252a10..f3db44cca9 100644 --- a/packages/db/tests/source-health-oracle.test.ts +++ b/packages/db/tests/source-health-oracle.test.ts @@ -5245,14 +5245,6 @@ describe(`on-demand demand grammar`, () => { `fail`, `crash`, ) - const demandReach = createOracleReach( - `source-health.on-demand`, - demandReachStates, - ) - let demandHistoriesRan = false - afterAll(() => { - if (demandHistoriesRan) demandReach.assertReached() - }) // A prefix that enters a declared reach state that random steps alone // rarely reach at the base run count; random steps follow it. const demandReachPrefixes: ReadonlyArray> = [ @@ -5299,6 +5291,21 @@ describe(`on-demand demand grammar`, () => { `restart`, ], ] + // Each prefix also runs as a fixed example, first in every run and on top + // of the random histories, with a direct subscriber. The reach check fails + // for a declared state that these examples do not enter. + const demandExamples = demandReachPrefixes.map( + (prefix): [Array, boolean] => [[...prefix], true], + ) + const demandReach = createOracleReach( + `source-health.on-demand`, + demandReachStates, + { fixedHistories: demandExamples.length }, + ) + let demandHistoriesRan = false + afterAll(() => { + if (demandHistoriesRan) demandReach.assertReached() + }) oracleReplayTest(fcTest, `source-health.on-demand`).prop( [ fc @@ -5312,12 +5319,17 @@ describe(`on-demand demand grammar`, () => { ]), fc.boolean(), ], - oraclePropertyOptions(60, `source-health.on-demand`), + oraclePropertyOptions(60, `source-health.on-demand`, { + examples: demandExamples, + }), )( `generated demand histories refine the model`, (steps, directSubscriber) => { demandHistoriesRan = true - demandReach.beginHistory() + // fast-check passes an example's input by reference. + demandReach.beginHistory({ + fixed: demandExamples.some(([example]) => example === steps), + }) return withDeadline( runDemandHistory(steps, directSubscriber, demandReach), ) @@ -7013,14 +7025,16 @@ describe(`lazy join demand grammar`, () => { ending: { at: `subscription-status`, dependents }, }), ) - const lazyReach = createOracleReach( - `source-health.lazy-demand`, - lazyReachStates, - ) - let lazyHistoriesRan = false - afterAll(() => { - if (lazyHistoriesRan) lazyReach.assertReached() - }) + // A load, then a reload whose load fails. + const failedReload: { steps: ReadonlyArray } = { + steps: [{ kind: `settle-ok` }, { kind: `reload` }, { kind: `settle-fail` }], + } + const reloadNotices: ReadonlyArray = [ + `source-health`, + `query-health`, + `loading-subset`, + `subscription-status`, + ] // Prefixes that enter a declared reach state that random steps alone rarely // reach at the base run count, with the driver options the state needs. // Random steps follow each one. @@ -7056,13 +7070,7 @@ describe(`lazy join demand grammar`, () => { { kind: `reload` }, ], }, - { - steps: [ - { kind: `settle-ok` }, - { kind: `reload` }, - { kind: `settle-fail` }, - ], - }, + failedReload, { steps: [{ kind: `truncate` }, { kind: `truncate` }] }, { steps: [{ kind: `settle-ok` }, { kind: `fail` }, { kind: `truncate` }], @@ -7128,6 +7136,91 @@ describe(`lazy join demand grammar`, () => { holdChildReady: false, }, ] + // The driver options a lazy history runs under, by name. + type LazyOptions = { + asyncChildReady: boolean + generatedHoldChildReady: boolean + directSubscriber: boolean + withEffect: boolean + windowed: boolean + deferredStart: `plain` | `reload-in-first-load` | undefined + ending: LazyEnding | undefined + } + const lazyOptions: fc.Arbitrary = fc.record({ + asyncChildReady: fc.boolean(), + generatedHoldChildReady: fc.boolean(), + directSubscriber: fc.boolean(), + withEffect: fc.boolean(), + windowed: fc.boolean(), + deferredStart: fc.constantFrom( + undefined, + `plain` as const, + `reload-in-first-load` as const, + ), + ending: fc.option( + fc.record({ + at: fc.constantFrom(...reloadNotices), + dependents: fc.boolean(), + }), + { freq: 2, nil: undefined }, + ), + }) + // A history's steps: an optional reach prefix, then generated steps. + type LazySteps = { + prefix: (typeof lazyReachPrefixes)[number] | null + rest: Array + } + const lazyExample = ( + prefix: LazySteps[`prefix`], + rest: LazySteps[`rest`], + options: Partial = {}, + ): [LazySteps, LazyOptions] => [ + { prefix, rest }, + { + asyncChildReady: false, + generatedHoldChildReady: false, + directSubscriber: true, + withEffect: false, + windowed: false, + deferredStart: undefined, + ending: undefined, + ...options, + }, + ] + // Each prefix also runs as a fixed example, first in every run and on top + // of the random histories, and so does each driver option that a declared + // state needs. Each example has a direct subscriber and no other option it + // does not name. The reach check fails for a declared state that these + // examples do not enter. + const lazyExamples = [ + ...lazyReachPrefixes.map((prefix) => lazyExample(prefix, [])), + // The provider deletes a row after a reload's run ended. + lazyExample(failedReload, [{ kind: `remote-delete`, key: 1 }]), + lazyExample(null, [{ kind: `settle-ok` }], { withEffect: true }), + lazyExample(null, [{ kind: `settle-ok` }], { + deferredStart: `reload-in-first-load`, + }), + // A cleanup at each notice of a reload. A reload changes loadingSubset + // only while the first load is still open. + ...reloadNotices.map((at) => + lazyExample( + null, + at === `loading-subset` ? [] : [{ kind: `settle-ok` }], + { + ending: { at, dependents: false }, + }, + ), + ), + ] + const lazyReach = createOracleReach( + `source-health.lazy-demand`, + lazyReachStates, + { fixedHistories: lazyExamples.length }, + ) + let lazyHistoriesRan = false + afterAll(() => { + if (lazyHistoriesRan) lazyReach.assertReached() + }) oracleReplayTest(fcTest, `source-health.lazy-demand`).prop( [ fc @@ -7135,47 +7228,34 @@ describe(`lazy join demand grammar`, () => { fc.option(fc.constantFrom(...lazyReachPrefixes), { freq: 4 }), fc.array(step, { maxLength: 10 }), ) - .map(([prefix, rest]) => ({ prefix, rest })), - fc.boolean(), - fc.boolean(), - fc.boolean(), - fc.boolean(), - fc.boolean(), - fc.constantFrom( - undefined, - `plain` as const, - `reload-in-first-load` as const, - ), - fc.option( - fc.record({ - at: fc.constantFrom( - `source-health`, - `query-health`, - `loading-subset`, - `subscription-status`, - ), - dependents: fc.boolean(), - }), - { freq: 2, nil: undefined }, - ), + .map(([prefix, rest]): LazySteps => ({ prefix, rest })), + lazyOptions, ], - oraclePropertyOptions(100, `source-health.lazy-demand`), + oraclePropertyOptions(100, `source-health.lazy-demand`, { + examples: lazyExamples, + }), )( `generated lazy demand histories refine the model`, ( - { prefix, rest }, - asyncChildReady, - generatedHoldChildReady, - directSubscriber, - withEffect, - windowed, - deferredStart, - ending, + lazySteps, + { + asyncChildReady, + generatedHoldChildReady, + directSubscriber, + withEffect, + windowed, + deferredStart, + ending, + }, ) => { + const { prefix, rest } = lazySteps const steps = [...(prefix?.steps ?? []), ...rest] const holdChildReady = prefix?.holdChildReady ?? generatedHoldChildReady lazyHistoriesRan = true - lazyReach.beginHistory() + // fast-check passes an example's input by reference. + lazyReach.beginHistory({ + fixed: lazyExamples.some(([example]) => example === lazySteps), + }) return withDeadline( runLazyHistory(steps, { asyncChildReady,