diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c0de3f..47b5b5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,79 @@ While pre-1.0, the public API may change between 0.x releases. ## [Unreleased] +### Changed + +- **BREAKING (raw clients only):** the server rejects an insert or update + whose `cols` carry the primary key with a value different from the op + `key`, answering with a `VALIDATION` rejection before `authorize` runs. + Before, such a row was stored under the `cols` pk, so its optimistic row + never reconciled (ADR-0001 D9). The collection client always sends + matching keys and is unaffected. An insert that omits the pk from `cols` + is still accepted (ADR-0025 amendment). + +### Fixed + +- **A subclass `tickMs` override now takes effect.** The coalescer read the + tick before subclass fields were set, so it always used 50 ms (ADR-0015 + amendment). +- **No empty ticks after a flush, close or error.** Flushing the last socket + with pending deltas, or closing or erroring a socket, now cancels the flush + timer when nothing else is pending, so the DO can hibernate sooner. +- **Expired dedup receipts are swept on every answered `mut` or `call`**, + including rejections, replays and commands that write nothing. The sweep + runs at most once per `dedupRetentionMs`, and always on the first receipt + after a wake. Before, it ran only with compaction after drained writes, so + a DO that only rejected or ran write-free commands grew `_sync_seen_tx` + without bound (ADR-0009 amendment). +- **Bounded on-demand snapshots place NULLs as TanStack does.** The server's + `ORDER BY` now emits `NULLS FIRST` or `NULLS LAST` from each clause's + `nulls` (default first), so a descending window over a nullable column no + longer drops its NULL rows. Locale collation is still pinned as an expected + failure (ADR-0023 amendment). +- **Bounded reads break ties by pk, ascending**, matching TanStack's row-key + tie-break, so a `limit` over tied rows picks the same rows on the server and + the client (ADR-0023 amendment). +- **`isNull` and `isUndefined` join the predicate floor**, so a + `nulls: "last"` window on `@tanstack/db` 0.9 can page past its first page. + Before, its cursor fetch was refused and read as empty (ADR-0013 amendment). +- **Dedup retention is a true lower bound.** Receipts are stamped in whole + seconds, so the sweep could remove one up to 999 ms early. +- **A fetch page no longer overtakes a buffered delta.** The server flushes + the socket's pending deltas before it reads a page, the same as it does + for snapshots (ADR-0023 amendment). +- **Server type declarations no longer need `@cloudflare/workers-types`.** + The shipped `.d.ts` files use the ambient Worker SQL types, so a Worker + typed with `wrangler types` and `skipLibCheck: false` no longer fails with + TS2307. The pack smoke test now checks both type setups (ADR-0022 + amendment). + +### Tests + +- Known bugs are now pinned as expected failures (`it.fails`, tagged + `bugbash `), so each fix flips its tests: + - The single client cursor skips changes when the socket drops during a + resubscribe round (F1): 30 reconnect-under-drop scenarios, 16 of them red. + - Known limitations L1 (no incarnation epoch, ADR-0011) and L2 (mid-tick + eviction, ADR-0009). + - Refusals read as empty results (F5): 14 wire and app scenarios. + - Dedup is scoped by txId alone (L3, ADR-0012 D4). +- Full-row live deltas for narrow and no-op updates (#28) are pinned as + current behaviour. +- Real-path on-demand scenarios pin the ADR-0023 fixes that were only covered + by fake transports: unload mid-snapshot, a below-floor reset across several + subsets (bites at the 0.8 floor), `.offset()` windows, and late or + overlaid pages. +- The abandoned-socket fetch test waits for the page to be held instead of + sleeping 30 ms, which timed out under full-suite load. + +### Documentation + +- README gains a Known limitations section: DO SQLite can't bind `bigint`, + and INTEGER values above 2^53 round on read (workerd#4195, #10). ADR-0001 + D17 is qualified to match. +- ADR-0012 D3 points to ADR-0014, which changed how `handleCall` sanitizes + authorize errors. + ## [0.7.0] — 2026-09-25 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 7250916..2f7d452 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -83,9 +83,14 @@ Each is load-bearing and guarded by an ADR — read it before you touch the area - **One decision → one ADR.** Conventional commits (`!` = breaking); end with the Co-Authored-By trailer. Commit/push only when asked; `main` is the release branch. -- **Run plans and risky changes past codex (`gpt-5.6-sol`) as an adversary** - before committing — it has caught real bugs here (the cursor-fetch race, the +- **Run plans and risky changes past codex as an adversary** before + committing — it has caught real bugs here (the cursor-fetch race, the scroll-back `DuplicateKeySyncError`). Use the strongest available model; a - weaker adversary is a weaker review. Pipe input in (`git diff | codex exec - … -`) or close stdin (`codex exec "…" < /dev/null`) — an open stdin hangs - codex forever. + weaker adversary is a weaker review. Run it as a bb subthread in your own + environment, not `codex exec`: + `bb thread spawn --project "$BB_PROJECT_ID" --parent-self --environment + "$BB_ENVIRONMENT_ID" --provider codex --model gpt-6-sol --reasoning-level + high --title "adversarial review: …" --prompt-file `. The prompt + names the diff (`git diff --no-ext-diff ...HEAD`), says read-only (no + edits, no commits), and asks for findings via `bb thread tell + "$BB_THREAD_ID"`. diff --git a/README.md b/README.md index cb05d53..f8466d7 100644 --- a/README.md +++ b/README.md @@ -457,6 +457,16 @@ show. The conventions, what's verified and how, and the full --- +## Known limitations + +- **Large integers in DO SQLite:** workerd `SqlStorage` cannot bind JS `bigint`, + and INTEGER values above `Number.MAX_SAFE_INTEGER` round on read before + syncing. For exact values, store decimal strings in TEXT columns and convert + at the application edge ([workerd#4195](https://github.com/cloudflare/workerd/issues/4195), + [issue #10](https://github.com/grrowl/tanstack-durable-object-sync/issues/10)). + +--- + ## Non-goals - **Multi-DO transactions.** A transaction touches collections in one DO. diff --git a/docs/adr/0001-sync-architecture.md b/docs/adr/0001-sync-architecture.md index 9f96c89..e29f0b5 100644 --- a/docs/adr/0001-sync-architecture.md +++ b/docs/adr/0001-sync-architecture.md @@ -180,3 +180,13 @@ interleave so every milestone is end-to-end demonstrable. - We inherit the entire client reactive layer from TanStack DB and the proven single-stream discipline from Electric, while keeping full DO sovereignty over data. + +## Clarification — 2026-09-29: D17 and SQLite bigint precision + +D17 describes the tagged codec, not arbitrary SQLite column storage. It +preserves `bigint` on the wire and in library-encoded TEXT (such as persisted +predicates and dedup results); workerd `SqlStorage` cannot bind a JS `bigint`, +and reading INTEGER values above `Number.MAX_SAFE_INTEGER` rounds them before +the codec sees them ([workerd#4195](https://github.com/cloudflare/workerd/issues/4195), +[issue #10](https://github.com/grrowl/tanstack-durable-object-sync/issues/10)). +Use TEXT columns for exact large integers and convert at the application edge. diff --git a/docs/adr/0009-changelog-time-retention.md b/docs/adr/0009-changelog-time-retention.md index 236a26b..f341d22 100644 --- a/docs/adr/0009-changelog-time-retention.md +++ b/docs/adr/0009-changelog-time-retention.md @@ -187,3 +187,38 @@ untouched (`dedupRetentionMs` stays separate). batch (`appliedSeq` advances on the next delivery without a contiguity check). This exists today, independent of retention; pruning neither causes nor worsens it. Not addressed here. + +## Amendment — 2026-09-29: the dedup sweep rides receipts, not compaction + +§1 put `sweepDedup` inside `maybeCompact`, which runs only when a drain finds +CDC rows. But `_sync_seen_tx` rows come from answered receipts, not drains. A DO +that only rejects mutations, replays receipts, or runs commands that write +nothing adds a row per frame and never sweeps (bug bash 2026-09-29, F7). + +`sweepDedup` moves out of `maybeCompact`. It now runs after every answered +`mut`/`call` receipt (new outcome or replay), after the reply is sent, behind +one gate held in memory: at most one sweep per `dedupRetentionMs`. The gate +starts at 0, so the first receipt after construction or a hibernation wake +always sweeps. No new knob, no idle timer, no alarm, and no write per receipt +beyond `recordTx`. + +- **Why in memory.** A persisted counter would add an UPSERT to every receipt. + An in-memory counter would reset on each wake, so a quiet DO that hibernates + between receipts would never reach its threshold. A time gate that reopens on + wake has neither problem. +- **Why time, not a count.** `DELETE … WHERE ts < ?` scans the whole table + (there is no index on `ts`). Under a count cadence the scans per hour grow with + traffic, and so does each scan. Under a time gate a DO that stays awake scans + once per window, whatever the traffic. A DO that wakes per receipt scans once + per wake, but it is quiet, so its table is small. +- **Why the window is `dedupRetentionMs`.** Retention is a lower bound: a retry + inside the window is deduped (ADR-0002 C5, ADR-0021). Sweeping less often only + keeps rows longer. An expired receipt now lives at most one more window, so the + table holds at most about two windows of receipts. +- **Server-originated writes** (`runSyncedWrite`) no longer sweep dedup. They add + no receipts, so they add no rows to sweep. +- A DO that receives nothing sweeps nothing. Its table does not grow either. + +Tests: `tests/dedup-sweep.test.ts` drives each receipt path through real +sockets and polls the table. It also pins the gate: closed within the window, +reopened by a wake (`evictDurableObject`) or by elapsed time. diff --git a/docs/adr/0012-wire-input-hardening.md b/docs/adr/0012-wire-input-hardening.md index 519a6d1..2ac2ff3 100644 --- a/docs/adr/0012-wire-input-hardening.md +++ b/docs/adr/0012-wire-input-hardening.md @@ -102,3 +102,10 @@ a maintainer design decision. - **Test coverage**: `tests/wire-hardening.test.ts` (5 tests) pins all four invariants; `tests/error-paths.test.ts` was updated to assert the new generic error text for execute failures. + +## Amendment — 2026-09-29: D3 command authorization + +[ADR-0014 D2](./0014-object-sync-schema.md) revises D3 above: command +`authorize` errors now surface to clients like mutation `authorize` errors; +only command `execute` errors are sanitized as `EXECUTE_FAILED`. The shared +sanitized catch described in D3 is historical, not the current contract. diff --git a/docs/adr/0013-predicate-floor-one-evaluator.md b/docs/adr/0013-predicate-floor-one-evaluator.md index 7054417..8800539 100644 --- a/docs/adr/0013-predicate-floor-one-evaluator.md +++ b/docs/adr/0013-predicate-floor-one-evaluator.md @@ -92,3 +92,29 @@ operator that lands in one floor and not the other. floor-operator agreement; `tests/subscriptions.test.ts` pins the JS-floor guard at the unit level; `tests/sql-compiler.test.ts` pins `ne` rejection and `not(eq)` lowering. + +## Amendment 2026-09-29: `isNull` and `isUndefined` join the floor (bugbash F4-cursor) + +On @tanstack/db 0.9, an indexed window ordered `nulls: "last"` scrolls with a +cursor whose `whereFrom` is `gt(v) OR isNull(col) OR isUndefined(col)` (`lt` for +descending). Both null checks were off-floor, so the cursor fetch was refused as +an empty page and the window never grew. + +D2's rule still holds: each addition is a verified-agreeing operator. Checked +against `evaluators.ts` in both 0.8.6 and 0.9.2 (identical): + +- `isNull(x)` is `x === null`. SQL: `"col" IS NULL`. The delta and catch-up paths + evaluate rows hydrated from SQLite, where a NULL column is always `null` + (whether the write omitted it or sent `null`), so the two agree. +- `isUndefined(x)` is `x === undefined`. A stored column is never undefined (SQLite + has no such value; hydration yields `null`), so it is false for every stored row. + SQL: the constant `0`, like the empty `in`. The operand is still validated as a + plain column reference. Residual: `=== undefined` is TRUE for a property the row + lacks, i.e. a column the table does not have; there SQL says false. Unknown + columns are out of contract (the schema is author-owned and shared with the + client, and SQLite reads an unresolvable quoted name as a string literal, so + no operator fails loud on one). + +The floor is now `{ eq, gt, gte, lt, lte, like, in, isNull, isUndefined, and, or, +not }`. `tests/predicate-parity.test.ts` pins both operators, and the composed +cursor shape, across the snapshot and delta paths, per D2. diff --git a/docs/adr/0015-syncable-mixin.md b/docs/adr/0015-syncable-mixin.md index 5b1a3b3..1664793 100644 --- a/docs/adr/0015-syncable-mixin.md +++ b/docs/adr/0015-syncable-mixin.md @@ -227,3 +227,11 @@ tddc design gap; a PR against Actors is possible in principle but out of scope. and ADR-0019's `tests/hibernation.test.ts` now exercises the restore under real `evictDurableObject` cycles — which promptly exposed that the restore was incomplete (sockets survived, subscriptions didn't). See ADR-0019.]* + +## Amendment — 2026-09-29: subclass tick initialization + +The coalescer reads `tickMs` when arming an on-demand flush, not while the mixin +constructor runs: subclass field initializers have not run during `super()`. +This preserves the protected-field tuning contract and creates no idle timer. +`tests/tick-override.test.ts` pins the real DO path; the slow-tick cursor-barrier +and replay tests now run with their intended 30-second coalescer window. diff --git a/docs/adr/0022-db-version-support-policy.md b/docs/adr/0022-db-version-support-policy.md index f4ec650..6640030 100644 --- a/docs/adr/0022-db-version-support-policy.md +++ b/docs/adr/0022-db-version-support-policy.md @@ -142,8 +142,21 @@ test green at the new floor, and a recorded reason. (go-to-definition lands on source, not a missing file): about +54 kB packed. - Known gap, not fixed here: the server `.d.ts` files import `@cloudflare/workers-types`, which the package does not declare. The smoke - consumer installs it — the documented setup (README, examples) — so the test - models that setup rather than proving the import is safe without it. + consumer installs it — the example-app setup (not specified in README) — + so the test models that setup rather than proving the import is safe without it. - GitHub disables scheduled workflows after 60 days without repository activity, and sends their failure notifications to whoever last edited the cron line. A quiet repo must re-enable the job. + +## Amendment — 2026-09-29: Worker type sources in packed consumers + +The known gap above is closed. Server declarations use ambient Cloudflare +`SqlStorage` / `SqlStorageValue` types instead of importing +`@cloudflare/workers-types`. A Worker may supply these globals either through +`wrangler types` (`worker-configuration.d.ts`) or through +`@cloudflare/workers-types`; the library no longer forces one source or silently +degrades SQL handler types to `any` when consumers skip library checks. The +pack smoke test type-checks the packed package with `skipLibCheck: false` in +both setups, including a fresh consumer without workers-types installed. The +previous reference to README as documenting an explicit workers-types install +was incorrect: only the examples included that package. diff --git a/docs/adr/0023-on-demand-watches-acquisitions-holds.md b/docs/adr/0023-on-demand-watches-acquisitions-holds.md index d7267d9..08f5104 100644 --- a/docs/adr/0023-on-demand-watches-acquisitions-holds.md +++ b/docs/adr/0023-on-demand-watches-acquisitions-holds.md @@ -238,3 +238,53 @@ mount for orders 0.9 cannot express as a cursor. fail loud. - SQL `ORDER BY` ignores `nulls` and locale collation, so a bounded snapshot can choose a different top-k than the client's comparator. + +## Amendment — 2026-09-29: fetch flush barrier + +The fetch/coalescer follow-up above is resolved: `#handleFetch` now flushes the +requesting socket's buffered deltas before reading its page, as the subscription +snapshot path already does. A covered load therefore cannot settle with an old +held row while its newer delta remains buffered. The page remains an atomic +read and does not advance the client cursor; the preceding delta boundary +commits the update first. `tests/fetch-flush-barrier.test.ts` pins both the +wire order and the held row's value at load settlement through the real DO, +transport, and on-demand collection. + +## Amendment — 2026-09-29: bounded snapshot NULL placement + +The `nulls` half of the ORDER BY follow-up is fixed: each SQLite sort term +explicitly uses the requested `NULLS FIRST` or `NULLS LAST`, and `NULLS FIRST` +when the clause names none. `@tanstack/db` 0.8.6 and 0.9.2 both default to +`nulls: "first"` in either direction, while SQLite's default puts NULLs last +in a `DESC` sort, so a descending window used to drop its NULL rows. +`tests/order-by-nulls.test.ts` compares the real DO's bounded snapshot with +TanStack's own answer. The locale-collation half remains open (pinned as +`it.fails`), as does top-k selection among NULL ties crossing the limit, which +SQLite does not break by row key as TanStack does. + +Cursor pages are client-built `where` expressions, so this fix does not touch +them. 0.8.6's `buildCursor` ignores `nulls` (plain `gt`/`lt`, `eq` for ties). +0.9.2 builds no cursor at a NULL boundary: a page request becomes a prefix load +(`orderBy` + `limit`, no cursor), and a boundary-tie request loads the full +subset. At a non-NULL boundary with `nulls: "last"`, its `whereFrom` is +`gt(v)` (`lt(v)` descending) `OR isNull OR isUndefined`, and `isNull` is +outside the predicate floor (ADR-0013): the fetch is refused as an empty page +and the window stops growing. Pinned as `it.fails`. + +## Amendment — 2026-09-29: row-key tie-break and the nulls-last cursor + +Two of the open items above are closed. + +- **Ties.** Every bounded read ends its `ORDER BY` with the collection's pk, + ascending whatever the direction. `@tanstack/db` breaks value ties by row key + ascending (`createKeyedComparator` → `compareKeys` in db-ivm; identical in + 0.8.6's db-ivm 0.1.19 and 0.9.2's 0.1.22), comparing strings with `<`. The + term is `COLLATE BINARY` explicitly, since a pk may declare another collation + (NOCASE would order `a` before `B`). BINARY matches JS code-unit order except + for astral versus U+E000–U+FFFF characters (JS compares UTF-16 code units, + SQLite UTF-8 bytes): a known residual divergence. The snapshot, RPC snapshot, and fetch paths all build their + SQL in `compileSubsetQuery`, so one change covers them. + `tests/order-by-ties.test.ts` drives the sub and fetch paths. +- **Nulls-last cursor.** `isNull` and `isUndefined` joined the predicate floor + (ADR-0013 amendment, same date), so 0.9.2's `gt(v) OR isNull OR isUndefined` + cursor compiles and the window grows. diff --git a/docs/adr/0025-empty-op-key-rejection.md b/docs/adr/0025-empty-op-key-rejection.md index caef159..3bf9a4a 100644 --- a/docs/adr/0025-empty-op-key-rejection.md +++ b/docs/adr/0025-empty-op-key-rejection.md @@ -145,3 +145,25 @@ Before this change the same never-settling handler already hung its own frame and kept the DO active. Tests: `tests/inflight-txid.test.ts` (with `GatedTestDO` in the test worker). + +## Amendment — 2026-09-29: supplied row primary key must agree with op key + +For a `mut` insert or update, when `cols` owns the collection's `pk` field, +reject the entire batch with `VALIDATION` if its value differs from `op.key`. +This includes `""`, `null`, and non-string values. Run the gate after the +dedup lookup, before any `authorize` or `execute`, and record the rejection +as a receipt. Do not overwrite `cols[pk]`: ADR-0014's validation is a gate, +not a parser, and rewriting the value would hide a mismatch from the author +while breaking ADR-0001 D9's optimistic-id == confirmed-id contract. An +update that supplies a different pk could likewise move a row under a +different identity in a handler that writes the patch, so it is rejected too. + +An insert whose `cols` omits the pk is not rejected by this gate: no value +exists to compare. ADR-0014 describes a full row for insert, but its schema +validation is opt-in and the author controls `execute`; a handler may use +`op.key` to populate the SQL pk. Requiring `cols[pk]` would change that +contract. The author still owns row completeness and the actual SQL write; +this gate only prevents an explicitly conflicting identity. Delete has no +`cols` and remains covered by the existing non-empty op-key check. + +Tests: `tests/mutation-pk-consistency.test.ts`. diff --git a/scripts/pack-smoke.mjs b/scripts/pack-smoke.mjs index 33417ad..55c6cf9 100644 --- a/scripts/pack-smoke.mjs +++ b/scripts/pack-smoke.mjs @@ -14,8 +14,8 @@ // rendering an SSR snapshot from a Durable Object. The vitest pool cannot // see module-scope restrictions (it evaluates inside a request); this can. // -// The consumer installs @cloudflare/workers-types itself — the documented setup -// (README, examples). Our server .d.ts files import it without declaring it. +// Type-check both supported Worker type setups: generated Wrangler globals +// without workers-types installed, and the explicit workers-types package. import { execFileSync } from "node:child_process" import { cpSync, existsSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs" @@ -66,18 +66,27 @@ try { for (const spec of [...new Set(specs)]) { const dir = join(work, `consumer-${spec.replace(/[^\w.-]/g, "_")}`) - for (const f of ["src", "tsconfig.worker.json", "tsconfig.browser.json", "wrangler.json"]) { + for (const f of ["src", "tsconfig.worker.json", "tsconfig.worker.generated.json", "tsconfig.browser.json", "wrangler.json"]) { cpSync(join(fixture, f), join(dir, f), { recursive: true }) } writeFileSync(join(dir, "package.json"), JSON.stringify({ name: "tddc-smoke-consumer", private: true, type: "module" })) const tag = `[@tanstack/db ${spec}]` step(`${tag} install`, () => { run("npm", ["install", "--no-audit", "--no-fund", "--no-package-lock", tarball, - `@tanstack/db@${spec}`, `@cloudflare/workers-types@${workersTypes}`], dir) + `@tanstack/db@${spec}`], dir) + if (existsSync(join(dir, "node_modules/@cloudflare/workers-types"))) { + throw new Error("generated-types consumer unexpectedly installed @cloudflare/workers-types") + } const got = JSON.parse(readFileSync(join(dir, "node_modules/@tanstack/db/package.json"), "utf8")).version if (/^\d+\.\d+\.\d+$/.test(spec) && got !== spec) throw new Error(`installed @tanstack/db ${got}, wanted ${spec}`) console.log(` resolved @tanstack/db ${got}`) }) + step(`${tag} types: Wrangler-generated Worker program (bundler)`, () => { + run("node", [wranglerBin, "types", "worker-configuration.d.ts", "--config", "wrangler.json"], dir) + typecheck(dir, ["-p", "tsconfig.worker.generated.json"]) + }) + step(`${tag} install @cloudflare/workers-types`, () => + run("npm", ["install", "--no-audit", "--no-fund", "--no-package-lock", `@cloudflare/workers-types@${workersTypes}`], dir)) step(`${tag} types: worker program (bundler)`, () => typecheck(dir, ["-p", "tsconfig.worker.json"])) step(`${tag} types: worker program (nodenext)`, () => typecheck(dir, ["-p", "tsconfig.worker.json", "--module", "nodenext", "--moduleResolution", "nodenext"])) diff --git a/scripts/pack-smoke/src/schema.ts b/scripts/pack-smoke/src/schema.ts index 8dbabd7..78be4e8 100644 --- a/scripts/pack-smoke/src/schema.ts +++ b/scripts/pack-smoke/src/schema.ts @@ -23,7 +23,9 @@ export const chatSchema = sync.schema({ }, execute: ({ op, sql }) => { const m: Message = op.cols - sql.exec("INSERT INTO messages VALUES (?, ?, ?, ?)", m.id, m.author, m.content, m.created_at) + type TypedSql = 0 extends (1 & Sql) ? never : Sql + const typedSql: TypedSql = sql + typedSql.exec("INSERT INTO messages VALUES (?, ?, ?, ?)", m.id, m.author, m.content, m.created_at) }, }, delete: { execute: ({ op, sql }) => void sql.exec("DELETE FROM messages WHERE id = ?", op.key) }, diff --git a/scripts/pack-smoke/tsconfig.worker.generated.json b/scripts/pack-smoke/tsconfig.worker.generated.json new file mode 100644 index 0000000..6d52160 --- /dev/null +++ b/scripts/pack-smoke/tsconfig.worker.generated.json @@ -0,0 +1,6 @@ +{ + "extends": "./tsconfig.worker.json", + "compilerOptions": { + "types": ["./worker-configuration.d.ts"] + } +} diff --git a/src/server/broadcast.ts b/src/server/broadcast.ts index bb26d5f..a8df592 100644 --- a/src/server/broadcast.ts +++ b/src/server/broadcast.ts @@ -4,8 +4,8 @@ // streaming token field) to one delta per tick. // // Hibernation-native: the flush timer is armed on demand by enqueue() and -// cleared by flushAll(). A quiet broadcaster holds no timer, so the DO is free -// to hibernate (held timers would prevent it). +// cleared when the live sockets have no pending work. A quiet broadcaster holds +// no timer, so the DO is free to hibernate (held timers would prevent it). // // The C1 ordering invariant (ADR-0002): the originating socket of a mutation is // flushed via flushOne() BEFORE its `committed` receipt, so its deltas precede @@ -30,7 +30,7 @@ export class Broadcaster { constructor( private readonly rawSend: RawSend, - private readonly tickMs: number = 50, + private readonly tickMs: () => number = () => 50, ) {} /** True while a tick flush is pending — exposed for hibernation assertions. */ @@ -74,11 +74,26 @@ export class Broadcaster { } m.clear() this.rawSend(ws, { t: "uptodate", seq: this.latestCursor }) + this.clearTimerIfIdle() + } + + discard(ws: WebSocket): void { + this.pending.delete(ws) + this.clearTimerIfIdle() } flushAll(wss: Iterable): void { - for (const ws of wss) this.flushOne(ws) this.clearTimer() + for (const ws of wss) this.flushOne(ws) + } + + private clearTimerIfIdle(): void { + if (this.flushTimer && this.getAllWs) { + for (const socket of this.getAllWs()) { + if (this.pending.get(socket)?.size) return + } + this.clearTimer() + } } private armFlush(): void { @@ -87,7 +102,7 @@ export class Broadcaster { this.flushTimer = null const get = this.getAllWs if (get) this.flushAll(get()) - }, this.tickMs) + }, this.tickMs()) } private clearTimer(): void { diff --git a/src/server/changes.ts b/src/server/changes.ts index 4b2c8a5..c7e5875 100644 --- a/src/server/changes.ts +++ b/src/server/changes.ts @@ -9,8 +9,6 @@ // No before-image column: move-in/move-out is computed at emit time from the // current row + the subscription predicate (ADR-0002 C4). -import type { SqlStorage, SqlStorageValue } from "@cloudflare/workers-types" - export const SYNC_PREFIX = "_sync_" export type ChangeOp = "insert" | "update" | "delete" diff --git a/src/server/dedup.ts b/src/server/dedup.ts index 2b5bb23..3e9998e 100644 --- a/src/server/dedup.ts +++ b/src/server/dedup.ts @@ -6,7 +6,6 @@ // current client can still retry an old txId, so this table is swept on its // own time-based horizon (M7), not tied to the compaction floor. -import type { SqlStorage } from "@cloudflare/workers-types" import { decode as decodeValue, encode as encodeValue } from "../wire/codec.ts" export interface SeenTx { @@ -69,7 +68,12 @@ export function decodeResult(stored: string | null): unknown { * Drop dedup entries older than `olderThanMs`. Retention is INDEPENDENT of the * changelog (ADR-0002 C5): sized to the maximum client retry/outbox window, not * the compaction floor — a fully-current client can still retry an old txId. + * + * `ts` is stamped in whole seconds (`unixepoch()*1000`), truncated, so it can + * trail the true write time by up to 999 ms. The extra 1000 ms in the cutoff + * keeps `olderThanMs` a true lower bound (a receipt is never swept before it + * has aged that long); the cost is retention of up to 1 s longer. */ export function sweepDedup(sql: SqlStorage, olderThanMs: number, nowMs: number): void { - sql.exec("DELETE FROM _sync_seen_tx WHERE ts < ?", nowMs - olderThanMs) + sql.exec("DELETE FROM _sync_seen_tx WHERE ts < ?", nowMs - olderThanMs - 1000) } diff --git a/src/server/mixin.ts b/src/server/mixin.ts index 7e6515e..5c82694 100644 --- a/src/server/mixin.ts +++ b/src/server/mixin.ts @@ -20,7 +20,6 @@ // reach SQLite through `this.ctx.storage.sql`. import { DurableObject } from "cloudflare:workers" -import type { SqlStorage, SqlStorageValue } from "@cloudflare/workers-types" import { createFrameCodec, type FrameCodec } from "../wire/frame-codec.ts" import type { ClientFrame, ServerFrame } from "../wire/frames.ts" import { @@ -190,6 +189,10 @@ export function Syncable() { readonly #codec: FrameCodec = createFrameCodec() readonly #subs = new SubscriptionRegistry() #writesSinceCompaction = 0 + /** When this instance last swept expired dedup receipts. In memory on + * purpose: 0 after construction or a hibernation wake, so the first receipt + * then always sweeps (ADR-0009 amendment 2026-09-29). */ + #lastDedupSweepAt = 0 readonly #broadcaster: Broadcaster readonly #liveWs = new Set() /** txIds whose mut/call handler is running now, each mapped to a promise that @@ -214,7 +217,7 @@ export function Syncable() { // ONLY our tagged sockets so the broadcaster never touches a host socket. const restore = this.#isBareDO ? this.ctx.getWebSockets() : this.ctx.getWebSockets(SYNC_TAG) for (const ws of restore) this.#liveWs.add(ws) - this.#broadcaster = new Broadcaster((ws, frame) => this.#send(ws, frame), this.tickMs) + this.#broadcaster = new Broadcaster((ws, frame) => this.#send(ws, frame), () => this.tickMs) this.#broadcaster.start(() => this.#liveWs) const self = this this.#api = { @@ -385,6 +388,7 @@ export function Syncable() { const coll = this.#registry.collections.get(req.collection) if (!coll) throw new Error(`readSyncSnapshot: unknown collection '${req.collection}'`) const query = compileSubsetQuery(req.collection, { + pk: coll.pk, where: req.where, orderBy: req.orderBy, limit: req.limit, @@ -559,6 +563,7 @@ export function Syncable() { * `registerSync` (no `initSchema`, no `_sync_subs`): it can have had no * subscriptions, so there is nothing durable to delete. */ #dropSocketSubs(ws: WebSocket): void { + this.#broadcaster.discard(ws) this.#subs.removeAll(ws) if (this.#compiled) { const sid = this.#socketIdFor(ws) @@ -652,6 +657,7 @@ export function Syncable() { * — a concurrent mutation is either reflected in it or arrives as a delta * AFTER it, never split across the two reads (ADR-0003). */ #handleFetch(ws: WebSocket, frame: Extract): void { + this.#broadcaster.flushOne(ws) const coll = this.#registry.collections.get(frame.collection) if (!coll) { this.#send(ws, { t: "page", fetchId: frame.fetchId, rows: [], seq: "0" }) @@ -675,6 +681,7 @@ export function Syncable() { // No cursor: a plain bounded `where` read. if (frame.cursor != null) { const tq = compileSubsetQuery(frame.collection, { + pk: coll.pk, where: andPredicates(frame.where, frame.cursor.whereCurrent), orderBy: frame.orderBy, }) @@ -682,6 +689,7 @@ export function Syncable() { } const nextWhere = frame.cursor != null ? andPredicates(frame.where, frame.cursor.whereFrom) : frame.where const nq = compileSubsetQuery(frame.collection, { + pk: coll.pk, where: nextWhere, orderBy: frame.orderBy, limit: frame.limit, @@ -727,6 +735,16 @@ export function Syncable() { return this.#rejectTx(ws, f.txId, "mutation op key must be a non-empty string", "VALIDATION") } + // A supplied pk must be the optimistic key, never a different row's + // identity. Check before author handlers; do not rewrite cols (ADR-0025). + const collection = this.#registry.collections.get(f.collection) + if (collection && f.ops.some((op) => + (op.type === "insert" || op.type === "update") && + op.cols != null && Object.hasOwn(op.cols, collection.pk) && op.cols[collection.pk] !== op.key + )) { + return this.#rejectTx(ws, f.txId, "mutation cols primary key must match op key", "VALIDATION") + } + const user = this.#userFor(ws) // Authorize every op BEFORE the transaction (may be async). @@ -856,7 +874,10 @@ export function Syncable() { async #oncePerTx(ws: WebSocket, txId: string, run: () => Promise): Promise { for (;;) { const seen = lookupTx(this.#sql, txId) - if (seen) return this.#replayReceipt(ws, txId, seen) + if (seen) { + this.#replayReceipt(ws, txId, seen) + return this.#maybeSweepDedup() + } const running = this.#inflight.get(txId) if (!running) break await running @@ -869,6 +890,18 @@ export function Syncable() { this.#inflight.delete(txId) settle() } + this.#maybeSweepDedup() + } + + /** Every answered receipt may add a `_sync_seen_tx` row, so the sweep rides + * receipts, after the reply, at most once per `dedupRetentionMs`. An expired + * receipt lives at most one more window. Not a timer: a quiet DO sweeps + * nothing and adds nothing (ADR-0009 amendment 2026-09-29). */ + #maybeSweepDedup(): void { + const now = Date.now() + if (now - this.#lastDedupSweepAt < this.dedupRetentionMs) return + this.#lastDedupSweepAt = now + sweepDedup(this.#sql, this.dedupRetentionMs, now) } #rejectTx(ws: WebSocket, txId: string, message: string, code?: string): void { @@ -972,7 +1005,7 @@ export function Syncable() { /** * Opportunistic GC: every `compactionEvery` drained mutations, collapse the - * change log to latest-op-per-key and sweep expired dedup entries. Deferred + * change log to latest-op-per-key and prune it by age. Deferred * via `ctx.waitUntil` so it rides just after a burst of work — it never * blocks a mutation's response, and (unlike an alarm) never wakes an idle DO. */ @@ -983,7 +1016,6 @@ export function Syncable() { (async (): Promise => { compactChanges(this.#sql) pruneChanges(this.#sql, this.changelogRetentionMs, Date.now()) - sweepDedup(this.#sql, this.dedupRetentionMs, Date.now()) // Orphaned subscription rows (socket died without webSocketClose — // hard termination): hygiene rides compaction, nothing polls // (ADR-0019 D4). Every id-tagged socket carries SYNC_TAG, so the @@ -1024,6 +1056,7 @@ export function Syncable() { let query: { sql: string; params: Array } try { query = compileSubsetQuery(frame.collection, { + pk: coll.pk, where: frame.where, orderBy: frame.orderBy, limit: frame.limit, diff --git a/src/server/registry.ts b/src/server/registry.ts index 7a3cdff..0254d16 100644 --- a/src/server/registry.ts +++ b/src/server/registry.ts @@ -20,7 +20,6 @@ // commands keyed by name) — see `compileSchema`. The wire/dispatch semantics are // unchanged; this file only changes how the author DECLARES them. -import type { SqlStorage } from "@cloudflare/workers-types" import type { MutOp, RowOp } from "../wire/frames.ts" import { SYNC_PREFIX } from "./changes.ts" import type { StandardSchemaV1 } from "./standard-schema.ts" diff --git a/src/server/sql-compiler.ts b/src/server/sql-compiler.ts index e0a1b34..5af3389 100644 --- a/src/server/sql-compiler.ts +++ b/src/server/sql-compiler.ts @@ -90,6 +90,17 @@ function compileExpr(node: unknown, params: Array): string { if (args.length !== 1) throw new UnsupportedPredicateError("'not' expects 1 argument") return `(NOT ${compileExpr(args[0], params)})` } + if (name === "isNull" || name === "isUndefined") { + if (args.length !== 1) throw new UnsupportedPredicateError(`'${name}' expects 1 argument`) + const col = column(args[0]) + // A stored column is NULL, never undefined: @tanstack/db's `isUndefined` + // (=== undefined) is false for every row hydrated from SQLite, and `isNull` + // (=== null) is exactly IS NULL. Needed for the nulls-last cursor's whereFrom + // (`gt OR isNull OR isUndefined`). ADR-0013 amendment 2026-09-29. + // Only a column the table lacks is undefined in JS (true there, false here); the + // schema is author-owned and shared with the client, so that is out of contract. + return name === "isNull" ? `${col} IS NULL` : "0" + } if (name === "in") { const arr = isNode(args[1]) && args[1].type === "val" ? (args[1] as { value: unknown }).value : undefined if (args.length !== 2 || !Array.isArray(arr)) { @@ -104,7 +115,7 @@ function compileExpr(node: unknown, params: Array): string { } throw new UnsupportedPredicateError( `operator '${name}' is not supported for server-side filtering ` + - `(floor: eq, gt, gte, lt, lte, like, in, and, or, not)`, + `(floor: eq, gt, gte, lt, lte, like, in, isNull, isUndefined, and, or, not)`, ) } @@ -125,33 +136,41 @@ export function compileWhere(where: unknown): { sql: string; params: Array } - compareOptions?: { direction?: string } + compareOptions?: { direction?: string; nulls?: string } col?: unknown dir?: unknown } if (o.expression?.type === "ref" && Array.isArray(o.expression.path) && o.expression.path.length > 0) { const path = o.expression.path - return { col: path[path.length - 1]!, desc: o.compareOptions?.direction === "desc" } + const nullsLast = o.compareOptions?.nulls === "last" + return { col: path[path.length - 1]!, desc: o.compareOptions?.direction === "desc", nullsLast } } - if (typeof o.col === "string") return { col: o.col, desc: o.dir === "desc" } + if (typeof o.col === "string") return { col: o.col, desc: o.dir === "desc", nullsLast: false } throw new UnsupportedPredicateError(`unsupported orderBy clause: ${JSON.stringify(item)}`) } -function compileOrderBy(orderBy: unknown): string { +function compileOrderBy(orderBy: unknown, pk: string): string { if (!Array.isArray(orderBy) || orderBy.length === 0) return "" - return orderBy - .map((item) => { - const { col, desc } = orderByTerm(item) - if (!IDENT.test(col)) throw new UnsupportedPredicateError(`invalid orderBy column: ${col}`) - return `"${col}" ${desc ? "DESC" : "ASC"}` - }) - .join(", ") + if (!IDENT.test(pk)) throw new UnsupportedPredicateError(`invalid pk column: ${pk}`) + const terms = orderBy.map((item) => { + const { col, desc, nullsLast } = orderByTerm(item) + if (!IDENT.test(col)) throw new UnsupportedPredicateError(`invalid orderBy column: ${col}`) + return `"${col}" ${desc ? "DESC" : "ASC"} NULLS ${nullsLast ? "LAST" : "FIRST"}` + }) + // Final tie-break by pk, ascending whatever the order direction: @tanstack/db + // breaks value ties by row key ascending (db-ivm `createKeyedComparator`, same in + // db-ivm 0.1.19 and 0.1.22), comparing strings with `<`. Forced to BINARY: the pk + // may declare another collation (NOCASE), which would order `a` before `B`. Without it a LIMIT over + // tied rows picks by scan order and the window differs from the client's own. + terms.push(`"${pk}" COLLATE BINARY ASC`) + return terms.join(", ") } function nonNegInt(n: unknown, label: string): number { @@ -162,6 +181,8 @@ function nonNegInt(n: unknown, label: string): number { } export interface SubsetQuery { + /** The collection's pk column: the final ORDER BY tie-break. */ + pk: string where?: unknown orderBy?: unknown limit?: number @@ -180,7 +201,7 @@ export function compileSubsetQuery(tbl: string, opts: SubsetQuery): { sql: strin params.push(...w.params) } - const orderBy = compileOrderBy(opts.orderBy) + const orderBy = compileOrderBy(opts.orderBy, opts.pk) if (orderBy) { sql += ` ORDER BY ${orderBy}` } else { diff --git a/src/server/subscriptions.ts b/src/server/subscriptions.ts index 9bb4192..4d7f87b 100644 --- a/src/server/subscriptions.ts +++ b/src/server/subscriptions.ts @@ -10,7 +10,6 @@ // client's operator semantics exactly (no second predicate implementation). import { compileSingleRowExpression, toBooleanPredicate } from "@tanstack/db" -import type { SqlStorage } from "@cloudflare/workers-types" import { decode as decodeValue, encode as encodeValue } from "../wire/codec.ts" import { UnsupportedPredicateError } from "./sql-compiler.ts" diff --git a/src/server/sync-do.ts b/src/server/sync-do.ts index 353c1de..62984f0 100644 --- a/src/server/sync-do.ts +++ b/src/server/sync-do.ts @@ -15,7 +15,6 @@ // non-trivial host reach `this.ctx.storage.sql` directly (ADR-0015). import { DurableObject } from "cloudflare:workers" -import type { SqlStorage } from "@cloudflare/workers-types" import { Syncable } from "./mixin.ts" import type { CompiledSync, SyncSchema } from "./registry.ts" diff --git a/tests/broadcaster.test.ts b/tests/broadcaster.test.ts index 4d6bf1e..41a884a 100644 --- a/tests/broadcaster.test.ts +++ b/tests/broadcaster.test.ts @@ -11,7 +11,7 @@ import type { ServerFrame } from "../src/wire/frames.ts" function setup() { const sent: Array<{ ws: unknown; frame: ServerFrame }> = [] - const b = new Broadcaster((ws, frame) => sent.push({ ws, frame }), 50) + const b = new Broadcaster((ws, frame) => sent.push({ ws, frame }), () => 50) const ws = {} as unknown as WebSocket return { b, ws, sent } } diff --git a/tests/cursor-known-limitations.test.ts b/tests/cursor-known-limitations.test.ts new file mode 100644 index 0000000..357eb1f --- /dev/null +++ b/tests/cursor-known-limitations.test.ts @@ -0,0 +1,184 @@ +import { env, evictDurableObject, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { type SubHandler, WebSocketTransport, type WebSocketLike } from "../src/client/transport.ts" + +// WHY: two ACCEPTED limitations of the single cursor, pinned as `it.fails` so a +// future fix has a target. Both end the same way: the client's cursor claims a +// position whose changes it never applied, and nothing ever repairs it. +// +// - L2 (ADR-0009 "Out of scope"): the DO is torn down while a delta sits in the +// coalescer and the hibernatable socket survives. No reconnect fires, and the +// next delivery advances the cursor past the lost write (no contiguity check). +// - L1 (ADR-0011 "Known limitations"): no incarnation epoch. After a storage +// reset the new change log reuses seqs; a client cursor from before the reset +// passes the retention floor check and catches up silently wrong. +// +// Each asserts convergence (client rows == DO rows), not a mechanism, so any +// fix (a delivered delta, a detected gap + resnapshot, an epoch reset) passes. + +const stubFor = (r: string) => env.SYNC_DO.get(env.SYNC_DO.idFromName(r)) +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((r) => setTimeout(r, 5)) + } +} +/** Poll up to `ms` for `pred`, without failing: the assertion that follows says what diverged. */ +async function settle(pred: () => boolean, ms = 500): Promise { + await waitFor(pred, ms).catch(() => {}) +} +async function serverWrite(r: string, sql: string, ...args: Array): Promise { + await runInDurableObject(stubFor(r), (instance, s) => { + s.storage.sql.exec(sql, ...args) + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + }) +} +async function serverIds(r: string): Promise> { + return runInDurableObject(stubFor(r), (_i, s) => + Array.from(s.storage.sql.exec<{ id: string }>("SELECT id FROM messages ORDER BY id")).map((x) => x.id), + ) +} + +/** A raw sub that keeps the rows it was told about. */ +function rowSink(): { rows: Map; terminals: Array; handler: SubHandler } { + const rows = new Map() + const terminals: Array = [] + return { + rows, + terminals, + handler: { + onSnap: (k, row) => rows.set(String(k), row), + onSnapEnd: () => terminals.push("snap-end"), + onDelta: (op, k, cols) => (op === "delete" ? rows.delete(String(k)) : rows.set(String(k), cols)), + onUptodate: (own) => terminals.push(own ? "uptodate(own)" : "uptodate"), + onReset: () => { + terminals.push("reset") + rows.clear() + }, + }, + } +} + +/** Limitation pins that got as far as their convergence check (see the last test). */ +const reached = new Set() + +describe("single-cursor known limitations", () => { + // ADR-0009 "Out of scope": mid-tick eviction dropping pending deltas. + it.fails("L2: a write buffered when the DO is torn down still reaches the connected client", async () => { + const r = `ckl-l2-${crypto.randomUUID()}` + let closed = false + const t = new WebSocketTransport({ + url: `https://example.com/sync/${r}`, + open: async () => { + const res = await SELF.fetch(`https://example.com/sync/${r}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket! + ws.accept() + ws.addEventListener("close", () => (closed = true)) + return ws as unknown as WebSocketLike + }, + }) + try { + const sink = rowSink() + await t.subscribe("s", "messages", sink.handler) + await waitFor(() => sink.terminals.includes("snap-end")) + + // SIMULATION, not a production path: we cannot make workerd tear the DO + // down inside the ≤tickMs window, so we stand in for "died before its + // armed tick fired" by handing the coalescer a flush timer that never + // fires (swallowed setTimeout), then evict. `evictDurableObject` drops + // the instance (and the buffered delta with it) and, by default + // (`webSockets: "hibernate"`), keeps the socket open. Whether production + // ever evicts with an armed timer while keeping hibernatable sockets is + // a platform question (bugbash report A, L2). + await runInDurableObject(stubFor(r), (instance, s) => { + s.storage.sql.exec("INSERT INTO messages(id,body) VALUES('x','buffered')") + const real = globalThis.setTimeout + ;(globalThis as { setTimeout: unknown }).setTimeout = () => 0 + try { + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + } finally { + globalThis.setTimeout = real + } + }) + await evictDurableObject(stubFor(r)) + // Premise: x was still buffered when the instance died. A fix that removes + // the buffering window altogether makes L2 moot and fails here: then + // retire this pin rather than weaken it. + expect(sink.rows.has("x")).toBe(false) + expect(closed).toBe(false) // premise: the socket survived, so no reconnect will fire + + // Delivery resumes on the surviving socket: a later write reaches it. + await serverWrite(r, "INSERT INTO messages(id,body) VALUES('y','later')") + await waitFor(() => sink.rows.has("y")) + const server = await serverIds(r) + await settle(() => JSON.stringify([...sink.rows.keys()].sort()) === JSON.stringify(server)) + + const client = [...sink.rows.keys()].sort() + reached.add("L2") + expect(client).toEqual(server) + } finally { + t.close() + } + }) + + // ADR-0011 "Known limitations": no incarnation epoch. + it.fails("L1: a reconnect after a storage reset does not keep the old incarnation's rows", async () => { + const r = `ckl-l1-${crypto.randomUUID()}` + let gate: Promise = Promise.resolve() + let release: () => void = () => {} + const t = new WebSocketTransport({ + url: `https://example.com/sync/${r}`, + reconnectDelay: () => 10, + open: async () => { + await gate + const res = await SELF.fetch(`https://example.com/sync/${r}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket! + ws.accept() + return ws as unknown as WebSocketLike + }, + }) + try { + const sink = rowSink() + for (let i = 1; i <= 5; i++) await serverWrite(r, "INSERT INTO messages(id,body) VALUES(?, 'gen1')", `old${i}`) + await t.subscribe("s", "messages", sink.handler) + await waitFor(() => sink.terminals.includes("snap-end")) + expect(sink.rows.size).toBe(5) + const cursor = BigInt(t.appliedCursor) + + // Drop; hold the reconnect while storage is reset and repopulated. + gate = new Promise((res) => (release = res)) + await runInDurableObject(stubFor(r), (_i, s) => { + for (const w of s.getWebSockets()) w.close(1000, "drop") + }) + await waitFor(() => (t as unknown as { ws: unknown }).ws === null) + await runInDurableObject(stubFor(r), async (_i, s) => { + await s.storage.deleteAll() // an operator reset; a point-in-time restore has the same shape + }) + await evictDurableObject(stubFor(r)) // the next access re-runs the constructor on empty storage + for (let i = 1; i <= 7; i++) await serverWrite(r, "INSERT INTO messages(id,body) VALUES(?, 'gen2')", `new${i}`) + const server = await serverIds(r) + const maxSeq = await runInDurableObject(stubFor(r), (_i, s) => + BigInt(Array.from(s.storage.sql.exec<{ m: number }>("SELECT max(seq) AS m FROM _sync_changes"))[0]!.m), + ) + expect(maxSeq).toBeGreaterThan(cursor) // premise: the new log reaches past the old cursor + + const mark = sink.terminals.length + release() + await waitFor(() => sink.terminals.slice(mark).some((e) => e === "uptodate(own)" || e === "snap-end")) + await settle(() => JSON.stringify([...sink.rows.keys()].sort()) === JSON.stringify(server)) + + const client = [...sink.rows.keys()].sort() + reached.add("L1") + expect(client).toEqual(server) + } finally { + t.close() + } + }) + + it("harness: every limitation pin ran to its convergence check", () => { + // A pin that threw before its convergence check passed `it.fails` for the + // wrong reason; this is what catches it. + expect([...reached].sort()).toEqual(["L1", "L2"]) + }) +}) diff --git a/tests/cursor-resubscribe-round.test.ts b/tests/cursor-resubscribe-round.test.ts new file mode 100644 index 0000000..4622292 --- /dev/null +++ b/tests/cursor-resubscribe-round.test.ts @@ -0,0 +1,412 @@ +import { createCollection, createLiveQueryCollection, gte, lt } from "@tanstack/db" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { doCollectionOptions } from "../src/client/do-collection.ts" +import { WebSocketTransport, type WebSocketLike } from "../src/client/transport.ts" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ClientFrame, ServerFrame } from "../src/wire/frames.ts" +import type { TestApi } from "./test-worker.ts" + +// WHY: after a reconnect, every bootstrapped sub resumes from the ONE shared +// cursor, and the DO answers each with its own catch-up ending in a sub-scoped +// `uptodate` at the CURRENT seq (which covers every table). Any cursor advance +// in the middle of that resubscribe round claims positions that the subs still +// waiting for their catch-up never applied. If the socket drops right there, +// the next round resumes those subs past their offline changes, and the loss is +// permanent and silent (bugbash F1; ADR-0002's single cursor, ADR-0016's +// resubscribe, ADR-0023 D7 covers only subs that never bootstrapped). +// +// The suite is the target for the fix, not a design for it: one transport +// carries four subs (eager `messages`, eager `files`, and two on-demand watches +// on `messages` with disjoint predicates). While the socket is down the DO +// applies a change set that only some subs see. The reconnect round is then cut +// at one boundary (before any frame, mid a sub's catch-up, after a sub's +// terminal, after a live broadcast boundary), and after one clean reconnect +// every collection must equal the DO's rows. Scenarios the single cursor gets +// wrong today are `it.fails` (bugbash F1). A correct fix turns them all green; +// then drop the `isRed` switch and run every scenario as a plain `it`. + +const codec = createFrameCodec() +const room = (): string => `crr-${crypto.randomUUID()}` +const stubFor = (r: string) => env.SYNC_DO.get(env.SYNC_DO.idFromName(r)) +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((r) => setTimeout(r, 5)) + } +} +/** A server-originated synced write: capture, then drain + broadcast (ADR-0006). */ +async function serverExec(r: string, statements: Array): Promise { + await runInDurableObject(stubFor(r), (instance, s) => { + for (const sql of statements) s.storage.sql.exec(sql) + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + }) +} + +// --- A socket we can cut at a chosen frame --------------------------------- + +/** What one opened socket did: the subs it sent (in order, which is the order + * the DO serves their catch-ups) and the frames it delivered. */ +interface Sock { + subs: Array<{ subId: string; collection: string }> + frames: Array + dropped: boolean +} +/** Where to cut a socket. `at` sees each inbound frame before delivery: "after" + * delivers it and then drops, "before" drops without delivering it. A drop + * loses a SUFFIX of what the client sent (the DO only ever processes a + * prefix), so `muteSubsFrom` loses every outbound frame from that `sub` on. */ +interface Cut { + at: (f: ServerFrame, s: Sock) => "before" | "after" | false + muteSubsFrom?: number +} + +function wrap(ws: WebSocket, s: Sock, cut: Cut | undefined): WebSocketLike { + const onMessage: Array<(ev: { data?: unknown }) => void> = [] + const onClose: Array<(ev: { data?: unknown }) => void> = [] + let muted = false + const drop = (): void => { + if (s.dropped) return + s.dropped = true + try { + ws.close(1000, "drop") + } catch { + /* already closed */ + } + for (const l of onClose) l({ code: 1006 } as { data?: unknown }) + } + ws.addEventListener("message", (e) => { + if (s.dropped) return + const data = (e as MessageEvent).data + const f = codec.decode(data as ArrayBuffer) as ServerFrame + const where = cut?.at(f, s) ?? false + if (where === "before") return drop() + s.frames.push(f) + for (const l of onMessage) l({ data }) + if (where === "after") drop() + }) + ws.addEventListener("close", () => drop()) + return { + send: (d) => { + if (s.dropped || muted) return + const f = codec.decode(d as ArrayBuffer) as ClientFrame + if (f.t === "sub") { + s.subs.push({ subId: f.subId, collection: f.collection }) + if (cut?.muteSubsFrom !== undefined && s.subs.length > cut.muteSubsFrom) { + muted = true + return + } + } + ws.send(d as ArrayBuffer) + }, + close: () => drop(), + addEventListener: (type, l) => { + if (type === "message") onMessage.push(l) + else if (type === "close") onClose.push(l) + }, + removeEventListener: () => {}, + } +} + +// --- Topology, change sets, boundaries ------------------------------------- + +/** The resubscribe round, in the order the transport re-sends the subs. */ +const ROUND = ["eager messages", "eager files", "on-demand a*", "on-demand c*"] as const +const TABLE_OF = ["messages", "files", "messages", "messages"] as const +/** The two on-demand watches: `lt(body, "b")` (a*) and `gte(body, "c")` (c*); b* is in neither. */ +const inA = (body: string): boolean => body < "b" +const inC = (body: string): boolean => body >= "c" + +const SEED = [ + "INSERT INTO messages(id,body) VALUES('m1','a1'),('m2','a2'),('m3','b3'),('m4','b4'),('m5','c5'),('m6','c6')", + "INSERT INTO files(id,name) VALUES('f1','one'),('f2','two'),('f3','three')", +] + +interface ChangeSet { + sql: Array + /** ROUND indexes whose rows the change set changes (the subs that must see it). */ + affects: Array +} +const CHANGE_SETS: Record = { + // Every sub sees something: insert, update, delete, and moves in and out of + // both on-demand subsets (a→c, b→a, c→b); files insert/update/delete. + mixed: { + sql: [ + "INSERT INTO messages(id,body) VALUES('m7','a7')", + "UPDATE messages SET body='a1x' WHERE id='m1'", + "DELETE FROM messages WHERE id='m5'", + "UPDATE messages SET body='c2' WHERE id='m2'", + "UPDATE messages SET body='a3' WHERE id='m3'", + "UPDATE messages SET body='b6' WHERE id='m6'", + "INSERT INTO files(id,name) VALUES('f4','four')", + "UPDATE files SET name='one!' WHERE id='f1'", + "DELETE FROM files WHERE id='f2'", + ], + affects: [0, 1, 2, 3], + }, + // Only the second sub sees anything; the others' catch-ups are empty. + "files only": { + sql: [ + "INSERT INTO files(id,name) VALUES('f4','four')", + "UPDATE files SET name='one!' WHERE id='f1'", + "DELETE FROM files WHERE id='f2'", + ], + affects: [1], + }, + // Only moves between the on-demand subsets (plus the eager mirror of them). + "on-demand moves": { + sql: [ + "UPDATE messages SET body='c2' WHERE id='m2'", // out of a*, into c* + "UPDATE messages SET body='b6' WHERE id='m6'", // out of c* + "UPDATE messages SET body='a3' WHERE id='m3'", // into a* + ], + affects: [0, 2, 3], + }, +} +/** A live write that lands mid-round (only the first sub is registered by then). */ +const LIVE_WRITE = ["INSERT INTO messages(id,body) VALUES('m8','a8')"] + +type Boundary = + | { kind: "none" } + | { kind: "before-any" } + | { kind: "mid"; i: number } + | { kind: "after"; i: number } + | { kind: "broadcast" } + +const own = (f: ServerFrame, subId: string | undefined): boolean => + subId !== undefined && (f.t === "uptodate" || f.t === "snap-end") && f.sub === subId + +function cutFor(b: Boundary): Cut | undefined { + switch (b.kind) { + case "none": + return undefined + case "before-any": + return { at: () => "before" } + case "mid": + // after the FIRST row of sub i's catch-up (each applicable one has >1), + // or of its snapshot, should a repair resnapshot instead + return { at: (f, s) => ((f.t === "d" || f.t === "snap") && f.sub === s.subs[b.i]?.subId ? "after" : false) } + case "after": + return { at: (f, s) => (own(f, s.subs[b.i]?.subId) ? "after" : false) } + case "broadcast": + // Sub 0's `sub` reached the DO, the rest were still in flight when the + // socket died. A live write then reaches sub 0 as a coalescer tick + // (`d` + broadcast `uptodate`), and the socket drops after that boundary. + return { at: (f) => (f.t === "uptodate" && f.sub === undefined ? "after" : false), muteSubsFrom: 1 } + } +} + +/** Does the single cursor lose changes here today? A cut loses a sub's + * catch-up only once something earlier in the round advanced the cursor. */ +function isRed(b: Boundary, set: ChangeSet): boolean { + switch (b.kind) { + case "none": + case "before-any": + return false + case "mid": + return b.i > 0 && set.affects.some((j) => j >= b.i) + case "after": + return set.affects.some((j) => j > b.i) + case "broadcast": + return true // the live write (a8) is itself a change the unfinished a* watch misses + } +} + +function label(b: Boundary): string { + switch (b.kind) { + case "none": + return "no mid-round drop (control)" + case "before-any": + return "drop before any catch-up frame" + case "mid": + return `drop mid ${ROUND[b.i]}'s catch-up` + case "after": + return `drop after ${ROUND[b.i]}'s catch-up terminal` + case "broadcast": + return `drop after a live broadcast boundary, mid-round` + } +} + +// --- Scenario --------------------------------------------------------------- + +type Row = { id: string; v: string } +const sorted = (rows: Array): Array => [...rows].sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)) + +/** Red scenarios that got as far as their convergence check. A red scenario + * that fails EARLIER (a broken harness, a boundary that never fired) would + * otherwise be indistinguishable from the bug under `it.fails`. */ +const reached = new Set() +const redNames: Array = [] + +async function scenario(name: string, setName: string, b: Boundary): Promise { + const set = CHANGE_SETS[setName]! + const r = room() + const sockets: Array = [] + const cuts = new Map() + let gate: Promise = Promise.resolve() + let release: () => void = () => {} + const t = new WebSocketTransport({ + url: `https://example.com/sync/${r}`, + reconnectDelay: () => 5, + open: async () => { + await gate + const res = await SELF.fetch(`https://example.com/sync/${r}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket! + ws.accept() + const s: Sock = { subs: [], frames: [], dropped: false } + const cut = cuts.get(sockets.length) + sockets.push(s) + return wrap(ws, s, cut) + }, + }) + try { + await serverExec(r, SEED) + const em = createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (m) => m.id })) + const ef = createCollection(doCollectionOptions({ transport: t, table: "files", getKey: (f) => f.id })) + const od = createCollection( + doCollectionOptions({ transport: t, table: "messages", getKey: (m) => m.id, syncMode: "on-demand" }), + ) + const qa = createLiveQueryCollection((q) => q.from({ m: od }).where(({ m }) => lt(m.body, "b"))) + const qc = createLiveQueryCollection((q) => q.from({ m: od }).where(({ m }) => gte(m.body, "c"))) + // Sequential, so the transport's handler order (the round's order) is ROUND. + await em.preload() + await ef.preload() + await qa.preload() + await qc.preload() + + const client = (): Record> => ({ + [ROUND[0]]: sorted(em.toArray.map((m) => ({ id: m.id, v: m.body }))), + [ROUND[1]]: sorted(ef.toArray.map((f) => ({ id: f.id, v: f.name }))), + [ROUND[2]]: sorted(qa.toArray.map((m) => ({ id: m.id, v: m.body }))), + [ROUND[3]]: sorted(qc.toArray.map((m) => ({ id: m.id, v: m.body }))), + "on-demand held rows": sorted(od.toArray.map((m) => ({ id: m.id, v: m.body }))), + }) + const server = async (): Promise>> => { + const [msgs, files] = await runInDurableObject(stubFor(r), (_i, s) => [ + Array.from(s.storage.sql.exec<{ id: string; body: string }>("SELECT id, body FROM messages")).map((x) => ({ + id: x.id, + v: x.body, + })), + Array.from(s.storage.sql.exec<{ id: string; name: string }>("SELECT id, name FROM files")).map((x) => ({ + id: x.id, + v: x.name, + })), + ]) + return { + [ROUND[0]]: sorted(msgs), + [ROUND[1]]: sorted(files), + [ROUND[2]]: sorted(msgs.filter((m) => inA(m.v))), + [ROUND[3]]: sorted(msgs.filter((m) => inC(m.v))), + "on-demand held rows": sorted(msgs.filter((m) => inA(m.v) || inC(m.v))), + } + } + const converged = async (): Promise => JSON.stringify(client()) === JSON.stringify(await server()) + await waitFor(() => sockets[0]!.subs.length === 4) + expect(sockets[0]!.subs.map((s) => s.collection)).toEqual([...TABLE_OF]) + expect(client()).toEqual(await server()) // bootstrapped, converged + + // Drop; hold the reconnect until the offline changes are in. + gate = new Promise((res) => (release = res)) + await runInDurableObject(stubFor(r), (_i, s) => { + for (const w of s.getWebSockets()) w.close(1000, "drop") + }) + await waitFor(() => sockets[0]!.dropped) + await serverExec(r, set.sql) + const cut = cutFor(b) + if (cut) cuts.set(1, cut) + release() + + // The round on socket 1, cut at the boundary (or not). + await waitFor(() => sockets.length >= 2) + const round = sockets[1]! + if (b.kind === "broadcast") { + await waitFor(() => round.frames.some((f) => own(f, round.subs[0]?.subId))) + await serverExec(r, LIVE_WRITE) + } + if (b.kind === "none") { + await waitFor(() => round.subs.length === 4 && round.subs.every((s) => round.frames.some((f) => own(f, s.subId)))) + } else { + await waitFor(() => round.dropped) + } + // The cut landed where the scenario says, judged by what was delivered — + // not by the cursor, which a fix may well hold back. + const last = round.frames.at(-1) as { t: string; sub?: string } | undefined + const ownDelivered = (i: number): boolean => round.frames.some((f) => own(f, round.subs[i]?.subId)) + switch (b.kind) { + case "before-any": + expect(round.frames).toEqual([]) + break + case "mid": + expect([last?.t === "d" || last?.t === "snap", last?.sub, ownDelivered(b.i)]).toEqual([true, round.subs[b.i]!.subId, false]) + break + case "after": + expect(own(last as ServerFrame, round.subs[b.i]!.subId)).toBe(true) + break + case "broadcast": + expect([last?.t, last?.sub, [1, 2, 3].some(ownDelivered)]).toEqual(["uptodate", undefined, false]) + break + } + + // One clean reconnect: wait for every sub's own terminal on it. + if (b.kind !== "none") await waitFor(() => sockets.length >= 3) + const final = sockets[b.kind === "none" ? 1 : 2]! + await waitFor(() => final.subs.length === 4 && final.subs.every((s) => final.frames.some((f) => own(f, s.subId)))) + // Convergence is eventual (live queries recompute after the commit); bound it. + const deadline = Date.now() + 250 + while (!(await converged()) && Date.now() < deadline) await new Promise((res) => setTimeout(res, 10)) + expect(final.dropped).toBe(false) + expect(sockets.length).toBe(b.kind === "none" ? 2 : 3) + + const got = client() + const want = await server() + reached.add(name) + expect(got).toEqual(want) + + // Holding the cursor back forever would also converge (every round replays + // from the start): the round must release it, and live delivery resume. + const seqNow = await runInDurableObject(stubFor(r), (_i, s) => + String(Array.from(s.storage.sql.exec<{ m: number }>("SELECT max(seq) AS m FROM _sync_changes"))[0]!.m), + ) + expect(t.appliedCursor).toBe(seqNow) + await serverExec(r, ["INSERT INTO files(id,name) VALUES('f9','live')"]) + await waitFor(() => ef.has("f9")) + await waitFor(() => BigInt(t.appliedCursor) > BigInt(seqNow)) + } finally { + t.close() + } +} + +describe("single cursor across a resubscribe round cut by a drop (bugbash F1)", () => { + for (const [setName, set] of Object.entries(CHANGE_SETS)) { + const boundaries: Array = [ + { kind: "none" }, + { kind: "before-any" }, + ...ROUND.map((_, i) => ({ kind: "after" as const, i })), + // A mid-catch-up cut needs a catch-up with more than one delta: only + // subs on a table the change set touches have one. + ...ROUND.map((_, i) => ({ kind: "mid" as const, i })).filter((m) => + set.sql.some((s) => s.includes(` ${TABLE_OF[m.i]}`)), + ), + { kind: "broadcast" }, + ] + describe(`offline change set: ${setName}`, () => { + for (const b of boundaries) { + const name = `${setName}: ${label(b)}` + if (isRed(b, set)) { + redNames.push(name) + // bugbash F1: a cursor advance mid-round skips the unfinished subs' catch-up. + it.fails(`${label(b)}: every collection converges to the DO`, () => scenario(name, setName, b)) + } else { + it(`${label(b)}: every collection converges to the DO`, () => scenario(name, setName, b)) + } + } + }) + } + + it("harness: every red scenario ran to its convergence check", () => { + // A red scenario that threw before its convergence check passed `it.fails` + // for the wrong reason; this is what catches it. + expect(redNames.filter((n) => !reached.has(n))).toEqual([]) + }) +}) diff --git a/tests/dedup-identity-scope.test.ts b/tests/dedup-identity-scope.test.ts new file mode 100644 index 0000000..2d7b2ea --- /dev/null +++ b/tests/dedup-identity-scope.test.ts @@ -0,0 +1,90 @@ +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ClientFrame, ServerFrame } from "../src/wire/frames.ts" + +// bugbash L3 — accepted limitation, deferred by ADR-0012 D4 ("Dedup identity +// binding deferred"). WHY: `_sync_seen_tx` is keyed by txId alone, so the +// receipt belongs to whichever socket presented the txId FIRST, whoever that +// was. Two halves, both `it.fails` until an identity-scoped dedup lands. They +// encode the scoping D4 sketches (a `dedupScope(user)` hook): another identity's +// txId is simply new in your scope, so your frame runs. A design that instead +// refuses cross-identity collisions would need to re-state them. +// 1. Read (the half D4 names): another identity presenting your txId gets +// your stored receipt, including a command's result. +// 2. Pre-claim (wider than D4's text): another identity that uses a txId +// FIRST makes your later write with it answer `committed` without running. +// Both need the txId; built-in clients mint CSPRNG UUIDs and no peer frame +// carries one, which is why D4 rates the risk low. Predictable app-supplied +// transaction ids, or txIds leaked through telemetry, raise it. + +const codec = createFrameCodec() + +async function openAs(room: string, user: string): Promise { + const res = await SELF.fetch(`https://example.com/sync/${room}`, { headers: { Upgrade: "websocket", "x-user": user } }) + const ws = res.webSocket! + ws.accept() + return ws +} + +function receipt(ws: WebSocket, f: ClientFrame & { txId: string }): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error(`no receipt for ${f.txId}`)), 2000) + const onMsg = (e: MessageEvent): void => { + const x = codec.decode(e.data as ArrayBuffer) as ServerFrame + if ((x.t === "committed" || x.t === "rejected") && x.txId === f.txId) { + clearTimeout(timer) + ws.removeEventListener("message", onMsg) + resolve(x) + } + } + ws.addEventListener("message", onMsg) + ws.send(codec.encode(f)) + }) +} + +const rowIds = (room: string): Promise> => + runInDurableObject(env.SYNC_DO.get(env.SYNC_DO.idFromName(room)), (_i, s) => + Array.from(s.storage.sql.exec<{ id: string }>("SELECT id FROM messages ORDER BY id")).map((r) => r.id), + ) + +describe("dedup is scoped to the identity that owns the txId (ADR-0012 D4, deferred)", () => { + it("control: the SAME identity retrying a txId gets its stored result replayed", async () => { + const room = `l3-control-${crypto.randomUUID()}` + const alice = await openAs(room, "alice") + const alice2 = await openAs(room, "alice") // a reconnect: a new socket, same identity + await receipt(alice, { t: "call", txId: "tx-1", name: "echo", args: { secret: "first" } }) + const retry = await receipt(alice2, { t: "call", txId: "tx-1", name: "echo", args: { secret: "second" } }) + alice.close() + alice2.close() + expect(retry).toMatchObject({ t: "committed", result: { echoed: { secret: "first" } } }) + }) + + // bugbash L3 (read half) — ADR-0012 D4. + it.fails("another identity presenting your txId does not receive your stored result", async () => { + const room = `l3-read-${crypto.randomUUID()}` + const alice = await openAs(room, "alice") + const bob = await openAs(room, "bob") + await receipt(alice, { t: "call", txId: "tx-shared", name: "echo", args: { secret: "alice-only" } }) + const b = await receipt(bob, { t: "call", txId: "tx-shared", name: "echo", args: { secret: "bob" } }) + alice.close() + bob.close() + // today: bob gets {committed, result:{echoed:{secret:"alice-only"}}} + expect(b).toMatchObject({ t: "committed", result: { echoed: { secret: "bob" } } }) + }) + + // bugbash L3 (pre-claim half) — wider than ADR-0012 D4's text. + it.fails("another identity using a txId first does not make your later write report committed unexecuted", async () => { + const room = `l3-preclaim-${crypto.randomUUID()}` + const alice = await openAs(room, "alice") + const bob = await openAs(room, "bob") + await receipt(bob, { t: "mut", txId: "tx-pre", collection: "messages", ops: [{ type: "insert", key: "bob-row", cols: { id: "bob-row", body: "b" } }] }) + const a = await receipt(alice, { t: "mut", txId: "tx-pre", collection: "messages", ops: [{ type: "insert", key: "alice-row", cols: { id: "alice-row", body: "a" } }] }) + const rows = await rowIds(room) + alice.close() + bob.close() + // today: alice gets bob's `committed` receipt and rows are ["bob-row"]. + expect(a.t).toBe("committed") + expect(rows).toEqual(["alice-row", "bob-row"]) + }) +}) diff --git a/tests/dedup-retention-bound.test.ts b/tests/dedup-retention-bound.test.ts new file mode 100644 index 0000000..5ba08db --- /dev/null +++ b/tests/dedup-retention-bound.test.ts @@ -0,0 +1,48 @@ +import { env, runInDurableObject } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { recordTx, sweepDedup } from "../src/server/dedup.ts" + +// WHY (bugbash F7 cutoff, ADR-0002 C5 / ADR-0021): a receipt is stamped with +// `unixepoch()*1000` (whole seconds, truncated), so its stamp can be up to 999 ms +// EARLIER than the true write time. Sweeping on `ts < now - retention` then deletes +// a receipt whose true age is as little as `retention - 999 ms`, breaking the +// promise that a retry inside `dedupRetentionMs` replays instead of re-executing. +// Retention must be a true lower bound. The times are passed in, so this is +// deterministic; the stamp itself comes from the real `recordTx`. + +const RETENTION = 3_600_000 + +async function stampOf(txId: string, then: (sql: SqlStorage, ts: number) => void): Promise { + const stub = env.SYNC_DO.get(env.SYNC_DO.idFromName(`dedup-bound-${crypto.randomUUID()}`)) + await runInDurableObject(stub, (_i, state) => { + const sql = state.storage.sql + recordTx(sql, txId, false, null, "x", null, null) + const ts = sql.exec<{ ts: number }>("SELECT ts FROM _sync_seen_tx WHERE tx_id = ?", txId).one().ts + then(sql, ts) + }) +} + +const present = (sql: SqlStorage, txId: string): boolean => + sql.exec("SELECT 1 FROM _sync_seen_tx WHERE tx_id = ?", txId).toArray().length > 0 + +describe("dedup sweep keeps retention a true lower bound (bugbash F7)", () => { + it("stamps whole seconds, so the true write time is anywhere in [ts, ts + 999 ms]", async () => { + await stampOf("whole-seconds", (_sql, ts) => expect(ts % 1000).toBe(0)) + }) + + it("does not sweep a receipt whose true age is under retention, even at worst-case truncation", async () => { + await stampOf("young", (sql, ts) => { + const trueWrite = ts + 999 // worst case: stamped 999 ms before it happened + sweepDedup(sql, RETENTION, trueWrite + RETENTION - 1) // true age: retention - 1 ms + expect(present(sql, "young")).toBe(true) + }) + }) + + it("does sweep once the true age exceeds retention at best-case stamping, by at most one second more", async () => { + await stampOf("old", (sql, ts) => { + // Best case: stamped exactly when it happened. A second past retention it must go. + sweepDedup(sql, RETENTION, ts + RETENTION + 1000 + 1) + expect(present(sql, "old")).toBe(false) + }) + }) +}) diff --git a/tests/dedup-sweep.test.ts b/tests/dedup-sweep.test.ts new file mode 100644 index 0000000..5cf243a --- /dev/null +++ b/tests/dedup-sweep.test.ts @@ -0,0 +1,167 @@ +import { env, evictDurableObject, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ClientFrame, ServerFrame } from "../src/wire/frames.ts" + +// WHY (bugbash F7, ADR-0009 amendment 2026-09-29): every answered `mut`/`call` +// adds a `_sync_seen_tx` row, so the sweep must ride answered receipts, not +// drained writes. A DO that only rejects, replays, or runs write-free commands +// otherwise grows the table forever. The gate is time-based and held in memory: +// at most one sweep per `dedupRetentionMs`, and the first receipt after a +// construction or hibernation wake always sweeps. These tests drive real +// sockets and poll the table for the effect (ADR-0009's lesson), never calling +// `sweepDedup` directly. +// +// SYNC_DO keeps the shipped 1 h retention, so "the gate is closed" is a +// race-free assertion there. MAINT_DO overrides retention to 1 s so the gate +// can be seen to reopen with time alone. + +const codec = createFrameCodec() +const HOUR = 3_600_000 +const stubFor = (path: "sync" | "maint", room: string) => { + const ns = path === "sync" ? env.SYNC_DO : env.MAINT_DO + return ns.get(ns.idFromName(room)) +} + +async function openWs(path: "sync" | "maint", room: string): Promise { + const res = await SELF.fetch(`https://example.com/${path}/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket + if (!ws) throw new Error("no webSocket") + ws.accept() + return ws +} + +function sendForReceipt(ws: WebSocket, frame: Extract): Promise { + const receipt = new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error(`no receipt for ${frame.txId}`)), 2000) + const onMsg = (event: MessageEvent): void => { + const f = codec.decode(event.data as ArrayBuffer) as ServerFrame + if ((f.t === "committed" || f.t === "rejected") && f.txId === frame.txId) { + clearTimeout(timer) + ws.removeEventListener("message", onMsg) + resolve(f) + } + } + ws.addEventListener("message", onMsg) + }) + ws.send(codec.encode(frame)) + return receipt +} + +// "FORBIDDEN" makes the messages:insert authorize throw, so the mut is rejected. +const insert = (ws: WebSocket, txId: string, body: string): Promise => + sendForReceipt(ws, { + t: "mut", txId, collection: "messages", ops: [{ type: "insert", key: txId, cols: { id: txId, body } }], + }) + +const seenIds = (stub: DurableObjectStub): Promise => + runInDurableObject(stub, (_i, state) => + Array.from(state.storage.sql.exec<{ tx_id: string }>("SELECT tx_id FROM _sync_seen_tx")).map((r) => r.tx_id), + ) + +/** Seed an already-expired receipt. Touches storage only, never the gate. */ +const seedExpired = (stub: DurableObjectStub, txId: string): Promise => + runInDurableObject(stub, (_i, state) => { + state.storage.sql.exec( + "INSERT INTO _sync_seen_tx(tx_id,ok,cursor,error,error_code,result,ts) VALUES(?,0,NULL,'old',NULL,NULL,?)", + txId, + Date.now() - 2 * HOUR, + ) + }) + +async function waitForSweep(stub: DurableObjectStub, expiredId: string, freshId: string): Promise { + const deadline = Date.now() + 2000 + while (Date.now() < deadline) { + const ids = await seenIds(stub) + if (!ids.includes(expiredId) && ids.includes(freshId)) return + await new Promise((r) => setTimeout(r, 10)) + } + const ids = await seenIds(stub) + expect(ids).not.toContain(expiredId) + expect(ids).toContain(freshId) +} + +describe("dedup sweep rides answered receipts (bugbash F7)", () => { + it("a rejected mutation sweeps expired receipts", async () => { + const room = `sweep-reject-${crypto.randomUUID()}` + const stub = stubFor("sync", room) + const ws = await openWs("sync", room) + await seedExpired(stub, "old") + expect((await insert(ws, "rej-1", "FORBIDDEN")).t).toBe("rejected") + await waitForSweep(stub, "old", "rej-1") + ws.close() + }) + + it("a successful write-free command sweeps expired receipts", async () => { + const room = `sweep-echo-${crypto.randomUUID()}` + const stub = stubFor("sync", room) + const ws = await openWs("sync", room) + await seedExpired(stub, "old") + expect((await sendForReceipt(ws, { t: "call", txId: "echo-1", name: "echo", args: 1 })).t).toBe("committed") + await waitForSweep(stub, "old", "echo-1") + ws.close() + }) + + it("a drained write sweeps expired receipts, without waiting for compaction", async () => { + // MAINT_DO compacts every 3 drains; one write proves the sweep no longer + // rides compaction. + const room = `sweep-write-${crypto.randomUUID()}` + const stub = stubFor("maint", room) + const ws = await openWs("maint", room) + await seedExpired(stub, "old") + expect((await insert(ws, "write-1", "fine")).t).toBe("committed") + await waitForSweep(stub, "old", "write-1") + ws.close() + }) + + it("a replayed receipt sweeps expired receipts", async () => { + const room = `sweep-replay-${crypto.randomUUID()}` + const stub = stubFor("sync", room) + const ws = await openWs("sync", room) + expect((await insert(ws, "old", "FORBIDDEN")).t).toBe("rejected") + await runInDurableObject(stub, (_i, state) => { + state.storage.sql.exec("UPDATE _sync_seen_tx SET ts = ts - ?", 2 * HOUR) + }) + expect((await insert(ws, "fresh", "FORBIDDEN")).t).toBe("rejected") + // The wake reopens the gate, so the replay is the receipt that sweeps. It + // replays the stored outcome first, then deletes the now-expired row. + await evictDurableObject(stub) + expect((await insert(ws, "old", "fine")).t).toBe("rejected") + await waitForSweep(stub, "old", "fresh") + ws.close() + }) + + it("sweeps at most once per retention window, and again after a hibernation wake", async () => { + const room = `sweep-wake-${crypto.randomUUID()}` + const stub = stubFor("sync", room) + const ws = await openWs("sync", room) + expect((await insert(ws, "rej-1", "FORBIDDEN")).t).toBe("rejected") // sweeps; gate closes + await seedExpired(stub, "old") + expect((await insert(ws, "rej-2", "FORBIDDEN")).t).toBe("rejected") + // Not yet: the gate is closed for an hour. The sweep would run synchronously + // after this receipt, so the row surviving here is a race-free "no". + expect(await seenIds(stub)).toContain("old") + // A wake starts a fresh in-memory gate, so a DO that hibernates between + // receipts still sweeps. + await evictDurableObject(stub) + expect((await insert(ws, "rej-3", "FORBIDDEN")).t).toBe("rejected") + await waitForSweep(stub, "old", "rej-3") + ws.close() + }) + + it("sweeps again once the retention window has passed, without a wake", async () => { + const room = `sweep-reopen-${crypto.randomUUID()}` + const stub = stubFor("maint", room) + const ws = await openWs("maint", room) + expect((await insert(ws, "rej-1", "FORBIDDEN")).t).toBe("rejected") // sweeps; gate closes + await seedExpired(stub, "old") + // Mark the instance: if it were evicted during the sleep, the wake would + // reopen the gate and this test would pass without testing elapsed time. + await runInDurableObject(stub, (instance) => void ((instance as { marker?: string }).marker = room)) + await new Promise((r) => setTimeout(r, 1100)) // MAINT_DO dedupRetentionMs = 1 s + expect((await insert(ws, "rej-2", "FORBIDDEN")).t).toBe("rejected") + await waitForSweep(stub, "old", "rej-2") + expect(await runInDurableObject(stub, (instance) => (instance as { marker?: string }).marker)).toBe(room) + ws.close() + }) +}) diff --git a/tests/fetch-flush-barrier.test.ts b/tests/fetch-flush-barrier.test.ts new file mode 100644 index 0000000..40711f0 --- /dev/null +++ b/tests/fetch-flush-barrier.test.ts @@ -0,0 +1,70 @@ +import { createCollection, createLiveQueryCollection, eq } from "@tanstack/db" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { expect, it } from "vitest" +import { doCollectionOptions } from "../src/client/do-collection.ts" +import { WebSocketTransport, type WebSocketLike } from "../src/client/transport.ts" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ServerFrame } from "../src/wire/frames.ts" +import type { TestApi } from "./test-worker.ts" + +const codec = createFrameCodec() + +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((resolve) => setTimeout(resolve, 5)) + } +} + +async function serverExec(room: string, sql: string): Promise { + await runInDurableObject(env.SLOW_DO.get(env.SLOW_DO.idFromName(room)), (instance, state) => { + state.storage.sql.exec(sql) + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + }) +} + +it("flushes a held row's buffered delta before a covered fetch settles", async () => { + const room = `fetch-barrier-${crypto.randomUUID()}` + const frames: Array = [] + const transport = new WebSocketTransport({ + url: `https://example.com/slow/${room}`, + open: async () => { + const response = await SELF.fetch(`https://example.com/slow/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = response.webSocket! + ws.accept() + ws.addEventListener("message", (event) => { + frames.push(codec.decode((event as MessageEvent).data as ArrayBuffer) as ServerFrame) + }) + return ws as unknown as WebSocketLike + }, + }) + try { + await serverExec(room, "INSERT INTO messages(id,body) VALUES('a','old')") + const messages = createCollection( + doCollectionOptions({ transport, table: "messages", getKey: (message) => message.id, syncMode: "on-demand" }), + ) + const all = createLiveQueryCollection((query) => query.from({ message: messages })) + await all.preload() + await waitFor(() => messages.get("a")?.body === "old") + + await serverExec(room, "UPDATE messages SET body='new' WHERE id='a'") + const mark = frames.length + expect(frames.some((frame) => frame.t === "d")).toBe(false) + const one = createLiveQueryCollection((query) => query.from({ message: messages }).where(({ message }) => eq(message.id, "a"))) + await one.preload() + + expect(messages.get("a")?.body).toBe("new") + const after = frames.slice(mark) + const page = after.findIndex((frame) => frame.t === "page") + const delta = after.findIndex((frame) => frame.t === "d" && frame.key === "a") + const boundary = after.findIndex((frame) => frame.t === "uptodate") + expect(page).toBeGreaterThanOrEqual(0) + expect(delta).toBeGreaterThanOrEqual(0) + expect(delta).toBeLessThan(boundary) + expect(boundary).toBeLessThan(page) + expect(after[page]).toMatchObject({ rows: [{ id: "a", body: "new" }] }) + } finally { + transport.close() + } +}) diff --git a/tests/flush-one-timer.test.ts b/tests/flush-one-timer.test.ts new file mode 100644 index 0000000..5ec3de6 --- /dev/null +++ b/tests/flush-one-timer.test.ts @@ -0,0 +1,125 @@ +import { describe, expect, it } from "vitest" +import { Broadcaster } from "../src/server/broadcast.ts" +import type { ServerFrame } from "../src/wire/frames.ts" + +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((resolve) => setTimeout(resolve, 5)) + } +} + +describe("flushOne timer hygiene", () => { + it("discards a dead socket's buffer without cancelling another socket's tick", async () => { + const sent: Array<{ ws: WebSocket; frame: ServerFrame }> = [] + const departed = {} as WebSocket + const remaining = {} as WebSocket + const broadcaster = new Broadcaster((ws, frame) => sent.push({ ws, frame }), () => 20) + broadcaster.start(() => [departed, remaining]) + try { + broadcaster.enqueue(departed, { subId: "s", key: "a", op: "delete" }, "1") + broadcaster.enqueue(remaining, { subId: "s", key: "b", op: "delete" }, "2") + broadcaster.discard(departed) + broadcaster.discard(departed) + broadcaster.flushOne(departed) + expect(sent).toHaveLength(0) + expect(broadcaster.isFlushScheduled).toBe(true) + await waitFor(() => sent.some((entry) => entry.ws === remaining && entry.frame.t === "uptodate")) + expect(sent.map((entry) => entry.ws)).toEqual([remaining, remaining]) + expect(broadcaster.isFlushScheduled).toBe(false) + } finally { + broadcaster.stop() + } + }) + + it("discards the last buffer, disarms its tick, and rearms for new work", async () => { + const sent: Array = [] + const ws = {} as WebSocket + const broadcaster = new Broadcaster((_ws, frame) => sent.push(frame), () => 20) + broadcaster.start(() => [ws]) + try { + broadcaster.enqueue(ws, { subId: "s", key: "old", op: "delete" }, "1") + broadcaster.discard(ws) + expect(broadcaster.isFlushScheduled).toBe(false) + broadcaster.flushOne(ws) + expect(sent).toHaveLength(0) + + broadcaster.enqueue(ws, { subId: "s", key: "new", op: "delete" }, "2") + expect(broadcaster.isFlushScheduled).toBe(true) + await waitFor(() => sent.some((frame) => frame.t === "uptodate")) + expect(sent.filter((frame) => frame.t === "d").map((frame) => frame.key)).toEqual(["new"]) + expect(broadcaster.isFlushScheduled).toBe(false) + } finally { + broadcaster.stop() + } + }) + + it("cancels the last pending socket's tick instead of running an empty flush", async () => { + const sent: Array = [] + const ws = {} as WebSocket + let tickFlushes = 0 + const broadcaster = new Broadcaster((_ws, frame) => sent.push(frame), () => 20) + broadcaster.start(() => { + tickFlushes++ + return [ws] + }) + try { + expect(broadcaster.isFlushScheduled).toBe(false) + broadcaster.enqueue(ws, { subId: "s", key: "a", op: "insert", cols: { id: "a" } }, "1") + expect(broadcaster.isFlushScheduled).toBe(true) + broadcaster.flushOne(ws) + const armedAfterFlush = broadcaster.isFlushScheduled + const scansAfterFlush = tickFlushes + expect(sent.map((frame) => frame.t)).toEqual(["d", "uptodate"]) + await new Promise((resolve) => setTimeout(resolve, 60)) + expect(armedAfterFlush).toBe(false) + expect(tickFlushes).toBe(scansAfterFlush) + expect(sent).toHaveLength(2) + } finally { + broadcaster.stop() + } + }) + + it("keeps another socket's tick, then rearms after becoming idle", async () => { + const sent: Array<{ ws: WebSocket; frame: ServerFrame }> = [] + const first = {} as WebSocket + const second = {} as WebSocket + const broadcaster = new Broadcaster((ws, frame) => sent.push({ ws, frame }), () => 20) + broadcaster.start(() => [first, second]) + try { + broadcaster.enqueue(first, { subId: "s", key: "a", op: "insert", cols: { id: "a" } }, "1") + broadcaster.enqueue(second, { subId: "s", key: "b", op: "insert", cols: { id: "b" } }, "2") + broadcaster.flushOne(first) + expect(broadcaster.isFlushScheduled).toBe(true) + expect(sent.some((entry) => entry.ws === second)).toBe(false) + await waitFor(() => sent.some((entry) => entry.ws === second && entry.frame.t === "uptodate")) + expect(broadcaster.isFlushScheduled).toBe(false) + expect(sent.filter((entry) => entry.ws === second).map((entry) => entry.frame.t)).toEqual(["d", "uptodate"]) + + broadcaster.enqueue(first, { subId: "s", key: "c", op: "delete" }, "3") + expect(broadcaster.isFlushScheduled).toBe(true) + await waitFor(() => sent.some((entry) => entry.frame.t === "d" && entry.frame.key === "c")) + expect(broadcaster.isFlushScheduled).toBe(false) + } finally { + broadcaster.stop() + } + }) + + it("ignores departed sockets when a live socket's flush decides whether to keep the tick", () => { + const first = {} as WebSocket + const departed = {} as WebSocket + const live = new Set([first, departed]) + const broadcaster = new Broadcaster(() => {}, () => 20) + broadcaster.start(() => live) + try { + broadcaster.enqueue(first, { subId: "s", key: "a", op: "delete" }, "1") + broadcaster.enqueue(departed, { subId: "s", key: "b", op: "delete" }, "2") + live.delete(departed) + broadcaster.flushOne(first) + expect(broadcaster.isFlushScheduled).toBe(false) + } finally { + broadcaster.stop() + } + }) +}) diff --git a/tests/full-row-deltas.test.ts b/tests/full-row-deltas.test.ts new file mode 100644 index 0000000..fa0155d --- /dev/null +++ b/tests/full-row-deltas.test.ts @@ -0,0 +1,89 @@ +import type { SqlStorage } from "@cloudflare/workers-types" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ServerFrame } from "../src/wire/frames.ts" + +// Issue #28 part (a), bugbash G28a. Characterization, not the wire contract +// (`d.cols` is typed as a partial patch and the client merges it as one): +// today a live `d` for an update carries the WHOLE current row, however little +// changed — including nothing. The full row is what lets several existing +// decisions stay simple: +// - capture records only (key, op) and rows hydrate fresh at drain, so a +// column an author adds later flows with no trigger change (ADR-0007; +// schema-evolution.test.ts); +// - the always-emit rule sends a matching row's current state to every sub +// with no per-sub membership, and the client applies an update for a key +// it lacks as an upsert — move-in, including a bounded (limit) sub's rows +// outside its window (ADR-0002 C4). A column-only patch would upsert a +// partial row there; +// - live deltas are self-contained (ADR-0009's premise), so the broadcaster +// may keep only the latest delta per key per tick, and a missed delta +// heals at the key's next write. +// The cost is bandwidth: row size × write rate × subscribers. ADR-0018 D3 +// warns above 1 MiB and names column projection the real fix. A projection +// change supersedes this test (with its own ADR). + +const codec = createFrameCodec() +const BIG = "x".repeat(300_000) + +type Frame = { f: ServerFrame; bytes: number } + +async function setup(room: string): Promise<{ ws: WebSocket; frames: Array; stub: DurableObjectStub }> { + const stub = env.SYNC_DO.get(env.SYNC_DO.idFromName(room)) + await runInDurableObject(stub, (inst, s) => { + s.storage.sql.exec("ALTER TABLE messages ADD COLUMN n INTEGER") // a small column beside the big one + s.storage.sql.exec("INSERT INTO messages(id, body, n) VALUES ('big', ?, 0)", BIG) + ;(inst as unknown as { sync: { drainAndBroadcast(): void } }).sync.drainAndBroadcast() + }) + const res = await SELF.fetch(`https://example.com/sync/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket! + ws.accept() + const frames: Array = [] + ws.addEventListener("message", (e) => { + frames.push({ f: codec.decode(e.data as ArrayBuffer) as ServerFrame, bytes: (e.data as ArrayBuffer).byteLength }) + }) + ws.send(codec.encode({ t: "sub", subId: "s1", collection: "messages" })) + await waitFor(() => frames.some(({ f }) => f.t === "snap-end")) + return { ws, frames, stub } +} + +function serverWrite(stub: DurableObjectStub, stmt: string): Promise { + return runInDurableObject(stub, (inst) => { + ;(inst as unknown as { sync: { runSyncedWrite(fn: (sql: SqlStorage) => void): void } }).sync.runSyncedWrite((sql) => { + sql.exec(stmt) + }) + }) +} + +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((r) => setTimeout(r, 5)) + } +} + +const delta = (frames: Array): Frame | undefined => frames.find(({ f }) => f.t === "d") + +describe("#28 (a): an update's live delta is the full current row", () => { + it("a narrow UPDATE of a small column re-sends the unchanged 300 KB column", async () => { + const { ws, frames, stub } = await setup(`g28-narrow-${crypto.randomUUID()}`) + await serverWrite(stub, "UPDATE messages SET n = 1 WHERE id = 'big'") + await waitFor(() => delta(frames) !== undefined) + ws.close() + const d = delta(frames)! + expect(d.f).toMatchObject({ t: "d", sub: "s1", key: "big", op: "update", cols: { id: "big", body: BIG, n: 1 } }) + expect(d.bytes).toBeGreaterThan(300_000) + }) + + it("an UPDATE that changes nothing still emits the full row", async () => { + const { ws, frames, stub } = await setup(`g28-noop-${crypto.randomUUID()}`) + await serverWrite(stub, "UPDATE messages SET body = body WHERE id = 'big'") + await waitFor(() => delta(frames) !== undefined) + ws.close() + const d = delta(frames)! + expect(d.f).toMatchObject({ t: "d", key: "big", op: "update", cols: { id: "big", body: BIG, n: 0 } }) + expect(d.bytes).toBeGreaterThan(300_000) + }) +}) diff --git a/tests/maybe-compact.test.ts b/tests/maybe-compact.test.ts index 7f35b9f..b9a12cd 100644 --- a/tests/maybe-compact.test.ts +++ b/tests/maybe-compact.test.ts @@ -5,7 +5,8 @@ import type { ClientFrame, ServerFrame } from "../src/wire/frames.ts" // WHY: maybeCompact is the opportunistic housekeeping path — it fires only every // `compactionEvery` drained mutations and runs compaction + retention prune + -// dedup sweep under `ctx.waitUntil`. It was previously covered only by calling +// orphan-sub sweep under `ctx.waitUntil` (the dedup sweep rides receipts +// instead; see dedup-sweep.test.ts). It was previously covered only by calling // the underlying functions directly; the WIRING (does the threshold gate it? // does it actually run compaction and the ADR-0009 prune?) was untested. These // drive the real WS dispatch → drain → maybeCompact → waitUntil path on a DO diff --git a/tests/mutation-pk-consistency.test.ts b/tests/mutation-pk-consistency.test.ts new file mode 100644 index 0000000..cd8e1a4 --- /dev/null +++ b/tests/mutation-pk-consistency.test.ts @@ -0,0 +1,192 @@ +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { defineSync } from "../src/server/registry.ts" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ClientFrame, MutOp, ServerFrame } from "../src/wire/frames.ts" + +const codec = createFrameCodec() +const receipt = (frame: ServerFrame): boolean => frame.t === "committed" || frame.t === "rejected" +const sync = defineSync() +const slugSchema = sync.schema({ + collections: { + aliases: sync.collection<{ slug: string; body: string }>({ + pk: "slug", + mutations: { + insert: { + execute: ({ op, sql }) => { + sql.exec("INSERT INTO aliases(slug, body) VALUES (?, ?)", op.cols.slug, op.cols.body) + }, + }, + }, + }), + }, +}) + +async function openWs(room: string): Promise { + const response = await SELF.fetch(`https://example.com/sync/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = response.webSocket! + ws.accept() + return ws +} + +function send(ws: WebSocket, frame: ClientFrame): void { + ws.send(codec.encode(frame)) +} + +function collectUntil(ws: WebSocket, done: (frame: ServerFrame) => boolean): Promise> { + return new Promise((resolve, reject) => { + const frames: Array = [] + const timer = setTimeout(() => reject(new Error(`timeout; got [${frames.map((frame) => frame.t).join(",")}]`)), 2000) + const onMessage = (event: MessageEvent): void => { + const frame = codec.decode(event.data as ArrayBuffer) as ServerFrame + frames.push(frame) + if (done(frame)) { + clearTimeout(timer) + ws.removeEventListener("message", onMessage) + resolve(frames) + } + } + ws.addEventListener("message", onMessage) + }) +} + +async function state(room: string): Promise<{ rows: Array<{ id: string; body: string }>; changes: number }> { + return runInDurableObject(env.SYNC_DO.get(env.SYNC_DO.idFromName(room)), (_instance, storage) => ({ + rows: Array.from(storage.storage.sql.exec("SELECT id, body FROM messages ORDER BY id")) as Array<{ id: string; body: string }>, + changes: (Array.from(storage.storage.sql.exec("SELECT COUNT(*) AS n FROM _sync_changes"))[0] as { n: number }).n, + })) +} + +async function mutWithWatcher(room: string, txId: string, ops: Array) { + const watcher = await openWs(room) + send(watcher, { t: "sub", subId: "w", collection: "messages" }) + await collectUntil(watcher, (frame) => frame.t === "snap-end") + const writer = await openWs(room) + const sentinel = `${txId}-sentinel` + const watched = collectUntil(watcher, (frame) => frame.t === "d" && frame.key === sentinel) + const before = await state(room) + send(writer, { t: "mut", txId, collection: "messages", ops }) + const result = (await collectUntil(writer, receipt)).at(-1)! + const after = await state(room) + + send(writer, { t: "mut", txId: sentinel, collection: "messages", ops: [{ type: "insert", key: sentinel, cols: { id: sentinel, body: "sentinel" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)!.t).toBe("committed") + const deltas = (await watched).filter((frame) => frame.t === "d" && frame.key !== sentinel) + writer.close() + watcher.close() + return { result, before, after, deltas } +} + +describe("mutation pk agrees with the op key (ADR-0025)", () => { + for (const [label, id] of [["different", "b"], ["empty", ""], ["null", null], ["numeric", 7]] as const) { + it(`rejects insert with ${label} cols.id without a write or delta`, async () => { + const room = `pk-insert-${crypto.randomUUID()}` + const { result, before, after, deltas } = await mutWithWatcher(room, `pk-${label}`, [ + { type: "insert", key: "a", cols: { id, body: "x" } }, + ]) + expect(result).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + expect(after).toEqual(before) + expect(deltas).toEqual([]) + }) + } + + it("rejects before the author's authorization runs", async () => { + const room = `pk-auth-${crypto.randomUUID()}` + const writer = await openWs(room) + send(writer, { t: "mut", txId: "pk-auth", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "b", body: "FORBIDDEN" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + writer.close() + }) + + it("rejects an update carrying a different pk, leaving the target untouched", async () => { + const room = `pk-update-${crypto.randomUUID()}` + const writer = await openWs(room) + send(writer, { t: "mut", txId: "seed", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "a", body: "seed" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)!.t).toBe("committed") + writer.close() + + const { result, before, after, deltas } = await mutWithWatcher(room, "pk-update", [ + { type: "update", key: "a", cols: { id: "b", body: "changed" } }, + ]) + expect(result).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + expect(after).toEqual(before) + expect(deltas).toEqual([]) + }) + + it("accepts an update that supplies the same pk", async () => { + const room = `pk-matching-update-${crypto.randomUUID()}` + const writer = await openWs(room) + send(writer, { t: "mut", txId: "seed", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "a", body: "seed" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)!.t).toBe("committed") + writer.close() + + const { result, after, deltas } = await mutWithWatcher(room, "pk-match", [ + { type: "update", key: "a", cols: { id: "a", body: "changed" } }, + ]) + expect(result.t).toBe("committed") + expect(after.rows).toEqual([{ id: "a", body: "changed" }]) + expect(deltas).toMatchObject([{ t: "d", key: "a" }]) + }) + + it("leaves an omitted insert pk to the author handler", async () => { + const room = `pk-omitted-${crypto.randomUUID()}` + const writer = await openWs(room) + send(writer, { t: "mut", txId: "pk-omitted", collection: "messages", ops: [{ type: "insert", key: "a", cols: { body: "FORBIDDEN" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)).toMatchObject({ t: "rejected", error: { message: "forbidden body" } }) + expect((await state(room)).rows).toEqual([]) + writer.close() + }) + + it("refuses the whole batch if a later insert conflicts", async () => { + const room = `pk-batch-${crypto.randomUUID()}` + const { result, before, after, deltas } = await mutWithWatcher(room, "pk-batch", [ + { type: "insert", key: "good", cols: { id: "good", body: "ok" } }, + { type: "insert", key: "a", cols: { id: "b", body: "bad" } }, + ]) + expect(result).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + expect(after).toEqual(before) + expect(deltas).toEqual([]) + }) + + it("replays a recorded commit before checking a duplicate's mismatched pk", async () => { + const room = `pk-dedup-${crypto.randomUUID()}` + const writer = await openWs(room) + send(writer, { t: "mut", txId: "pk-dedup", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "a", body: "seed" } }] }) + const first = (await collectUntil(writer, receipt)).at(-1)! + expect(first.t).toBe("committed") + send(writer, { t: "mut", txId: "pk-dedup", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "b", body: "changed" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)).toMatchObject({ t: "committed", txId: "pk-dedup", seq: (first as { seq: string }).seq }) + expect((await state(room)).rows).toEqual([{ id: "a", body: "seed" }]) + writer.close() + }) + + it("persists a pk rejection and replays it for a corrected duplicate", async () => { + const room = `pk-rejected-dedup-${crypto.randomUUID()}` + const writer = await openWs(room) + send(writer, { t: "mut", txId: "pk-rejected", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "b", body: "bad" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + send(writer, { t: "mut", txId: "pk-rejected", collection: "messages", ops: [{ type: "insert", key: "a", cols: { id: "a", body: "good" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + expect((await state(room)).rows).toEqual([]) + writer.close() + }) + + it("compares the registered collection pk, not a hard-coded id field", async () => { + const room = `pk-slug-${crypto.randomUUID()}` + const writer = await openWs(room) + await runInDurableObject(env.SYNC_DO.get(env.SYNC_DO.idFromName(room)), (instance, storage) => { + storage.storage.sql.exec("CREATE TABLE aliases(slug TEXT PRIMARY KEY, body TEXT)") + ;(instance as unknown as { registerSync: (schema: typeof slugSchema) => void }).registerSync(slugSchema) + }) + + send(writer, { t: "mut", txId: "pk-slug-ok", collection: "aliases", ops: [{ type: "insert", key: "a", cols: { slug: "a", id: "wrong", body: "ok" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)!.t).toBe("committed") + send(writer, { t: "mut", txId: "pk-slug-bad", collection: "aliases", ops: [{ type: "insert", key: "c", cols: { slug: "d", id: "c", body: "bad" } }] }) + expect((await collectUntil(writer, receipt)).at(-1)).toMatchObject({ t: "rejected", error: { code: "VALIDATION" } }) + const rows = await runInDurableObject(env.SYNC_DO.get(env.SYNC_DO.idFromName(room)), (_instance, storage) => + Array.from(storage.storage.sql.exec("SELECT slug, body FROM aliases ORDER BY slug")), + ) + expect(rows).toEqual([{ slug: "a", body: "ok" }]) + writer.close() + }) +}) diff --git a/tests/on-demand-scenarios.test.ts b/tests/on-demand-scenarios.test.ts new file mode 100644 index 0000000..638ce21 --- /dev/null +++ b/tests/on-demand-scenarios.test.ts @@ -0,0 +1,302 @@ +import * as db from "@tanstack/db" +import { BTreeIndex, createCollection, createLiveQueryCollection, eq } from "@tanstack/db" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { doCollectionOptions } from "../src/client/do-collection.ts" +import { WebSocketTransport, type WebSocketLike } from "../src/client/transport.ts" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { TestApi } from "./test-worker.ts" + +// WHY: realistic on-demand sessions, end to end (live queries → loadSubset → +// real WebSocketTransport → real DO), pinning ADR-0023 fixes that were only +// pinned by fake transports or not at all (bug bash 2026-09-29, K8: C3, C5, C7, +// C8). Each assertion names the bug it guards. Run at both ends of the +// @tanstack/db matrix (ADR-0022): some regressions bite on one version only, +// and the comments say which. + +// @tanstack/db 0.9.0 removed the subset-algebra helpers, so their presence +// marks the 0.8.x floor. Used only where upstream itself differs. +const db08 = "isWhereSubset" in db + +const room = (tag: string): string => `ods-${tag}-${crypto.randomUUID()}` +const stubFor = (r: string) => env.SYNC_DO.get(env.SYNC_DO.idFromName(r)) + +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((r) => setTimeout(r, 5)) + } +} +const sleep = (ms: number): Promise => new Promise((r) => setTimeout(r, ms)) + +/** Raw server-side write + broadcast (the firehose path other clients use). */ +async function serverExec(r: string, sql: string, ...args: Array): Promise { + await runInDurableObject(stubFor(r), (instance, s) => { + s.storage.sql.exec(sql, ...args) + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + }) +} +/** Seed `bodies` as messages m00.., drained BEFORE any subscriber exists. */ +async function seed(r: string, bodies: Array): Promise { + await runInDurableObject(stubFor(r), (instance, s) => { + bodies.forEach((b, i) => s.storage.sql.exec("INSERT INTO messages(id,body) VALUES(?,?)", `m${String(i).padStart(2, "0")}`, b)) + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + }) +} +const pad = (n: number): string => String(n).padStart(2, "0") +const dropSocket = (r: string) => + runInDurableObject(stubFor(r), (_i, state) => { + for (const sock of state.getWebSockets()) sock.close(1000, "drop") + }) + +interface Gate { + hold: boolean + queue: Array<() => void> + /** Hold only frames of these types (default: every frame). */ + only?: ReadonlyArray +} +const codec = createFrameCodec() +/** A real transport to the room. `frames` records every sub the client opens, + * `pages` every fetch it sends, `paged` the ids of every row a page brought; + * `gate` holds inbound frames (in order). */ +function transportFor( + r: string, + opts: { frames?: Array; pages?: Array>; paged?: Set; gate?: Gate } = {}, +): WebSocketTransport { + const t = new WebSocketTransport({ + url: `https://example.com/sync/${r}`, + reconnectDelay: () => 0, + open: async () => { + const res = await SELF.fetch(`https://example.com/sync/${r}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket + if (!ws) throw new Error("no webSocket") + ws.accept() + const gate = opts.gate + if (!gate) return ws as unknown as WebSocketLike + return { + send: (d) => ws.send(d as never), + close: (c, reason) => ws.close(c, reason), + addEventListener: (type, fn) => { + if (type !== "message") return ws.addEventListener(type as never, fn as never) + ws.addEventListener("message", (ev) => { + const held = gate.hold && (!gate.only || gate.only.includes(codec.decode(ev.data as ArrayBuffer).t)) + if (held) gate.queue.push(() => fn(ev as never)) + else fn(ev as never) + }) + }, + removeEventListener: () => {}, + } satisfies WebSocketLike + }, + }) + if (opts.frames) { + const frames = opts.frames + const sub = t.subscribe.bind(t) + t.subscribe = (...a) => { + frames.push(a[0]) + return sub(...a) + } + } + if (opts.pages) { + const { pages, paged } = opts + const fetch = t.fetch.bind(t) + t.fetch = (frame) => { + const p = fetch(frame) + pages.push(p.then((rows) => rows.forEach((row) => paged?.add((row as { id: string }).id)))) + return p + } + } + return t +} +const onDemand = (t: WebSocketTransport) => + createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (m) => m.id, syncMode: "on-demand" })) +type Windowed = { utils: { setWindow: (w: { offset?: number; limit?: number }) => true | Promise } } +const bodies = (q: { toArray: Array<{ body: string }> }): Array => q.toArray.map((m) => m.body) +const ids = (q: { toArray: Array<{ id: string }> }): Array => q.toArray.map((m) => m.id).sort() + +describe("filtered lists on a flaky socket", () => { + it("a filter abandoned mid-snapshot leaves nothing; a below-floor reconnect reloads every open filter", async () => { + const r = room("lists") + const gate: Gate = { hold: false, queue: [], only: ["snap-end"] } + const frames: Array = [] + const t = transportFor(r, { frames, gate }) + await seed(r, ["a", "a", "b", "b", "c", "c"]) // m00 m01: a · m02 m03: b · m04 m05: c + const messages = onDemand(t) + const qa = createLiveQueryCollection((q) => q.from({ m: messages }).where(({ m }) => eq(m.body, "a"))) + const qb = createLiveQueryCollection((q) => q.from({ m: messages }).where(({ m }) => eq(m.body, "b"))) + await qa.preload() + await qb.preload() + await waitFor(() => qa.size === 2 && qb.size === 2) + + // C3: open a third filter; its snap rows reach the handler (inside the open + // sync transaction) but its snap-end is held; the user navigates away. + gate.hold = true + const qc = createLiveQueryCollection((q) => q.from({ m: messages }).where(({ m }) => eq(m.body, "c"))) + void qc.preload().catch(() => {}) + await waitFor(() => gate.queue.length === 1) // snap m04, m05 dispatched ahead of it + expect(messages.isLoadingSubset).toBe(true) + await qc.cleanup() + // bugbash C3: the release settles the load (do-collection unloadSubset's + // `acq.settle()`); pre-fix it stayed loading forever. + await waitFor(() => !messages.isLoadingSubset) + gate.hold = false + for (const deliver of gate.queue.splice(0)) deliver() // the stale snap-end: no handler now + // A later commit boundary on another filter (a live delta to qa). + await serverExec(r, "INSERT INTO messages(id,body) VALUES(?,?)", "m06", "a") + await waitFor(() => qa.size === 3) + await sleep(30) + // bugbash C3: the partial snapshot was dropped with its hold (unpin's + // `dropHolds`), not committed by the release or by qa's boundary. + expect([...messages.keys()].sort()).toEqual(["m00", "m01", "m02", "m03", "m06"]) + + // C5: changes the client never sees (written without a broadcast, as if the + // socket had missed them), then the changelog is pruned below the client's + // cursor and the socket drops: the reconnect's resubscribes are both reset. + const subsBefore = frames.length + await runInDurableObject(stubFor(r), (_i, s) => { + s.storage.sql.exec("DELETE FROM messages WHERE id = ?", "m01") + s.storage.sql.exec("INSERT INTO messages(id,body) VALUES(?,?)", "x1", "b") + s.storage.sql.exec("DELETE FROM _sync_changes") + }) + await dropSocket(r) + // bugbash C5: every open filter reloads the current server rows. The fix + // (`truncateAll` on a bootstrapped sub's reset: one truncate, every watch + // retired, a new generation) makes core's truncate replay reopen each demand + // with a fresh sub. Reverting it to a plain truncate fails this on the 0.8.x + // floor ONLY: 0.8 answers its load-then-release replay from the refcount (no + // new sub), while 0.9's release-then-reload replay rescues it. + await waitFor(() => qa.size === 2 && qb.size === 3, 4000) + await sleep(100) + expect(ids(qa)).toEqual(["m00", "m06"]) + expect(ids(qb)).toEqual(["m02", "m03", "x1"]) + expect([...messages.keys()].sort()).toEqual(["m00", "m02", "m03", "m06", "x1"]) + expect(frames.length - subsBefore).toBe(2) // one fresh sub per open filter, no storm + t.close() + }) +}) + +describe("offset windows through a real live query", () => { + for (const indexed of [false, true]) { + it(`page 2 of a sorted list shows its own rows beside page 1 and the reverse order (index: ${indexed})`, async () => { + const r = room(`offset-${indexed}`) + const t = transportFor(r) + await seed(r, Array.from({ length: 20 }, (_, i) => pad(i + 1))) + const messages = onDemand(t) + // Lexical: an order core can express to the source, so it asks for each + // window exactly (a locale sort is recovered by loading the whole source + // on 0.9, which would hide a shared snapshot). + const asc = { direction: "asc", stringSort: "lexical" } as const + const desc = { direction: "desc", stringSort: "lexical" } as const + if (indexed) messages.createIndex((m) => m.body, { indexType: BTreeIndex, options: { compareOptions: { stringSort: "lexical" } } }) + const page1 = createLiveQueryCollection((q) => q.from({ m: messages }).orderBy(({ m }) => m.body, asc).limit(3)) + await page1.preload() + await waitFor(() => page1.size === 3) + // Core folds a query's offset into the request's limit ({limit: 8}): the + // adapter never sees a bare offset (it rejects one loudly). + const page2 = createLiveQueryCollection((q) => q.from({ m: messages }).orderBy(({ m }) => m.body, asc).offset(5).limit(3)) + await page2.preload() + // The same page from the other end: page 2's where and limit, other order. + const last2 = createLiveQueryCollection((q) => q.from({ m: messages }).orderBy(({ m }) => m.body, desc).offset(5).limit(3)) + await last2.preload() + // bugbash C7: each window is its own request (identity is where + orderBy + // + limit, `requestKey`). Pre-fix page 2 shared page 1's 3-row snapshot + // and showed nothing; without orderBy in the key, last2 would share page + // 2's ascending snapshot and show 03..01. + await waitFor(() => page2.size === 3 && last2.size === 3) + await sleep(30) + expect(bodies(page1)).toEqual(["01", "02", "03"]) + expect(bodies(page2)).toEqual(["06", "07", "08"]) + expect(bodies(last2)).toEqual(["15", "14", "13"]) + // An offset jump: 0.9 fills it. 0.8.x does not (upstream: no request for + // an unindexed window; one cursor request {limit: 3, offset: 8} for an + // indexed one, which any exact adapter answers with 09..11), so skip there. + if (db08) return t.close() + const p = (page2 as unknown as Windowed).utils.setWindow({ offset: 10, limit: 3 }) + if (p !== true) await p + await waitFor(() => bodies(page2).join() === "11,12,13") + expect(bodies(page1)).toEqual(["01", "02", "03"]) + t.close() + }) + } +}) + +describe("a paged list: left mid-scroll, then back with a colliding create", () => { + it("a page landing after unmount installs nothing; a page row is written under an optimistic overlay", async () => { + const r = room("paged") + const gate: Gate = { hold: false, queue: [] } + const pages: Array> = [] + const paged = new Set() + const t = transportFor(r, { pages, paged, gate }) + await seed(r, Array.from({ length: 20 }, (_, i) => pad(i + 1))) // m00 "01" .. m19 "20" + const messages = onDemand(t) + // Growing this window pages from the DO: on 0.8.6 a default-sort indexed + // window grows by CURSOR request (loadMore); on 0.9.2 a lexical one by + // covered tie + prefix fetches (a default locale sort is loaded whole at + // mount there, so nothing would page). Both install through installPage. + const sort = db08 ? ({ direction: "desc" } as const) : ({ direction: "desc", stringSort: "lexical" } as const) + if (db08) messages.createIndex((m) => m.body, { indexType: BTreeIndex }) + else messages.createIndex((m) => m.body, { indexType: BTreeIndex, options: { compareOptions: { stringSort: "lexical" } } }) + const mount = async () => { + const win = createLiveQueryCollection((q) => q.from({ m: messages }).orderBy(({ m }) => m.body, sort).limit(5)) + await win.preload() + await waitFor(() => win.size === 5 && !messages.isLoadingSubset) // incl. 0.9's boundary-tie fetch + return win + } + const grow = (win: unknown, limit: number) => Promise.resolve((win as Windowed).utils.setWindow({ offset: 0, limit })) + + // C8 late page: scroll (the page is held), then leave before it lands. + const win1 = await mount() + gate.only = ["page"] + gate.hold = true + void grow(win1, 10).catch(() => {}) // core aborts it on cleanup + await waitFor(() => gate.queue.length >= 1) // a page is in flight + await win1.cleanup() // the watch is released, its rows dropped + await waitFor(() => messages.size === 0) + gate.hold = false + for (const deliver of gate.queue.splice(0)) deliver() // the late page + await Promise.all(pages.splice(0)) + await sleep(50) + // bugbash C8 (late-page half): nothing holds these rows, so none install. + // The guard differs by version: 0.8.6's in-flight request is a cursor page + // (loadMore's `o.signal?.aborted`), 0.9.2's are covered fetches + // (fetchCovered's `acq.released`); reverting either fails its version. + expect([...messages.keys()]).toEqual([]) + + // C8 overlay: back on the list, the user posts a new item as m10 (it tops + // the list; the id already exists on the DO, so the insert will be + // rejected; `rejected` is held), then scrolls: the page carrying the + // server's m10 lands under the overlay. (A draft sorting BELOW the loaded + // rows would not do: 0.8.6 pages its index cursor from the optimistic row.) + const win2 = await mount() + expect(messages.get("m10")).toBeUndefined() // "11": below the window, not loaded + gate.only = ["rejected"] + gate.hold = true + const tx = messages.insert({ id: "m10", body: "99-draft" }) + tx.isPersisted.promise.catch(() => {}) + paged.clear() // win1's late page carried m10 too + void grow(win2, 11).catch(() => {}) // six visible (draft + 20..16), so the page reaches "11" + await waitFor(() => paged.has("m10")) + await sleep(20) // installPage ran (it continues the same fetch resolution) + // Hold later pages: on 0.9 the rollback (a source change) starts core's + // full-source recovery fetch, which would re-deliver m10 and mask a + // regression here with one extra round trip. + gate.only = ["page"] + for (const deliver of gate.queue.splice(0)) deliver() // rejected: the overlay rolls back + await expect(tx.isPersisted.promise).rejects.toThrow() + // bugbash C8 (overlay half): installPage writes insert-if-absent against + // SYNCED state (`syncedHas`). Pre-fix it checked `collection.get`, which + // sees the optimistic row, so the server row was skipped and was gone once + // the overlay rolled back. + await waitFor(() => (messages.get("m10") as { body?: string } | undefined)?.body === "11") + gate.hold = false + for (const deliver of gate.queue.splice(0)) deliver() + // The window shows the server's m10 in place of the draft. Upstream-only + // gate: 0.8.6 live queries miss a sync commit that lands with an optimistic + // insert's rollback of the same key (reproduced without this adapter; its + // collection is right, the query keeps the draft), so check 0.9 only. + // Filling the 11th slot the rollback vacated is core's business too. + if (db08) return t.close() + await waitFor(() => bodies(win2).slice(0, 10).join() === "20,19,18,17,16,15,14,13,12,11") + t.close() + }) +}) diff --git a/tests/order-by-nulls.test.ts b/tests/order-by-nulls.test.ts new file mode 100644 index 0000000..f667844 --- /dev/null +++ b/tests/order-by-nulls.test.ts @@ -0,0 +1,232 @@ +import * as db from "@tanstack/db" +import { BTreeIndex, createCollection, createLiveQueryCollection, localOnlyCollectionOptions } from "@tanstack/db" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { doCollectionOptions } from "../src/client/do-collection.ts" +import { WebSocketTransport, type WebSocketLike } from "../src/client/transport.ts" +import type { TestApi } from "./test-worker.ts" + +const version = "isWhereSubset" in db ? "0.8.x" : "0.9.x" +const room = (tag: string): string => `bbD-${tag}-${crypto.randomUUID()}` +const stubFor = (r: string) => env.SYNC_DO.get(env.SYNC_DO.idFromName(r)) +const sleep = (ms: number): Promise => new Promise((r) => setTimeout(r, ms)) +async function waitFor(predicate: () => boolean): Promise { + const deadline = Date.now() + 3000 + while (!predicate()) { + if (Date.now() > deadline) throw new Error("timed out waiting for snapshot") + await sleep(10) + } +} + +type Row = { id: string; body: string | null } + +async function seed(r: string, rows: Array): Promise { + await runInDurableObject(stubFor(r), (instance, s) => { + for (const row of rows) s.storage.sql.exec("INSERT INTO messages(id,body) VALUES(?,?)", row.id, row.body) + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + }) +} + +function transportFor(r: string, boundedSnapshots: Array>): WebSocketTransport { + const t = new WebSocketTransport({ + url: `https://example.com/sync/${r}`, + reconnectDelay: () => 0, + open: async () => { + const res = await SELF.fetch(`https://example.com/sync/${r}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket + if (!ws) throw new Error("no webSocket") + ws.accept() + return ws as unknown as WebSocketLike + }, + }) + const sub = t.subscribe.bind(t) + t.subscribe = (subId, coll, handler, where, orderBy, limit, since) => { + if (limit !== K) return sub(subId, coll, handler, where, orderBy, limit, since) + const snapshot: Array = [] + return sub(subId, coll, { + ...handler, + onSnap(key, row) { + snapshot.push(row as Row) + handler.onSnap(key, row) + }, + onSnapEnd() { + boundedSnapshots.push([...snapshot]) + handler.onSnapEnd() + }, + onRestart() { + snapshot.length = 0 + handler.onRestart?.() + }, + }, where, orderBy, limit, since) + } + return t +} + +// NULLs + strings whose locale and lexical orders AGREE ("01".."10"): isolates `nulls`. +const nullRows: Array = [ + ...Array.from({ length: 10 }, (_, i) => ({ id: `m${i}`, body: String(i + 1).padStart(2, "0") })), + { id: "n1", body: null }, + { id: "n2", body: null }, +] +// Mixed case, no NULLs: isolates collation. Byte order: B D F a c e; locale: a B c D e F. +const caseRows: Array = ["apple", "Banana", "cherry", "Date", "eel", "Fig"].map((b, i) => ({ id: `c${i}`, body: b })) + +type Opts = { direction: "asc" | "desc"; nulls?: "first" | "last"; stringSort?: "lexical" | "locale" } + +const cases: Array<{ name: string; rows: Array; opts: Opts }> = [] +for (const stringSort of ["lexical", undefined] as const) { + for (const direction of ["asc", "desc"] as const) { + for (const nulls of [undefined, "first", "last"] as const) { + cases.push({ name: `nulls ${direction} nulls=${nulls ?? "default"} sort=${stringSort ?? "default(locale)"}`, rows: nullRows, opts: { direction, nulls, stringSort } }) + } + } +} +for (const stringSort of ["locale", undefined, "lexical"] as const) { + for (const direction of ["asc", "desc"] as const) { + cases.push({ name: `case ${direction} sort=${stringSort ?? "default(locale)"}`, rows: caseRows, opts: { direction, stringSort } }) + } +} + +const K = 3 +const bodies = (q: { toArray: Array<{ body: string | null }> }) => q.toArray.map((m) => m.body) +const strip = (o: Opts): Record => Object.fromEntries(Object.entries(o).filter(([, v]) => v !== undefined)) + +describe(`F4 bounded snapshot vs TanStack comparator (@tanstack/db ${version})`, () => { + for (const indexed of [false, true]) { + for (const c of cases) { + // bugbash F4-locale: SQLite BINARY cannot reproduce JS localeCompare. + // 0.9.x masks the bad bounded SQL result by subsequently fetching the full subset. + const locale = c.rows === caseRows && c.opts.stringSort !== "lexical" + let bounded: Array | undefined + let expected: Array | undefined + it(`${c.name} index=${indexed}`, async () => { + const opts = strip(c.opts) as Opts + // TanStack's own answer over all rows. + const local = createCollection(localOnlyCollectionOptions({ id: `l-${crypto.randomUUID()}`, getKey: (r) => r.id, initialData: c.rows })) + const want = createLiveQueryCollection((q) => q.from({ m: local }).orderBy(({ m }) => m.body, opts).limit(K)) + await want.preload() + + const r = room("f4") + await seed(r, c.rows) + const boundedSnapshots: Array> = [] + const t = transportFor(r, boundedSnapshots) + const coll = createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (m: Row) => m.id, syncMode: "on-demand" })) + if (indexed) { + coll.createIndex((m) => m.body, { indexType: BTreeIndex, options: { + compareOptions: { direction: opts.direction, nulls: opts.nulls ?? "first", stringSort: opts.stringSort ?? "locale" }, + } }) + } + const got = createLiveQueryCollection((q) => q.from({ m: coll }).orderBy(({ m }) => m.body, opts).limit(K)) + try { + await got.preload() + await waitFor(() => got.size === K && boundedSnapshots.length > 0) + const w = bodies(want) + expect(w.length).toBe(K) + bounded = boundedSnapshots[0]!.map((row) => row.body) + expected = w + if (!locale) expect(bounded).toEqual(w) + if (version === "0.9.x") { + await waitFor(() => JSON.stringify(bodies(got)) === JSON.stringify(w)) + } + if (!locale || version === "0.9.x") expect(bodies(got)).toEqual(w) + } finally { + t.close() + } + }) + if (locale) { + // bugbash F4-locale: the SQL snapshot alone is wrong, even when 0.9.x later recovers. + it.fails(`${c.name} index=${indexed} bounded locale parity`, () => { + expect(bounded).toEqual(expected) + }) + } + } + } + + const tiedRows: Array = ["z", "a", "m", "b"].map((id) => ({ id, body: null })) + let tiedSnapshot: Array | undefined + let tiedExpected: Array | undefined + it("captures a bounded NULL tie snapshot from the real DO", async () => { + const local = createCollection(localOnlyCollectionOptions({ + id: `ties-${crypto.randomUUID()}`, getKey: (row) => row.id, initialData: tiedRows, + })) + const want = createLiveQueryCollection((query) => query.from({ row: local }).orderBy(({ row }) => row.body, { + direction: "desc", nulls: "first", stringSort: "lexical", + }).limit(K)) + await want.preload() + tiedExpected = want.toArray.map((row) => row.id) + + const r = room("ties") + await seed(r, tiedRows) + const snapshots: Array> = [] + const transport = transportFor(r, snapshots) + try { + await transport.subscribe("ties", "messages", { + onSnap() {}, onSnapEnd() {}, onDelta() {}, onUptodate() {}, onReset() {}, + }, undefined, [{ expression: { type: "ref", path: ["body"] }, compareOptions: { + direction: "desc", nulls: "first", stringSort: "lexical", + } }], K) + await waitFor(() => snapshots.length > 0) + tiedSnapshot = snapshots[0]!.map((row) => row.id) + expect(tiedExpected).toHaveLength(K) + expect(tiedSnapshot).toHaveLength(K) + } finally { + transport.close() + } + }) + + // bugbash F4-ties: the bounded ORDER BY ends in the pk, TanStack's row-key tie-break. + it("selects the same three NULL rows as TanStack when ties cross the limit", () => { + expect(tiedSnapshot).toEqual(tiedExpected) + }) + + // 0.9.x scrolls an indexed, cursor-expressible window with a cursor fetch. + // For nulls: "last" its `whereFrom` is `gt(v) OR isNull OR isUndefined`, and + // isNull/isUndefined joined the server's predicate floor for this (ADR-0013 + // amendment 2026-09-29, bugbash F4-cursor): before, the fetch was refused as an + // empty page and the window never grew. nulls: "first" is the control: the same + // path, with a cursor the floor always compiled. 0.8.x never scrolls on + // setWindow (see on-demand-contracts), so there is nothing to pin. + for (const nulls of ["first", "last"] as const) { + const run = version === "0.8.x" ? it.skip : it + let grown: Array | undefined + let expected: Array | undefined + run(`scrolls an indexed asc nulls=${nulls} window with a cursor fetch`, async () => { + const opts = { direction: "asc", nulls, stringSort: "lexical" } as const + const all = nullRows.length + const local = createCollection(localOnlyCollectionOptions({ id: `s-${crypto.randomUUID()}`, getKey: (row) => row.id, initialData: nullRows })) + const want = createLiveQueryCollection((q) => q.from({ m: local }).orderBy(({ m }) => m.body, opts).limit(all)) + await want.preload() + + const r = room("scroll") + await seed(r, nullRows) + const t = transportFor(r, []) + let cursorPages = 0 + const fetch = t.fetch.bind(t) + t.fetch = async (frame) => { + const page = await fetch(frame) + if (frame.cursor) cursorPages++ + return page + } + const coll = createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (m: Row) => m.id, syncMode: "on-demand" })) + coll.createIndex((m) => m.body, { indexType: BTreeIndex, options: { compareOptions: opts } }) + const got = createLiveQueryCollection((q) => q.from({ m: coll }).orderBy(({ m }) => m.body, opts).limit(K)) + try { + await got.preload() + await waitFor(() => got.size === K) + const scrolled = (got as unknown as { utils: { setWindow(w: { offset: number; limit: number }): true | Promise } }).utils.setWindow({ offset: 0, limit: all }) + expect(scrolled).not.toBe(true) + await scrolled + expect(cursorPages).toBeGreaterThan(0) + grown = bodies(got) + expected = bodies(want) + expect(expected).toHaveLength(all) + } finally { + t.close() + } + }) + const parity = version === "0.8.x" ? it.skip : it + parity(`the scrolled asc nulls=${nulls} window matches TanStack's order`, () => { + expect(grown).toEqual(expected) + }) + } +}) diff --git a/tests/order-by-ties.test.ts b/tests/order-by-ties.test.ts new file mode 100644 index 0000000..c99c95c --- /dev/null +++ b/tests/order-by-ties.test.ts @@ -0,0 +1,115 @@ +// WHY (bugbash F4-ties): when more rows tie on the order value than the limit, +// WHICH rows land in the window is decided by the tie-break. @tanstack/db breaks +// ties by row key, ascending regardless of the order direction (db-ivm +// `createKeyedComparator` -> `compareKeys`, identical in 0.8.6/db-ivm 0.1.19 and +// 0.9.2/db-ivm 0.1.22). SQLite's top-k otherwise picks by scan order, so the +// server would ship a different window than the client's own live query holds. +// Every bounded path builds its ORDER BY through `compileSubsetQuery`; these tests +// drive the real sub and fetch paths and compare with TanStack's own answer. +import { createCollection, createLiveQueryCollection, localOnlyCollectionOptions } from "@tanstack/db" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ClientFrame, ServerFrame } from "../src/wire/frames.ts" + +const codec = createFrameCodec() +const K = 3 + +type Row = { id: string; body: string | null } + +async function openWs(room: string): Promise { + const res = await SELF.fetch(`https://example.com/sync/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket + if (!ws) throw new Error("no webSocket") + ws.accept() + return ws +} + +function collectUntil(ws: WebSocket, done: (f: ServerFrame) => boolean): Promise> { + return new Promise((resolve, reject) => { + const out: Array = [] + const timer = setTimeout(() => reject(new Error(`timeout; got [${out.map((f) => f.t).join(",")}]`)), 3000) + const onMsg = (e: MessageEvent): void => { + out.push(codec.decode(e.data as ArrayBuffer) as ServerFrame) + if (done(out[out.length - 1]!)) { + clearTimeout(timer) + ws.removeEventListener("message", onMsg) + resolve(out) + } + } + ws.addEventListener("message", onMsg) + }) +} + +const send = (ws: WebSocket, f: ClientFrame): void => ws.send(codec.encode(f)) +const ref = (col: string): unknown => ({ type: "ref", path: [col] }) +const val = (value: unknown): unknown => ({ type: "val", value }) +const fn = (name: string, ...args: Array): unknown => ({ type: "func", name, args }) + +// Inserted in an order that is NOT key order, so scan order != key order. "10" < "9" +// as strings: the tie-break is string comparison, not numeric. +const ids = ["z", "9", "a", "m", "10", "b"] + +async function seed(room: string, body: string | null): Promise> { + const rows = ids.map((id) => ({ id, body })) + await runInDurableObject(env.SYNC_DO.get(env.SYNC_DO.idFromName(room)), (_i, s) => { + for (const r of rows) s.storage.sql.exec("INSERT INTO messages(id,body) VALUES(?,?)", r.id, r.body) + }) + return rows +} + +// TanStack's own answer for the window over the same rows. +async function tanstackWindow(rows: Array, direction: "asc" | "desc", nulls: "first" | "last"): Promise> { + const local = createCollection(localOnlyCollectionOptions({ id: `t-${crypto.randomUUID()}`, getKey: (r) => r.id, initialData: rows })) + const q = createLiveQueryCollection((b) => + b.from({ m: local }).orderBy(({ m }) => m.body, { direction, nulls, stringSort: "lexical" }).limit(K), + ) + await q.preload() + return q.toArray.map((r) => r.id) +} + +const orderBy = (direction: "asc" | "desc", nulls: "first" | "last") => + [{ expression: ref("body"), compareOptions: { direction, nulls, stringSort: "lexical" } }] as never + +describe("bounded ORDER BY breaks ties by row key like @tanstack/db (bugbash F4-ties)", () => { + for (const direction of ["asc", "desc"] as const) { + for (const nulls of ["first", "last"] as const) { + it(`sub snapshot: all-NULL ties, ${direction} nulls=${nulls}`, async () => { + const room = `ties-sub-${crypto.randomUUID()}` + const rows = await seed(room, null) + const ws = await openWs(room) + send(ws, { t: "sub", subId: "s1", collection: "messages", orderBy: orderBy(direction, nulls), limit: K }) + const frames = await collectUntil(ws, (f) => f.t === "snap-end") + ws.close() + const got = frames.filter((f): f is Extract => f.t === "snap").map((f) => f.key as string) + expect(got).toEqual(await tanstackWindow(rows, direction, nulls)) + expect(got).toEqual(["10", "9", "a"]) + }) + + it(`fetch page: all-NULL ties, ${direction} nulls=${nulls}`, async () => { + const room = `ties-fetch-${crypto.randomUUID()}` + const rows = await seed(room, null) + const ws = await openWs(room) + send(ws, { t: "fetch", fetchId: "f1", collection: "messages", orderBy: orderBy(direction, nulls), limit: K }) + const [page] = await collectUntil(ws, (f) => f.t === "page") + ws.close() + const got = (page as Extract).rows.map((r) => (r as Row).id) + expect(got).toEqual(await tanstackWindow(rows, direction, nulls)) + }) + } + } + + it("fetch page after a cursor: the bounded next page picks tied rows by key", async () => { + const room = `ties-cursor-${crypto.randomUUID()}` + await seed(room, "v") + const ws = await openWs(room) + // whereFrom = gt("a"): every seeded row qualifies and all tie on body = "v". + send(ws, { + t: "fetch", fetchId: "f1", collection: "messages", orderBy: orderBy("asc", "first"), limit: K, + cursor: { whereFrom: fn("gt", ref("body"), val("a")) as never, whereCurrent: fn("eq", ref("body"), val("a")) as never }, + }) + const [page] = await collectUntil(ws, (f) => f.t === "page") + ws.close() + expect((page as Extract).rows.map((r) => (r as Row).id)).toEqual(["10", "9", "a"]) + }) +}) diff --git a/tests/predicate-parity.test.ts b/tests/predicate-parity.test.ts index fa8f767..0354dc0 100644 --- a/tests/predicate-parity.test.ts +++ b/tests/predicate-parity.test.ts @@ -186,6 +186,41 @@ describe("SQL/JS predicate parity (ADR-0013)", () => { expect(snap).toEqual(["lo", "up"]) }) + // isNull (bugbash F4-cursor): a nulls-last cursor's whereFrom is + // `gt(v) OR isNull(col) OR isUndefined(col)`. SQL IS NULL and JS `=== null` agree + // because the delta path evaluates rows hydrated from SQLite, where a NULL column + // is `null` whether the write omitted it or sent it explicitly. + it("isNull: only the NULL row matches on both paths", async () => { + const where = fn("isNull", ref("body")) + const snap = await snapshotMembers("pp-isnull-snap", where) + const delta = await deltaMembers("pp-isnull-delta", where) + const deltaExplicit = await deltaMembers("pp-isnull-delta-explicit", where, true) + expect(snap, `isNull mismatch — snap=${snap} delta=${delta}`).toEqual(delta) + expect(delta, `isNull null-spelling mismatch — omit=${delta} explicit=${deltaExplicit}`).toEqual(deltaExplicit) + expect(snap).toEqual(["n1"]) + }) + + // isUndefined: SQLite has no undefined and hydration yields null, so it matches + // nothing on either path — including the NULL row. + it("isUndefined: matches no stored row on either path, NULL included", async () => { + const where = fn("isUndefined", ref("body")) + const snap = await snapshotMembers("pp-isundef-snap", where) + const delta = await deltaMembers("pp-isundef-delta", where) + const deltaExplicit = await deltaMembers("pp-isundef-delta-explicit", where, true) + expect(snap, `isUndefined mismatch — snap=${snap} delta=${delta}`).toEqual(delta) + expect(delta, `isUndefined null-spelling mismatch — omit=${delta} explicit=${deltaExplicit}`).toEqual(deltaExplicit) + expect(snap).toEqual([]) + }) + + // The nulls-last cursor shape TanStack 0.9 emits, composed as a whole. + it("gt OR isNull OR isUndefined: the cursor whereFrom agrees across paths", async () => { + const where = fn("or", fn("gt", ref("body"), val("a")), fn("isNull", ref("body")), fn("isUndefined", ref("body"))) + const snap = await snapshotMembers("pp-cursor-snap", where) + const delta = await deltaMembers("pp-cursor-delta", where) + expect(snap, `cursor mismatch — snap=${snap} delta=${delta}`).toEqual(delta) + expect(snap).toEqual(["lo", "n1", "x"]) + }) + // in: SQL NULL IN (...) → NULL → n1 excluded; JS .includes(null) → false. "HELLO" not in list. it("in: NULL body excluded, exact match only (no case folding)", async () => { const where = fn("in", ref("body"), val(["hello", "x"])) diff --git a/tests/reconnect-bootstrap.test.ts b/tests/reconnect-bootstrap.test.ts index e7c233d..1f2c63a 100644 --- a/tests/reconnect-bootstrap.test.ts +++ b/tests/reconnect-bootstrap.test.ts @@ -182,7 +182,7 @@ describe("a fetch page from an abandoned socket", () => { () => "resolved", (e: Error) => `rejected: ${e.message}`, ) - await sleep(30) // the page is held on the old socket + await waitFor(() => gate.queue.length >= 1) // the page is held on the old socket t.seedCursor("1") // regress: forced reconnect onto a fresh socket const held = gate.queue.splice(0) gate.hold = false diff --git a/tests/refusals-fail-loud.test.ts b/tests/refusals-fail-loud.test.ts new file mode 100644 index 0000000..3815708 --- /dev/null +++ b/tests/refusals-fail-loud.test.ts @@ -0,0 +1,268 @@ +import { createCollection, createLiveQueryCollection, ilike } from "@tanstack/db" +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { doCollectionOptions } from "../src/client/do-collection.ts" +import { WebSocketTransport, type WebSocketLike } from "../src/client/transport.ts" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ClientFrame, ServerFrame } from "../src/wire/frames.ts" +import type { TestApi } from "./test-worker.ts" + +// bugbash F5 (design gap; the fix is a wire ADR, not written yet). WHY: every +// refusal cause (unknown collection, malformed cursor, unsupported predicate, +// sub cap) answers with the frame a genuinely EMPTY result gets — `page +// rows:[]` for a fetch, `reset` with no snapshot for a sub — so the app sees a +// successful empty load and a typo'd table or an off-floor query renders as +// "no rows". The only trace is a server-side console.error. +// +// These are the scenarios a "refusals fail loud" design must satisfy, one per +// cause and surface, each `it.fails` until that design lands. They assert only +// what any such design must deliver — a correlated reply that is neither the +// empty result's nor the below-floor `reset`; a load that rejects — not a frame +// name or error type; the ADR picks those, and its fix should tighten these to +// them. The passing tests pin what holds before AND after: an empty result +// stays a success, and every refusal settles promptly (so a hang cannot hide +// behind a `.fails`). +// +// Existing tests that pin "refusal = empty" as correct today; a fix must +// supersede them deliberately (not changed here): +// - sync-read.test.ts "subscribing to an unknown collection fails safe with reset" +// - wire-hardening.test.ts "subscription cap: third sub on 2-cap DO → reset; …" +// - on-demand.test.ts "resolves loadSubset on a rejected sub (reset, no snap-end) instead of hanging" +// - on-demand.test.ts "rejects a malformed cursor (missing a half) instead of scanning the table" (expects []) +// - on-demand-contracts.test.ts "a request covered by a pending watch that is then refused opens its own" (fake reset) +// - on-demand-contracts.test.ts "an unsupported predicate beside a valid subset: …" swallows bad.preload()'s outcome; +// its containment assertions should survive a fix. +// Precedent: the SSR read path already fails loud on an unknown collection +// (read-sync-snapshot.test.ts "throws on an unknown collection (fail loud, not empty-success)"). + +const codec = createFrameCodec() +const ref = (col: string) => ({ type: "ref", path: [col] }) +const val = (value: unknown) => ({ type: "val", value }) +const fn = (name: string, ...args: Array) => ({ type: "func", name, args }) +const ILIKE = fn("ilike", ref("body"), val("%b%")) // outside the SQL predicate floor (ADR-0013) +const HALF_CURSOR = { whereFrom: fn("gt", ref("body"), val("")) } // whereCurrent missing (ADR-0005) + +async function openWs(path: string): Promise { + const res = await SELF.fetch(`https://example.com${path}`, { headers: { Upgrade: "websocket" } }) + const ws = res.webSocket! + ws.accept() + return ws +} + +type Correlated = ServerFrame & { fetchId?: string; sub?: string; subId?: string } + +/** Send `f`, resolve with the first frame correlated to it that is not a row + * (`snap`/`d`). Never rejects: a missing reply resolves `undefined`. */ +function reply(ws: WebSocket, f: ClientFrame, id: string, timeoutMs = 2000): Promise { + return new Promise((resolve) => { + const timer = setTimeout(() => { + ws.removeEventListener("message", onMsg) + resolve(undefined) + }, timeoutMs) + const onMsg = (e: MessageEvent): void => { + const x = codec.decode(e.data as ArrayBuffer) as Correlated + if (x.t === "snap" || x.t === "d") return + if (x.fetchId === id || x.sub === id || x.subId === id) { + clearTimeout(timer) + ws.removeEventListener("message", onMsg) + resolve(x) + } + } + ws.addEventListener("message", onMsg) + ws.send(codec.encode(f)) + }) +} + +async function seed(ns: DurableObjectNamespace, room: string): Promise { + await runInDurableObject(ns.get(ns.idFromName(room)), (inst, s) => { + for (let i = 1; i <= 3; i++) s.storage.sql.exec("INSERT INTO messages(id, body) VALUES (?, ?)", `m${i}`, `b${i}`) + ;(inst as unknown as { sync: { drainAndBroadcast(): void } }).sync.drainAndBroadcast() + }) +} + +type FetchBody = Omit, "t" | "fetchId"> +type SubBody = Omit, "t" | "subId"> +type WireCase = { name: string; kind: "fetch" | "sub"; run: () => Promise } + +// Each case on a fresh LimitsTestDO (maxSubsPerSocket = 2) seeded with 3 rows. +async function onLimitsSocket(fn: (ws: WebSocket) => Promise): Promise { + const room = `f5-wire-${crypto.randomUUID()}` + await seed(env.LIMITS_DO, room) + const ws = await openWs(`/limits/${room}`) + try { + return await fn(ws) + } finally { + ws.close() + } +} +const fetchCase = (name: string, body: FetchBody): WireCase => ({ + name: `fetch refused for ${name}`, + kind: "fetch", + run: () => onLimitsSocket((ws) => reply(ws, { t: "fetch", fetchId: "f1", ...body }, "f1")), +}) +const subCase = (name: string, body: SubBody): WireCase => ({ + name: `sub refused for ${name}`, + kind: "sub", + run: () => onLimitsSocket((ws) => reply(ws, { t: "sub", subId: "s1", ...body }, "s1")), +}) +const WIRE_CASES: Array = [ + fetchCase("unknown collection", { collection: "nope" }), + fetchCase("malformed cursor (missing whereCurrent)", { collection: "messages", cursor: HALF_CURSOR as never }), + fetchCase("unsupported predicate (ilike)", { collection: "messages", where: ILIKE }), + subCase("unknown collection", { collection: "nope" }), + subCase("unsupported predicate (ilike)", { collection: "messages", where: ILIKE }), + { + name: "sub refused over maxSubsPerSocket", + kind: "sub", + run: () => + onLimitsSocket(async (ws) => { + await reply(ws, { t: "sub", subId: "s1", collection: "messages" }, "s1") + await reply(ws, { t: "sub", subId: "s2", collection: "files" }, "s2") + return reply(ws, { t: "sub", subId: "s3", collection: "validated" }, "s3") + }), + }, +] + +describe("F5 wire: a refusal is distinguishable from an empty result", () => { + it("control: a genuinely empty fetch is `page rows:[]` and a genuinely empty sub is `snap-end`", async () => { + const none = fn("eq", ref("body"), val("none")) + const [page, snap] = await onLimitsSocket(async (ws) => [ + await reply(ws, { t: "fetch", fetchId: "f-empty", collection: "messages", where: none }, "f-empty"), + await reply(ws, { t: "sub", subId: "s-empty", collection: "messages", where: none }, "s-empty"), + ]) + expect(page).toMatchObject({ t: "page", rows: [] }) + expect(snap?.t).toBe("snap-end") + }) + + it("every refusal gets a prompt, correlated reply (never silence)", async () => { + for (const c of WIRE_CASES) expect(await c.run(), c.name).toBeDefined() + }) + + for (const c of WIRE_CASES) { + it.fails(`${c.name}: the reply is neither the empty result nor a bare reset`, async () => { + const r = await c.run() + expect(r, "no correlated reply").toBeDefined() + if (c.kind === "fetch") { + const { fetchId: _id, seq: _seq, ...shape } = r as Correlated & { seq?: string } + expect(shape).not.toEqual({ t: "page", rows: [] }) // today: exactly this + } else { + expect(r!.t).not.toBe("snap-end") + expect(r!.t).not.toBe("reset") // today: `reset`, which also means "below the floor, resnapshot" + } + }) + } +}) + +function realTransport(path: string): WebSocketTransport { + return new WebSocketTransport({ + url: `https://example.com${path}`, + timeoutMs: 5000, + open: async () => (await openWs(path)) as unknown as WebSocketLike, + }) +} + +/** How a promise settles within `ms`; never throws. */ +async function outcome(p: Promise, ms = 1500): Promise { + return Promise.race([ + p.then( + () => "resolved", + (e: unknown) => `rejected: ${String(e)}`, + ), + new Promise((r) => setTimeout(() => r("pending"), ms)), + ]) +} + +// `status` is set for eager collections: a refused eager sub should fail the +// collection's readiness (markError), the route an eager receipt failure takes. +type AppCase = { name: string; run: () => Promise<{ o: string; status?: string }> } + +async function withTransport(path: "sync" | "limits", fn: (t: WebSocketTransport) => Promise): Promise { + const room = `f5-app-${crypto.randomUUID()}` + await seed(path === "sync" ? env.SYNC_DO : env.LIMITS_DO, room) + const t = realTransport(`/${path}/${room}`) + try { + return await fn(t) + } finally { + t.close() + } +} +const fetchApp = (name: string, body: FetchBody): AppCase => ({ + name: `transport.fetch with ${name} rejects instead of resolving []`, + run: () => + withTransport("sync", async (t) => { + await t.connect() + return { o: await outcome(t.fetch({ t: "fetch", fetchId: "f1", ...body })) } + }), +}) +const APP_CASES: Array = [ + { + name: "eager collection on an unknown table: preload() rejects and the collection is in error", + run: () => + withTransport("sync", async (t) => { + const c = createCollection(doCollectionOptions({ transport: t, table: "nope" as "messages", getKey: (r) => r.id })) + return { o: await outcome(c.preload()), status: c.status } + }), + }, + { + name: "eager collection with an unsupported static `where`: preload() rejects and the collection is in error", + run: () => + withTransport("sync", async (t) => { + const c = createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (r) => r.id, where: ILIKE })) + return { o: await outcome(c.preload()), status: c.status } // today: ready, size 0; the server holds 3 matching rows + }), + }, + { + name: "eager collection past the sub cap: its preload() rejects and it is in error", + run: () => + withTransport("limits", async (t) => { + // maxSubsPerSocket = 2; three collections share one socket + await createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (r) => r.id })).preload() + await createCollection(doCollectionOptions({ transport: t, table: "files", getKey: (r) => r.id })).preload() + const c = createCollection(doCollectionOptions({ transport: t, table: "validated", getKey: (r) => r.id })) + return { o: await outcome(c.preload()), status: c.status } + }), + }, + { + name: "on-demand live query on an unknown table: preload() rejects", + run: () => + withTransport("sync", async (t) => { + const c = createCollection(doCollectionOptions({ transport: t, table: "nope" as "messages", getKey: (r) => r.id, syncMode: "on-demand" })) + return { o: await outcome(createLiveQueryCollection((qb) => qb.from({ n: c })).preload()) } + }), + }, + { + name: "on-demand live query with an unsupported predicate (ilike): preload() rejects", + run: () => + withTransport("sync", async (t) => { + const messages = createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (m) => m.id, syncMode: "on-demand" })) + const q = createLiveQueryCollection((qb) => qb.from({ m: messages }).where(({ m }) => ilike(m.body, "%b%"))) + return { o: await outcome(q.preload()) } // today: ready, size 0; the server holds 3 matching rows + }), + }, + fetchApp("an unknown collection", { collection: "nope" }), + fetchApp("a malformed cursor", { collection: "messages", cursor: HALF_CURSOR as never }), + fetchApp("an unsupported predicate (ilike)", { collection: "messages", where: ILIKE }), +] + +describe("F5 app: a refused load rejects instead of settling ready + empty", () => { + it("control: an eager collection on a registered table preloads ready with its rows", async () => { + const { o, size } = await withTransport("sync", async (t) => { + const c = createCollection(doCollectionOptions({ transport: t, table: "messages", getKey: (r) => r.id })) + return { o: await outcome(c.preload()), size: c.size } + }) + expect(o).toBe("resolved") + expect(size).toBe(3) + }) + + it("every refused load settles promptly (never hangs)", async () => { + for (const c of APP_CASES) expect((await c.run()).o, c.name).not.toBe("pending") + }) + + for (const c of APP_CASES) { + it.fails(c.name, async () => { + const { o, status } = await c.run() + expect(o).toMatch(/^rejected/) // today: resolved + if (status !== undefined) expect(status).toBe("error") // today: ready + }) + } +}) diff --git a/tests/socket-close-timer.test.ts b/tests/socket-close-timer.test.ts new file mode 100644 index 0000000..6d849a6 --- /dev/null +++ b/tests/socket-close-timer.test.ts @@ -0,0 +1,66 @@ +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { expect, it, vi } from "vitest" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ServerFrame } from "../src/wire/frames.ts" + +const codec = createFrameCodec() + +async function waitFor(pred: () => boolean | Promise, timeoutMs = 3000): Promise { + const start = Date.now() + while (!(await pred())) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((resolve) => setTimeout(resolve, 5)) + } +} + +it.each(["close", "error"])("a socket %s cancels the only subscribed socket's buffered 30-second tick", async (event) => { + const room = `close-timer-${crypto.randomUUID()}` + const stub = env.SLOW_DO.get(env.SLOW_DO.idFromName(room)) + const response = await SELF.fetch(`https://example.com/slow/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = response.webSocket! + ws.accept() + const frames: Array = [] + ws.addEventListener("message", (event) => { + frames.push(codec.decode((event as MessageEvent).data as ArrayBuffer) as ServerFrame) + }) + const cancel = vi.spyOn(globalThis, "clearTimeout") + let pendingTimer: ReturnType | undefined + try { + ws.send(codec.encode({ t: "sub", subId: "s", collection: "messages" })) + await waitFor(() => frames.some((frame) => frame.t === "snap-end")) + pendingTimer = await runInDurableObject(stub, (instance, state) => { + const schedule = vi.spyOn(globalThis, "setTimeout") + try { + state.storage.sql.exec("INSERT INTO messages(id,body) VALUES('x','v')") + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + const tick = schedule.mock.calls.findIndex(([, delay]) => delay === 30_000) + expect(tick).toBeGreaterThanOrEqual(0) + return schedule.mock.results[tick]!.value as ReturnType + } finally { + schedule.mockRestore() + } + }) + expect(cancel.mock.calls.some(([timer]) => timer === pendingTimer)).toBe(false) + expect(frames.some((frame) => frame.t === "d")).toBe(false) + + if (event === "close") { + ws.close() + } else { + await runInDurableObject(stub, (instance, state) => { + const [serverWs] = state.getWebSockets() + expect(serverWs).toBeDefined() + ;(instance as unknown as { webSocketError(ws: WebSocket, error: unknown): void }) + .webSocketError(serverWs!, new Error("socket failed")) + }) + } + await waitFor(() => runInDurableObject(stub, (_instance, state) => + state.storage.sql.exec<{ count: number }>("SELECT COUNT(*) AS count FROM _sync_subs").one().count === 0, + )) + expect(cancel.mock.calls.some(([timer]) => timer === pendingTimer)).toBe(true) + expect(frames.some((frame) => frame.t === "d")).toBe(false) + } finally { + cancel.mockRestore() + ws.close() + if (pendingTimer !== undefined) await runInDurableObject(stub, () => clearTimeout(pendingTimer)) + } +}) diff --git a/tests/sql-compiler.test.ts b/tests/sql-compiler.test.ts index eb1ef2e..87f22cc 100644 --- a/tests/sql-compiler.test.ts +++ b/tests/sql-compiler.test.ts @@ -41,16 +41,27 @@ describe("IR -> SQL compiler (M6)", () => { expect(compileWhere(fn("in", ref("id"), val([])))).toEqual({ sql: `0`, params: [] }) }) - it("rejects operators outside the floor (ne, ilike, isNull, functions)", () => { + it("rejects operators outside the floor (ne, ilike, functions)", () => { // `ne` was removed from the floor (ADR-0013): @tanstack/db's evaluator has no // `ne` (only `not(eq(...))`), so accepting it in SQL desynced the snapshot and // delta paths. It must now be rejected like any other off-floor operator. expect(() => compileWhere(fn("ne", ref("body"), val("x")))).toThrow(UnsupportedPredicateError) expect(() => compileWhere(fn("ilike", ref("body"), val("x")))).toThrow(UnsupportedPredicateError) - expect(() => compileWhere(fn("isNull", ref("body")))).toThrow(UnsupportedPredicateError) expect(() => compileWhere(fn("upper", ref("body")))).toThrow(/not supported/) }) + it("compiles isNull to IS NULL and isUndefined to constant-false (ADR-0013 amendment)", () => { + expect(compileWhere(fn("isNull", ref("body")))).toEqual({ sql: `"body" IS NULL`, params: [] }) + expect(compileWhere(fn("isUndefined", ref("body")))).toEqual({ sql: `0`, params: [] }) + expect(compileWhere(fn("or", fn("gt", ref("body"), val("a")), fn("isNull", ref("body")), fn("isUndefined", ref("body"))))).toEqual({ + sql: `("body" > ? OR "body" IS NULL OR 0)`, + params: ["a"], + }) + // Still validated: a non-column operand is rejected, not silently constant-folded. + expect(() => compileWhere(fn("isUndefined", val(1)))).toThrow(/column reference/) + expect(() => compileWhere(fn("isNull", ref("a"), ref("b")))).toThrow(/1 argument/) + }) + it("the supported 'not equal' is not(eq(...)), which lowers correctly", () => { // The canonical not-equal a @tanstack/db client emits — both floors agree on it. expect(compileWhere(fn("not", fn("eq", ref("body"), val("x"))))).toEqual({ @@ -70,32 +81,51 @@ describe("IR -> SQL compiler (M6)", () => { it("builds a full subset SELECT with where/orderBy/limit/offset", () => { const q = compileSubsetQuery("messages", { + pk: "id", where: fn("eq", ref("body"), val("x")), orderBy: [{ col: "created_at", dir: "desc" }], limit: 10, offset: 20, }) - expect(q.sql).toBe(`SELECT * FROM "messages" WHERE "body" = ? ORDER BY "created_at" DESC LIMIT ? OFFSET ?`) + expect(q.sql).toBe(`SELECT * FROM "messages" WHERE "body" = ? ORDER BY "created_at" DESC NULLS FIRST, "id" COLLATE BINARY ASC LIMIT ? OFFSET ?`) expect(q.params).toEqual(["x", 10, 20]) }) it("emits LIMIT -1 when offset is given without a limit (SQLite requirement)", () => { - const q = compileSubsetQuery("t", { offset: 5 }) + const q = compileSubsetQuery("t", { pk: "id", offset: 5 }) expect(q.sql).toBe(`SELECT * FROM "t" ORDER BY rowid LIMIT -1 OFFSET ?`) expect(q.params).toEqual([5]) }) + it("preserves each orderBy clause's null placement", () => { + const q = compileSubsetQuery("t", { + pk: "id", + orderBy: [ + { expression: ref("body"), compareOptions: { direction: "desc", nulls: "last" } }, + { expression: ref("id"), compareOptions: { direction: "asc" } }, + ], + }) + expect(q.sql).toBe(`SELECT * FROM "t" ORDER BY "body" DESC NULLS LAST, "id" ASC NULLS FIRST, "id" COLLATE BINARY ASC`) + }) + + it("breaks value ties by pk ascending, whatever the order direction (bugbash F4-ties)", () => { + // @tanstack/db's createKeyedComparator: value first, then row key ascending. + const desc = compileSubsetQuery("t", { pk: "key", orderBy: [{ col: "body", dir: "desc" }], limit: 3 }) + expect(desc.sql).toBe(`SELECT * FROM "t" ORDER BY "body" DESC NULLS FIRST, "key" COLLATE BINARY ASC LIMIT ?`) + expect(() => compileSubsetQuery("t", { pk: "k; DROP", orderBy: [{ col: "body" }] })).toThrow(/pk column/) + }) + it("defaults to ORDER BY rowid when the client sends no orderBy (deterministic snapshot order)", () => { // Field-verified regression: a bare SELECT with no ORDER BY leaves row order // as an accident of SQLite's query plan. `rowid` is always available (D9 // forbids an INTEGER PRIMARY KEY pk) and matches insertion order among // currently-live rows. - const q = compileSubsetQuery("t", {}) + const q = compileSubsetQuery("t", { pk: "id" }) expect(q.sql).toBe(`SELECT * FROM "t" ORDER BY rowid`) }) it("rejects a negative limit/offset and an invalid orderBy column", () => { - expect(() => compileSubsetQuery("t", { limit: -1 })).toThrow(/non-negative/) - expect(() => compileSubsetQuery("t", { orderBy: [{ col: "a; DROP" }] })).toThrow(/orderBy column/) + expect(() => compileSubsetQuery("t", { pk: "id", limit: -1 })).toThrow(/non-negative/) + expect(() => compileSubsetQuery("t", { pk: "id", orderBy: [{ col: "a; DROP" }] })).toThrow(/orderBy column/) }) }) diff --git a/tests/subset-sql-edges.test.ts b/tests/subset-sql-edges.test.ts new file mode 100644 index 0000000..9f2039c --- /dev/null +++ b/tests/subset-sql-edges.test.ts @@ -0,0 +1,22 @@ +import { env, runInDurableObject } from "cloudflare:test" +import { describe, expect, it } from "vitest" +import { compileSubsetQuery } from "../src/server/sql-compiler.ts" + +// WHY (codex review of the F4-ties fix): the edge runs through real SQLite, since +// the bug is in what SQLite does with the compiled text. + +const inDo = (run: (sql: SqlStorage) => void): Promise => + runInDurableObject(env.SYNC_DO.get(env.SYNC_DO.idFromName(`edges-${crypto.randomUUID()}`)), (_i, s) => run(s.storage.sql)) + +describe("compiled subset SQL edges", () => { + it("the pk tie-break is BINARY even when the pk column declares another collation", async () => { + // @tanstack/db compares row keys with `<` (code-unit order): "B" < "a". A pk + // declared COLLATE NOCASE would otherwise order a, B. + await inDo((sql) => { + sql.exec("CREATE TABLE ci(id TEXT PRIMARY KEY COLLATE NOCASE, v TEXT)") + for (const id of ["a", "B", "c"]) sql.exec("INSERT INTO ci(id,v) VALUES(?,NULL)", id) + const q = compileSubsetQuery("ci", { pk: "id", orderBy: [{ col: "v", dir: "asc" }], limit: 2 }) + expect(sql.exec<{ id: string }>(q.sql, ...q.params).toArray().map((r) => r.id)).toEqual(["B", "a"]) + }) + }) +}) diff --git a/tests/test-worker.ts b/tests/test-worker.ts index e1d68ca..e6ae243 100644 --- a/tests/test-worker.ts +++ b/tests/test-worker.ts @@ -228,11 +228,13 @@ export class UnregisteredDO extends SyncDurableObject {} /** Same collections as SyncTestDO, but with a tiny compaction threshold so the * opportunistic `maybeCompact` housekeeping (compaction + retention prune + - * dedup sweep) fires after a few writes instead of 200 — lets tests exercise - * the real drain → maybeCompact → waitUntil path. Keeps the default 2-day - * `changelogRetentionMs` so the retention wiring is tested as shipped. */ + * orphan-sub sweep) fires after a few writes instead of 200 — lets tests + * exercise the real drain → maybeCompact → waitUntil path. Keeps the default + * 2-day `changelogRetentionMs` so the retention wiring is tested as shipped. + * A 1 s `dedupRetentionMs` lets the dedup sweep gate reopen within a test. */ export class MaintTestDO extends SyncTestDO { protected override readonly compactionEvery = 3 + protected override readonly dedupRetentionMs = 1000 } /** Same collections as SyncTestDO with an effectively-infinite coalescer tick: diff --git a/tests/tick-override.test.ts b/tests/tick-override.test.ts new file mode 100644 index 0000000..6524951 --- /dev/null +++ b/tests/tick-override.test.ts @@ -0,0 +1,48 @@ +import { env, runInDurableObject, SELF } from "cloudflare:test" +import { expect, it, vi } from "vitest" +import { createFrameCodec } from "../src/wire/frame-codec.ts" +import type { ServerFrame } from "../src/wire/frames.ts" + +const codec = createFrameCodec() + +async function waitFor(pred: () => boolean, timeoutMs = 3000): Promise { + const start = Date.now() + while (!pred()) { + if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout") + await new Promise((resolve) => setTimeout(resolve, 5)) + } +} + +it("honours a subclass tickMs field without flushing at the constructor's default tick", async () => { + const room = `tick-override-${crypto.randomUUID()}` + const response = await SELF.fetch(`https://example.com/slow/${room}`, { headers: { Upgrade: "websocket" } }) + const ws = response.webSocket! + ws.accept() + const frames: Array = [] + ws.addEventListener("message", (event) => { + frames.push(codec.decode((event as MessageEvent).data as ArrayBuffer) as ServerFrame) + }) + try { + ws.send(codec.encode({ t: "sub", subId: "s", collection: "messages" })) + await waitFor(() => frames.some((frame) => frame.t === "snap-end")) + const delays = await runInDurableObject(env.SLOW_DO.get(env.SLOW_DO.idFromName(room)), (instance, state) => { + const schedule = vi.spyOn(globalThis, "setTimeout") + try { + state.storage.sql.exec("INSERT INTO messages(id,body) VALUES('x','v')") + ;(instance as unknown as { drainAndBroadcast(): void }).drainAndBroadcast() + return schedule.mock.calls.map(([, delay]) => delay) + } finally { + schedule.mockRestore() + } + }) + expect(delays).toContain(30_000) + await new Promise((resolve) => setTimeout(resolve, 1000)) + expect(frames.some((frame) => frame.t === "d")).toBe(false) + + ws.send(codec.encode({ t: "sub", subId: "barrier", collection: "files" })) + await waitFor(() => frames.some((frame) => frame.t === "snap-end" && frame.sub === "barrier")) + expect(frames.find((frame) => frame.t === "d")).toMatchObject({ key: "x", cols: { body: "v" } }) + } finally { + ws.close() + } +})