Bug bash 2026-09-29: v0.7.0 fixes and coverage

CHANGELOG.md

@@ -8,6 +8,79 @@ While pre-1.0, the public API may change between 0.x releases.
 
 ## [Unreleased]
 
+### Changed
+
+- **BREAKING (raw clients only):** the server rejects an insert or update
+  whose `cols` carry the primary key with a value different from the op
+  `key`, answering with a `VALIDATION` rejection before `authorize` runs.
+  Before, such a row was stored under the `cols` pk, so its optimistic row
+  never reconciled (ADR-0001 D9). The collection client always sends
+  matching keys and is unaffected. An insert that omits the pk from `cols`
+  is still accepted (ADR-0025 amendment).
+
+### Fixed
+
+- **A subclass `tickMs` override now takes effect.** The coalescer read the
+  tick before subclass fields were set, so it always used 50 ms (ADR-0015
+  amendment).
+- **No empty ticks after a flush, close or error.** Flushing the last socket
+  with pending deltas, or closing or erroring a socket, now cancels the flush
+  timer when nothing else is pending, so the DO can hibernate sooner.
+- **Expired dedup receipts are swept on every answered `mut` or `call`**,
+  including rejections, replays and commands that write nothing. The sweep
+  runs at most once per `dedupRetentionMs`, and always on the first receipt
+  after a wake. Before, it ran only with compaction after drained writes, so
+  a DO that only rejected or ran write-free commands grew `_sync_seen_tx`
+  without bound (ADR-0009 amendment).
+- **Bounded on-demand snapshots place NULLs as TanStack does.** The server's
+  `ORDER BY` now emits `NULLS FIRST` or `NULLS LAST` from each clause's
+  `nulls` (default first), so a descending window over a nullable column no
+  longer drops its NULL rows. Locale collation is still pinned as an expected
+  failure (ADR-0023 amendment).
+- **Bounded reads break ties by pk, ascending**, matching TanStack's row-key
+  tie-break, so a `limit` over tied rows picks the same rows on the server and
+  the client (ADR-0023 amendment).
+- **`isNull` and `isUndefined` join the predicate floor**, so a
+  `nulls: "last"` window on `@tanstack/db` 0.9 can page past its first page.
+  Before, its cursor fetch was refused and read as empty (ADR-0013 amendment).
+- **Dedup retention is a true lower bound.** Receipts are stamped in whole
+  seconds, so the sweep could remove one up to 999 ms early.
+- **A fetch page no longer overtakes a buffered delta.** The server flushes
+  the socket's pending deltas before it reads a page, the same as it does
+  for snapshots (ADR-0023 amendment).
+- **Server type declarations no longer need `@cloudflare/workers-types`.**
+  The shipped `.d.ts` files use the ambient Worker SQL types, so a Worker
+  typed with `wrangler types` and `skipLibCheck: false` no longer fails with
+  TS2307. The pack smoke test now checks both type setups (ADR-0022
+  amendment).
+
+### Tests
+
+- Known bugs are now pinned as expected failures (`it.fails`, tagged
+  `bugbash <ID>`), so each fix flips its tests:
+  - The single client cursor skips changes when the socket drops during a
+    resubscribe round (F1): 30 reconnect-under-drop scenarios, 16 of them red.
+  - Known limitations L1 (no incarnation epoch, ADR-0011) and L2 (mid-tick
+    eviction, ADR-0009).
+  - Refusals read as empty results (F5): 14 wire and app scenarios.
+  - Dedup is scoped by txId alone (L3, ADR-0012 D4).
+- Full-row live deltas for narrow and no-op updates (#28) are pinned as
+  current behaviour.
+- Real-path on-demand scenarios pin the ADR-0023 fixes that were only covered
+  by fake transports: unload mid-snapshot, a below-floor reset across several
+  subsets (bites at the 0.8 floor), `.offset()` windows, and late or
+  overlaid pages.
+- The abandoned-socket fetch test waits for the page to be held instead of
+  sleeping 30 ms, which timed out under full-suite load.
+
+### Documentation
+
+- README gains a Known limitations section: DO SQLite can't bind `bigint`,
+  and INTEGER values above 2^53 round on read (workerd#4195, #10). ADR-0001
+  D17 is qualified to match.
+- ADR-0012 D3 points to ADR-0014, which changed how `handleCall` sanitizes
+  authorize errors.
+
 ## [0.7.0] — 2026-09-25
 
 ### Added

CLAUDE.md

@@ -83,9 +83,14 @@ Each is load-bearing and guarded by an ADR — read it before you touch the area
 - **One decision → one ADR.** Conventional commits (`!` = breaking); end with the
   Co-Authored-By trailer. Commit/push only when asked; `main` is the release
   branch.
-- **Run plans and risky changes past codex (`gpt-5.6-sol`) as an adversary**
-  before committing — it has caught real bugs here (the cursor-fetch race, the
+- **Run plans and risky changes past codex as an adversary** before
+  committing — it has caught real bugs here (the cursor-fetch race, the
   scroll-back `DuplicateKeySyncError`). Use the strongest available model; a
-  weaker adversary is a weaker review. Pipe input in (`git diff | codex exec
-  … -`) or close stdin (`codex exec "…" < /dev/null`) — an open stdin hangs
-  codex forever.
+  weaker adversary is a weaker review. Run it as a bb subthread in your own
+  environment, not `codex exec`:
+  `bb thread spawn --project "$BB_PROJECT_ID" --parent-self --environment
+  "$BB_ENVIRONMENT_ID" --provider codex --model gpt-6-sol --reasoning-level
+  high --title "adversarial review: …" --prompt-file <file>`. The prompt
+  names the diff (`git diff --no-ext-diff <base>...HEAD`), says read-only (no
+  edits, no commits), and asks for findings via `bb thread tell
+  "$BB_THREAD_ID"`.

README.md

@@ -457,6 +457,16 @@ show. The conventions, what's verified and how, and the full
 
 ---
 
+## Known limitations
+
+- **Large integers in DO SQLite:** workerd `SqlStorage` cannot bind JS `bigint`,
+  and INTEGER values above `Number.MAX_SAFE_INTEGER` round on read before
+  syncing. For exact values, store decimal strings in TEXT columns and convert
+  at the application edge ([workerd#4195](https://github.com/cloudflare/workerd/issues/4195),
+  [issue #10](https://github.com/grrowl/tanstack-durable-object-sync/issues/10)).
+
+---
+
 ## Non-goals
 
 - **Multi-DO transactions.** A transaction touches collections in one DO.

docs/adr/0001-sync-architecture.md

@@ -180,3 +180,13 @@ interleave so every milestone is end-to-end demonstrable.
 - We inherit the entire client reactive layer from TanStack DB and the
   proven single-stream discipline from Electric, while keeping full DO
   sovereignty over data.
+
+## Clarification — 2026-09-29: D17 and SQLite bigint precision
+
+D17 describes the tagged codec, not arbitrary SQLite column storage. It
+preserves `bigint` on the wire and in library-encoded TEXT (such as persisted
+predicates and dedup results); workerd `SqlStorage` cannot bind a JS `bigint`,
+and reading INTEGER values above `Number.MAX_SAFE_INTEGER` rounds them before
+the codec sees them ([workerd#4195](https://github.com/cloudflare/workerd/issues/4195),
+[issue #10](https://github.com/grrowl/tanstack-durable-object-sync/issues/10)).
+Use TEXT columns for exact large integers and convert at the application edge.

docs/adr/0009-changelog-time-retention.md

@@ -187,3 +187,38 @@ untouched (`dedupRetentionMs` stays separate).
   batch (`appliedSeq` advances on the next delivery without a contiguity check).
   This exists today, independent of retention; pruning neither causes nor worsens
   it. Not addressed here.
+
+## Amendment — 2026-09-29: the dedup sweep rides receipts, not compaction
+
+§1 put `sweepDedup` inside `maybeCompact`, which runs only when a drain finds
+CDC rows. But `_sync_seen_tx` rows come from answered receipts, not drains. A DO
+that only rejects mutations, replays receipts, or runs commands that write
+nothing adds a row per frame and never sweeps (bug bash 2026-09-29, F7).
+
+`sweepDedup` moves out of `maybeCompact`. It now runs after every answered
+`mut`/`call` receipt (new outcome or replay), after the reply is sent, behind
+one gate held in memory: at most one sweep per `dedupRetentionMs`. The gate
+starts at 0, so the first receipt after construction or a hibernation wake
+always sweeps. No new knob, no idle timer, no alarm, and no write per receipt
+beyond `recordTx`.
+
+- **Why in memory.** A persisted counter would add an UPSERT to every receipt.
+  An in-memory counter would reset on each wake, so a quiet DO that hibernates
+  between receipts would never reach its threshold. A time gate that reopens on
+  wake has neither problem.
+- **Why time, not a count.** `DELETE … WHERE ts < ?` scans the whole table
+  (there is no index on `ts`). Under a count cadence the scans per hour grow with
+  traffic, and so does each scan. Under a time gate a DO that stays awake scans
+  once per window, whatever the traffic. A DO that wakes per receipt scans once
+  per wake, but it is quiet, so its table is small.
+- **Why the window is `dedupRetentionMs`.** Retention is a lower bound: a retry
+  inside the window is deduped (ADR-0002 C5, ADR-0021). Sweeping less often only
+  keeps rows longer. An expired receipt now lives at most one more window, so the
+  table holds at most about two windows of receipts.
+- **Server-originated writes** (`runSyncedWrite`) no longer sweep dedup. They add
+  no receipts, so they add no rows to sweep.
+- A DO that receives nothing sweeps nothing. Its table does not grow either.
+
+Tests: `tests/dedup-sweep.test.ts` drives each receipt path through real
+sockets and polls the table. It also pins the gate: closed within the window,
+reopened by a wake (`evictDurableObject`) or by elapsed time.

docs/adr/0012-wire-input-hardening.md

@@ -102,3 +102,10 @@ a maintainer design decision.
 - **Test coverage**: `tests/wire-hardening.test.ts` (5 tests) pins all four
   invariants; `tests/error-paths.test.ts` was updated to assert the new generic
   error text for execute failures.
+
+## Amendment — 2026-09-29: D3 command authorization
+
+[ADR-0014 D2](./0014-object-sync-schema.md) revises D3 above: command
+`authorize` errors now surface to clients like mutation `authorize` errors;
+only command `execute` errors are sanitized as `EXECUTE_FAILED`. The shared
+sanitized catch described in D3 is historical, not the current contract.

docs/adr/0013-predicate-floor-one-evaluator.md

@@ -92,3 +92,29 @@ operator that lands in one floor and not the other.
   floor-operator agreement; `tests/subscriptions.test.ts` pins the JS-floor guard at
   the unit level; `tests/sql-compiler.test.ts` pins `ne` rejection and `not(eq)`
   lowering.
+
+## Amendment 2026-09-29: `isNull` and `isUndefined` join the floor (bugbash F4-cursor)
+
+On @tanstack/db 0.9, an indexed window ordered `nulls: "last"` scrolls with a
+cursor whose `whereFrom` is `gt(v) OR isNull(col) OR isUndefined(col)` (`lt` for
+descending). Both null checks were off-floor, so the cursor fetch was refused as
+an empty page and the window never grew.
+
+D2's rule still holds: each addition is a verified-agreeing operator. Checked
+against `evaluators.ts` in both 0.8.6 and 0.9.2 (identical):
+
+- `isNull(x)` is `x === null`. SQL: `"col" IS NULL`. The delta and catch-up paths
+  evaluate rows hydrated from SQLite, where a NULL column is always `null`
+  (whether the write omitted it or sent `null`), so the two agree.
+- `isUndefined(x)` is `x === undefined`. A stored column is never undefined (SQLite
+  has no such value; hydration yields `null`), so it is false for every stored row.
+  SQL: the constant `0`, like the empty `in`. The operand is still validated as a
+  plain column reference. Residual: `=== undefined` is TRUE for a property the row
+  lacks, i.e. a column the table does not have; there SQL says false. Unknown
+  columns are out of contract (the schema is author-owned and shared with the
+  client, and SQLite reads an unresolvable quoted name as a string literal, so
+  no operator fails loud on one).
+
+The floor is now `{ eq, gt, gte, lt, lte, like, in, isNull, isUndefined, and, or,
+not }`. `tests/predicate-parity.test.ts` pins both operators, and the composed
+cursor shape, across the snapshot and delta paths, per D2.

docs/adr/0015-syncable-mixin.md

@@ -227,3 +227,11 @@ tddc design gap; a PR against Actors is possible in principle but out of scope.
   and ADR-0019's `tests/hibernation.test.ts` now exercises the restore under
   real `evictDurableObject` cycles — which promptly exposed that the restore
   was incomplete (sockets survived, subscriptions didn't). See ADR-0019.]*
+
+## Amendment — 2026-09-29: subclass tick initialization
+
+The coalescer reads `tickMs` when arming an on-demand flush, not while the mixin
+constructor runs: subclass field initializers have not run during `super()`.
+This preserves the protected-field tuning contract and creates no idle timer.
+`tests/tick-override.test.ts` pins the real DO path; the slow-tick cursor-barrier
+and replay tests now run with their intended 30-second coalescer window.

docs/adr/0022-db-version-support-policy.md

@@ -142,8 +142,21 @@ test green at the new floor, and a recorded reason.
   (go-to-definition lands on source, not a missing file): about +54 kB packed.
 - Known gap, not fixed here: the server `.d.ts` files import
   `@cloudflare/workers-types`, which the package does not declare. The smoke
-  consumer installs it — the documented setup (README, examples) — so the test
-  models that setup rather than proving the import is safe without it.
+  consumer installs it — the example-app setup (not specified in README) —
+  so the test models that setup rather than proving the import is safe without it.
 - GitHub disables scheduled workflows after 60 days without repository
   activity, and sends their failure notifications to whoever last edited the
   cron line. A quiet repo must re-enable the job.
+
+## Amendment — 2026-09-29: Worker type sources in packed consumers
+
+The known gap above is closed. Server declarations use ambient Cloudflare
+`SqlStorage` / `SqlStorageValue` types instead of importing
+`@cloudflare/workers-types`. A Worker may supply these globals either through
+`wrangler types` (`worker-configuration.d.ts`) or through
+`@cloudflare/workers-types`; the library no longer forces one source or silently
+degrades SQL handler types to `any` when consumers skip library checks. The
+pack smoke test type-checks the packed package with `skipLibCheck: false` in
+both setups, including a fresh consumer without workers-types installed. The
+previous reference to README as documenting an explicit workers-types install
+was incorrect: only the examples included that package.

docs/adr/0023-on-demand-watches-acquisitions-holds.md

@@ -238,3 +238,53 @@ mount for orders 0.9 cannot express as a cursor.
   fail loud.
 - SQL `ORDER BY` ignores `nulls` and locale collation, so a bounded snapshot
   can choose a different top-k than the client's comparator.
+
+## Amendment — 2026-09-29: fetch flush barrier
+
+The fetch/coalescer follow-up above is resolved: `#handleFetch` now flushes the
+requesting socket's buffered deltas before reading its page, as the subscription
+snapshot path already does. A covered load therefore cannot settle with an old
+held row while its newer delta remains buffered. The page remains an atomic
+read and does not advance the client cursor; the preceding delta boundary
+commits the update first. `tests/fetch-flush-barrier.test.ts` pins both the
+wire order and the held row's value at load settlement through the real DO,
+transport, and on-demand collection.
+
+## Amendment — 2026-09-29: bounded snapshot NULL placement
+
+The `nulls` half of the ORDER BY follow-up is fixed: each SQLite sort term
+explicitly uses the requested `NULLS FIRST` or `NULLS LAST`, and `NULLS FIRST`
+when the clause names none. `@tanstack/db` 0.8.6 and 0.9.2 both default to
+`nulls: "first"` in either direction, while SQLite's default puts NULLs last
+in a `DESC` sort, so a descending window used to drop its NULL rows.
+`tests/order-by-nulls.test.ts` compares the real DO's bounded snapshot with
+TanStack's own answer. The locale-collation half remains open (pinned as
+`it.fails`), as does top-k selection among NULL ties crossing the limit, which
+SQLite does not break by row key as TanStack does.
+
+Cursor pages are client-built `where` expressions, so this fix does not touch
+them. 0.8.6's `buildCursor` ignores `nulls` (plain `gt`/`lt`, `eq` for ties).
+0.9.2 builds no cursor at a NULL boundary: a page request becomes a prefix load
+(`orderBy` + `limit`, no cursor), and a boundary-tie request loads the full
+subset. At a non-NULL boundary with `nulls: "last"`, its `whereFrom` is
+`gt(v)` (`lt(v)` descending) `OR isNull OR isUndefined`, and `isNull` is
+outside the predicate floor (ADR-0013): the fetch is refused as an empty page
+and the window stops growing. Pinned as `it.fails`.
+
+## Amendment — 2026-09-29: row-key tie-break and the nulls-last cursor
+
+Two of the open items above are closed.
+
+- **Ties.** Every bounded read ends its `ORDER BY` with the collection's pk,
+  ascending whatever the direction. `@tanstack/db` breaks value ties by row key
+  ascending (`createKeyedComparator` → `compareKeys` in db-ivm; identical in
+  0.8.6's db-ivm 0.1.19 and 0.9.2's 0.1.22), comparing strings with `<`. The
+  term is `COLLATE BINARY` explicitly, since a pk may declare another collation
+  (NOCASE would order `a` before `B`). BINARY matches JS code-unit order except
+  for astral versus U+E000–U+FFFF characters (JS compares UTF-16 code units,
+  SQLite UTF-8 bytes): a known residual divergence. The snapshot, RPC snapshot, and fetch paths all build their
+  SQL in `compileSubsetQuery`, so one change covers them.
+  `tests/order-by-ties.test.ts` drives the sub and fetch paths.
+- **Nulls-last cursor.** `isNull` and `isUndefined` joined the predicate floor
+  (ADR-0013 amendment, same date), so 0.9.2's `gt(v) OR isNull OR isUndefined`
+  cursor compiles and the window grows.

docs/adr/0025-empty-op-key-rejection.md

@@ -145,3 +145,25 @@ Before this change the same never-settling handler already hung its own
 frame and kept the DO active.
 
 Tests: `tests/inflight-txid.test.ts` (with `GatedTestDO` in the test worker).
+
+## Amendment — 2026-09-29: supplied row primary key must agree with op key
+
+For a `mut` insert or update, when `cols` owns the collection's `pk` field,
+reject the entire batch with `VALIDATION` if its value differs from `op.key`.
+This includes `""`, `null`, and non-string values. Run the gate after the
+dedup lookup, before any `authorize` or `execute`, and record the rejection
+as a receipt. Do not overwrite `cols[pk]`: ADR-0014's validation is a gate,
+not a parser, and rewriting the value would hide a mismatch from the author
+while breaking ADR-0001 D9's optimistic-id == confirmed-id contract. An
+update that supplies a different pk could likewise move a row under a
+different identity in a handler that writes the patch, so it is rejected too.
+
+An insert whose `cols` omits the pk is not rejected by this gate: no value
+exists to compare. ADR-0014 describes a full row for insert, but its schema
+validation is opt-in and the author controls `execute`; a handler may use
+`op.key` to populate the SQL pk. Requiring `cols[pk]` would change that
+contract. The author still owns row completeness and the actual SQL write;
+this gate only prevents an explicitly conflicting identity. Delete has no
+`cols` and remains covered by the existing non-empty op-key check.
+
+Tests: `tests/mutation-pk-consistency.test.ts`.