record-adapter-do-sqlite: a Workers story for SQLite records - #215
Open
cuibonobo wants to merge 5 commits into
Open
record-adapter-do-sqlite: a Workers story for SQLite records#215cuibonobo wants to merge 5 commits into
cuibonobo wants to merge 5 commits into
Conversation
…ehavior for #161 Throwaway probes against the real Workers runtime (@cloudflare/vitest-pool-workers), not the shipped adapter. Findings, to inform the real implementation: - Raw BEGIN/COMMIT/ROLLBACK via sql.exec() is rejected outright. - DO does NOT auto-rollback a write when a later exception is thrown -- each sql.exec() auto-commits immediately, so no-op'ing BEGIN/COMMIT/ROLLBACK would silently break atomicity. - ctx.storage.transactionSync(fn) does roll back on throw, commits on success, and passes return values through synchronously. - PRAGMA foreign_keys = ON works and enforces; PRAGMA journal_mode = WAL is rejected (not authorized) -- expected, DO owns its own durability. - FK/unique violation error strings match what isForeignKeyViolation/ isUniqueConstraintViolation already check for. - Cursor exposes toArray()/one()/raw()/rowsRead/rowsWritten; parameter binding is spread-args, same convention as node:sqlite. Net conclusion: SharedSqlRecordLogic's exec('BEGIN')/'COMMIT'/'ROLLBACK' calls can't translate to DO as-is -- transactionSync's callback boundary doesn't match three independent string-based exec() calls. Needs a SqlExecutor.transaction<T>(fn) primitive instead, touching record-logic.ts and record-adapter-sqlite's executor, not just this new package. pnpm-workspace.yaml: allow workerd's postinstall build script (needed by wrangler/miniflare for the Workers test pool). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJoUkCo4dLjXsDsNDkYjbx
Implements StackRecordAdapter over Cloudflare Durable Objects' SQLite
storage (ctx.storage.sql) for Workers deployments with no Node runtime.
Reuses SharedSqlRecordLogic, the FTS5 schema/strategy, query builder,
cursor codec, and row mappers from @haverstack/sqlite-shared, via its new
./record subpath (the token-store and file-lock pieces stay Node-only and
unreachable from this adapter's bundle — a bare import of either survives
esbuild's tree-shaking as a dead-but-still-imported `node:crypto`/`node:fs`
module, which throws at load time in a Worker without nodejs_compat).
No lock file: a Durable Object id maps to exactly one running instance, so
the platform itself is the single-writer guarantee. No persist/flush step:
every write through ctx.storage.sql is durable by the time the call
returns.
sqlite-shared's SqlExecutor gains a transaction<T>(fn: () => T): T
primitive, replacing the raw BEGIN/COMMIT/ROLLBACK statements
record-logic.ts issued directly. A pre-implementation spike against the
real Workers runtime (@cloudflare/vitest-pool-workers) found DO SQLite
rejects those statements outright, and does not roll back a write on a
later exception the way an open SQL transaction would — its real
primitive is ctx.storage.transactionSync(fn), a callback boundary three
independent string-based exec() calls can't reach. record-adapter-sqlite's
executor implements transaction() as literal BEGIN/COMMIT/ROLLBACK around
fn(), behavior-identical to the code it replaces — its full 116-test suite
passes unchanged.
Tests run against the real Workers runtime, not `environment: 'node'` —
a targeted subset (CRUD, FK/unique constraint mapping, FTS5, cursor
pagination, associations, commitMigration), plus the load-bearing case for
the transaction refactor: a rejected patchContent must leave the FTS index
consistent with stored content, which only holds if the rollback actually
rolls back.
Also:
- docs/spec/adapters.md: new adapter in the backends table, capabilities
notes, and Concurrency & storage ownership section ("the DO is the
lock"); sqlite-shared's two entry points and transaction() rationale.
- README.md: package table and directory tree entries.
- scripts/verify-pack.mjs: notes why the new package is deliberately
absent from the Node-import EXPECTATIONS map (its entry point uses
ambient Durable Object globals that don't exist under plain Node).
- .npmrc: keep @types/chai (pulled in only by this package's vitest 4)
out of pnpm's shared @types hoist, so it can't collide with the chai
types vendored in @vitest/expect@2 that the rest of the repo uses.
- .gitignore: carve out an exception for hand-authored ambient .d.ts
files under tests/support/ (the blanket *.d.ts rule is for build
output; this package's cloudflare:test type reference isn't generated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJoUkCo4dLjXsDsNDkYjbx
🦋 Changeset detectedLatest commit: 9154fe3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
CI failed on every job (lint/build/test/format) with: [ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION] 1 lockfile entries failed verification: @cloudflare/workers-types@5.20260830.1 was published at 2026-08-30T01:19:42.000Z, within the minimumReleaseAge cutoff. That version is a transitive dependency of wrangler / @cloudflare/vitest- pool-workers, not something this PR asked for directly, but @cloudflare/ workers-types publishes a new calendar-versioned release roughly every 24h — floating on "latest" for it is structurally incompatible with a minimum-release-age policy, since the newest version is almost never older than the policy's window. Pinned it to 5.20260817.1 (~2 weeks aged) via a root pnpm.overrides entry, applying repo-wide regardless of which dependent requests it. That override reintroduces a real conflict this branch had already worked around once: adding @cloudflare/workers-types back as a direct devDependency (needed for tsup's isolated dts build of src/executor.ts, which uses the ambient DurableObjectStorage/SqlStorage/SqlStorageValue globals with no import and doesn't see wrangler's generated worker-configuration.d.ts) reintroduced the ambient Env/Cloudflare.Env duplicate-declaration conflict between the two type sources for the *test* tsconfig, which only needs wrangler's generated types and never needed workers-types at all. Fixed by scoping workers-types to tsup's dts step alone (dts.compilerOptions.types), rather than the shared tsconfig.json every tsc --noEmit run reads — the build gets the globals it needs, the test typecheck never sees the conflicting second copy. Verified with the same commands CI runs (format:check, build, test, lint, typecheck, and a frozen-lockfile install) across the whole workspace, plus confirmed no other lockfile entry is within the release-age window. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJoUkCo4dLjXsDsNDkYjbx
… for pnpm 11 Second round of the same CI failure — this repo's workflow pins pnpm/action-setup to major version 11, but my local pnpm was 10.33, which silently accepted config locations that v11 no longer reads: - package.json's "pnpm.overrides" field: v11 logs a warning and ignores it entirely, so the @cloudflare/workers-types pin from the previous fix never took effect in CI — [ERR_PNPM_LOCKFILE_CONFIG_MISMATCH], since the lockfile itself did carry the pinned resolution but the running config didn't request it. - .npmrc's hoist-pattern: v11 doesn't read it either (no warning, just silently not applied) — confirmed via `pnpm config get hoist-pattern` returning undefined despite the file being present. @types/chai was back in the shared hoist folder, reproducing the duplicate-identifier typecheck failure across every package that the previous branch state had (apparently only coincidentally) stopped showing. Both now live in pnpm-workspace.yaml, which v11 does read for both keys — verified with `pnpm config get hoist-pattern`/`get overrides` actually resolving, and a full clean install + frozen-lockfile install + build + lint + test + typecheck pass, all run with pnpm@11.24.0 specifically (matching what pnpm/action-setup resolves version: 11 to right now) rather than relying on local pnpm 10, which had masked both of these. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJoUkCo4dLjXsDsNDkYjbx
Resolves conflicts with #160 (blob-adapter-s3), which merged into main after this branch was created — both PRs added a row to the same adapter-backends tables in README.md and docs/spec/adapters.md. Kept both additions in each table; scripts/verify-pack.mjs and the README package-tree section merged cleanly on their own. Lockfile regenerated via `pnpm install --no-frozen-lockfile` (pnpm 11.24.0, matching CI) rather than hand-resolved, per the repo's own tooling. Verified with format:check, build, lint, typecheck, and test across the full merged workspace (blob-adapter-s3 and record-adapter-do-sqlite both green), plus a from-scratch --frozen-lockfile install with pnpm 11.24.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@haverstack/record-adapter-do-sqlite, aStackRecordAdapterover Cloudflare Durable Objects' SQLite storage — a Workers-native record adapter with no Node runtime dependency, closing record-adapter-do-sqlite — a Workers story that fits the sync SqlExecutor and the ownership rule #161.SharedSqlRecordLogic, the FTS5 schema/strategy, query builder, cursor codec, and row mappers from@haverstack/sqlite-shared, via a new./recordsubpath that excludes the Node-only token-store and file-lock code (a bare import of either survives esbuild's tree-shaking as a dead-but-still-importednode:crypto/node:fsmodule, which throws at load time in a Worker withoutnodejs_compat).ctx.storage.sqlis durable by the time the call returns.SqlExecutor(in@haverstack/sqlite-shared) gains atransaction<T>(fn: () => T): Tprimitive, replacing the rawBEGIN/COMMIT/ROLLBACKstatementsrecord-logic.tsissued directly. A pre-implementation spike against the real Workers runtime found DO SQLite rejects those statements outright, and does not roll back a write on a later exception the way an open SQL transaction would — its real primitive isctx.storage.transactionSync(fn), a callback boundary three independent string-basedexec()calls can't reach.record-adapter-sqlite's executor implementstransaction()as literalBEGIN/COMMIT/ROLLBACKaroundfn(), behavior-identical to the code it replaces.Spec
Yes —
docs/spec/adapters.mdupdated:record-adapter-sqlite's capabilities viaSharedSqlRecordLogic).@haverstack/sqlite-shared's two entry points (.vs./record) and why the split exists.SqlExecutor.transaction()as the interface's transaction primitive, and why it replaced rawBEGIN/COMMIT/ROLLBACK.record-adapter-do-sqlite— "the DO is the lock."README.md's package table and directory tree also updated.Verification
pnpm run format:check && pnpm run lint && pnpm test && pnpm run build && pnpm run typecheck— all green across the entire workspace (11 packages), not just the touched ones.record-adapter-sqlite's full existing suite (116 tests) passes unchanged after thetransaction()refactor — confirms it's behavior-preserving.record-adapter-do-sqlite's new suite (16 tests) runs against the real Workers runtime (@cloudflare/vitest-pool-workers), notenvironment: 'node'— a targeted subset (CRUD, FK/unique constraint mapping, FTS5, cursor pagination, associations,commitMigration), plus the load-bearing case for the transaction refactor: a rejectedpatchContentmust leave the FTS index consistent with stored content, which only holds if the rollback actually rolls back.PRAGMA journal_mode = WALrejected, FK/unique error strings match, FTS5 available, cursor/binding shape) came from a spike against the real runtime, not documentation assumptions.Notes for reviewers
record-adapter-do-sqliteis deliberately absent fromscripts/verify-pack.mjs'sEXPECTATIONSmap (see the comment added there) — its entry point uses ambient Durable Object globals that don't exist under plain Node, so a bareimport()there would fail for reasons unrelated to packaging correctness. It's still packed and installed into the throwaway consumer, so a missingfilesentry or an accidentally-unbundled@haverstack/sqlite-sharedwould still be caught..npmrcto keep@types/chai(pulled in only by this package's vitest 4 /@cloudflare/vitest-pool-workers) out of pnpm's shared@typeshoist folder, where it was colliding with the chai types vendored in@vitest/expect@2that the rest of the repo uses on vitest 2 — a real cross-package typecheck regression during development, fixed before it could land..gitignoregained a narrow exception for hand-authored ambient.d.tsfiles undertests/support/(this package'scloudflare:testtype reference) — the blanket*.d.tsrule is meant for build output, and was silently dropping this file.Generated by Claude Code