Add blob-adapter-s3: StackBlobAdapter over the S3 API - #213
Merged
Conversation
Closes the gap combineAdapters() names as its own motivating example: S3/R2-backed blob storage previously had no shipped adapter, so any server deployment wanting object storage had to implement StackBlobAdapter itself. blob-adapter-s3 mirrors blob-adapter-disk's content-addressed design (SHA-256 keys, same C3 error contract) and supports S3-compatible endpoints like Cloudflare R2 via `endpoint` + `forcePathStyle`. Uses @aws-sdk/client-s3 for correct SigV4 signing rather than a hand-rolled implementation. Tests mock the S3 client with a small stateful in-memory fake (aws-sdk-client-mock has no server-side state of its own), mirroring blob-adapter-disk's test list. Wires the new package into README, docs/spec/adapters.md, CONTRIBUTING.md, scripts/verify-pack.mjs, and the changeset workflow's package dropdown.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
@haverstack/blob-adapter-s3, aStackBlobAdapterover the S3 API for S3 and S3-compatible stores (Cloudflare R2 via its S3-compatible endpoint). Closes #160.blob-adapter-diskwas previously the only shippedStackBlobAdapter, even thoughcombineAdapters()names SQLite + S3 as its own motivating example. This package closes that gap:S3BlobAdaptermirrorsblob-adapter-disk's content-addressed design — SHA-256 hex keys, same C3 error contract (StackNotFoundErrorfor a missing key,StackQueryErrorfor a malformedfileId).endpoint+forcePathStyleoptions route through the same client to R2 or any other S3-compatible store, per the issue's ask.listFiles()(paginatedListObjectsV2) soStack.collectAttachmentGarbage()can find bare-bytes orphans in object storage, same as the disk adapter.maxAttachmentBytesisn't declared here — that field lives onAdapterCapabilities, whichcombineAdapters()always sources from the pairedStackRecordAdapter, not the blob half. Documented this explicitly indocs/spec/adapters.mdsince the issue's own checklist item ("decide whethermaxAttachmentBytesstaysnullhere") assumed the blob adapter owns it.PutObjectis atomic per key, soputAttachmentjust does aHeadObjectexistence check (to skip a redundant upload) thenPutObject— no rename trick needed, and a concurrent write of identical content to the same key is harmless by construction.Uses
@aws-sdk/client-s3rather than a hand-rolled fetch + SigV4 implementation — this repo's adapters otherwise have zero non-@haverstack/*runtime dependencies, but getting AWS request signing and S3's XML error/listing parsing right by hand isn't worth the risk versus the maintained SDK.Also wired the package into the repo's package-list surfaces: README (package table, capability matrix, packages tree),
docs/spec/adapters.md(adapter backends table, and turned itsS3BlobAdaptercode example from "hypothetical" into a real one),CONTRIBUTING.md's package tree,scripts/verify-pack.mjs, and the.github/workflows/changeset.ymlpackage dropdown (now the 9th manually-listed package — updated the "ninth package" note inCONTRIBUTING.mdto "tenth" for the next one).Out of scope per the issue's own cross-refs:
putAttachment/getAttachmentkeep today's whole-Uint8Arrayshape and don't architecturally tie hashing to buffering, so a later streaming variant shouldn't have to fight this shape.listFiles()follows the existing optional-method convention (StackBlobAdapter.listFiles?()), same asblob-adapter-disk.Not included, and needs a manual step outside this repo: the actual first
npm publish+ npm trusted-publisher configuration for@haverstack/blob-adapter-s3, perCONTRIBUTING.md§ Releasing ("a ninth [now tenth] package needs one manual publish... then its own trusted publisher"). No changeset was added for this PR since changesets doesn't manage a package's first-ever publish, only version bumps of already-published packages.Spec
Yes — updated
docs/spec/adapters.md§ Adapter backends: addedblob-adapter-s3to the adapter table, replaced theS3BlobAdaptercode example (previously marked// hypothetical) with the real package, and added a paragraph clarifying thatmaxAttachmentBytesis declared by the pairedStackRecordAdapter/combineAdapters(), not by a blob-only adapter.Verification
All green, including
verify-pack.mjs, which packs@haverstack/blob-adapter-s3into a tarball, installs it into a project outside the workspace, and confirmsS3BlobAdapterimports from it — catching the case where@aws-sdk/client-s3resolves via the workspace link but isn't actually a declared dependency.blob-adapter-s3's own test suite (18 tests) mocks@aws-sdk/client-s3withaws-sdk-client-mock, backed by a small in-memory fake bucket (the mock library itself has no server-side state), and tracksblob-adapter-disk's test list: dedup, not-found, invalid-fileId, delete-is-idempotent, andlistFiles()(empty/populated/ignores-non-hash-keys/reflects-deletes/carries-modifiedAt).No live AWS or R2 calls in CI or locally — everything runs against the mock.
Notes for reviewers
@aws-sdk/client-s3over a zero-dependency hand-rolled SigV4 client for correctness (signing and S3 XML parsing are easy to get subtly wrong), at the cost of being the first package in this repo with a non-@haverstack/*, non-tooling runtime dependency (@aws-sdk/client-s3+ its@smithy/*tree).clientconstructor option is an escape hatch for callers who need control over retry policy, credential providers, or middleware beyond whatregion/endpoint/forcePathStyle/credentialsexpose — and it's what the test suite uses to inject the mocked client.Generated by Claude Code