store-owned v2 chunked export digest: PublishDigested with dir parallel hash and s3 upload tee

[CMGS]

Summary

Replace the v1 template content digest with a v2 chunked-manifest digest owned by the store backends:

• PublishDigested computes the digest inside the backend publish path: dir hashes before installing a generation, while s3 hashes the same source read used for upload.
• Fetch returns the digest with the committed meta and export.
• The digest leaves pool-owned meta.json.
• Plain Publish remains unchanged for checkpoint and archive records that do not need a digest.

PR #61 is merged, but no tagged release contains the digest API yet. This issue replaces v1 before the next release; it does not add a migration framework for an API that has not shipped.

Motivation

The current pool-owned v1 hash adds a full export read before every publish:

• s3 reads each export byte once for the digest and again for upload.
• dir changed from a generation install plus meta-pointer commit into a serial full-tree hash before publication.
• cocoon's local snapshot export normally reflinks files, so there is no engine-side byte stream to hash without adding another read.

The digest must remain an independently recomputable content identity. It identifies the exact exported regular-file byte tree, not semantic VM state. Republishing the same staged bytes yields the same identity; a new Promote that re-captures a running VM may produce different export bytes, including snapshot-envelope identifiers, and therefore a different digest.

Instar is an intended consumer. Binding the value into Instar task/run identity is a separate downstream change and is not an acceptance criterion for this issue.

Digest format v2

The chunk size is fixed at 16 MiB. A different chunk size is a different digest format.

For every regular file:

chunk[i] = sha256(file[i*16MiB : min(size, (i+1)*16MiB)])

An empty file has zero chunks. Chunk sums may be computed concurrently, but they MUST be appended to the manifest in strictly increasing chunk-index/file-offset order. Worker completion order MUST NOT affect the digest. Each chunk sum in the manifest is the raw 32-byte SHA-256 value.

Files are sorted by their slash-relative path bytes. The path length is its byte length.

digest = "sha256:" + hex(sha256(
  "sandbox-template-export-v2\x00"
  for each regular file in ascending path-byte order:
    'f' + u64be(len(path)) + path +
    u64be(size) + u64be(nchunks) +
    chunk[0] + ... + chunk[nchunks-1]
))

Directories, modes, mtimes, ownership, and sparse allocation are excluded. A non-regular entry is an error. Digest chunk boundaries are independent of s3 transfer part sizes and of individual Read calls.

Fixed vectors:

• tree { nested/a.bin = "a", z.bin = "z" } → sha256:ad65b315de70e494c767969e65fc5de65c73846c4ebcdc7051abe08b056637ad
• tree { empty.bin = "" } → sha256:e3a1004a091d5fbaf95fb995f6999355a060a04949e1c3c9d8e886abcd933251
• tree { boundary.bin = "x" repeated 16 MiB followed by "y" } → sha256:da8dd289e7d66408bd7d202642f180f2c22bc508cf0fe1d6e9f0e0b7967f6a5e

Interface

Publish(ctx, staging, id) error
PublishDigested(ctx, staging, id) (string, error)
Fetch(ctx, id) (dir string, meta []byte, digest string, release func(), err error)

Publish does not compute or expose a digest. PublishDigested is used only for templates.

A separate Digest method is not added: on s3 it would require another full source read instead of sharing the upload read.

Visibility, failure, and retry contract

A digest is visible only with the complete generation it describes:

• PublishDigested returns a digest only after the backend commits the new record.
• Hash, export upload, or sidecar failure before the commit leaves the previous committed generation visible.
• An ambiguous s3 commit-marker result may leave either the previous generation or the fully uploaded replacement visible; it must never expose a mixed generation.
• A later successful retry commits one complete replacement.
• Fetch returns meta, export, and digest from the same committed generation.
• Records written by plain Publish, and pre-v2 records with no backend digest, return an empty digest.
• Re-publishing changed export bytes returns and fetches the replacement generation's digest; a previous digest must not leak into a plain publish.

The existing ownership and single-writer contracts remain unchanged.

Digest assembly

The framing and assembly live once in store/digest.go. The shared assembler owns path-byte sorting, framing, chunk-count validation, and leaf assembly. Backends supply file size and chunk sums indexed by file and chunk index.

Directory workers write results into preallocated index slots; they do not append in completion order. The s3 reader feeds the same chunk vocabulary from sequential byte offsets.

Backend storage

Digest storage is backend-internal. S3 adds no object-store round trip to Fetch:

• dir stores digest-<gen> beside the generation directories, where <gen> is derived from the committed meta bytes. It installs the sidecar before the atomic meta.json rename. Fetch reads meta once, resolves export-<gen> and digest-<gen> from it, and treats a missing sidecar as empty. The generation sweep retains the current sidecar and ages superseded sidecars with their exports.
• s3 stores the digest as user metadata on the meta.json object. That object remains the last-uploaded commit marker. Fetch extracts the digest from the response of its existing meta GetObject; it must not add a HeadObject or second GetObject.

Backend implementations

dir

PublishDigested:

1. Enumerates the staged export/ regular files.
2. Hashes chunks with a fixed-size, per-call worker pool and small streaming buffers.
3. Assembles the manifest by path and chunk index.
4. Installs the immutable export-<gen> directory and atomically writes digest-<gen>.
5. Commits meta.json last with an atomic rename.

A hash failure occurs before generation install. An export or sidecar failure before the meta rename leaves the previous committed generation visible; uncommitted residue remains invisible and the normal generation sweep reclaims it. A fresh retry can commit one complete replacement.

The pool keeps its existing per-template mutation lock from the owner re-check through PublishDigested. A same-process claim of the same key may therefore wait for the dir hash during a re-promote. This is an accepted latency tradeoff for the simple single-stage implementation; measure it, but do not add a prepare/commit protocol unless it violates an observed latency target.

s3

The pinned transfermanager already reads the source Body sequentially into part buffers; it does not use ReaderAt to read the source concurrently. The hashing wrapper should expose Read, but it is not changing the SDK from a concurrent-source path.

Because the wrapper hides io.Seeker, the upload input MUST set ContentLength from the opened file's stat result. This preserves the transfermanager's size-aware part planning for large objects. Part uploads remain concurrent.

The chunker must follow byte offsets, not SDK part or Read boundaries. Request retries rewind the already-buffered part body and must not hash source bytes twice. A failed file upload discards its incomplete chunk results. Only after every export upload and the manifest complete does s3 upload the meta commit marker with the digest metadata.

Pool changes

• Delete templateRecord.ContentDigest.
• Delete the pool-side exportContentDigest.
• Under the existing template mutation lock, re-check the owner, write pool meta, call PublishDigested, and return its committed digest.
• resolveGolden takes the digest returned by Fetch.
• Checkpoint/archive paths keep plain Publish; callers that do not consume the new Fetch field ignore it.

Upgrade behavior

The v1 digest stream and meta field are removed rather than supported in parallel. Existing records without backend digest storage return an empty digest until re-promoted. Downstream consumers must continue treating an empty digest as unavailable, not as an identity equal to another empty digest.

Acceptance

Format and assembly:

• Keep the two original fixed vectors and the 16 MiB + 1 vector above.
• Cover 16 MiB - 1, exactly 16 MiB, and 16 MiB + 1.
• Force chunk workers to complete out of order and prove assembly still follows chunk index.
• Prove dir and s3 produce the same digest when their read boundaries differ.

Store contract, against dir and s3:

• PublishDigested returns the fixed digest and Fetch returns the same value.
• Plain Publish followed by Fetch returns an empty digest.
• A replacement hash or export failure before commit leaves the old meta/export/digest fetchable; a later retry commits the new complete generation and digest.
• Non-regular input or hash/upload failure does not commit a digest.

S3-specific:

• fakeS3 retains user metadata and counts requests.
• Cache hit and miss each use the existing single meta GetObject, with no added HeadObject.
• Explicit ContentLength, arbitrary source read sizes, upload retry, and failed export-before-marker are covered.
• A real-endpoint A/B records PublishDigested versus plain Publish throughput; no fallback implementation is added without an observed regression.

Dir-specific:

• Worker and buffer counts are bounded per call.
• Existing Fetch pinning and republish tests also assert that export, meta, and digest stay generation-consistent.
• The current digest sidecar is never swept; superseded sidecars follow the same grace as their export generations.
• A sidecar failure before meta commit preserves the previous generation, and a fresh retry commits the complete replacement.
• Measure same-key claim latency during re-promote; this is performance evidence, not a correctness blocker.

Final gates: focused store/pool tests, go test -race, dual-GOOS lint/asl, and an end-to-end Promote/Claim flow with a non-empty matching digest.

Out of scope

• Instar task/run integration.
• Semantic VM-state normalization.
• Dirty-chunk incremental digests.

[CMGS]

Freeze review passed. Independently verified against the pinned toolchain:

• All three fixed vectors recomputed byte-exact from the spec text alone (two-file, empty-file, and the 16 MiB + 1 boundary tree).
• transfermanager v0.2.14 source: upload consumes the source Body strictly sequentially (nextReader -> readFillBuf into pooled part buffers; no ReaderAt path), part retries rewind the buffered bytes.Reader and never re-read the source, and UploadObjectInput.ContentLength feeds initSize's part planning exactly when the body hides io.Seeker - all three SDK behavior claims in the body hold as written.
• No tagged release contains the digest API (v0.1.4 is the latest tag; 0a21882 is untagged), so the no-compat scope is accurate.
• The dir hash runs before the exclusive flock and the same-process-only wait claim matches the lock structure on main.

No changes requested; freezing as the implementation contract.