[zibi-pplx]

Persist evidence for the logical bytes actually read into compression/encryption and forward it through sandbox-manager finalization into the catalog. Existing restore verification can consume the logical digests; the recovery layer can also require stable metadata and verified object identity.

Common-jobs atomically merges a versioned completion result into the current payload under attempt/lease fencing, preserving concurrent intent updates. Legacy, malformed, unknown, or stale evidence remains optional for ordinary transfers. Owned immutable raw staging is hashed in place; legacy raw inputs get a private uploaded copy. Evidence is emitted only after EOF and object verification, and ambiguous roles cannot produce certified evidence. Additive protobuf fields preserve existing field numbers and readers.

Validation: 62 common-jobs tests, 219 volume-manager tests, 30 snapshot-upload tests, and the catalog evidence/finalization test passed. The concurrent-payload regression failed before correction. A 16 MiB synthetic opt-in measurement covered owned raw, copied raw, compressed, and encrypted uploads; added hashing CPU was roughly 56–83 ms in this debug run, with no extra raw staging copy for owned input. Workspace all-target/all-feature Clippy and affected aarch64 Linux musl compilation/lint pass. Public diff review and Gitleaks passed.

Stack (land in order): #318 schema → #319 materialization ownership → #345 backup caller identity → #346 upload evidence → #283 retained-artifact recovery.

Rebuilt against d1abc66fe48bccaaa715e09593f875f7c6c4d11d. The schema is separate because current CI requires migration-only PRs. All existing PRs are reused.

Scope boundary: no file overlap with #828 (2a4988b6), #829 (1b696768), or #837 (58ceff04). Both compositions—#828+#829 and #837+#829—passed aarch64 Linux musl Clippy and all 98 selected snapshot/teardown tests each. These are component tests, not live Firecracker recovery evidence. Capture, encrypted sealing, teardown handoff, and source cleanup remain with those workstreams. Linux/KVM held-volume recovery, manager-restart/handoff failure, and live restore of the same artifact have not been validated here; issuance must remain disabled until that deployment evidence exists.

AI assistance: Codex contributed implementation, tests, and this description.

[shrikant-pplx]

Reviewed together with the rest of stack #345 → #318 → #319 → #346, in two passes (correctness, then security). No P0/P1/P2 findings. Two P3s are inline.

The producer side holds up:

• The digest wraps the only reader each transform uses (raw copy, zstd file, zstd streaming, AEAD, zstd+AEAD).
• A proof is only published after EOF.
• Proofs and completion are written in one UPDATE, fenced on attempts (and on lease_owner for leased jobs).

Notes that are not defects in this diff:

• Nothing in the stack reads logical_artifact_digests yet. The step where sandbox-manager forwards them to a recovery grant is future work, so "recovery approvals must bind the bytes actually uploaded" isn't enforced anywhere yet.
• The proof is computed and stored by the uploading node itself. It shows that an honest node bound the bytes it uploaded; it does not independently verify a node's claim. A future grant consumer should treat it that way, for example by re-hashing or otherwise checking the stored object.
• Test gaps (P3):
  • No BackgroundWorker-level test checks that a JobCompletion payload is persisted, plus a stale-attempt case.
  • Nothing covers the size-mismatch guard (logical digest size differs from uploaded source size).
  • No snapshot.json push test shows the metadata binding reaching the proof.
  • Empty files aren't covered.

1. Fixes the stated problem and root cause? Partly. The producer does what it claims, but nothing consumes the proofs yet.
2. Covers every affected caller and path? Partly. All file push paths and the single worker completion site are covered. btrfs-send is excluded, as documented.
3. Every change necessary, no scope creep? Yes. Adding the attempts check to the leased path changes completion for the build manager. That's safe, because the build manager never changes attempts mid-run.
4. Do the tests reproduce the failure and prove the fix and edge cases? Partly. The db test reproduces the stale-attempt completion, and the reader, streaming-retry and compatibility cases are covered. See the gaps above and the raw-path comment inline.
5. Regression? No, apart from the inline P3 duplicate-artifact case. The added hashing CPU hasn't been measured.

[shrikant-pplx]

P3 [check] Duplicate artifacts are accepted at submit, then fail the job permanently after the upload

PushFilesAsync only checks role uniqueness for encrypted requests (service.rs: if req.encryption.is_some() { validate_entry_roles(..) }). Even then, two roles can resolve to the same effective snapshot_artifact.

Every entry uploads and passes HEAD verification, and the staged sources are deleted. Then this check fails the job permanently, and every resubmission fails the same way. The same request succeeded before this PR. Today only the CLI calls PushFilesAsync.

Fix (either):

• Reject duplicate effective artifacts with invalid_argument at submit, next to validate_entry_roles.
• Leave duplicates out of the proofs instead of failing a verified upload.

[shrikant-pplx]

P3 [func] The raw path copies sources that are already immutable

For new jobs, upload_source() is the staged path, an immutable copy that the volume manager owns. Hashing that file in place and uploading the same file would bind the same bytes without the extra full-size temp file. The copy is only needed for legacy jobs that have no staged_path. As written, every staged raw push needs 2× disk on the staging volume while it uploads. The description lists this cost but says removing it needs a new streaming implementation, which it doesn't for staged sources.

Separately, the test (compress: false) doesn't prove the upload comes from the owned copy: MockObjectStore only counts put_calls. It would still pass if the raw path hashed the copy and then uploaded upload_path.

Fix:

• Copy only when staged_path is None.
• Have the mock record the uploaded path and bytes, and assert that the uploaded bytes hash to the published digest.

[mabdelraouf-pplx]

🤖 Posted by Claude Code acting on @mabdelraouf-pplx's behalf.

Approving. I agree with the existing review. The points below are non-blocking follow-ups.

1. Proof generation can fail a verified upload. After the HEAD check passes, execute_push can still turn success into an error: source not read to completion, size mismatch, poisoned lock, or duplicate artifacts. On the last attempt the staged source is deleted and the transfer fails. Nothing consumes the proofs yet, and consumers must already refuse missing evidence. Consider omitting the proof and logging a warning instead. This would also cover the duplicate-artifact P3. I traced every current reader (zstd copy_encode and read::Encoder, encrypt_stream_blocking, and the streaming SourceReader); all reach a 0-byte read, so I don't think this can happen today.
2. The proof doesn't identify the stored object. LogicalArtifactDigest has no wire size or ETag, though the verification HEAD already returns both. Push overwrites unconditionally, so a later write to the same key changes the object without invalidating the proof. Recording wire size and ETag (and version id, where the store has one) would let a grant consumer check the object is unchanged.
3. SHA-256 now runs on every snapshot upload and hasn't been measured. Encrypt-only pushes deliberately skip codec admission, and SHA-256 is likely slower than AES-GCM there. On the codec paths it also holds the permit longer. The existing production-shaped benchmark could measure this.
4. JobCompletion.payload replaces the whole stored payload with the copy taken when execution started. That is safe for the volume manager, which never edits a running job's payload. Other users of the jobs crate do edit running payloads (update_active_job_payload), so either document this contract on JobCompletion or store evidence separately from the intent payload.
5. job_to_proto warns on every proof decode failure. After a rollback, each Wait/List poll logs again. Consider rate-limiting the warning or lowering it to debug.

Verified as fine: the leased claim returns the job after attempts is incremented, so the new attempt fence holds. Executor clones share the codec, producer and buffer semaphores. Older binaries ignore the new payload field.

_{Session: b5a56bdf-dcac-44f1-b12f-55a62b174be9}