release security: bind the qualified wheel to the tagged source tree

[pengfei-threemoonslab]

Summary

The wheel published to PyPI is never bound to the tagged source tree. The release job does not build what it ships — it downloads a pre-built wheel from a repository variable URL and publishes that file.

Evidence

.github/workflows/release.yml establishes three bindings:

1. tag ↔ pyproject.toml version — the "Verify tag matches package version" step compares GITHUB_REF_NAME against the checkout's version.
2. qualification ↔ wheel bytes — scripts/run_safety_qualification.py::inspect_wheel records the wheel's sha256, and the sigstore bundle signs the qualification JSON containing it.
3. tag ↔ wheel version string — verify_safety_qualification_release.py:70: tag == f"v{wheel_version}", where wheel_version comes from the wheel's own METADATA.

What no step establishes is tagged commit ↔ wheel contents. inspect_wheel reads only Name, Version, and a digest:

name = str(metadata.get("Name", "")).strip()
version = str(metadata.get("Version", "")).strip()
...
return canonical_name, version, sha256_file(path)

Any wheel that declares Name: agents-shipgate and Version: 0.16.0 satisfies every check, regardless of what source produced it.

Meanwhile the "Lint and test" step runs ruff, compileall, and pytest against the checkout, while "Publish to PyPI" ships the downloaded wheel. The pipeline therefore tests one artifact and publishes a different one, with nothing asserting they correspond.

Impact

• The trusted-publishing identity vouches for an artifact of unverified provenance.
• A compromised or stale SAFETY_QUALIFICATION_WHEEL_URL publishes whatever it points at, as long as the metadata matches.
• The green test run on the release commit is not evidence about the shipped bytes.
• For a project whose entire thesis is deterministic, content-addressed verification, this is the one place the chain is not content-addressed back to source.

To be clear about severity bounds: the pypi environment is protected and the URLs are repository variables, so this requires maintainer-level access or a compromise of the hosting location to exploit. It is a missing control, not an open door.

Proposed fix

Build the wheel in-job from the tagged checkout and require byte equality with the qualified wheel:

1. python -m build --wheel (or uv build) from the release checkout.
2. cmp the locally built wheel against the downloaded qualified wheel, or compare sha256 against the digest recorded in the qualification artifact.
3. Fail closed on mismatch, before uv publish.

This closes the provenance gap and, as a side effect, turns every release into a reproducible-build check — a claim this project is unusually well positioned to make.

If bit-for-bit reproducibility is not achievable yet (timestamps, wheel ordering), the interim control is to compare the unpacked content tree rather than the archive bytes, and to file the reproducibility gap separately rather than skipping the check.

Acceptance criteria

• The release job builds a wheel from the tagged checkout.
• Publication is gated on that wheel matching the qualified wheel; mismatch fails before uv publish.
• A test or dry-run proves the mismatch path fails closed.
• Documentation states which artifact is authoritative and how provenance is established.

Related

• release policy: decide the approved evidence bar for pre-1.0 tags #341, release: separate verification from publication and make partial publish recoverable #343, release: align test selection with CI and set a measured timeout #344, release: add a non-publishing rehearsal path #355, and release: generate and sign an SBOM for the shipped wheel only #356 — other release-readiness workstreams from the same review.
• Root cause of this shipping unnoticed: the pipeline has no non-publishing rehearsal path (release: add a non-publishing rehearsal path #355) and has never run with these steps — they were added after v0.15.0.

[pengfei-threemoonslab]

PM scope clarification: keep the acceptance method-neutral. Reproducible byte equality between the tagged build and qualified wheel is preferred. If wheel archives are not yet bit-for-bit reproducible, a normalized unpacked-content comparison or an equivalently source-bound build attestation is an acceptable explicit interim control. Publication must still fail closed on mismatch, and any remaining reproducibility gap should be tracked separately.