From 1379881b61d9607fd4f42aff2ff3cc8ee768a61c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 23 Sep 2026 03:50:02 +0900 Subject: [PATCH 01/21] feat(ci): fail-closed pre-publish dependency license and Strix gate origin/main has no pre-publish dependency gate. The only license signal is scripts/ci/sbom_inventory_aggregator.py, a scheduled informational org SBOM roll-up that flags GPL/AGPL/NOASSERTION for governance -- not per-dependency, not fail-closed, not bound to a release head. That gap blocks fast-mlsirm's 0.11.5 PyPI release and contextual-orchestrator's VCS-pin removal. Add the gate as a reusable workflow_call workflow plus its tested decision engine: - spdx_license_policy.py parses SPDX 2.3 expressions (AND/OR/WITH/parens/plus) and applies policy to the parsed tree, never by substring matching. GPL, LGPL and AGPL are denied in every version and in both -only and -or-later spellings; an exception never rescues a denied base; missing, NOASSERTION, NONE, UNKNOWN, custom, LicenseRef-* and unparseable expressions fail closed. A dual license passes only with an explicit non-denied selection and a written rationale, which is copied into the artifact provenance. Bundled LICENSE/COPYING/NOTICE text is substring-scanned, which is correct for prose, so MIT metadata over GPL text fails as a disagreement. - release_dependency_gate.py enumerates both ecosystems from captured inputs and refuses any asymmetry between the hash-pinned Python lock and the build environment, or between Cargo.lock and the resolved build graph including build-dependencies and cfg()-gated targets. It verifies each captured source hash against its pin, evaluates static and dynamic native linking targets against an explicit auditable platform-runtime soname allowlist, and runs deterministic archive-escape and install-hook detectors. - Strix evidence is accepted only as a machine-readable per-dependency binding covering one isolated synthetic fixture each. A textual "0 findings" is rejected; a missing or malformed binding is a failure, never neutral. The trusted binder is resolved next to the gate script's own directory, adopting strix_quick_gate.sh's trusted-path semantics in new code without editing that file (PR #2291 owns its one-line repair). - On success the gate seals exactly the six members verify_exact_artifact_sbom_handoff.py expects and emits all 17 inputs of exact-artifact-sbom-attestation.yml as workflow outputs, so provenance covers exactly the bytes that were gated. - release_dependency_capture_raw.sh runs the runner-only tools and writes their output verbatim; every decision lives in the unit-tested Python that reads it. No anyio pin and no requirements-strix-ci* file is touched (#2278 owns that lane); the gate adds no Python dependency. Refs #2342. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01FvosNg4GVUjaV5UfrimrsX --- .../release-dependency-license-strix-gate.yml | 351 ++++ ...3-release-dependency-license-strix-gate.md | 53 + scripts/ci/release_dependency_capture_raw.sh | 249 +++ scripts/ci/release_dependency_gate.py | 1439 +++++++++++++++++ scripts/ci/spdx_license_policy.py | 482 ++++++ tests/test_release_dependency_gate.py | 686 ++++++++ ...elease_dependency_gate_capture_and_seal.py | 777 +++++++++ ...lease_dependency_gate_workflow_contract.py | 187 +++ tests/test_spdx_license_policy.py | 197 +++ 9 files changed, 4421 insertions(+) create mode 100644 .github/workflows/release-dependency-license-strix-gate.yml create mode 100644 CHANGELOG.d/20260923-release-dependency-license-strix-gate.md create mode 100755 scripts/ci/release_dependency_capture_raw.sh create mode 100644 scripts/ci/release_dependency_gate.py create mode 100644 scripts/ci/spdx_license_policy.py create mode 100644 tests/test_release_dependency_gate.py create mode 100644 tests/test_release_dependency_gate_capture_and_seal.py create mode 100644 tests/test_release_dependency_gate_workflow_contract.py create mode 100644 tests/test_spdx_license_policy.py diff --git a/.github/workflows/release-dependency-license-strix-gate.yml b/.github/workflows/release-dependency-license-strix-gate.yml new file mode 100644 index 0000000000..82933a60e4 --- /dev/null +++ b/.github/workflows/release-dependency-license-strix-gate.yml @@ -0,0 +1,351 @@ +name: Release Dependency License and Strix Gate + +# Central pre-publish dependency gate (issue #2342). A release workflow calls +# this BEFORE it publishes anything. There is no neutral outcome: the gate +# either succeeds or the release is refused. The organization's scheduled SBOM +# roll-up (scripts/ci/sbom_inventory_aggregator.py) is informational governance +# reporting and is deliberately not reused here. +# +# On success the job's outputs are exactly the inputs of +# .github/workflows/exact-artifact-sbom-attestation.yml, so the caller chains: +# +# gate: +# uses: ContextualWisdomLab/.github/.github/workflows/release-dependency-license-strix-gate.yml@ +# secrets: inherit +# attest: +# needs: gate +# uses: ContextualWisdomLab/.github/.github/workflows/exact-artifact-sbom-attestation.yml@ +# with: +# source_repository: ${{ needs.gate.outputs.source_repository }} +# ... +# +# Provenance then covers exactly the bytes the gate examined. + +on: + workflow_call: + inputs: + source_repository: + description: Release repository in owner/name form. + required: true + type: string + source_sha: + description: Exact release head commit SHA. + required: true + type: string + ecosystems: + description: Comma-separated ecosystems to enumerate (python and/or cargo). + required: true + type: string + python_lock_path: + description: Hash-pinned Python lock installed into the build environment. + required: false + type: string + default: "" + cargo_manifest_path: + description: Release Cargo.toml whose Cargo.lock and build graph are gated. + required: false + type: string + default: "" + build_artifact_name: + description: Caller artifact holding the exact wheel and sdist to publish. + required: true + type: string + wheel_filename: + description: Exact wheel filename inside the build artifact. + required: true + type: string + sdist_filename: + description: Exact source distribution filename inside the build artifact. + required: true + type: string + evidence_artifact_name: + description: Name of the sealed evidence artifact handed to attestation. + required: false + type: string + default: release-dependency-sealed-evidence + secrets: + BYTEZ_API_KEY: + required: true + NVIDIA_NIM_API_KEY: + required: true + NVIDIA_NIM_API_KEY_SUB: + required: true + OPENROUTER_API_KEY: + required: true + OPENAI_API_KEY: + required: true + outputs: + source_repository: + description: Gated release repository. + value: ${{ jobs.gate.outputs.source_repository }} + source_sha: + description: Gated release head SHA. + value: ${{ jobs.gate.outputs.source_sha }} + evidence_artifact_id: + description: Immutable same-run sealed evidence artifact ID. + value: ${{ jobs.gate.outputs.evidence_artifact_id }} + evidence_artifact_name: + description: Sealed evidence artifact name. + value: ${{ jobs.gate.outputs.evidence_artifact_name }} + evidence_artifact_digest: + description: Sealed evidence artifact digest in sha256: form. + value: ${{ jobs.gate.outputs.evidence_artifact_digest }} + wheel_filename: + description: Exact gated wheel filename. + value: ${{ jobs.gate.outputs.wheel_filename }} + wheel_sha256: + description: SHA-256 of the exact gated wheel. + value: ${{ jobs.gate.outputs.wheel_sha256 }} + wheel_sbom_filename: + description: CycloneDX SBOM filename for the gated wheel. + value: ${{ jobs.gate.outputs.wheel_sbom_filename }} + wheel_sbom_sha256: + description: SHA-256 of the gated wheel's CycloneDX SBOM. + value: ${{ jobs.gate.outputs.wheel_sbom_sha256 }} + sdist_filename: + description: Exact gated source distribution filename. + value: ${{ jobs.gate.outputs.sdist_filename }} + sdist_sha256: + description: SHA-256 of the exact gated source distribution. + value: ${{ jobs.gate.outputs.sdist_sha256 }} + sdist_sbom_filename: + description: CycloneDX SBOM filename for the gated source distribution. + value: ${{ jobs.gate.outputs.sdist_sbom_filename }} + sdist_sbom_sha256: + description: SHA-256 of the gated source distribution's CycloneDX SBOM. + value: ${{ jobs.gate.outputs.sdist_sbom_sha256 }} + source_identity_sha256: + description: SHA-256 of the sealed source-identity.json. + value: ${{ jobs.gate.outputs.source_identity_sha256 }} + checksum_sha256: + description: SHA-256 of the sealed checksums.sha256. + value: ${{ jobs.gate.outputs.checksum_sha256 }} + predicate_type: + description: Canonical CycloneDX in-toto predicate type. + value: ${{ jobs.gate.outputs.predicate_type }} + cyclonedx_schema: + description: Canonical CycloneDX 1.7 schema URL. + value: ${{ jobs.gate.outputs.cyclonedx_schema }} + +permissions: + contents: read + +env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true + +jobs: + gate: + name: Gate every resolved dependency before publishing + runs-on: ubuntu-24.04 + timeout-minutes: 360 + permissions: + contents: read + outputs: + source_repository: ${{ steps.seal.outputs.source_repository }} + source_sha: ${{ steps.seal.outputs.source_sha }} + evidence_artifact_id: ${{ steps.sealed-evidence.outputs.artifact-id }} + evidence_artifact_name: ${{ steps.seal.outputs.evidence_artifact_name }} + evidence_artifact_digest: sha256:${{ steps.sealed-evidence.outputs.artifact-digest }} + wheel_filename: ${{ steps.seal.outputs.wheel_filename }} + wheel_sha256: ${{ steps.seal.outputs.wheel_sha256 }} + wheel_sbom_filename: ${{ steps.seal.outputs.wheel_sbom_filename }} + wheel_sbom_sha256: ${{ steps.seal.outputs.wheel_sbom_sha256 }} + sdist_filename: ${{ steps.seal.outputs.sdist_filename }} + sdist_sha256: ${{ steps.seal.outputs.sdist_sha256 }} + sdist_sbom_filename: ${{ steps.seal.outputs.sdist_sbom_filename }} + sdist_sbom_sha256: ${{ steps.seal.outputs.sdist_sbom_sha256 }} + source_identity_sha256: ${{ steps.seal.outputs.source_identity_sha256 }} + checksum_sha256: ${{ steps.seal.outputs.checksum_sha256 }} + predicate_type: ${{ steps.seal.outputs.predicate_type }} + cyclonedx_schema: ${{ steps.seal.outputs.cyclonedx_schema }} + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Materialize immutable trusted gate + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + repository: ContextualWisdomLab/.github + ref: ${{ github.workflow_sha }} + path: trusted-gate + persist-credentials: false + sparse-checkout: | + scripts/ci/release_dependency_gate.py + scripts/ci/release_dependency_capture_raw.sh + scripts/ci/spdx_license_policy.py + scripts/ci/strix_evidence_binding.py + scripts/ci/contextual_orchestrator_review_sidecar.sh + scripts/ci/load_contextual_orchestrator_token.sh + requirements-strix-ci-hashes.txt + sparse-checkout-cone-mode: false + + - name: Check out the exact release head + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + repository: ${{ inputs.source_repository }} + ref: ${{ inputs.source_sha }} + path: release-source + persist-credentials: false + + - name: Set up the release build interpreter + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + + - name: Install the release dependency closure from its hash-pinned lock + if: ${{ inputs.python_lock_path != '' }} + env: + PYTHON_LOCK_PATH: ${{ inputs.python_lock_path }} + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + python3 -m pip install --require-hashes --only-binary=:all: \ + -r "release-source/${PYTHON_LOCK_PATH}" + + - name: Download the exact distributions the caller intends to publish + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: ${{ inputs.build_artifact_name }} + path: release-distributions + + - name: Collect raw resolved-dependency evidence from both ecosystems + env: + ECOSYSTEMS: ${{ inputs.ecosystems }} + PYTHON_LOCK_PATH: ${{ inputs.python_lock_path }} + CARGO_MANIFEST_PATH: ${{ inputs.cargo_manifest_path }} + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + python_lock="" + cargo_manifest="" + if [ -n "$PYTHON_LOCK_PATH" ]; then + python_lock="${PWD}/release-source/${PYTHON_LOCK_PATH}" + fi + if [ -n "$CARGO_MANIFEST_PATH" ]; then + cargo_manifest="${PWD}/release-source/${CARGO_MANIFEST_PATH}" + fi + bash trusted-gate/scripts/ci/release_dependency_capture_raw.sh \ + --raw-root "${RUNNER_TEMP}/raw" \ + --capture-root "${RUNNER_TEMP}/capture" \ + --ecosystems "$ECOSYSTEMS" \ + --python-lock "$python_lock" \ + --cargo-manifest "$cargo_manifest" + + - name: Assemble per-dependency evidence and isolated synthetic fixtures + env: + SOURCE_REPOSITORY: ${{ inputs.source_repository }} + SOURCE_SHA: ${{ inputs.source_sha }} + ECOSYSTEMS: ${{ inputs.ecosystems }} + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + jq -n \ + --arg repository "$SOURCE_REPOSITORY" \ + --arg sha "$SOURCE_SHA" \ + --arg ecosystems "$ECOSYSTEMS" \ + '{source_repository: $repository, source_sha: $sha, + ecosystems: ($ecosystems | split(","))}' \ + > "${RUNNER_TEMP}/capture/release.json" + python3 -I trusted-gate/scripts/ci/release_dependency_gate.py capture \ + --raw "${RUNNER_TEMP}/raw" \ + --capture "${RUNNER_TEMP}/capture" + + - name: Provision the zero-cost review gateway for Strix + env: + BYTEZ_API_KEY: ${{ secrets.BYTEZ_API_KEY }} + NVIDIA_NIM_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }} + NVIDIA_NIM_API_KEY_SUB: ${{ secrets.NVIDIA_NIM_API_KEY_SUB }} + OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + bash trusted-gate/scripts/ci/contextual_orchestrator_review_sidecar.sh + + - name: Install the pinned Strix toolchain + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + python3 -m pip install --require-hashes --only-binary=:all: \ + -r trusted-gate/requirements-strix-ci-hashes.txt + + - name: Run Strix against one isolated synthetic fixture per dependency + env: + STRIX_LLM: contextual-orchestrator/orchestrator/free + STRIX_LLM_DEFAULT_PROVIDER: contextual_orchestrator + STRIX_REASONING_EFFORT: none + SOURCE_SHA: ${{ inputs.source_sha }} + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + source trusted-gate/scripts/ci/load_contextual_orchestrator_token.sh + mkdir -p "${RUNNER_TEMP}/capture/strix/bindings" + for fixture in "${RUNNER_TEMP}"/capture/strix/fixtures/*.json; do + slug="$(basename "$fixture" .json)" + workspace="${RUNNER_TEMP}/strix-workspace/${slug}" + mkdir -p "$workspace" + cp "$fixture" "$workspace/fixture.json" + ( cd "$workspace" && strix --target . --non-interactive ) + vulnerabilities="" + if [ -d "$workspace/strix_runs" ]; then + vulnerabilities="$(find "$workspace/strix_runs" -type f \ + -name 'vulnerabilities.json' -print -quit)" + fi + if [ -z "$vulnerabilities" ]; then + echo "::warning::Strix produced no machine-readable findings file for ${slug};" \ + "the gate will refuse the release with STRIX_BINDING_MISSING." + continue + fi + jq -n \ + --argjson fixture "$(cat "$fixture")" \ + --argjson findings "$(cat "$vulnerabilities")" \ + --arg sha "$SOURCE_SHA" \ + --arg digest "$(cat "${fixture%.json}.sha256")" \ + '{ + schema: "cwl.release-dependency-strix-binding/1", + dependency: $fixture.dependency, + fixture: { + id: ($fixture.dependency.ecosystem + "/" + $fixture.dependency.name + + "@" + $fixture.dependency.version), + sha256: $digest, + scenarios: ($fixture.scenarios | keys) + }, + source_sha: $sha, + findings: $findings, + verdict: (if ($findings | length) == 0 + then "no_exploitable_findings" else "findings_present" end) + }' > "${RUNNER_TEMP}/capture/strix/bindings/${slug}.json" + done + + - name: Refuse the release unless every dependency passes + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + python3 -I trusted-gate/scripts/ci/release_dependency_gate.py gate \ + --capture "${RUNNER_TEMP}/capture" \ + --report "${RUNNER_TEMP}/gate-report.json" + + - name: Seal exactly the gated bytes for attestation + id: seal + env: + EVIDENCE_ARTIFACT_NAME: ${{ inputs.evidence_artifact_name }} + WHEEL_FILENAME: ${{ inputs.wheel_filename }} + SDIST_FILENAME: ${{ inputs.sdist_filename }} + shell: bash --noprofile --norc -e -o pipefail {0} + run: | + python3 -I trusted-gate/scripts/ci/release_dependency_gate.py seal \ + --report "${RUNNER_TEMP}/gate-report.json" \ + --wheel "release-distributions/${WHEEL_FILENAME}" \ + --sdist "release-distributions/${SDIST_FILENAME}" \ + --evidence-root "${RUNNER_TEMP}/sealed-evidence" \ + --evidence-artifact-name "$EVIDENCE_ARTIFACT_NAME" + + - name: Export the sealed evidence as one immutable same-run artifact + id: sealed-evidence + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.0 + with: + name: ${{ inputs.evidence_artifact_name }} + path: ${{ runner.temp }}/sealed-evidence + if-no-files-found: error + + - name: Export the per-dependency gate report + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.0 + with: + name: release-dependency-gate-report + path: ${{ runner.temp }}/gate-report.json + if-no-files-found: error diff --git a/CHANGELOG.d/20260923-release-dependency-license-strix-gate.md b/CHANGELOG.d/20260923-release-dependency-license-strix-gate.md new file mode 100644 index 0000000000..9ca283e206 --- /dev/null +++ b/CHANGELOG.d/20260923-release-dependency-license-strix-gate.md @@ -0,0 +1,53 @@ +### Central pre-publish dependency gate: parsed license denial, resolved-graph reconciliation, per-dependency Strix bindings + +- `origin/main` had **no** fail-closed pre-publish dependency gate. The only license signal was + `scripts/ci/sbom_inventory_aggregator.py`, a *scheduled, informational* org SBOM roll-up that + flags GPL/AGPL/NOASSERTION for governance: it is not per-dependency, not fail-closed, and not + bound to a release head. That gap blocked fast-mlsirm's 0.11.5 PyPI release and + contextual-orchestrator's VCS-pin removal (#2342). +- New reusable `workflow_call` workflow `.github/workflows/release-dependency-license-strix-gate.yml` + runs **before** a release workflow publishes. It has no `continue-on-error`, no `if: always()`, + no neutral outcome, and no bypass; `permissions` is `contents: read` at both workflow and job + scope, and every action is pinned to the same commits `exact-artifact-sbom-attestation.yml` uses. + The decision code is materialized from `ContextualWisdomLab/.github` at `github.workflow_sha` + into `trusted-gate/`, so a caller's tree can never supply it. +- New `scripts/ci/spdx_license_policy.py` is a recursive-descent SPDX 2.3 expression parser + (`AND`/`OR`/`WITH`/parentheses/legacy `+`). Policy is applied to the parsed tree, never by + substring matching: GPL, LGPL, and AGPL are denied in every version and in both the `-only` and + `-or-later` spellings, an exception never rescues a denied base (`GPL-2.0-only WITH + Classpath-exception-2.0` stays denied), and `missing`, `NOASSERTION`, `NONE`, `UNKNOWN`, + `custom`, `LicenseRef-*`, and any unparseable expression fail closed. A dual-licensed dependency + passes only when a non-denied operand is explicitly selected with a written rationale, which is + copied into the artifact provenance as a CycloneDX component property. Bundled `LICENSE`, + `COPYING`, and `NOTICE` text *is* substring-scanned — correct for prose — so metadata claiming + MIT while shipping GPL text fails as a disagreement. +- New `scripts/ci/release_dependency_gate.py` enumerates both ecosystems and refuses any + asymmetry: the hash-pinned Python lock against `pip inspect` of the build environment + (`LOCK_ENV_MISMATCH`), and `Cargo.lock` against the full resolved build graph including + build-dependencies and every `cfg()`-gated target (`CARGO_LOCK_GRAPH_MISMATCH`, + `CARGO_CHECKSUM_MISSING`). It records name, version, source hash, license, license source, and + distribution inclusion per dependency; verifies the captured source hash against the pin + (`SOURCE_HASH_MISMATCH`); evaluates static and dynamic linking targets of shipped native + libraries against an explicit, auditable platform-runtime soname allowlist (glibc, the GCC + runtime-library-exception libraries, `libpython`) so a real compiled wheel can pass at all; and + runs deterministic archive-escape and install-hook detectors (`ARCHIVE_PATH_ESCAPE`, + `INSTALL_HOOK`). +- Strix evidence is accepted **only** as a machine-readable binding, one isolated synthetic + fixture per dependency, simulating file parsing, install hooks, archive traversal, native library + loading, credential/network attempts, and known-vulnerability surface. A textual "0 findings" or + "No exploitable vulnerabilities detected" is rejected (`STRIX_TEXTUAL_PASS_REJECTED`), and a + missing or malformed binding is a failure, never neutral (`STRIX_BINDING_MISSING`, + `STRIX_BINDING_MALFORMED`, `STRIX_BINDING_UNBOUND`). The trusted binder is resolved next to the + gate script's **own** directory, adopting `strix_quick_gate.sh`'s trusted-path semantics in new + code without touching that file (PR #2291 owns its one-line repair). +- On success the gate seals exactly the six members + `scripts/ci/verify_exact_artifact_sbom_handoff.py` expects — wheel, sdist, their CycloneDX 1.7 + SBOMs, `source-identity.json`, `checksums.sha256` — and emits all 17 inputs of + `exact-artifact-sbom-attestation.yml` as workflow outputs, so provenance covers exactly the bytes + that were gated. `tests/test_release_dependency_gate_capture_and_seal.py` proves the sealed + directory is accepted verbatim by that verifier. +- `scripts/ci/release_dependency_capture_raw.sh` runs the runner-only tools (`pip inspect`, + `pip download`, `cargo metadata --locked`, `cargo fetch`, archive listing, `readelf -d`) and + writes their output verbatim; every decision lives in the unit-tested Python that reads it. The + gate adds no Python dependency and does not touch any `anyio` pin or `requirements-strix-ci*` + (#2278 owns that lane). Refs #2342. diff --git a/scripts/ci/release_dependency_capture_raw.sh b/scripts/ci/release_dependency_capture_raw.sh new file mode 100755 index 0000000000..02133149b6 --- /dev/null +++ b/scripts/ci/release_dependency_capture_raw.sh @@ -0,0 +1,249 @@ +#!/usr/bin/env bash +# Collect raw pre-publish dependency evidence for the central release gate (#2342). +# +# This script only *runs tools and writes their output verbatim*. Every decision +# — license policy, lock/environment reconciliation, archive-escape and +# install-hook detection, Strix binding validation — lives in the unit-tested +# scripts/ci/release_dependency_gate.py, which reads what this writes. Keeping +# the split that way means no untested shell ever decides whether a release may +# publish. +# +# It requires a runner: pip, cargo, readelf, and network access to the indexes. +# It is therefore exercised in GitHub Actions only; see +# .github/workflows/release-dependency-license-strix-gate.yml. +# +# Output layout (consumed by `release_dependency_gate.py capture` and `gate`): +# +# /python/lock.txt the hash-pinned lock that was installed +# /python/installed.json pip inspect of the build environment +# /cargo/Cargo.lock the committed Cargo lock +# /cargo/metadata.json cargo metadata --format-version 1 --locked +# //metadata.json declared identity + license fields +# //source.sha256 sha256 of the distribution as fetched +# //members.txt "\t\t" per member +# //licenses/* bundled LICENSE/COPYING/NOTICE verbatim +# //hooks/* setup.py / build.rs sources verbatim +# //native.json dynamic/static link targets per shipped .so +# //parsed_inputs.txt file names the dependency parses + +set -euo pipefail + +RAW_ROOT="" +CAPTURE_ROOT="" +ECOSYSTEMS="" +PYTHON_LOCK="" +CARGO_MANIFEST="" + +while [ "$#" -gt 0 ]; do + case "$1" in + --raw-root) RAW_ROOT="$2"; shift 2 ;; + --capture-root) CAPTURE_ROOT="$2"; shift 2 ;; + --ecosystems) ECOSYSTEMS="$2"; shift 2 ;; + --python-lock) PYTHON_LOCK="$2"; shift 2 ;; + --cargo-manifest) CARGO_MANIFEST="$2"; shift 2 ;; + *) echo "ERROR: unknown argument $1" >&2; exit 2 ;; + esac +done + +if [ -z "$RAW_ROOT" ] || [ -z "$CAPTURE_ROOT" ] || [ -z "$ECOSYSTEMS" ]; then + echo "ERROR: --raw-root, --capture-root and --ecosystems are required." >&2 + exit 2 +fi + +mkdir -p "$RAW_ROOT" "$CAPTURE_ROOT" + +# Record one archive's members as "\t\t". Symlink and +# hardlink targets are preserved verbatim so the gate can detect escapes. +record_members() { + local archive="$1" destination="$2" + case "$archive" in + *.whl | *.zip) + unzip -Z1 "$archive" | while IFS= read -r member; do + printf 'file\t%s\t\n' "$member" + done + ;; + *) + tar -tvf "$archive" | while IFS= read -r line; do + local permissions name link type + permissions="${line%% *}" + name="$(printf '%s' "$line" | sed -E 's/^.* [0-9]{2}:[0-9]{2} //')" + link="" + type="file" + case "$permissions" in + l*) type="symlink"; link="${name#* -> }"; name="${name%% -> *}" ;; + h*) type="hardlink"; link="${name#* link to }"; name="${name%% link to *}" ;; + d*) type="directory" ;; + esac + printf '%s\t%s\t%s\n' "$type" "$name" "$link" + done + ;; + esac >"$destination" +} + +# Record every bundled license-like file verbatim, flattened into one directory. +record_license_files() { + local root="$1" destination="$2" + mkdir -p "$destination" + find "$root" -maxdepth 4 -type f \ + \( -iname 'LICENSE*' -o -iname 'COPYING*' -o -iname 'NOTICE*' \) -print0 | + while IFS= read -r -d '' found; do + cp -- "$found" "$destination/$(printf '%s' "${found#"$root"/}" | tr '/' '_')" + done +} + +# Record install/build hook sources verbatim so the gate can inspect them. +record_hook_sources() { + local root="$1" destination="$2" + mkdir -p "$destination" + find "$root" -maxdepth 3 -type f \ + \( -name 'setup.py' -o -name 'build.rs' -o -name 'conanfile.py' \) -print0 | + while IFS= read -r -d '' found; do + cp -- "$found" "$destination/$(printf '%s' "${found#"$root"/}" | tr '/' '_')" + done +} + +# Record dynamic NEEDED entries and shipped static archives for native libraries. +record_native_libraries() { + local root="$1" destination="$2" + local entries="[]" + while IFS= read -r library; do + local needed + needed="$(readelf -d "$library" 2>/dev/null | + sed -n 's/.*(NEEDED).*\[\(.*\)\]/\1/p' | + jq -R . | jq -s .)" + entries="$(jq --arg path "${library#"$root"/}" --argjson needed "${needed:-[]}" \ + '. + [{"path": $path, "needed": $needed, "static_archives": []}]' <<<"$entries")" + done < <(find "$root" -type f \( -name '*.so' -o -name '*.so.*' -o -name '*.pyd' \)) + printf '%s\n' "$entries" >"$destination" +} + +capture_python() { + local lock="$1" + mkdir -p "$CAPTURE_ROOT/python" + cp -- "$lock" "$CAPTURE_ROOT/python/lock.txt" + python3 -m pip inspect --local >"$CAPTURE_ROOT/python/installed.json" + + local download_root + download_root="$(mktemp -d)" + python3 -m pip download --no-deps --no-build-isolation \ + --dest "$download_root" -r "$lock" >/dev/null + + while IFS=$'\t' read -r name version; do + local slug distribution extracted target + slug="pypi__$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]' | tr '._' '--')__$version" + target="$RAW_ROOT/$slug" + mkdir -p "$target" + distribution="$(find "$download_root" -maxdepth 1 -type f \ + -iname "$(printf '%s' "$name" | tr '.-' '__')-${version}*" | head -n 1)" + if [ -z "$distribution" ]; then + echo "ERROR: no fetched distribution for ${name}==${version}" >&2 + exit 2 + fi + sha256sum "$distribution" | cut -d' ' -f1 >"$target/source.sha256" + record_members "$distribution" "$target/members.txt" + extracted="$(mktemp -d)" + case "$distribution" in + *.whl) unzip -qq -o "$distribution" -d "$extracted" ;; + *) tar -xf "$distribution" -C "$extracted" ;; + esac + record_license_files "$extracted" "$target/licenses" + record_hook_sources "$extracted" "$target/hooks" + record_native_libraries "$extracted" "$target/native.json" + find "$extracted" -maxdepth 3 -type f -name '*.py' -printf '%P\n' | + LC_ALL=C sort >"$target/parsed_inputs.txt" + printf '{}\n' >"$target/bundled_library_licenses.json" + python3 -m pip show "$name" | + jq -R -s --arg name "$name" --arg version "$version" ' + split("\n") + | map(select(length > 0)) + | { + ecosystem: "pypi", + name: $name, + version: $version, + license_expression: ( + map(select(startswith("License-Expression: "))) + | first // "" | sub("^License-Expression: "; "") + ), + license: ( + map(select(startswith("License: "))) + | first // "" | sub("^License: "; "") + ), + classifiers: ( + map(select(startswith("Classifier: License "))) + | map(sub("^Classifier: "; "")) + ), + distribution_inclusion: ["sdist", "wheel"], + known_vulnerabilities: [] + }' >"$target/metadata.json" + rm -rf "${extracted:?}" + done < <(jq -r '.installed[] | [.metadata.name, .metadata.version] | @tsv' \ + "$CAPTURE_ROOT/python/installed.json") + rm -rf "${download_root:?}" +} + +capture_cargo() { + local manifest="$1" manifest_dir + manifest_dir="$(dirname -- "$manifest")" + mkdir -p "$CAPTURE_ROOT/cargo" + cp -- "$manifest_dir/Cargo.lock" "$CAPTURE_ROOT/cargo/Cargo.lock" + cargo metadata --format-version 1 --locked --manifest-path "$manifest" \ + >"$CAPTURE_ROOT/cargo/metadata.json" + cargo fetch --locked --manifest-path "$manifest" >/dev/null + + while IFS=$'\t' read -r name version license; do + local slug target crate extracted + slug="cargo__${name}__${version}" + target="$RAW_ROOT/$slug" + mkdir -p "$target" + crate="$(find "${CARGO_HOME:-$HOME/.cargo}/registry/cache" -type f \ + -name "${name}-${version}.crate" | head -n 1)" + if [ -z "$crate" ]; then + echo "ERROR: no fetched crate for ${name} ${version}" >&2 + exit 2 + fi + sha256sum "$crate" | cut -d' ' -f1 >"$target/source.sha256" + record_members "$crate" "$target/members.txt" + extracted="$(mktemp -d)" + tar -xf "$crate" -C "$extracted" + record_license_files "$extracted" "$target/licenses" + record_hook_sources "$extracted" "$target/hooks" + printf '[]\n' >"$target/native.json" + printf '{}\n' >"$target/bundled_library_licenses.json" + find "$extracted" -maxdepth 3 -type f -name '*.rs' -printf '%P\n' | + LC_ALL=C sort >"$target/parsed_inputs.txt" + jq -n --arg name "$name" --arg version "$version" --arg license "$license" '{ + ecosystem: "cargo", + name: $name, + version: $version, + license_expression: $license, + license: "", + classifiers: [], + distribution_inclusion: ["wheel"], + known_vulnerabilities: [] + }' >"$target/metadata.json" + rm -rf "${extracted:?}" + done < <(jq -r '.packages[] | select(.source != null) | [.name, .version, (.license // "")] | @tsv' \ + "$CAPTURE_ROOT/cargo/metadata.json") +} + +case ",${ECOSYSTEMS}," in +*,python,*) + if [ -z "$PYTHON_LOCK" ] || [ ! -f "$PYTHON_LOCK" ]; then + echo "ERROR: --python-lock must name the hash-pinned lock that was installed." >&2 + exit 2 + fi + capture_python "$PYTHON_LOCK" + ;; +esac + +case ",${ECOSYSTEMS}," in +*,cargo,*) + if [ -z "$CARGO_MANIFEST" ] || [ ! -f "$CARGO_MANIFEST" ]; then + echo "ERROR: --cargo-manifest must name the release Cargo.toml." >&2 + exit 2 + fi + capture_cargo "$CARGO_MANIFEST" + ;; +esac + +echo "Raw dependency capture complete: $(find "$RAW_ROOT" -mindepth 1 -maxdepth 1 -type d | wc -l) dependencies." diff --git a/scripts/ci/release_dependency_gate.py b/scripts/ci/release_dependency_gate.py new file mode 100644 index 0000000000..4309a850af --- /dev/null +++ b/scripts/ci/release_dependency_gate.py @@ -0,0 +1,1439 @@ +#!/usr/bin/env python3 +"""Fail-closed pre-publish dependency gate for org releases (issue #2342). + +``scripts/ci/sbom_inventory_aggregator.py`` is a *scheduled, informational* org +SBOM roll-up: it flags GPL/AGPL/NOASSERTION components for governance, but it +is not per-dependency, not fail-closed, and not bound to a release head. This +module is the missing gate. It runs in +``.github/workflows/release-dependency-license-strix-gate.yml`` **before** a +release workflow publishes anything, and it either exits ``0`` or refuses the +release. There is no neutral outcome, no allow-failure, and no bypass. + +Design: the gate is a pure function over *captured* inputs. Workflow steps run +``pip inspect``, ``cargo metadata --locked``, archive listing, ``readelf -d``, +and Strix; each writes a file into a capture directory. This module only reads +files. That split keeps every deterministic decision unit-testable without a +runner and makes the Actions-only parts explicit instead of simulated. + +Capture layout (produced by the workflow, consumed here):: + + / + release.json source repository/SHA + artifact names + python/lock.txt the hash-pinned lock that was installed + python/installed.json `pip inspect` of the build environment + cargo/Cargo.lock the committed Cargo lock + cargo/metadata.json `cargo metadata --format-version 1 --locked` + evidence/.json per-dependency captured evidence + strix/bindings/.json per-dependency Strix structured binding + license-selections.json optional dual-license selections + +Every resolved dependency of both ecosystems must appear in the lock *and* in +the environment/build graph; any asymmetry fails ``LOCK_ENV_MISMATCH`` or +``CARGO_LOCK_GRAPH_MISMATCH``. No dependency is exempt: bootstrap tools such as +``pip`` are pinned in this organization's own ``*-hashes.txt`` files, so a lock +that omits an installed distribution is a defect, not a special case. + +Strix evidence is accepted **only** as a machine-readable binding. A textual +"0 findings" or "No exploitable vulnerabilities detected" is rejected +(``STRIX_TEXTUAL_PASS_REJECTED``), and a missing or malformed binding is a +failure rather than a neutral result. The binding's fail-closed shape and error +type follow ``scripts/ci/strix_evidence_binding.py``, which is imported from +this script's **own** directory so the gate behaves identically wherever the +trusted verifier is materialized. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import sys +import tomllib +import uuid +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Iterable, Mapping, Sequence + +try: + from scripts.ci.spdx_license_policy import ( + LICENSE_MISSING, + LicenseDecision, + evaluate_license_expression, + scan_license_text, + spdx_from_classifiers, + ) +except ImportError: # pragma: no cover - direct `python3 -I