Skip to content

Feat/heremetic build v0.3 - #373

Open
AadarshLN wants to merge 1 commit into
darnitdevorg:mainfrom
AadarshLN:feat/heremetic-build-v0.3
Open

Feat/heremetic build v0.3#373
AadarshLN wants to merge 1 commit into
darnitdevorg:mainfrom
AadarshLN:feat/heremetic-build-v0.3

Conversation

@AadarshLN

Copy link
Copy Markdown
Contributor

Summary

RE-02.01 (HermeticBuild) v0.2 treated any CI text mentioning witness run as a strong PASS signal — but that only proves the tool ran, not what it observed. This PR moves to v0.3: real, cryptographically verified attestation evidence.

  • Verified attestation, not text grep: Fetches Witness/runtime-trace attestation artifacts from the repo's latest successful CI run via the gh CLI, then verifies the DSSE envelope with Sigstore against the repo's GitHub Actions OIDC identity before trusting anything it claims
  • Two predicate shapes handled: the newer runtime-trace/v0.1 predicate's network array is authoritative (empty = clean, non-empty = dirty); Witness's own command-run attestation has no network field, so falls back to scanning process cmdline/program for the same suspicious patterns as the grep heuristic — a best-effort hint, not proof
  • Dirty evidence outranks the grep: a verified attestation that does record network activity is fed into the violation list ahead of the CI-text scan — stronger evidence than a line match
  • Mention alone no longer passes: witness run appearing in CI YAML with nothing verified behind it is no longer sufficient for PASS (previously it was)
  • Fails safe, always: missing gh CLI, no auth, no matching CI run/artifact, sigstore not installed, or a verification failure all degrade to "no attestation evidence" — never to failing the audit outright
  • Distinguishable failure reasons: evidence now differentiates "gh not authenticated" from "no successful CI run" from "no matching artifacts," instead of one generic "nothing found" message
  • Opt-out toggle: verify_witness_attestations = false in the TOML pass config skips the network round-trip entirely, for air-gapped audits or repos without a usable gh login

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

If this PR modifies the darnit framework (packages/darnit/):

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Control/TOML Changes Checklist

If this PR modifies controls or TOML configuration:

  • Control metadata defined in TOML (not Python code)
  • SARIF fields (description, severity, help_url) included where appropriate
  • Ran validation to confirm TOML schema compliance

Testing

  • Tests pass locally (uv run pytest tests/ -v)
  • Added tests for new functionality (if applicable)
  • Linting passes (uv run ruff check .)

Additional Notes

  • 137 tests passed locally
  • added Sigstore-verified Witness/runtime-trace attestation checking to RE-02.01, replacing the v0.2 heuristic that treated any CI-text mention of "witness run" as a strong PASS signal.

@mlieberman85

Copy link
Copy Markdown
Contributor

Thanks for the PR. Agreed on the general premise here. Just a couple of small fixes and a rebase and I think it should be good to go.

Must-fix

  1. Rebase. The +1/-1 change to .github/dot-project-spec-hash.txt overlaps with fix(context): reconcile .project/ reader with CNCF spec drift (closes #372) #374 (just merged); the delta will disappear cleanly on rebase against main. That's what's making the PR show DIRTY.

  2. CI workflow approval. Only Kusari Inspector ran on this PR — the standard Test / Lint / Type Check jobs need a maintainer approval before they fire on a first-time contributor's fork PR. I'll approve so we can see the 137 tests actually run in-repo, not just locally.

Should-fix

  1. _MAX_ARTIFACT_FILES = 5 silently caps the search at 5 alphabetically-sorted candidates. If a repo has 6+ attestation artifacts and the verifiable one sorts 6th, we miss it. Either raise the cap (10-20 is still cheap), or prioritize by filename heuristic (*.att.json / *.bundle.json / *.sigstore.json before generic *.json).

  2. Discoverability of verify_witness_attestations. The opt-out toggle is documented in the Python docstring but not in reproducibility.toml. A maintainer reading the TOML has no way to see it. Please add a comment in the RE-02.01 pass section naming the option and its default so the option is greppable from config.

Nits (LOW, non-blocking)

  • _SUSPICIOUS_CMDLINE_PATTERNS duplicates _SUSPICIOUS_PATTERNS. The comment defends this as "different data source" (process cmdlines vs CI text), which is fair — but a single tuple with two names would keep them in sync automatically as the list evolves.
  • _strip_comment isn't quote-aware. Fine for the current heuristic; consider an inline # TODO noting a shlex-based tokenizer as a future-proofing option if false positives surface on shell/YAML lines with # inside strings.
  • scripts/audit_hermetic_real_repos.py could use a header comment pointing at the design doc for RE-02.01 so future maintainers know what it validates.

Positives worth calling out

  • Constitution II alignment is explicit: grep-clean → INCONCLUSIVE (not PASS), and the docstring says why. That's the conservative-by-default posture the constitution requires.
  • Failure-mode labeling (_AUTH_ERROR_HINTS distinguishing "gh not logged in" from "no matching artifacts" from "no successful CI run") is much more useful than a generic "no witness evidence" reason string.
  • witness_result.network_clean is False correctly feeds into the violations list ahead of the grep — verified positive evidence outranks negative-evidence heuristics.
  • 109 new tests, well-partitioned by unit/behavior boundary. test_witness_takes_priority_over_nix locks a subtle ordering invariant that would otherwise regress silently.

…ticity

Replace the CI-text grep for "witness run" with real verification: fetch
Witness/runtime-trace attestation artifacts from the repo's latest
successful CI run via gh, cryptographically verify the DSSE envelope
against the repo's GitHub Actions OIDC identity via Sigstore, and only
treat an empty verified network log as a strong PASS signal. A verified
attestation that does record network activity is fed into the violation
list. Add a verify_witness_attestations config toggle and distinguish
gh-not-authenticated from no-artifacts-found in evidence.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mlieberman85
mlieberman85 force-pushed the feat/heremetic-build-v0.3 branch from 7d821ae to a3b178c Compare August 16, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants