feat(harness): every coverage difference between the engines is registered per rule, both ways (GT-716 AC3) - #795
Merged
Conversation
…tered per rule, both ways (GT-716 AC3) `68-validate-engine-verdict-parity.mjs` holds the engines to agreement on the rules BOTH decide and prints what only one decides as two counts gated on nothing. ADR-0041 never promised equal reach; this does not ask for it. It asks that every coverage difference be a diff somebody reads. - `73-validate-engine-coverage-parity.mjs`, a sibling of 68 reusing its `deriveOutcomes` precedence: both engines run on the repository root and on a satellite the guard creates with `evolith init` in a temporary directory (`--core` pointed at the root), because the sign flips between the two — here the native engine decides 138 ADR-conformance rules no policy names; there the policies that decide anything read facets nobody supplied. - Every rule decided by exactly one engine is held to `engine-coverage-parity.baseline.json`: per scenario, per direction, per rule, with the reason the OTHER engine gave. Measured first — the class its report states and, for the OPA side, the facets its skip row names — then derived: an id no reachable policy emits, or a policy reading facets `opa-input-builder.ts` never emits (absent on a bare run whatever their provenance: `layers` is observed in nature and supplied in practice, `input.repository` is produced by nothing). A native-side class stated on the OPA side is filed as `opa-gave-no-reason` (the enforcer-routed HXA-01/02/04/05 skip without a class of their own), not as handler debt. - The ratchet closes both ways: an unregistered rule fails, a registered rule both engines now decide fails, an entry whose class changed fails. `--write` regenerates the file for review; `--json` publishes the counts. Measured 2026-09-20 — repository: 220 native-only (164 no-policy-in-bundle, 52 supplied-facet-absent, 4 opa-gave-no-reason) and 8 opa-only (7 unimplemented-native, 1 handler declined); init satellite: 50 native-only (35 / 11 / 4) and 4 opa-only (2 needs-runtime, 1 needs-external-system, 1 handler declined). ≈17 s + ≈3 s, in the `Test` job after 68. Falsifier, both outputs recorded: the criterion's own probe (drop an import from `main.rego`) cannot run since GT-675 — the bundle build refuses a reachable policy nobody imports — so the equivalent is renaming `OBS-EVD-03` in `telemetry-evidence.rego`: the guard went red on both scenarios naming the id (`opaOnly OBS-EVD-03 … no longer coverage-only`), and green once restored. Guard 42 classifies it INSTRUMENTED (89 guards); guard 43 observed it red on the empty fixture (65/65); 11 unit tests. `known-limitations` and the OPA README now say CI registers coverage differences per rule rather than merely allowing them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: aarroyo <beyondnet.peru@gmail.com>
…er-rule ratchet now Ticks criterion 3 in both catalogs with the measurement and the recorded falsifier (the criterion's own probe cannot run since GT-675; the equivalent one — renaming OBS-EVD-03 in its policy — turned the guard red on both scenarios naming the id), appends the closure to the board row and adds a `Last Updated` line. Counters unchanged (688 / 715). 08, 01, 04 and 46 green. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: aarroyo <beyondnet.peru@gmail.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
📊 Bilingual Coverage ImpactPR Changes
Repository Coverage
To create skeletons: node .harness/scripts/generate-es-skeleton.mjs <file.md>Generated by GitHub Actions |
…cked artifacts flipped six rules between machines The first CI run of the coverage ratchet disagreed with the laptop that wrote its baseline: EM-Y-01 and QT-01 were decided natively here (a `coverage/` directory from a local jest run) and skipped there; DRIFT-01 was decided here (git history) and skipped on a shallow clone; MCP-01..03 the other way round; and OBS-EVD-01..03 changed class because the CLI's bundled corpus copy predated `facets.json` and classified them by the old defaults. Both scenarios now read the Core from an export of the tracked files (`git ls-files`, local modifications included) plus the compiled `policy.wasm`: no coverage directories, no dists, no `.git`, on every machine alike. A rule whose native verdict needs one of those is skipped identically everywhere, which is the fact the baseline should carry. The export also corrected a measurement this branch had recorded: the 138 ADR-conformance rules were never "decided by native alone" — the working-tree run resolved their decision-record references against the CLI's bundled copy and failed all 138 falsely; against the export they are documentation on both engines, decided by neither. Repository: 82 native-only (52 supplied-facet- absent, 26 no-policy-in-bundle, 4 opa-gave-no-reason) and 7 opa-only; init satellite: 49 and the same 7. Catalog, board and the guard's header say so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: aarroyo <beyondnet.peru@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
68-validate-engine-verdict-parity.mjsholds the engines to agreement on the rules both decide and prints what only one decides as two counts gated on nothing. ADR-0041 never promised equal reach; this PR does not ask for it. It asks that every coverage difference be a diff somebody reads. GT-716 criterion 3.73-validate-engine-coverage-parity.mjs, a sibling of 68 reusing itsderiveOutcomesprecedence. Both engines run on an export of the tracked tree and on a satellite the guard creates withevolith initin a temporary directory (--corepointed at the export) — because the two do not skip the same rules: here native handlers decide the DoD, compliance-baseline, manifesto and taxonomy rules whose policies read a context nobody supplied; there almost everything either engine decides is decided by one of them.engine-coverage-parity.baseline.json: per scenario, per direction, per rule, with the reason the other engine gave. Measured first — the class its report states and, for the OPA side, the facets its skip row names — then derived: an id no reachable policy emits, or a policy reading facetsopa-input-builder.tsnever emits (absent on a bare run whatever their provenance:layersis observed in nature and supplied in practice;input.repositoryis produced by nothing). A native-side class stated on the OPA side is filed asopa-gave-no-reason(the enforcer-routedHXA-01/02/04/05skip without a class of their own), not as handler debt.--writeregenerates the file for review;--jsonpublishes the counts. Runs in theTestjob after 68 (≈17 s + ≈3 s).Measured (2026-09-20)
supplied-facet-absent, 26no-policy-in-bundle, 4opa-gave-no-reasonMCP-01..03unimplemented-native,OBS-EVD-01/02needs-runtime,OBS-EVD-03needs-external-system,MCP-05handler declinedinitsatelliteBoth scenarios read the Core from an export of the tracked tree plus the compiled bundle (
afc804bc). The first CI run disagreed with the laptop that wrote the baseline: a localcoverage/directory decidedEM-Y-01/QT-01, git history decidedDRIFT-01, the CLI's bundled corpus copy predatedfacets.json, and the working-tree run resolved the 138 ADR-conformance rules' decision-record references against that copy and failed them all falsely. No coverage directories, no dists, no.git, on every machine alike.Falsifier, both outputs recorded. The criterion's own probe — removing an import from
main.rego— cannot run since GT-675: the bundle build refuses a reachable policy nobody imports. The equivalent probe is renamingOBS-EVD-03intelemetry-evidence.regoso the bundle stops deciding it: the guard went red on both scenarios naming the id (❌ repository: 1 registered entry(ies) are no longer coverage-only — opaOnly OBS-EVD-03), and green again once restored.Guard 42 classifies it INSTRUMENTED (89 guards); guard 43 observed it red on the empty fixture (65/65); 11 unit tests.
Docs and board
known-limitationsand the OPA README now say CI registers coverage differences per rule rather than merely allowing them. AC3 ticked in both catalogs with the measurement and the recorded falsifier; closure appended to the row;Last Updatedline; counters unchanged (688 / 715). AC4–AC5 stay open.Verified locally
73tests 11/11 and the guard green against its baseline; 42, 43, 01, 04, 08, 46 (fixed point), 66.Advances GT-716.
🤖 Generated with Claude Code