Repository navigation
test: pin each e2e data overlay's effect on the validator that gates its file - #1269
Merged
mrbobbytables merged 1 commit intoOct 10, 2026
Merged
Conversation
…its file tests/e2e-data-fixture-integrity.test.mjs establishes that every committed overlay names a real data file, still applies, still changes it and still describes the arm it covers. None of that asks what the merged document is: an overlay can apply cleanly and still produce a document that scripts/validate-<name>.mjs would reject. Several deliberately do -- the degraded arms in the variant build are only reachable from a document the collector is not supposed to produce -- but a deliberate rejection and one acquired by accident look identical from CI, because no test runs a validator over a merged fixture at all. Records the exact delta instead: for each overlay, the set of validator errors the merged document produces that the unmodified file does not. Errors are compared against a baseline run over the real data/ tree so that sandbox-only findings (static/ is not mirrored) cancel out. An overlay that starts, stops, or changes how it is rejected now fails by name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: quality <quality@hive.kubestellar.io>
Contributor
Author
|
Important Held for human review by the hive's ACMM level gate. This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the Hive will keep the |
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test Improvement
Adds
tests/e2e-data-fixture-validity.test.mjs, which pins what each committeddata overlay does to the validator that gates the file it patches.
tests/e2e-data-fixture-integrity.test.mjsalready establishes that an overlaynames a real data file, still applies, still changes it, and describes the arm it
covers. None of that asks what the merged document is: an overlay can apply
cleanly and still produce a document
scripts/validate-<name>.mjswould reject,and no test in the repository runs a validator over a merged fixture at all.
Several overlays are rejected on purpose — the degraded arms the variant build
exists for are only reachable from a document the collector is not supposed to
produce. So this guard records the rejections rather than forbidding them. For
each overlay it asserts the exact set of validator errors the merged document adds
relative to the unmodified
data/tree, with the reason on the record. An overlaythat starts, stops, or changes how it is rejected fails by name.
One overlay's stated contract is already wrong against the validator and nothing
reports it:
tests/e2e/fixtures/data/members.jsonsays its appended record "isvalid per
MEMBERSHIP_STATUSESinscripts/validate-members.mjs", while themerged document is rejected with
coverage-fixture-org: member-and-contributor status must carry both source roles.The table records that, so the next edit to either side has to account for it.
Errors are compared as a delta against a baseline run of the same validator
over the real tree. The sandbox (
runScriptInSandbox) mirrorsscripts/anddata/but notstatic/, so asset-existence rules report against both runs alikeand cancel out; what remains is a property of the overlay.
Scope
Exactly one new file,
tests/e2e-data-fixture-validity.test.mjs. No productioncode, no workflow, no change to any existing test or fixture.
Deliberately not added to
tests/e2e-data-fixture-integrity.test.mjs: thatfile is held by open PR #1260 (
every declared e2e fixture build is visited by a spec), which this PR does not touch. It also does not touchpackage.jsonortests/coverage-gate-thresholds.test.mjs(#1250),tests/uri-safety.test.mjs(#1262),
tests/e2e/data-fixtures.spec.jsortests/e2e/interactions.spec.js(#1266), or
tests/e2e-coverage-source-boundary-segments.test.mjs(#1252).Verification
Run locally at
365e11f+ this commit, node v26.10.0, TZ=UTC:node --test tests/e2e-data-fixture-validity.test.mjs— 15 pass, 0 fail, 2.5s.npm run test:unit:coverage:check— exits0. The new file reports100.00 | 100.00;src filesstays100.00 | 100.00 (9015/9015 lines, 2714/2714 regions)andall filesregions move96.13 -> 96.14.npx prettier --checkon the new file passes.data-variant/metrics.json'sset.generatedAtto a valid timestamp (thesilently-repaired-overlay case) fails
data-variant/metrics.json affects validate-metrics.mjs exactly as recorded,and only that case. The fixture was restored; it is not part of this diff.
Playwright's chromium could not be launched in this environment, so no end-to-end
percentage is claimed anywhere in this PR. None is needed: the fixtures are harness
input, which the browser run never loads as source.
Related Issue
Closes #1268
Filed by quality agent (hold-gated mode). Human review required.
— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88