Skip to content

test: pin each e2e data overlay's effect on the validator that gates its file - #1269

Merged
mrbobbytables merged 1 commit into
mainfrom
quality/test-e2e-fixture-validator-conformance
Oct 10, 2026
Merged

mrbobbytables merged 1 commit into
mainfrom
quality/test-e2e-fixture-validator-conformance

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Adds tests/e2e-data-fixture-validity.test.mjs, which pins what each committed
data overlay does to the validator that gates the file it patches.

tests/e2e-data-fixture-integrity.test.mjs already establishes that an overlay
names a real data file, still applies, still changes it, and describes the arm it
covers. None of that asks what the merged document is: an overlay can apply
cleanly and still produce a document scripts/validate-<name>.mjs would reject,
and no test in the repository runs a validator over a merged fixture at all.

Several overlays are rejected on purpose — the degraded arms the variant build
exists for are only reachable from a document the collector is not supposed to
produce. So this guard records the rejections rather than forbidding them. For
each overlay it asserts the exact set of validator errors the merged document adds
relative to the unmodified data/ tree, with the reason on the record. An overlay
that starts, stops, or changes how it is rejected fails by name.

One overlay's stated contract is already wrong against the validator and nothing
reports it: tests/e2e/fixtures/data/members.json says its appended record "is
valid per MEMBERSHIP_STATUSES in scripts/validate-members.mjs", while the
merged document is rejected with
coverage-fixture-org: member-and-contributor status must carry both source roles.
The table records that, so the next edit to either side has to account for it.

Errors are compared as a delta against a baseline run of the same validator
over the real tree. The sandbox (runScriptInSandbox) mirrors scripts/ and
data/ but not static/, so asset-existence rules report against both runs alike
and cancel out; what remains is a property of the overlay.

Scope

Exactly one new file, tests/e2e-data-fixture-validity.test.mjs. No production
code, no workflow, no change to any existing test or fixture.

Deliberately not added to tests/e2e-data-fixture-integrity.test.mjs: that
file is held by open PR #1260 (every declared e2e fixture build is visited by a spec), which this PR does not touch. It also does not touch package.json or
tests/coverage-gate-thresholds.test.mjs (#1250), tests/uri-safety.test.mjs
(#1262), tests/e2e/data-fixtures.spec.js or tests/e2e/interactions.spec.js
(#1266), or tests/e2e-coverage-source-boundary-segments.test.mjs (#1252).

Verification

Run locally at 365e11f + this commit, node v26.10.0, TZ=UTC:

  • node --test tests/e2e-data-fixture-validity.test.mjs — 15 pass, 0 fail, 2.5s.
  • npm run test:unit:coverage:check — exits 0. The new file reports
    100.00 | 100.00; src files stays 100.00 | 100.00 (9015/9015 lines, 2714/2714 regions) and all files regions move 96.13 -> 96.14.
  • npx prettier --check on the new file passes.
  • The guard was shown to go red rather than always pass: setting
    data-variant/metrics.json's set.generatedAt to a valid timestamp (the
    silently-repaired-overlay case) fails
    data-variant/metrics.json affects validate-metrics.mjs exactly as recorded,
    and only that case. The fixture was restored; it is not part of this diff.

Playwright's chromium could not be launched in this environment, so no end-to-end
percentage is claimed anywhere in this PR. None is needed: the fixtures are harness
input, which the browser run never loads as source.

Related Issue

Closes #1268


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

…its file

tests/e2e-data-fixture-integrity.test.mjs establishes that every committed
overlay names a real data file, still applies, still changes it and still
describes the arm it covers. None of that asks what the merged document is:
an overlay can apply cleanly and still produce a document that
scripts/validate-<name>.mjs would reject.

Several deliberately do -- the degraded arms in the variant build are only
reachable from a document the collector is not supposed to produce -- but a
deliberate rejection and one acquired by accident look identical from CI,
because no test runs a validator over a merged fixture at all.

Records the exact delta instead: for each overlay, the set of validator
errors the merged document produces that the unmodified file does not.
Errors are compared against a baseline run over the real data/ tree so that
sandbox-only findings (static/ is not mirrored) cancel out. An overlay that
starts, stops, or changes how it is rejected now fails by name.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] no test runs a validator over a merged e2e data overlay, so a fixture's invalidity cannot be told from drift

1 participant