Repository navigation
test(e2e): give each fixture overlay directory its own coverage build - #1203
Conversation
The end-to-end fixture mechanism supported exactly two builds: the ordinary coverage build, and one "variant" build selected by E2E_COVERAGE_VARIANT=1. That ceiling is reached. ReferenceArchitectures' absent-revision guard needs sources.architectures.revision cleared while metrics.generatedAt stays present, and the variant build already clears generatedAt to reach the same component's ': null' syncDate arm -- an arm that lives inside the guard. One build cannot hold both shapes, so covering either arm un-covers the other. Replace the boolean with a name: every tests/e2e/fixtures/data-<name>/ directory declares one build, compiled into build/e2e-coverage-<name> under base URL /e2e-coverage-<name>/ with E2E_COVERAGE_BUILD=<name>. The build list is the directory listing rather than a registry, so adding a build is adding a directory. An unknown name is an error rather than a silent fall-back to the ordinary build, which would otherwise serve a site that covers nothing new. The overlay engine, the base fixtures and the coverage report's containment fold are unchanged -- the report already resolves scripts from the whole build/ tree, so it needed no edit. tests/e2e/fixtures/data-variants/ moves to data-variant/ so the directory, the build name, the base URL and the output directory are all derived from one name. No route changes. Then the first consumer of the third build: data-no-revision/ clears sources.architectures.revision, and a spec pairs /architectures with /e2e-coverage-no-revision/architectures, asserting the provenance line renders on the first and is absent on the second. Verified at this revision: npm run build:e2e:coverage compiles all three sites, and the rendered architectures/index.html carries the provenance line with a <code>3ddf917</code> revision in build/ and build/e2e-coverage-variant/ and carries neither in build/e2e-coverage-no-revision/, while the catalog section itself still renders there. npm run test:unit is 2083/2083 and npm run check is clean. Signed-off-by: quality <quality@hive.kubestellar.io>
|
Important Held for human review by the hive's ACMM level gate. This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the Hive will keep the |
|
Reviewed against current main (post-#1201/#1204). The design composes cleanly with everything merged today: it deliberately leaves One blocker: the required Validate repository check fails.
The PR body verified The root cause is structural: the module calls if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main();
}(see Requested revision:
|
tests/tools/e2e-coverage-builds.mjs was added as a bare top-level script, so nothing could import it and no coverage record ever named it. The Validate repository gate runs the unit coverage reporter with --require-source-files, which requires every file under tests/tools/ to be exercised, and failed the run with "1 source file(s) were never measured". Give the driver the entrypoint-guard shape the sibling tools already use (tests/tools/e2e-coverage-run.mjs, tests/tools/e2e-coverage-report.mjs): export buildCommand() and main(), run main() only when the file is the process entrypoint, and return an exit code instead of calling process.exit from inside the loop. Behaviour is unchanged -- the same npm invocation, the same derived out-dir and BASE_URL, the same stop-at-first-failure. tests/e2e-coverage-builds.test.mjs covers it: a spawn stub for the ordering, the signalled-build (status null) arm and the no-fixture-directory arm, and two CLI runs with a stub npm first on PATH so the entrypoint guard and the exit code it forwards are exercised rather than described. The file now reports 100.00% lines / 100.00% regions and the gate exits 0. Signed-off-by: quality <quality@hive.kubestellar.io>
| }); | ||
| return { | ||
| status: result.status, | ||
| calls: spawnSync('cat', [log], { encoding: 'utf8' }).stdout ?? '', |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Test Improvement
The end-to-end fixture mechanism supported exactly two builds: the ordinary
coverage build, and one "variant" build selected by
E2E_COVERAGE_VARIANT=1.That ceiling is reached, and #1172 is the proof.
ReferenceArchitectures'absent-revision guard needs
sources.architectures.revisioncleared whilemetrics.generatedAtstays present — and the variant build already clearsgeneratedAtto reach the same component's: nullsyncDate arm, an arm thatlives inside the guard. One build cannot hold both shapes, so covering either
arm un-covers the other. An additive overlay cannot reach it either:
addworks for a keyed collection, and
sources.architectures.revisionis a scalaron a singleton with nothing to sit beside.
The mechanism: a directory per build
Every
tests/e2e/fixtures/data-<name>/directory now declares one build.npm run build:e2e:coveragecompiles it intobuild/e2e-coverage-<name>underbase URL
/e2e-coverage-<name>/withE2E_COVERAGE_BUILD=<name>, layered ontop of
tests/e2e/fixtures/data/. Onedocusaurus serveoffers every site, onePlaywright run visits them all, and the report unions what each reached.
(
coverageBuildNames()), so adding a build is adding a directory and nothinghas to be kept in step with it. The loop lives in
tests/tools/e2e-coverage-builds.mjsrather than in a package.json shellone-liner for exactly that reason.
E2E_COVERAGE_BUILDis an error, not a silent fall-back to theordinary build — a typo would otherwise compile a site that serves, passes a
smoke test, and covers nothing new.
tests/e2e/fixtures/data-variants/moves todata-variant/so the directory,the build name, the base URL and the output directory are all derived from one
name. No route changes:
/e2e-coverage-variant/is unchanged, so everyexisting variant spec keeps its base path.
set/add/append/setWhere,fail-loud on shape drift), the base fixtures, and
tests/tools/e2e-coverage-report.mjs— the containment fold from fix(coverage): fold e2e regions whose spans cross across artifacts #1163 isalready build-count-agnostic and the report resolves scripts from the whole
build/tree, so it needed no edit..github/workflows/ci.ymlneeded no edit either: the e2e-coverage job runsnpm run build:e2e:coverage, which is where the loop now lives.Cost: each build is a full Docusaurus compile, so the e2e-coverage job's wall
time grows linearly in the number of these directories. This PR takes it from two
to three.
CONTRIBUTING.mdand the module preamble both say so, and both say anew build is for a shape that provably conflicts with every existing one — not
the first reach for a branch an additive overlay could cover.
The first consumer
tests/e2e/fixtures/data-no-revision/metrics.jsonclearssources.architectures.revision(leavinggeneratedAtpresent, so the builddiffers from the variant in exactly the field under test), and
tests/e2e/reference-architectures-no-revision.spec.jspairs/architectureswith
/e2e-coverage-no-revision/architectures: the provenance line and its<code>revision render on the first, the whole line is absent on the second.The real route is asserted alongside, because on its own an assertion that a
page omits something passes just as well when the page is broken.
Verification
npm run build:e2e:coveragecompiles all three sites. The renderedarchitectures/index.htmlof each:Last synced frombuild/3ddf917on September 28, 2026build/e2e-coverage-variant/3ddf917''arm)build/e2e-coverage-no-revision/The catalog section (
aria-label="Reference architecture catalog") still rendersin the no-revision build, so this is the guard returning
nullrather than thepage failing.
npm run test:unitis 2085/2085 andnpm run checkis clean atthis revision.
Honest limitation: the Playwright specs themselves were not executed here —
this environment has no usable browser (
chrome: error while loading shared libraries: libglib-2.0.so.0, and no privilege to install it). Their assertionswere verified line by line against the SSR'd HTML above, and the locator pattern
is the one
tests/e2e/data-variants.spec.jsalready proves against the samepage. CI's
End-to-end coveragejob is the real check.Coverage evidence
src/components/ReferenceArchitectures/index.jsis at100% lines and regions in
npm run test:unit:coverage.e2e-coverageid11492367935, run37641752914, job
End-to-end coverage, head03cfcfe:report.txtlists region 13 uncovered.Covered by unit but not e2e, so priority 2 under the coverage-evidence
rules. (Region 34 in that same line is the region-attribution issue tracked in
[quality] e2e region union does not fold single-line '||' fallback arms across the real and variant builds, so correct variant tests lower the gate #1066 / [quality] e2e region union still counts three demonstrably-executed single-line arms uncovered; their covered twin differs in START column, which #1066's proposed key would miss #1079 and is not what this changes.)
Related Issues
Closes #1187
Closes #1172
Refs #1097 — the RadarReports empty-corpus overlay and spec it asks for already
landed (
tests/e2e/fixtures/data-variant/radar-reports.json,tests/e2e/radar-reports-variant.spec.js), so the browser does take both arms;its remaining box is the two regions leaving the report's uncovered list, which
is gated on the region-attribution fold in #1066 / #1079 and is not something a
third build can reach. Left open deliberately.
— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88 requested_by=@mrbobbytables