Skip to content

test(e2e): give each fixture overlay directory its own coverage build - #1203

Merged
mrbobbytables merged 4 commits into
mainfrom
quality/test-e2e-named-fixture-builds
Oct 9, 2026
Merged

mrbobbytables merged 4 commits into
mainfrom
quality/test-e2e-named-fixture-builds

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

The end-to-end fixture mechanism supported exactly two builds: the ordinary
coverage build, and one "variant" build selected by E2E_COVERAGE_VARIANT=1.
That ceiling is reached, and #1172 is the proof. ReferenceArchitectures'
absent-revision guard needs sources.architectures.revision cleared while
metrics.generatedAt stays present — and the variant build already clears
generatedAt to reach the same component's : null syncDate arm, an arm that
lives inside the guard. One build cannot hold both shapes, so covering either
arm un-covers the other. An additive overlay cannot reach it either: add
works for a keyed collection, and sources.architectures.revision is a scalar
on a singleton with nothing to sit beside.

The mechanism: a directory per build

Every tests/e2e/fixtures/data-<name>/ directory now declares one build.
npm run build:e2e:coverage compiles it into build/e2e-coverage-<name> under
base URL /e2e-coverage-<name>/ with E2E_COVERAGE_BUILD=<name>, layered on
top of tests/e2e/fixtures/data/. One docusaurus serve offers every site, one
Playwright run visits them all, and the report unions what each reached.

  • The build list is the directory listing, not a registry
    (coverageBuildNames()), so adding a build is adding a directory and nothing
    has to be kept in step with it. The loop lives in
    tests/tools/e2e-coverage-builds.mjs rather than in a package.json shell
    one-liner for exactly that reason.
  • An unknown E2E_COVERAGE_BUILD is an error, not a silent fall-back to the
    ordinary build — a typo would otherwise compile a site that serves, passes a
    smoke test, and covers nothing new.
  • tests/e2e/fixtures/data-variants/ moves to data-variant/ so the directory,
    the build name, the base URL and the output directory are all derived from one
    name. No route changes: /e2e-coverage-variant/ is unchanged, so every
    existing variant spec keeps its base path.
  • Unchanged on purpose: the overlay engine (set/add/append/setWhere,
    fail-loud on shape drift), the base fixtures, and
    tests/tools/e2e-coverage-report.mjs — the containment fold from fix(coverage): fold e2e regions whose spans cross across artifacts #1163 is
    already build-count-agnostic and the report resolves scripts from the whole
    build/ tree, so it needed no edit.
  • .github/workflows/ci.yml needed no edit either: the e2e-coverage job runs
    npm run build:e2e:coverage, which is where the loop now lives.

Cost: each build is a full Docusaurus compile, so the e2e-coverage job's wall
time grows linearly in the number of these directories. This PR takes it from two
to three. CONTRIBUTING.md and the module preamble both say so, and both say a
new build is for a shape that provably conflicts with every existing one — not
the first reach for a branch an additive overlay could cover.

The first consumer

tests/e2e/fixtures/data-no-revision/metrics.json clears
sources.architectures.revision (leaving generatedAt present, so the build
differs from the variant in exactly the field under test), and
tests/e2e/reference-architectures-no-revision.spec.js pairs /architectures
with /e2e-coverage-no-revision/architectures: the provenance line and its
<code> revision render on the first, the whole line is absent on the second.
The real route is asserted alongside, because on its own an assertion that a
page omits something passes just as well when the page is broken.

Verification

npm run build:e2e:coverage compiles all three sites. The rendered
architectures/index.html of each:

build Last synced from revision sync date
build/ yes 3ddf917 on September 28, 2026
build/e2e-coverage-variant/ yes 3ddf917 absent ('' arm)
build/e2e-coverage-no-revision/ no — —

The catalog section (aria-label="Reference architecture catalog") still renders
in the no-revision build, so this is the guard returning null rather than the
page failing. npm run test:unit is 2085/2085 and npm run check is clean at
this revision.

Honest limitation: the Playwright specs themselves were not executed here —
this environment has no usable browser (chrome: error while loading shared libraries: libglib-2.0.so.0, and no privilege to install it). Their assertions
were verified line by line against the SSR'd HTML above, and the locator pattern
is the one tests/e2e/data-variants.spec.js already proves against the same
page. CI's End-to-end coverage job is the real check.

Coverage evidence

Related Issues

Closes #1187
Closes #1172
Refs #1097 — the RadarReports empty-corpus overlay and spec it asks for already
landed (tests/e2e/fixtures/data-variant/radar-reports.json,
tests/e2e/radar-reports-variant.spec.js), so the browser does take both arms;
its remaining box is the two regions leaving the report's uncovered list, which
is gated on the region-attribution fold in #1066 / #1079 and is not something a
third build can reach. Left open deliberately.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88 requested_by=@mrbobbytables

The end-to-end fixture mechanism supported exactly two builds: the ordinary
coverage build, and one "variant" build selected by E2E_COVERAGE_VARIANT=1.
That ceiling is reached. ReferenceArchitectures' absent-revision guard needs
sources.architectures.revision cleared while metrics.generatedAt stays
present, and the variant build already clears generatedAt to reach the same
component's ': null' syncDate arm -- an arm that lives inside the guard. One
build cannot hold both shapes, so covering either arm un-covers the other.

Replace the boolean with a name: every tests/e2e/fixtures/data-<name>/
directory declares one build, compiled into build/e2e-coverage-<name> under
base URL /e2e-coverage-<name>/ with E2E_COVERAGE_BUILD=<name>. The build list
is the directory listing rather than a registry, so adding a build is adding
a directory. An unknown name is an error rather than a silent fall-back to
the ordinary build, which would otherwise serve a site that covers nothing
new. The overlay engine, the base fixtures and the coverage report's
containment fold are unchanged -- the report already resolves scripts from
the whole build/ tree, so it needed no edit.

tests/e2e/fixtures/data-variants/ moves to data-variant/ so the directory,
the build name, the base URL and the output directory are all derived from
one name. No route changes.

Then the first consumer of the third build: data-no-revision/ clears
sources.architectures.revision, and a spec pairs /architectures with
/e2e-coverage-no-revision/architectures, asserting the provenance line
renders on the first and is absent on the second.

Verified at this revision: npm run build:e2e:coverage compiles all three
sites, and the rendered architectures/index.html carries the provenance line
with a <code>3ddf917</code> revision in build/ and build/e2e-coverage-variant/
and carries neither in build/e2e-coverage-no-revision/, while the catalog
section itself still renders there. npm run test:unit is 2083/2083 and
npm run check is clean.

Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive hivecommons-hive Bot added the hold label Oct 8, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

@mrbobbytables

Copy link
Copy Markdown
Member

Reviewed against current main (post-#1201/#1204). The design composes cleanly with everything merged today: it deliberately leaves tests/tools/e2e-coverage-report.mjs untouched, and the new same-script contradiction fold from #1204 keys witnesses by script URL, so each build's chunks stay independent — no interaction. The End-to-end coverage job also cleared the ratcheted floor of 95 with all three builds. The data-variants → data-variant rename leaves only comment-level references elsewhere (e.g. tests/e2e-coverage-report.test.mjs:802), which are cosmetic.

One blocker: the required Validate repository check fails.

npm run test:unit:coverage:check exits 1:

1 source file(s) were never measured; --require-source-files requires every file
under scripts/ and src/ and tests/tools/ to be exercised.

Never measured (1):
  tests/tools/e2e-coverage-builds.mjs

The PR body verified npm run test:unit (2085/2085), but that skips the coverage gate — --require-source-files demands every file under tests/tools/ be exercised by some unit test, and the per-file floor from #1191 then requires it to reach 97% source regions.

The root cause is structural: the module calls main() unconditionally at import and spawns full Docusaurus builds, so no unit test can import it. Every other CLI tool in tests/tools/ uses the entry-point guard pattern:

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
  main();
}

(see tests/tools/coverage-report.mjs:813, e2e-coverage-report.mjs:915, e2e-coverage-run.mjs:207).

Requested revision:

  1. Guard the entry point as above and export the pieces (main plus the per-build loop), with the spawn injectable so tests don't compile real sites.
  2. Add unit tests covering: the happy path (one build command per coverageBuildNames() entry, with the right E2E_COVERAGE_BUILD/BASE_URL/out-dir), the empty-directory-list error exit, and the nonzero-spawn-status exit (including the status === null arm).
  3. Re-verify with npm run test:unit:coverage:check — the full gate, not test:unit — so the never-measured and per-file-region floors are actually exercised.

tests/tools/e2e-coverage-builds.mjs was added as a bare top-level script, so
nothing could import it and no coverage record ever named it. The Validate
repository gate runs the unit coverage reporter with --require-source-files,
which requires every file under tests/tools/ to be exercised, and failed the
run with "1 source file(s) were never measured".

Give the driver the entrypoint-guard shape the sibling tools already use
(tests/tools/e2e-coverage-run.mjs, tests/tools/e2e-coverage-report.mjs):
export buildCommand() and main(), run main() only when the file is the
process entrypoint, and return an exit code instead of calling process.exit
from inside the loop. Behaviour is unchanged -- the same npm invocation, the
same derived out-dir and BASE_URL, the same stop-at-first-failure.

tests/e2e-coverage-builds.test.mjs covers it: a spawn stub for the ordering,
the signalled-build (status null) arm and the no-fixture-directory arm, and
two CLI runs with a stub npm first on PATH so the entrypoint guard and the
exit code it forwards are exercised rather than described. The file now
reports 100.00% lines / 100.00% regions and the gate exits 0.

Signed-off-by: quality <quality@hive.kubestellar.io>
});
return {
status: result.status,
calls: spawnSync('cat', [log], { encoding: 'utf8' }).stdout ?? '',
mrbobbytables and others added 2 commits October 9, 2026 12:21
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 9, 2026
@mrbobbytables
mrbobbytables added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit 4e1ee5a Oct 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

2 participants