Repository navigation
Open
Labels
area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.CI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Description
User Story
As a contributor, I want CI to pick the E2E tests and environments that matter for my change, so a PR gets a useful E2E signal quickly without running every row.
Description
A PR either runs no E2E tests (Branch Checks) or every row once someone adds test:e2e (Branch E2E): about 30 tmachine rows plus the legacy lanes. Most PRs need a few suites on a few environments. A docs change needs none. A Podman driver change needs the Podman rows, not k3s.
Proposed design: an AI-assisted selector step, following the pattern in codex-compatibility.yml and codex-security.yml.
- Inputs: the PR diff, plus a catalog of rows and tests generated from the CI matrix source file (ci(e2e): show which environments run which test suites at each CI stage #5678) and
openshell-conformance list. Each test carries a short description and the crates and paths it covers. - Output: a list of rows and, where useful, a test filter for each. Each pick has a one-line reason, and the selection is written to the job summary.
- Guardrails:
- A small smoke set always runs (for example, Docker conformance smoke and lifecycle).
- Changes to
tests/, testkit, the runner, tmachine or the workflows run everything. - If the selector errors, times out or returns something invalid, CI runs everything.
main, Release Dev and Release Qualification always run their full set.- A label such as
test:e2e-fulloverrides the selection.
- Measuring it: compare selections against the full runs on
main. Count the failures onmainthat the selection for the PR that caused them would have skipped. Publish the miss rate before using the selection as a gate.
Open questions:
- Start with the selection in Branch E2E, behind the label, or run a selected subset on every PR?
- Which model and credentials, and the cost per PR?
- Would a plain rule table (paths → rows) give most of the benefit? Build that as the fallback or baseline either way.
Context
- Part of test: retire the old e2e test system onto testkit #5281. Depends on the CI matrix source file from ci(e2e): show which environments run which test suites at each CI stage #5678, which provides the catalog.
- Selection must not weaken release gates. It only changes what runs before merge.
Definition of Done
- Branch E2E (or Branch Checks, per the open question) runs the rows and tests the selector picks. The picks and their reasons are in the job summary.
- Every guardrail above is in place and has a test.
- A miss-rate report against
mainruns covers at least two weeks before the selection becomes a required gate. - The testing docs describe how selection works and how to force a full run.
Metadata
Metadata
Assignees
Labels
area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.CI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Activity