Skip to content

ci(e2e): pick the E2E tests a pull request needs from its diff #5679

Open

Description

@matthewgrossman

User Story

As a contributor, I want CI to pick the E2E tests and environments that matter for my change, so a PR gets a useful E2E signal quickly without running every row.

Description

A PR either runs no E2E tests (Branch Checks) or every row once someone adds test:e2e (Branch E2E): about 30 tmachine rows plus the legacy lanes. Most PRs need a few suites on a few environments. A docs change needs none. A Podman driver change needs the Podman rows, not k3s.

Proposed design: an AI-assisted selector step, following the pattern in codex-compatibility.yml and codex-security.yml.

  1. Inputs: the PR diff, plus a catalog of rows and tests generated from the CI matrix source file (ci(e2e): show which environments run which test suites at each CI stage #5678) and openshell-conformance list. Each test carries a short description and the crates and paths it covers.
  2. Output: a list of rows and, where useful, a test filter for each. Each pick has a one-line reason, and the selection is written to the job summary.
  3. Guardrails:
    • A small smoke set always runs (for example, Docker conformance smoke and lifecycle).
    • Changes to tests/, testkit, the runner, tmachine or the workflows run everything.
    • If the selector errors, times out or returns something invalid, CI runs everything.
    • main, Release Dev and Release Qualification always run their full set.
    • A label such as test:e2e-full overrides the selection.
  4. Measuring it: compare selections against the full runs on main. Count the failures on main that the selection for the PR that caused them would have skipped. Publish the miss rate before using the selection as a gate.

Open questions:

  • Start with the selection in Branch E2E, behind the label, or run a selected subset on every PR?
  • Which model and credentials, and the cost per PR?
  • Would a plain rule table (paths → rows) give most of the benefit? Build that as the fallback or baseline either way.

Context

Definition of Done

  • Branch E2E (or Branch Checks, per the open question) runs the rows and tests the selector picks. The picks and their reasons are in the job summary.
  • Every guardrail above is in place and has a test.
  • A miss-rate report against main runs covers at least two weeks before the selection becomes a required gate.
  • The testing docs describe how selection works and how to force a full run.

Activity

  1. added
    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.
    topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions