Skip to content

ci(e2e): show which environments run which test suites at each CI stage #5678

Open

Description

@matthewgrossman

User Story

As a maintainer reading CI results, I want one table that shows which environments run which test suites at each CI stage, so I can tell what a green PR proves and what only runs on main or at release time.

Description

Today the answer is spread across five workflow files and a hard-coded JSON list. It takes reading YAML to learn, for example, that:

  • Branch Checks (every PR push and the merge queue) runs no test environment. Its only conformance step is tasks/scripts/check-conformance-program.sh, which checks that the test lists match.

  • Branch E2E (the test:e2e label, plus test:e2e-gpu and test:e2e-kubernetes) runs every row in e2e-suites.yml:

    • conformance: 6 rows, including the external Docker driver;
    • features: 14 rows;
    • drivers/podman: 8 rows;
    • podman-e2e-integration;
    • the legacy Docker, Podman, VM and Kubernetes lanes.
  • Release Dev (every push to main) and Release Qualification run:

    • conformance (5 rows, without the external driver);
    • features;
    • the legacy docker-e2e (arm64) and vm-e2e lanes.

    They don't run drivers/podman or the legacy Kubernetes and Podman lanes, so those run only when someone adds a label to a PR.

The rows live in a rows= JSON string inside .github/workflows/e2e-suites.yml. Callers pick rows with the family and external-drivers inputs. Nothing lists them by stage.

Proposed design:

  1. Move the rows to one source file (for example tests/ci-matrix.toml or YAML). Each row names its environment, installer, suite and runner, plus a stages field (branch-e2e, release-dev, release-qualification, and later branch-checks).
  2. e2e-suites.yml reads the rows from that file and selects them by a stage input instead of family and external-drivers.
  3. A generated table (stage × environment × suite) goes in the testing docs, with a check in Branch Checks that fails if the table is stale.
  4. Legacy host lanes (VM, GPU, Kubernetes, Podman scripts) appear in the same table as rows of their own type until they're deleted, so the table is complete.

Context

Definition of Done

  • One source file defines every tmachine row and the stages it runs in. Workflows read it; no workflow hard-codes a row list.
  • The testing docs show a generated stage × environment × suite table, including the legacy host lanes while they exist.
  • Branch Checks fails when the table and the source file disagree.
  • The rows that run at each stage are unchanged by the move, or each change is listed and approved in the PR.

Activity

  1. added
    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.
    topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions