From 254e5cf474c74f2058bd1a11e8442c2aa61a81f1 Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Fri, 18 Sep 2026 15:05:21 +0200 Subject: [PATCH 1/9] docs(testing): define nix and tmachine target state Signed-off-by: Evan Lezar --- CI.md | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 94 insertions(+), 2 deletions(-) diff --git a/CI.md b/CI.md index 6dac0a4bf5..a5b20a7796 100644 --- a/CI.md +++ b/CI.md @@ -1,11 +1,103 @@ # CI -This document describes how OpenShell's continuous integration works for pull requests, with a focus on what contributors need to do to get their PR tested. +This document describes the current OpenShell continuous integration system, +with a focus on what contributors need to do to get a pull request tested. It +documents implemented workflow behavior rather than the complete desired test +architecture. -For local test commands see [TESTING.md](TESTING.md). For PR conventions see [CONTRIBUTING.md](CONTRIBUTING.md). +For the target testing model and canonical local entry points, see +[TESTING.md](TESTING.md). For pull request conventions, see +[CONTRIBUTING.md](CONTRIBUTING.md). ## Overview +CI implements the testing layers defined in `TESTING.md` through separate +workflow families: + +| Testing responsibility | Current CI implementation | +|---|---| +| Static, build, unit, component, and SDK checks | `Branch Checks` across Linux x86_64, Linux ARM64, and macOS ARM64, with additional language-specific jobs | +| Runtime and cross-component validation | Label-selected `Branch E2E Checks` for Docker, Podman, Kubernetes, VM, GPU, MCP, Python, and managed or standalone drivers | +| Nix and `tmachine` validation | Release Dev builds exact-revision artifacts and test archives, then runs CLI conformance in disposable Ubuntu/Docker and Fedora/Podman guests | +| Windows compatibility | Opt-in Windows MSVC x64 and ARM64 jobs; main and manual runs also build release binaries | +| Integrated revision validation | Required statuses on merge-group SHAs before `main` advances | +| Published artifact validation | Release workflows and post-publish release canaries | +| Security analysis | Required dependency and deployment gates plus informational reports described below | + +This is not yet the full target state. The project is migrating integration and +Linux installation testing from `mise` tasks and workflow-local shell harnesses +to Nix-built artifacts and `tmachine` wherever the runner can represent the +environment. Today that path is limited to release conformance. `TESTING.md` +defines the desired gates, matrices, migration phases, and deferred decisions. +This document should describe a migration step as complete only after the +workflow enforces it. + +### Nix and tmachine migration + +The desired CI path builds candidate artifacts through the flake, installs them +into a scenario declared in `tests/config.nix`, and runs a reusable testsuite +from `tests/suites` through `nix run .#tmachine`. GitHub Actions explicitly +lists scenario and testsuite pairs so it can run them in parallel, but it must +not recreate machine setup or installation behavior in workflow steps. + +`.github/workflows/conformance.yml` is the first implementation of this model. +It downloads exact-revision binaries and OCI image artifacts, uses +`nix run .#build-artifacts-test-archives` to package the test suite, and fans out +`nix run .#tmachine -- test ` on KVM-enabled runners. The +workflow caches prepared disks, while `tmachine` gives each test a fresh writable +overlay. Release Dev currently calls the reusable workflow for these pairs: + +| Scenario | Testsuite | +|---|---| +| `ubuntu-docker-rootful` | `conformance` | +| `fedora-podman-rootful` | `conformance` | +| `fedora-podman-rootless` | `conformance` | + +The desired green-change gate has these parts: + +- always-required Rust formatting, linting, unit tests, conditional-compilation + checks, dependency policy, and repository security checks in Nix environments; +- Rust source validation on Linux x86_64, Linux ARM64, and macOS ARM64; +- conditional SDK validation, with shared protobuf changes selecting every + affected SDK; +- conditional package validation, with transitive inputs selecting every + affected package format; +- conformance on Fedora/rootful Podman, Fedora/rootless Podman, Ubuntu/Docker, + one Kubernetes environment, and the VM driver; +- every feature-specific and driver-specific suite on at least one + representative compatible environment; and +- one representative external-driver compatibility scenario. + +GPU and Windows suites remain opt-in through labels. The Kubernetes distribution +and representative external-driver environment are not yet selected. + +Migration should proceed in this order: + +1. Express the candidate build or archive as a Nix output in + `tests/artifacts.nix`. +2. Add reusable assertions under `tests/suites` and installation behavior under + `tests/ansible`. +3. Add or extend a machine scenario in `tests/config.nix`; do not duplicate its + setup in a workflow matrix or shell script. +4. Verify that the `tmachine` path provides the same behavior coverage as the + legacy path. Parallel execution is optional rather than a migration + requirement. +5. Make the `tmachine` pair the required PR and merge-queue path, then remove the + redundant workflow steps, shell harness, and `mise` task. + +Release validation should install candidate RPM, DEB, Snap, and Helm artifacts +through tmachine and run conformance before publication. It should reuse the +same representative environment model without creating a full cross-product. +The existing native Homebrew path remains in place while candidate-artifact +Homebrew validation is deferred. + +Direct workflows remain appropriate for native Windows or macOS behavior, GPU +hardware, external Kubernetes topologies, and virtualization combinations that +`tmachine` cannot faithfully expose. Such jobs should consume Nix-built +exact-revision artifacts where possible, publish the same evidence, and state +the capability that prevents migration. Adding a new host-managed system test +requires that justification. + PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal `pull-request/` branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR. When a PR is not mirrored automatically, anyone with Write, Maintain, or Admin From 4d15f7b1b5a7e2840c48049cb8380af8a38b57c1 Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Fri, 18 Sep 2026 16:51:43 +0200 Subject: [PATCH 2/9] docs(testing): clarify current test paths Signed-off-by: Evan Lezar --- CI.md | 59 ++++++++--------------------------------------------------- 1 file changed, 8 insertions(+), 51 deletions(-) diff --git a/CI.md b/CI.md index a5b20a7796..13315de9e8 100644 --- a/CI.md +++ b/CI.md @@ -32,13 +32,7 @@ defines the desired gates, matrices, migration phases, and deferred decisions. This document should describe a migration step as complete only after the workflow enforces it. -### Nix and tmachine migration - -The desired CI path builds candidate artifacts through the flake, installs them -into a scenario declared in `tests/config.nix`, and runs a reusable testsuite -from `tests/suites` through `nix run .#tmachine`. GitHub Actions explicitly -lists scenario and testsuite pairs so it can run them in parallel, but it must -not recreate machine setup or installation behavior in workflow steps. +### Current tmachine implementation `.github/workflows/conformance.yml` is the first implementation of this model. It downloads exact-revision binaries and OCI image artifacts, uses @@ -53,50 +47,13 @@ overlay. Release Dev currently calls the reusable workflow for these pairs: | `fedora-podman-rootful` | `conformance` | | `fedora-podman-rootless` | `conformance` | -The desired green-change gate has these parts: - -- always-required Rust formatting, linting, unit tests, conditional-compilation - checks, dependency policy, and repository security checks in Nix environments; -- Rust source validation on Linux x86_64, Linux ARM64, and macOS ARM64; -- conditional SDK validation, with shared protobuf changes selecting every - affected SDK; -- conditional package validation, with transitive inputs selecting every - affected package format; -- conformance on Fedora/rootful Podman, Fedora/rootless Podman, Ubuntu/Docker, - one Kubernetes environment, and the VM driver; -- every feature-specific and driver-specific suite on at least one - representative compatible environment; and -- one representative external-driver compatibility scenario. - -GPU and Windows suites remain opt-in through labels. The Kubernetes distribution -and representative external-driver environment are not yet selected. - -Migration should proceed in this order: - -1. Express the candidate build or archive as a Nix output in - `tests/artifacts.nix`. -2. Add reusable assertions under `tests/suites` and installation behavior under - `tests/ansible`. -3. Add or extend a machine scenario in `tests/config.nix`; do not duplicate its - setup in a workflow matrix or shell script. -4. Verify that the `tmachine` path provides the same behavior coverage as the - legacy path. Parallel execution is optional rather than a migration - requirement. -5. Make the `tmachine` pair the required PR and merge-queue path, then remove the - redundant workflow steps, shell harness, and `mise` task. - -Release validation should install candidate RPM, DEB, Snap, and Helm artifacts -through tmachine and run conformance before publication. It should reuse the -same representative environment model without creating a full cross-product. -The existing native Homebrew path remains in place while candidate-artifact -Homebrew validation is deferred. - -Direct workflows remain appropriate for native Windows or macOS behavior, GPU -hardware, external Kubernetes topologies, and virtualization combinations that -`tmachine` cannot faithfully expose. Such jobs should consume Nix-built -exact-revision artifacts where possible, publish the same evidence, and state -the capability that prevents migration. Adding a new host-managed system test -requires that justification. +This workflow currently runs from Release Dev. It is not a required pull-request +or merge-queue gate. Branch E2E, GPU, Kubernetes, VM, Windows, and release-canary +workflows continue to provide the coverage that has not migrated to tmachine. + +The desired merge matrix, release-validation model, and migration sequence are +defined in [TESTING.md](TESTING.md). They describe target behavior and do not +become current CI behavior until the corresponding workflows implement them. PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal `pull-request/` branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR. From 3fb7687d5923f1f393a4dc7652fd6bd3255c0a0c Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Thu, 1 Oct 2026 14:18:46 +0200 Subject: [PATCH 3/9] docs(rfc): propose OpenShell testing strategy Signed-off-by: Evan Lezar --- CI.md | 53 +--- rfc/0016-testing-strategy/README.md | 359 ++++++++++++++++++++++++++++ 2 files changed, 361 insertions(+), 51 deletions(-) create mode 100644 rfc/0016-testing-strategy/README.md diff --git a/CI.md b/CI.md index 13315de9e8..6dac0a4bf5 100644 --- a/CI.md +++ b/CI.md @@ -1,60 +1,11 @@ # CI -This document describes the current OpenShell continuous integration system, -with a focus on what contributors need to do to get a pull request tested. It -documents implemented workflow behavior rather than the complete desired test -architecture. +This document describes how OpenShell's continuous integration works for pull requests, with a focus on what contributors need to do to get their PR tested. -For the target testing model and canonical local entry points, see -[TESTING.md](TESTING.md). For pull request conventions, see -[CONTRIBUTING.md](CONTRIBUTING.md). +For local test commands see [TESTING.md](TESTING.md). For PR conventions see [CONTRIBUTING.md](CONTRIBUTING.md). ## Overview -CI implements the testing layers defined in `TESTING.md` through separate -workflow families: - -| Testing responsibility | Current CI implementation | -|---|---| -| Static, build, unit, component, and SDK checks | `Branch Checks` across Linux x86_64, Linux ARM64, and macOS ARM64, with additional language-specific jobs | -| Runtime and cross-component validation | Label-selected `Branch E2E Checks` for Docker, Podman, Kubernetes, VM, GPU, MCP, Python, and managed or standalone drivers | -| Nix and `tmachine` validation | Release Dev builds exact-revision artifacts and test archives, then runs CLI conformance in disposable Ubuntu/Docker and Fedora/Podman guests | -| Windows compatibility | Opt-in Windows MSVC x64 and ARM64 jobs; main and manual runs also build release binaries | -| Integrated revision validation | Required statuses on merge-group SHAs before `main` advances | -| Published artifact validation | Release workflows and post-publish release canaries | -| Security analysis | Required dependency and deployment gates plus informational reports described below | - -This is not yet the full target state. The project is migrating integration and -Linux installation testing from `mise` tasks and workflow-local shell harnesses -to Nix-built artifacts and `tmachine` wherever the runner can represent the -environment. Today that path is limited to release conformance. `TESTING.md` -defines the desired gates, matrices, migration phases, and deferred decisions. -This document should describe a migration step as complete only after the -workflow enforces it. - -### Current tmachine implementation - -`.github/workflows/conformance.yml` is the first implementation of this model. -It downloads exact-revision binaries and OCI image artifacts, uses -`nix run .#build-artifacts-test-archives` to package the test suite, and fans out -`nix run .#tmachine -- test ` on KVM-enabled runners. The -workflow caches prepared disks, while `tmachine` gives each test a fresh writable -overlay. Release Dev currently calls the reusable workflow for these pairs: - -| Scenario | Testsuite | -|---|---| -| `ubuntu-docker-rootful` | `conformance` | -| `fedora-podman-rootful` | `conformance` | -| `fedora-podman-rootless` | `conformance` | - -This workflow currently runs from Release Dev. It is not a required pull-request -or merge-queue gate. Branch E2E, GPU, Kubernetes, VM, Windows, and release-canary -workflows continue to provide the coverage that has not migrated to tmachine. - -The desired merge matrix, release-validation model, and migration sequence are -defined in [TESTING.md](TESTING.md). They describe target behavior and do not -become current CI behavior until the corresponding workflows implement them. - PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal `pull-request/` branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR. When a PR is not mirrored automatically, anyone with Write, Maintain, or Admin diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md new file mode 100644 index 0000000000..3bec5593b9 --- /dev/null +++ b/rfc/0016-testing-strategy/README.md @@ -0,0 +1,359 @@ +--- +authors: + - "@elezar" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/3954 + - https://github.com/NVIDIA/OpenShell/pull/3460 +--- + +# RFC 0016 - OpenShell testing strategy + +## Summary + +Define a common testing strategy for OpenShell that separates the behavioral +contract being tested from the environment, installation, execution mechanism, +and CI policy used to validate it. Contributors should be able to place a test, +run it locally, and understand what its result establishes without consulting +several competing descriptions of conformance. + +Use Nix for reproducible build inputs and test artifacts, and tmachine for +integration and Linux installation environments it can represent. Keep general +conformance focused on public behavior exercised through the OpenShell CLI. +Feature, driver, disruption, load/scale, and SDK coverage retain explicit +boundaries. This is a proposal for incremental implementation, not a description +of gates already enforced by CI. + +## Motivation + +OpenShell tests have accumulated across crate-local tests, E2E binaries, +driver wrappers, installed-artifact suites, and CI workflows. A single source +binary can mix a portable behavioral contract, native runtime inspection, an +external service integration, and performance measurements. This makes both +ownership and coverage difficult to assess. + +Documentation has the same problem. TESTING.md is being asked to describe local +commands, a future execution model, conformance admission rules, migration work, +and release gates. Suite READMEs and implementation PRs independently define +parts of that strategy. Contributors cannot tell which document owns a decision +or whether a statement describes current behavior or a target state. + +The migration tracked by #3712 makes this concrete: the tmachine e2e-podman +suite supplies interim coverage, but the intended outcome is to move the source +tests into suites that own their behavioral intent and then remove the catch-all. +The strategy needs to support that migration without making its source inventory +the definition of OpenShell conformance. + +## Non-goals + +- Implement the proposed framework, capability API, or CI matrices in this PR. +- Finalize the destination of every existing E2E assertion; migration issues + retain that analysis and their task lists. +- Introduce named conformance profiles, hierarchical capability inference, or + generic capability parameters before concrete tests require them. +- Create a dedicated workload fixture image or dependency-declaration system + for scenarios initially. +- Establish a certification program, review board, or reporting service. +- Replace the SDK compatibility proposal in #3238 or standardize language SDK + APIs through the CLI runner. + +## Proposal + +### 1. Separate behavioral ownership from execution dimensions + +Each assertion has an intended contract and an appropriate owner. A binary can +be split when its assertions belong to different families. + +| Family | Contract and admission boundary | +| --- | --- | +| Unit and component integration | Internal logic, configuration selection, translation, and implementation mechanics; use the lowest effective layer. | +| General conformance | Public behavior demonstrated on at least two different drivers, with effective API capabilities for non-universal support. | +| Feature-specific | Public feature behavior requiring configured external integration, or currently implemented on only one driver; one representative driver suffices. | +| Driver-specific | Driver configuration, runtime and host integration, and implementation contracts; exercise applicable environments for that driver. | +| Disruption conformance | Portable continuity or recovery assertions using environment-specific disruption actuators; one working actuator suffices initially. | +| Load/scale | Throughput, latency, concurrency, saturation, and scaling measurements, reported separately from behavioral correctness. | +| SDK conformance | Behavior and compatibility of SDK implementations through their native interfaces, specified separately in #3238. | + +Security portability is a workstream spanning these families. Define a separate +security family only if multiple tests need common specialized infrastructure. +For example, bypass prevention is an intended universal guarantee, but the +existing seccomp-oriented probe needs separate analysis before its portable +migration. Core-dump protection similarly needs a portable contract and probe. + +The execution matrix has independent axes: + +| Axis | Examples | +| --- | --- | +| Platform | Host and workload OS and architecture | +| Driver | Docker, Podman, Kubernetes, VM, MXC | +| Environment | Rootful/rootless Podman, local runtime, Kubernetes cluster | +| Configuration | Gateway settings, in-process/external driver wiring | +| Installation | Candidate binaries/images, RPM, DEB, Snap, Helm, Homebrew | +| Suite | General conformance, a feature suite, a driver suite, disruption | + +Rootful and rootless Podman are environments of one driver. They do not satisfy +the two-driver admission rule. Driver tests should preserve their behavioral +contract across those environments. Matrix entries should target meaningful +risks without requiring the full cross-product of every axis. + +### 2. Define general conformance through observable contracts + +Conformance tests exercise a behavioral contract against an already configured +OpenShell gateway. They remain CLI-based until direct API access is strictly +required. Assertions should use observable outcomes and structured output; +incidental presentation text is not a conformance contract. Native runtime +inspection may provide best-effort failure diagnostics, but must not determine +whether general conformance passed. + +Tests must not update gateway startup configuration, driver flags, deployment +manifests, or gateway.toml. They may create and mutate public API-managed state, +including sandboxes, policies, providers, workspaces, and settings. Use unique +resource names and cleanup. Avoid conflicting global-state mutations within one +run. Restoring a pre-existing global setting is recommended, not a hard +conformance requirement; the global effects still need to be clear to callers. + +A short-lived service fully owned by a test or harness can be a conformance +fixture. An external service requiring gateway configuration is a feature-suite +prerequisite. Public capabilities must not become an inventory of third-party +services, credentials, or fixture configuration. Missing prerequisites for a +selected feature suite are setup errors. + +Start with existing general-purpose workload images and a small common toolset. +The harness supplies an appropriate image for each target. Missing fixture +tooling is an infrastructure error. Defer a dedicated image until concrete +incompatibilities justify it. A default-image scenario deliberately omits the +image and checks that a usable sandbox results; precise configuration selection +and registry resolution belong in lower-level or integration coverage. + +Two-driver validation is the minimum evidence for admitting general conformance, +not a limit on subsequent execution. Run it against every supported driver for +which infrastructure exists. Docker and Podman are acceptable initial targets. +Unavailable infrastructure is a coverage gap. A successful test against one +configuration does not establish success for all deployments of its driver. + +Disruption conformance has a separate execution family. Portable assertions +describe continuity or recovery; actuators induce gateway restart, runtime loss, +or another disruption. An absent actuator prevents selection for that environment; +a broken configured actuator is an infrastructure error. One actuator is enough +to begin migrating a portable disruption contract. + +Timing limits normally bound readiness, polling, and hangs. A performance +threshold belongs in load/scale unless the deadline is itself a specified public +contract. + +### 3. Distinguish capabilities, scenario selection, and results + +Use flat, namespaced boolean capability identifiers describing precise effective +product behavior. Namespaces can organize names without implying parent/child +support. Parameters and named profiles are deferred. A driver name, a fixture +prerequisite, and a selectable test-family prefix are not product capabilities. +In particular, the leaf selection introduced by #3768 is separate from API +capability reporting. + +The gateway reports support for its running configuration. Obtain the snapshot +through machine-readable CLI output backed by the public API. Reporting is a +hard requirement: failure to discover capabilities aborts the conformance run. +Each scenario encodes its expected behavior and whether that behavior is +mandatory. A missing mandatory capability fails; a missing optional capability +is unsupported. An advertised capability must satisfy its contract, including +when support for that capability is optional. + +Non-universal behavior supported by at least two drivers is a signal to extend +the public capability API. Prefer adding the capability and its concrete test +in the same PR. If an API extension is infeasible, use feature-specific or +driver-specific coverage. Both drivers used for general-conformance admission +must advertise the capability and pass its assertions. + +| Outcome | Meaning | +| --- | --- | +| Passed | The selected behavioral assertions held. | +| Failed | A mandatory capability is missing or the exercised contract was violated. | +| Unsupported | An optional capability was not advertised. | +| Skipped | A scenario was deliberately excluded from this invocation. | +| Error | Infrastructure, fixture, runner, or actuator failure prevented evaluation. | + +Keep these distinctions as lightweight as practical. An advisory CI job can +report behavioral failures without blocking a merge; that job policy must not +turn those failures into passing scenario results. Mock-MXC execution, for +example, needs to identify the mock target and cannot qualify the production +driver. A focused or incomplete invocation must not imply complete coverage. + +### 4. Keep execution reusable and select CI gates explicitly + +Nix pins source-check dependencies and builds candidate artifacts and test +archives. Tmachine provisions supported environments, applies installation and +configuration, and runs a selected suite. Tests should consume an already +configured target; general conformance must not require tmachine to have created +it. Native Windows/macOS, GPU, or other environments tmachine cannot represent +may use a suitable external provisioner with the same behavioral contracts. + +Keep environment definitions in tests/config.nix, artifact construction in +tests/artifacts.nix, and provisioning/installation in tests/ansible. Workflow +matrices select declared environment/suite combinations rather than duplicating +their setup logic. Existing mise commands remain usable migration entry points +until replacements provide equivalent coverage. + +After #3866 removes the standalone conformance executable, consolidate +crates/openshell-conformance under tests/suites/conformance. Preserve a shared +library for scenarios and execution helpers alongside the Cargo test entry +points that exercise the openshell CLI. Update workspace membership, lockfiles, +archive inputs, and CI references together. Extract utilities shared by feature +or disruption suites only when actual consumers require them. + +Source checks cover formatting, lint, unit tests, feature/platform compilation, +and existing dependency/security checks. The earlier target-state proposal used +Linux x86_64, Linux ARM64, and macOS ARM64 as the initial source matrix. SDK-local +changes run the relevant SDK checks; protobuf changes run all affected SDK +checks. Packaging selection includes transitive inputs and uses broader coverage +when impact cannot be determined safely. + +General conformance should run across available drivers, with expensive or +platform-constrained environments using appropriate dedicated or scheduled +workflows. Feature suites run where prerequisites exist; driver suites exercise +applicable environments; disruption suites require actuators. The exact required +merge checks, schedules, and release gates remain review decisions. The earlier +five-environment matrix is an initial candidate, not a permanent ceiling. + +Release validation installs exact candidate artifacts and then runs applicable +suites before publication. Representative RPM/Fedora, DEB/Ubuntu, Snap/Ubuntu, +and Helm/Kubernetes paths can reuse conformance. Native Homebrew qualification +uses its appropriate platform. Passing behavior after installation does not by +itself prove package ownership, service setup, upgrade, or uninstall contracts; +installation-specific assertions must cover those separately. Upgrade and +version-skew obligations remain explicit follow-up decisions. + +Start report attribution with the OpenShell Git SHA and the tmachine +configuration used for a run. Preserve enough configuration identity to recover +the effective configuration. Other provisioners supply equivalent target +identification. This is a placeholder, not a complete report schema; richer +capability, scenario, and artifact reporting can follow a concrete need. + +### 5. Give each document one responsibility + +| Document | Responsibility | +| --- | --- | +| This RFC and its PR | Strategy discussion, rationale, alternatives, unresolved decisions | +| TESTING.md | Contributor entry point, concise routing rules, currently working local commands | +| Proposed tests/CONFORMANCE.md | Conformance contracts, admission, capabilities, result semantics, and agreed policy | +| tests/suites/conformance/README.md | Authoring and running the suite, fixtures, library extension points, selection, cleanup | +| Other suite READMEs | Their fixtures, prerequisites, commands, and specialized contracts | +| CI.md | Implemented workflow triggers, matrices, gates, and artifact investigation | +| AGENTS.md and contributor skills | Short routing instructions linking to canonical guidance | +| #3954 and focused issues | Strategy follow-ups, migration tasks, dependencies, and completion tracking | + +The suite README owns the relocated library's contributor guidance. A separate +crate README need not duplicate it. SDK specifications remain owned by the SDK +proposal while sharing terminology and provisioning boundaries. Accepted policy +is published in the living guides as implementation lands; current references +must not describe proposed gates as already enforced. + +## Implementation plan + +1. Discuss this RFC through existing PR #3460 and track follow-ups in #3954. + Resolve policy questions independently from per-test migration details. +2. Reconcile #3768, #3864, and #3866 for leaf selection, obsolete parity removal, + and Cargo execution. Relocate the shared library in a focused follow-up, + coordinating the same files with Windows work in #3769. +3. Add effective capability reporting and stable CLI discovery. Migrations can + be developed in parallel, but qualification under this proposal requires the + mandatory reporting contract. Add further capability/test pairs together. +4. Rescope #3945 around the documentation ownership above. Align SDK RFC #3238 + with the shared strategy. Assess #2182 and #2873 for supersession and preserve + unique assertions; this RFC does not automatically close those PRs. +5. Migrate coherent behavioral intent in parallel by destination family. One PR + can add destination coverage and remove covered source tests. Delete empty + binaries and rename or split residual binaries. Require two-driver evidence + for general conformance, one representative driver for feature tests, and + one working actuator for disruption. Keep implementation tests at the lowest + effective layer. +6. Use #3712 for the Podman migration inventory, including security portability + investigations and load/scale routing. Retire the e2e-podman feature, follow-up + exclusion list, and temporary tmachine suite when all source intent is covered + or explicitly retired. Do not introduce inventory-check machinery unless + continued source churn demonstrates a need. +7. Expand destination-suite CI and candidate-installation validation, documenting + actual gates and coverage gaps as they become operational. Preserve existing + coverage until its intent has a validated destination; no mandatory overlap + period is needed solely for migration. + +## Risks + +- Optional capability reporting can hide regressions if an implementation simply + stops advertising support. Mandatory contracts remain test-owned; policy for + reviewing optional-support removal needs an explicit decision. +- Two Linux container drivers can pass a scenario containing OS assumptions. + Admission evidence is a starting point; broader platform validation and probe + review remain necessary. +- Global API state mutation can affect other workloads or test processes. + Document side effects, coordinate conflicts within a run, and recommend + restoration. Cross-process coordination is not designed here. +- Nix, provisioning, and test-archive changes create migration cost and merge + conflicts. Keep scenario contracts independent of provisioners and sequence + shared library changes before many concurrent migrations. +- Declaring a contract mandatory can turn an existing product limitation into a + visible failure. Decide the product contract explicitly rather than weakening + a test or treating setup failure as unsupported. + +## Alternatives + +### Continue expanding TESTING.md and suite READMEs independently + +This is the smallest immediate documentation change, but leaves strategy, +commands, and future gates interleaved. A single RFC and focused living +references give reviewers a place to resolve disagreements before publishing +authoritative guidance. + +### Introduce separate API and CLI conformance frameworks immediately + +The older #2182 and #2873 proposals offer direct API coverage, but add another +runner and overlapping scenarios before a CLI limitation requires them. Keep +general conformance CLI-based initially; SDK interface testing remains a +separate justified consumer under #3238. + +### Require every scenario on every configuration + +This gives a uniform baseline but excludes useful portable behavior that some +drivers do not implement. Mandatory scenarios plus precise optional capabilities +allow useful coverage while requiring advertised behavior to pass. + +### Establish profiles and a full certification process first + +Profiles, a versioned certification program, and formal promotion gates could +clarify compatibility guarantees, but would add policy and infrastructure before +the existing tests have been classified. Keep these decisions open and evolve +them from concrete contracts and consumers. + +## Prior art + +- [Kubernetes conformance testing](https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/conformance-tests.md) + separates stable behavioral requirements from execution infrastructure and + defines promotion, reliability, version compatibility, and normative test + descriptions. Those are useful policy questions for OpenShell; its mandatory + GA baseline and governance process are not adopted wholesale here. +- #2925, #3768, and #3866 establish reusable OpenShell CLI scenarios, independent + selection, and installed Cargo execution. They provide implementation pieces + without independently defining the entire strategy. +- #3107 introduced sandbox-continuity testing and #3342 reverted that + infrastructure. Separating portable assertions from disruption actuators + remains useful without committing to the former implementation. +- SDK RFC #3238 treats SDK-native behavior as its own compatibility surface. + Shared gateway provisioning does not require identical language runners. + +## Open questions + +- What constitutes a complete conformance claim, and how are suite versions + matched to gateway/CLI releases? Which version-skew guarantees are required? +- What maturity and reliability evidence is required beyond two-driver success? + How are promotion, material changes, and demotion reviewed without imposing + an unnecessary new governance process? +- What minimal normative description and stable identity must each scenario + carry, and how is removal of previously advertised support reviewed? +- Should general conformance explicitly require offline execution after artifact + provisioning, and what host privileges and fixture reachability assumptions + are allowed for externally provisioned gateways? +- Which source and integration jobs are required for merges and release + promotion? How should Windows, GPU, disruption, and load/scale runs be scheduled? +- Where should disruption and load/scale suites live, and what is the smallest + actuator interface needed by the first migration? +- Which fields beyond SHA and configuration become necessary when the first + report consumer needs completeness or cross-version comparisons? From d2c06441ca363384071e4ed136ffe80bbf0100b2 Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Thu, 1 Oct 2026 15:07:07 +0200 Subject: [PATCH 4/9] docs(rfc): clarify testing model and suite organisation Signed-off-by: Evan Lezar --- rfc/0016-testing-strategy/README.md | 214 +++++++++++++++++++--------- 1 file changed, 148 insertions(+), 66 deletions(-) diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md index 3bec5593b9..5b8d24f1fc 100644 --- a/rfc/0016-testing-strategy/README.md +++ b/rfc/0016-testing-strategy/README.md @@ -12,31 +12,34 @@ links: ## Summary Define a common testing strategy for OpenShell that separates the behavioral -contract being tested from the environment, installation, execution mechanism, -and CI policy used to validate it. Contributors should be able to place a test, -run it locally, and understand what its result establishes without consulting -several competing descriptions of conformance. +contract being tested from the environment, installation, client interface, +and CI policy used to validate it. Reuse tests of public behavior across +configured targets, while keeping implementation-specific checks and performance +measurements distinct. A passing test establishes the same contract regardless +of how its target was prepared. Use Nix for reproducible build inputs and test artifacts, and tmachine for -integration and Linux installation environments it can represent. Keep general -conformance focused on public behavior exercised through the OpenShell CLI. -Feature, driver, disruption, load/scale, and SDK coverage retain explicit -boundaries. This is a proposal for incremental implementation, not a description -of gates already enforced by CI. +integration and Linux installation environments it can represent. Keep test +contracts independent of those tools. Initially exercise general conformance +through the OpenShell CLI; using an SDK does not create a different test category. +This is a proposal for incremental implementation, not a description of gates +already enforced by CI. ## Motivation -OpenShell tests have accumulated across crate-local tests, E2E binaries, -driver wrappers, installed-artifact suites, and CI workflows. A single source -binary can mix a portable behavioral contract, native runtime inspection, an -external service integration, and performance measurements. This makes both -ownership and coverage difficult to assess. +How do we establish that OpenShell behaves as promised across supported +configurations without duplicating behavioral tests for every driver and +environment? Tests tied to particular environments make it difficult to +distinguish product requirements from implementation details, reuse coverage, +or determine what a passing suite establishes. -Documentation has the same problem. TESTING.md is being asked to describe local -commands, a future execution model, conformance admission rules, migration work, -and release gates. Suite READMEs and implementation PRs independently define -parts of that strategy. Contributors cannot tell which document owns a decision -or whether a statement describes current behavior or a target state. +OpenShell tests have accumulated across crate-local tests, E2E binaries, +driver wrappers, installed-artifact suites, and CI workflows. A single binary +can mix public behavior, native runtime inspection, an external integration, +and performance measurements. Without a shared model, contributors must either +duplicate this coverage for new targets or carry assumptions that do not apply +to them. Reviewers cannot reliably distinguish missing product support from +missing test infrastructure. The migration tracked by #3712 makes this concrete: the tmachine e2e-podman suite supplies interim coverage, but the intended outcome is to move the source @@ -49,30 +52,41 @@ the definition of OpenShell conformance. - Implement the proposed framework, capability API, or CI matrices in this PR. - Finalize the destination of every existing E2E assertion; migration issues retain that analysis and their task lists. -- Introduce named conformance profiles, hierarchical capability inference, or - generic capability parameters before concrete tests require them. -- Create a dedicated workload fixture image or dependency-declaration system - for scenarios initially. - Establish a certification program, review board, or reporting service. - Replace the SDK compatibility proposal in #3238 or standardize language SDK APIs through the CLI runner. ## Proposal -### 1. Separate behavioral ownership from execution dimensions +### 1. Define contracts and organise tests by purpose -Each assertion has an intended contract and an appropriate owner. A binary can -be split when its assertions belong to different families. +A behavioral contract specifies the expected observable result of an operation +under stated conditions. A test case exercises that behavior; an assertion +checks an observation against an expectation. A suite groups related test cases. +For example, a contract may require that a deleted sandbox is absent from the +sandbox list; the assertion checks that its identifier is absent. Assertions +that verify test setup do not each define a separate product contract. -| Family | Contract and admission boundary | +Group tests by the contracts they exercise, not by their current binary or +runner. Split binaries when they mix unrelated contracts or prerequisites. + +| Family | Purpose | | --- | --- | | Unit and component integration | Internal logic, configuration selection, translation, and implementation mechanics; use the lowest effective layer. | -| General conformance | Public behavior demonstrated on at least two different drivers, with effective API capabilities for non-universal support. | +| General conformance | Public behavioral contracts that hold across drivers and environments, independent of implementation details. | | Feature-specific | Public feature behavior requiring configured external integration, or currently implemented on only one driver; one representative driver suffices. | | Driver-specific | Driver configuration, runtime and host integration, and implementation contracts; exercise applicable environments for that driver. | | Disruption conformance | Portable continuity or recovery assertions using environment-specific disruption actuators; one working actuator suffices initially. | | Load/scale | Throughput, latency, concurrency, saturation, and scaling measurements, reported separately from behavioral correctness. | -| SDK conformance | Behavior and compatibility of SDK implementations through their native interfaces, specified separately in #3238. | + +The placement of portable, capability-dependent tests in general conformance +or feature-specific suites remains open until a concrete migration example +requires that decision. Capability dependence alone does not determine a family. + +The client interface is separate from the test family. A public contract tested +through an SDK can belong to general conformance just as it can through the CLI. +SDK-specific behavior, such as language-specific conversion and cancellation, +needs focused coverage; #3238 retains its SDK compatibility design scope. Security portability is a workstream spanning these families. Define a separate security family only if multiple tests need common specialized infrastructure. @@ -80,27 +94,12 @@ For example, bypass prevention is an intended universal guarantee, but the existing seccomp-oriented probe needs separate analysis before its portable migration. Core-dump protection similarly needs a portable contract and probe. -The execution matrix has independent axes: - -| Axis | Examples | -| --- | --- | -| Platform | Host and workload OS and architecture | -| Driver | Docker, Podman, Kubernetes, VM, MXC | -| Environment | Rootful/rootless Podman, local runtime, Kubernetes cluster | -| Configuration | Gateway settings, in-process/external driver wiring | -| Installation | Candidate binaries/images, RPM, DEB, Snap, Helm, Homebrew | -| Suite | General conformance, a feature suite, a driver suite, disruption | - -Rootful and rootless Podman are environments of one driver. They do not satisfy -the two-driver admission rule. Driver tests should preserve their behavioral -contract across those environments. Matrix entries should target meaningful -risks without requiring the full cross-product of every axis. - ### 2. Define general conformance through observable contracts Conformance tests exercise a behavioral contract against an already configured -OpenShell gateway. They remain CLI-based until direct API access is strictly -required. Assertions should use observable outcomes and structured output; +OpenShell gateway. The initial suite remains CLI-based until direct API access +is strictly required; this is an execution choice, not a classification rule. +Assertions should use observable outcomes and structured output; incidental presentation text is not a conformance contract. Native runtime inspection may provide best-effort failure diagnostics, but must not determine whether general conformance passed. @@ -160,9 +159,9 @@ when support for that capability is optional. Non-universal behavior supported by at least two drivers is a signal to extend the public capability API. Prefer adding the capability and its concrete test -in the same PR. If an API extension is infeasible, use feature-specific or -driver-specific coverage. Both drivers used for general-conformance admission -must advertise the capability and pass its assertions. +in the same PR. The eventual suite classification of optional portable contracts +is deferred, but their result semantics are not: absent optional support is +unsupported, and advertised behavior that violates its contract fails. | Outcome | Meaning | | --- | --- | @@ -180,6 +179,48 @@ driver. A focused or incomplete invocation must not imply complete coverage. ### 4. Keep execution reusable and select CI gates explicitly +A test run exercises selected test cases against a configured target and records +their results. Target preparation supplies the environment, installs artifacts, +and configures the gateway. Behavioral tests consume that target through a +client interface. CI policy selects runs and decides which results gate a merge +or release; it does not redefine the tested contracts. + +```mermaid +flowchart LR + subgraph preparation[Target preparation] + machine[Machine / base image] --> setup[Environment setup] + setup --> install[Install and configure OpenShell] + end + artifacts[Build artifacts] --> install + install --> target[Configured OpenShell target] + external[External provisioning] --> target + suite[Test suite] --> client[CLI or SDK] + client -->|exercises| target +``` + +In the current tmachine configuration, a `Machine` identifies a base image. +An `Environment` references a machine and supplies setup playbooks. A separate +`Installer` supplies installation playbooks and artifact inputs, and a +`Testsuite` supplies test playbooks and inputs. Selecting an environment, +installer, and suite composes a run without making test ownership depend on +target preparation. + +Runs vary along several dimensions; not every combination is meaningful: + +| Dimension | Examples | +| --- | --- | +| Platform | Host and workload OS and architecture | +| Driver | Docker, Podman, Kubernetes, VM, MXC | +| Environment | Local runtime, Kubernetes cluster, rootful/rootless Podman | +| Configuration | Gateway settings, in-process/external driver wiring | +| Installation | Candidate binaries/images, RPM, DEB, Snap, Helm, Homebrew | +| Client interface | CLI, language SDK | +| Suite | General conformance, feature-specific, driver-specific, disruption | + +For example, rootful and rootless Podman change the environment, not the driver +or expected contract. They do not supply two-driver admission evidence. Select +combinations for meaningful coverage rather than requiring a full cross-product. + Nix pins source-check dependencies and builds candidate artifacts and test archives. Tmachine provisions supported environments, applies installation and configuration, and runs a selected suite. Tests should consume an already @@ -228,7 +269,43 @@ the effective configuration. Other provisioners supply equivalent target identification. This is a placeholder, not a complete report schema; richer capability, scenario, and artifact reporting can follow a concrete need. -### 5. Give each document one responsibility +## Implementation plan + +### Build on the existing suite layout + +Retain the existing suite locations and extend them as contracts are migrated: + +```text +tests/ +├── config.nix # Existing tmachine definitions +├── artifacts.nix # Existing artifact construction +├── ansible/ # Existing provisioning and execution +├── CONFORMANCE.md # Proposed agreed conformance policy +└── suites/ + ├── conformance/ + │ ├── cli/ # Existing CLI test entry points + │ └── README.md # Proposed suite contributor guidance + ├── drivers/ + │ └── podman/ # Existing driver-specific tests + └── features/ + └── provider-refresh/ + └── keycloak/ # Existing external-integration tests +``` + +Move the shared `crates/openshell-conformance` library under +`tests/suites/conformance` alongside its existing CLI entry points; choose its +precise internal layout in that relocation. Extend `drivers/` and `features/` +with focused suites rather than new catch-all binaries. Unit and component +integration tests remain alongside their components, and SDK-native tests may +remain in their SDK trees. Folder placement does not define the behavioral +contract. Disruption and load/scale locations remain open pending their concrete +infrastructure needs. + +### Publish the agreed model + +Documentation records the testing model and its implementation; reorganising +documentation is not a substitute for implementing reusable suites and target +preparation. Publish agreed policy as implementation lands: | Document | Responsibility | | --- | --- | @@ -247,7 +324,7 @@ proposal while sharing terminology and provisioning boundaries. Accepted policy is published in the living guides as implementation lands; current references must not describe proposed gates as already enforced. -## Implementation plan +### Adopt incrementally 1. Discuss this RFC through existing PR #3460 and track follow-ups in #3954. Resolve policy questions independently from per-test migration details. @@ -296,12 +373,19 @@ must not describe proposed gates as already enforced. ## Alternatives -### Continue expanding TESTING.md and suite READMEs independently +### Maintain independent driver-specific E2E suites + +Each driver could retain a complete E2E suite tailored to its runtime. This +minimises initial migration, but duplicates public behavioral checks and allows +expectations to diverge. Reuse portable contracts across targets and reserve +driver-specific coverage for implementation and integration requirements. -This is the smallest immediate documentation change, but leaves strategy, -commands, and future gates interleaved. A single RFC and focused living -references give reviewers a place to resolve disagreements before publishing -authoritative guidance. +### Couple behavioral tests to target provisioning + +Each suite could provision and configure its own gateway. This simplifies local +setup for that suite, but makes it harder to validate installed artifacts or an +externally prepared gateway with the same tests. Separate target preparation +from behavioral testing while allowing a harness to orchestrate both. ### Introduce separate API and CLI conformance frameworks immediately @@ -310,18 +394,13 @@ runner and overlapping scenarios before a CLI limitation requires them. Keep general conformance CLI-based initially; SDK interface testing remains a separate justified consumer under #3238. -### Require every scenario on every configuration +### Require every tested behavior on every configuration This gives a uniform baseline but excludes useful portable behavior that some drivers do not implement. Mandatory scenarios plus precise optional capabilities -allow useful coverage while requiring advertised behavior to pass. - -### Establish profiles and a full certification process first - -Profiles, a versioned certification program, and formal promotion gates could -clarify compatibility guarantees, but would add policy and infrastructure before -the existing tests have been classified. Keep these decisions open and evolve -them from concrete contracts and consumers. +allow useful coverage while requiring advertised behavior to pass. Whether +optional portable contracts belong in general conformance or feature-specific +suites is deferred; this alternative concerns support requirements, not naming. ## Prior art @@ -341,6 +420,9 @@ them from concrete contracts and consumers. ## Open questions +- Should portable, capability-dependent tests belong in general conformance or + feature-specific suites? Resolve this using the first concrete migration + example that requires the distinction, without adding a category in advance. - What constitutes a complete conformance claim, and how are suite versions matched to gateway/CLI releases? Which version-skew guarantees are required? - What maturity and reliability evidence is required beyond two-driver success? From ae9b081fe7ee0da450145291bf2dbf29c93e4667 Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Thu, 1 Oct 2026 17:23:12 +0200 Subject: [PATCH 5/9] docs(rfc): record agreed conformance policies Signed-off-by: Evan Lezar --- rfc/0016-testing-strategy/README.md | 56 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md index 5b8d24f1fc..d45cefc725 100644 --- a/rfc/0016-testing-strategy/README.md +++ b/rfc/0016-testing-strategy/README.md @@ -104,6 +104,17 @@ incidental presentation text is not a conformance contract. Native runtime inspection may provide best-effort failure diagnostics, but must not determine whether general conformance passed. +General conformance assertions require only public CLI/API access and sandbox +operations. Host or runtime administrative access belongs to provisioning, +best-effort diagnostics, or disruption actuators, not general-conformance +pass/fail assertions. Unavailable diagnostic access must not change the result. + +Each test is the single source of truth for its behavioral contract. Keep its +stable identity, preconditions, expected behavior, mandatory/optional status, +and significant side effects alongside the test. Shared guidance defines these +conventions, not a duplicate catalogue of contracts. Adding a test must not +require manually updating its specification in another location. + Tests must not update gateway startup configuration, driver flags, deployment manifests, or gateway.toml. They may create and mutate public API-managed state, including sandboxes, policies, providers, workspaces, and settings. Use unique @@ -117,6 +128,12 @@ prerequisite. Public capabilities must not become an inventory of third-party services, credentials, or fixture configuration. Missing prerequisites for a selected feature suite are setup errors. +General conformance must not depend on public internet services for behavioral +assertions. Images and dependencies may be downloaded during provisioning; +gateway connectivity and test-owned services remain allowed. Demonstrating a +complete run with public internet access disabled is an incremental infrastructure +follow-up, not an immediate migration gate. + Start with existing general-purpose workload images and a small common toolset. The harness supplies an appropriate image for each target. Missing fixture tooling is an infrastructure error. Defer a dedicated image until concrete @@ -130,6 +147,14 @@ which infrastructure exists. Docker and Podman are acceptable initial targets. Unavailable infrastructure is a coverage gap. A successful test against one configuration does not establish success for all deployments of its driver. +Admission uses normal PR review and the required driver-validation evidence, +without a fixed soak period or separate promotion PR. Resolve known flakiness +rather than concealing it with retries. Migration, admission, and removal of +covered source tests may occur in the same PR. Changes to or removal of an +established contract also use normal PR review; explicitly distinguish a test +correction from a change to promised product behavior, including removal of +previously advertised support. + Disruption conformance has a separate execution family. Portable assertions describe continuity or recovery; actuators induce gateway restart, runtime loss, or another disruption. An absent actuator prevents selection for that environment; @@ -177,6 +202,13 @@ turn those failures into passing scenario results. Mock-MXC execution, for example, needs to identify the mock target and cannot qualify the production driver. A focused or incomplete invocation must not imply complete coverage. +A complete run evaluates the entire identified suite without filtering. A +filtered run is partial even if every selected test passes. Evaluating an +optional capability and recording unsupported accounts for that test without +making the run partial. Missing mandatory support fails. An unfiltered invocation +that stops before accounting for all tests is incomplete; completeness and +success are distinct, and errors must not be reported as a complete pass. + ### 4. Keep execution reusable and select CI gates explicitly A test run exercises selected test cases against a configured target and records @@ -261,7 +293,9 @@ and Helm/Kubernetes paths can reuse conformance. Native Homebrew qualification uses its appropriate platform. Passing behavior after installation does not by itself prove package ownership, service setup, upgrade, or uninstall contracts; installation-specific assertions must cover those separately. Upgrade and -version-skew obligations remain explicit follow-up decisions. +version-skew obligations remain explicit follow-up decisions. Initially, the +suite, CLI, and gateway use the same OpenShell revision. Version-skew testing is +deferred, and these results make no cross-version compatibility claim. Start report attribution with the OpenShell Git SHA and the tmachine configuration used for a run. Preserve enough configuration identity to recover @@ -351,13 +385,15 @@ must not describe proposed gates as already enforced. 7. Expand destination-suite CI and candidate-installation validation, documenting actual gates and coverage gaps as they become operational. Preserve existing coverage until its intent has a validated destination; no mandatory overlap - period is needed solely for migration. + period is needed solely for migration. Add enforced offline validation as an + infrastructure follow-up after provisioning supplies images and dependencies. ## Risks - Optional capability reporting can hide regressions if an implementation simply - stops advertising support. Mandatory contracts remain test-owned; policy for - reviewing optional-support removal needs an explicit decision. + stops advertising support. Mandatory contracts remain test-owned; review + removal of advertised support explicitly as a product-contract change, not + merely as a way to avoid failing tests. - Two Linux container drivers can pass a scenario containing OS assumptions. Admission evidence is a starting point; broader platform validation and probe review remain necessary. @@ -423,16 +459,8 @@ suites is deferred; this alternative concerns support requirements, not naming. - Should portable, capability-dependent tests belong in general conformance or feature-specific suites? Resolve this using the first concrete migration example that requires the distinction, without adding a category in advance. -- What constitutes a complete conformance claim, and how are suite versions - matched to gateway/CLI releases? Which version-skew guarantees are required? -- What maturity and reliability evidence is required beyond two-driver success? - How are promotion, material changes, and demotion reviewed without imposing - an unnecessary new governance process? -- What minimal normative description and stable identity must each scenario - carry, and how is removal of previously advertised support reviewed? -- Should general conformance explicitly require offline execution after artifact - provisioning, and what host privileges and fixture reachability assumptions - are allowed for externally provisioned gateways? +- What fixture reachability assumptions are needed for tests against externally + provisioned gateways, without requiring administrative access for assertions? - Which source and integration jobs are required for merges and release promotion? How should Windows, GPU, disruption, and load/scale runs be scheduled? - Where should disruption and load/scale suites live, and what is the smallest From b8def3bfb64486630e1e00bb24f260e90013539a Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Thu, 1 Oct 2026 17:29:20 +0200 Subject: [PATCH 6/9] docs(rfc): condense testing strategy proposal Signed-off-by: Evan Lezar --- rfc/0016-testing-strategy/README.md | 593 +++++++++------------------- 1 file changed, 189 insertions(+), 404 deletions(-) diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md index d45cefc725..fc88b99a97 100644 --- a/rfc/0016-testing-strategy/README.md +++ b/rfc/0016-testing-strategy/README.md @@ -11,211 +11,132 @@ links: ## Summary -Define a common testing strategy for OpenShell that separates the behavioral -contract being tested from the environment, installation, client interface, -and CI policy used to validate it. Reuse tests of public behavior across -configured targets, while keeping implementation-specific checks and performance -measurements distinct. A passing test establishes the same contract regardless -of how its target was prepared. - -Use Nix for reproducible build inputs and test artifacts, and tmachine for -integration and Linux installation environments it can represent. Keep test -contracts independent of those tools. Initially exercise general conformance -through the OpenShell CLI; using an SDK does not create a different test category. -This is a proposal for incremental implementation, not a description of gates -already enforced by CI. +Organise tests by the behavior they verify, independently of how OpenShell is +deployed. Reuse public-contract tests across drivers and environments, keep +implementation-specific coverage separate, and make each test's contract and +each run's coverage explicit. + +This proposal defines test categories, conformance rules, and an execution model +that builds on Nix and tmachine without requiring tests to depend on them. +Adopt it incrementally; it does not describe CI gates already in place. ## Motivation -How do we establish that OpenShell behaves as promised across supported -configurations without duplicating behavioral tests for every driver and -environment? Tests tied to particular environments make it difficult to -distinguish product requirements from implementation details, reuse coverage, -or determine what a passing suite establishes. - -OpenShell tests have accumulated across crate-local tests, E2E binaries, -driver wrappers, installed-artifact suites, and CI workflows. A single binary -can mix public behavior, native runtime inspection, an external integration, -and performance measurements. Without a shared model, contributors must either -duplicate this coverage for new targets or carry assumptions that do not apply -to them. Reviewers cannot reliably distinguish missing product support from -missing test infrastructure. - -The migration tracked by #3712 makes this concrete: the tmachine e2e-podman -suite supplies interim coverage, but the intended outcome is to move the source -tests into suites that own their behavioral intent and then remove the catch-all. -The strategy needs to support that migration without making its source inventory -the definition of OpenShell conformance. +Existing E2E binaries mix public behavior, runtime inspection, external +integrations, and performance measurements. This makes coverage difficult to +reuse across drivers and obscures whether a failure reflects the product or +its test infrastructure. + +A shared model lets contributors place tests by purpose and reviewers understand +what passing establishes. The Podman migration in #3712 is an initial consumer, +not the definition of OpenShell conformance. ## Non-goals -- Implement the proposed framework, capability API, or CI matrices in this PR. -- Finalize the destination of every existing E2E assertion; migration issues - retain that analysis and their task lists. -- Establish a certification program, review board, or reporting service. -- Replace the SDK compatibility proposal in #3238 or standardize language SDK - APIs through the CLI runner. +- Implement the framework, capability API, or CI matrices in this PR. +- Classify every existing test; retain that work in migration issues. +- Introduce certification, a review board, or version-skew testing. +- Replace the SDK compatibility design in #3238. ## Proposal -### 1. Define contracts and organise tests by purpose - -A behavioral contract specifies the expected observable result of an operation -under stated conditions. A test case exercises that behavior; an assertion -checks an observation against an expectation. A suite groups related test cases. -For example, a contract may require that a deleted sandbox is absent from the -sandbox list; the assertion checks that its identifier is absent. Assertions -that verify test setup do not each define a separate product contract. +### 1. Organise tests by contract -Group tests by the contracts they exercise, not by their current binary or -runner. Split binaries when they mix unrelated contracts or prerequisites. +A behavioral contract specifies an operation's expected observable result under +stated conditions. A test case exercises it, an assertion checks an observation, +and a suite groups related cases. For example, deleting a sandbox must remove +it from the sandbox list; an assertion checks that its identifier is absent. | Family | Purpose | | --- | --- | -| Unit and component integration | Internal logic, configuration selection, translation, and implementation mechanics; use the lowest effective layer. | -| General conformance | Public behavioral contracts that hold across drivers and environments, independent of implementation details. | -| Feature-specific | Public feature behavior requiring configured external integration, or currently implemented on only one driver; one representative driver suffices. | -| Driver-specific | Driver configuration, runtime and host integration, and implementation contracts; exercise applicable environments for that driver. | -| Disruption conformance | Portable continuity or recovery assertions using environment-specific disruption actuators; one working actuator suffices initially. | -| Load/scale | Throughput, latency, concurrency, saturation, and scaling measurements, reported separately from behavioral correctness. | - -The placement of portable, capability-dependent tests in general conformance -or feature-specific suites remains open until a concrete migration example -requires that decision. Capability dependence alone does not determine a family. - -The client interface is separate from the test family. A public contract tested -through an SDK can belong to general conformance just as it can through the CLI. -SDK-specific behavior, such as language-specific conversion and cancellation, -needs focused coverage; #3238 retains its SDK compatibility design scope. - -Security portability is a workstream spanning these families. Define a separate -security family only if multiple tests need common specialized infrastructure. -For example, bypass prevention is an intended universal guarantee, but the -existing seccomp-oriented probe needs separate analysis before its portable -migration. Core-dump protection similarly needs a portable contract and probe. - -### 2. Define general conformance through observable contracts - -Conformance tests exercise a behavioral contract against an already configured -OpenShell gateway. The initial suite remains CLI-based until direct API access -is strictly required; this is an execution choice, not a classification rule. -Assertions should use observable outcomes and structured output; -incidental presentation text is not a conformance contract. Native runtime -inspection may provide best-effort failure diagnostics, but must not determine -whether general conformance passed. - -General conformance assertions require only public CLI/API access and sandbox -operations. Host or runtime administrative access belongs to provisioning, -best-effort diagnostics, or disruption actuators, not general-conformance -pass/fail assertions. Unavailable diagnostic access must not change the result. - -Each test is the single source of truth for its behavioral contract. Keep its -stable identity, preconditions, expected behavior, mandatory/optional status, -and significant side effects alongside the test. Shared guidance defines these -conventions, not a duplicate catalogue of contracts. Adding a test must not -require manually updating its specification in another location. - -Tests must not update gateway startup configuration, driver flags, deployment -manifests, or gateway.toml. They may create and mutate public API-managed state, -including sandboxes, policies, providers, workspaces, and settings. Use unique -resource names and cleanup. Avoid conflicting global-state mutations within one -run. Restoring a pre-existing global setting is recommended, not a hard -conformance requirement; the global effects still need to be clear to callers. - -A short-lived service fully owned by a test or harness can be a conformance -fixture. An external service requiring gateway configuration is a feature-suite -prerequisite. Public capabilities must not become an inventory of third-party -services, credentials, or fixture configuration. Missing prerequisites for a -selected feature suite are setup errors. - -General conformance must not depend on public internet services for behavioral -assertions. Images and dependencies may be downloaded during provisioning; -gateway connectivity and test-owned services remain allowed. Demonstrating a -complete run with public internet access disabled is an incremental infrastructure -follow-up, not an immediate migration gate. - -Start with existing general-purpose workload images and a small common toolset. -The harness supplies an appropriate image for each target. Missing fixture -tooling is an infrastructure error. Defer a dedicated image until concrete -incompatibilities justify it. A default-image scenario deliberately omits the -image and checks that a usable sandbox results; precise configuration selection -and registry resolution belong in lower-level or integration coverage. - -Two-driver validation is the minimum evidence for admitting general conformance, -not a limit on subsequent execution. Run it against every supported driver for -which infrastructure exists. Docker and Podman are acceptable initial targets. -Unavailable infrastructure is a coverage gap. A successful test against one -configuration does not establish success for all deployments of its driver. - -Admission uses normal PR review and the required driver-validation evidence, -without a fixed soak period or separate promotion PR. Resolve known flakiness -rather than concealing it with retries. Migration, admission, and removal of -covered source tests may occur in the same PR. Changes to or removal of an -established contract also use normal PR review; explicitly distinguish a test -correction from a change to promised product behavior, including removal of -previously advertised support. - -Disruption conformance has a separate execution family. Portable assertions -describe continuity or recovery; actuators induce gateway restart, runtime loss, -or another disruption. An absent actuator prevents selection for that environment; -a broken configured actuator is an infrastructure error. One actuator is enough -to begin migrating a portable disruption contract. - -Timing limits normally bound readiness, polling, and hangs. A performance -threshold belongs in load/scale unless the deadline is itself a specified public -contract. - -### 3. Distinguish capabilities, scenario selection, and results - -Use flat, namespaced boolean capability identifiers describing precise effective -product behavior. Namespaces can organize names without implying parent/child -support. Parameters and named profiles are deferred. A driver name, a fixture -prerequisite, and a selectable test-family prefix are not product capabilities. -In particular, the leaf selection introduced by #3768 is separate from API -capability reporting. - -The gateway reports support for its running configuration. Obtain the snapshot -through machine-readable CLI output backed by the public API. Reporting is a -hard requirement: failure to discover capabilities aborts the conformance run. -Each scenario encodes its expected behavior and whether that behavior is -mandatory. A missing mandatory capability fails; a missing optional capability -is unsupported. An advertised capability must satisfy its contract, including -when support for that capability is optional. - -Non-universal behavior supported by at least two drivers is a signal to extend -the public capability API. Prefer adding the capability and its concrete test -in the same PR. The eventual suite classification of optional portable contracts -is deferred, but their result semantics are not: absent optional support is -unsupported, and advertised behavior that violates its contract fails. - -| Outcome | Meaning | +| Unit and component integration | Internal logic and implementation mechanics, at the lowest effective layer. | +| General conformance | Public behavioral contracts across drivers and environments. | +| Feature-specific | Features requiring configured external integration or currently implemented on only one driver. | +| Driver-specific | Driver configuration, runtime and host integration, and implementation contracts. | +| Disruption conformance | Portable continuity or recovery behavior under deliberately induced disruption. | +| Load/scale | Performance and scaling measurements, separate from behavioral correctness. | + +The client interface is not a test category: CLI and SDK tests can exercise the +same public contract. SDK-specific semantics retain focused coverage. Security +tests span these families; introduce a separate family only if tests need shared +specialized infrastructure. Performance thresholds belong in load/scale unless +the deadline is itself a public contract. + +### 2. Define conformance through public behavior + +Each test owns its contract: keep its stable identity, preconditions, expected +behavior, mandatory/optional status, and significant side effects alongside the +test. Do not maintain a duplicate specification elsewhere. + +General conformance uses an already configured gateway, initially through the +CLI until direct API access is necessary. Assert observable behavior and +structured output, not incidental presentation or native runtime details. +Pass/fail requires only public CLI/API access and sandbox operations. +Administrative access belongs to provisioning, diagnostics, or disruption +actuators; unavailable diagnostics must not affect results. + +Tests must not change gateway startup configuration. They may mutate public +API-managed state, using unique names and cleanup and avoiding conflicting +global-setting changes within a run. Document global effects; restoring prior +global settings is recommended, not mandatory. + +Test-owned short-lived services are allowed. External services requiring gateway +configuration belong in feature-specific suites. Behavioral assertions must not +depend on public internet services; provisioning may download dependencies and +images. Use existing workload images and a small common toolset, supplied by the +harness. Missing prerequisites are setup errors. Dedicated fixture images and +enforced offline validation are follow-ups, not migration gates. + +General-conformance admission requires evidence from at least two drivers; +Docker and Podman suffice initially. Then run against every supported driver +with available infrastructure, recording gaps. Feature tests need one +representative driver. Disruption tests need one working actuator initially; +an absent actuator prevents selection, while a broken configured actuator is +an infrastructure error. + +Use normal PR review, without a soak period or separate promotion PR. Resolve +known flakiness rather than hiding it with retries. Changes or removals must +distinguish test corrections from changes to promised behavior, including +withdrawal of advertised support. + +### 3. Make applicability and coverage explicit + +The gateway must report effective capabilities for its running configuration +through the public API and machine-readable CLI output. Discovery failure aborts +conformance. Start with flat, namespaced booleans; defer hierarchy, parameters, +and profiles. Capabilities describe product behavior, not test selectors, +credentials, or external-service prerequisites. + +Portable, non-universal behavior is a reason to extend capability reporting; +prefer adding the capability and its test together. Whether those tests belong +in general conformance or feature-specific suites remains open until a concrete +migration example requires a decision. + +| Result | Meaning | | --- | --- | -| Passed | The selected behavioral assertions held. | -| Failed | A mandatory capability is missing or the exercised contract was violated. | -| Unsupported | An optional capability was not advertised. | -| Skipped | A scenario was deliberately excluded from this invocation. | -| Error | Infrastructure, fixture, runner, or actuator failure prevented evaluation. | - -Keep these distinctions as lightweight as practical. An advisory CI job can -report behavioral failures without blocking a merge; that job policy must not -turn those failures into passing scenario results. Mock-MXC execution, for -example, needs to identify the mock target and cannot qualify the production -driver. A focused or incomplete invocation must not imply complete coverage. - -A complete run evaluates the entire identified suite without filtering. A -filtered run is partial even if every selected test passes. Evaluating an -optional capability and recording unsupported accounts for that test without -making the run partial. Missing mandatory support fails. An unfiltered invocation -that stops before accounting for all tests is incomplete; completeness and -success are distinct, and errors must not be reported as a complete pass. - -### 4. Keep execution reusable and select CI gates explicitly - -A test run exercises selected test cases against a configured target and records -their results. Target preparation supplies the environment, installs artifacts, -and configures the gateway. Behavioral tests consume that target through a -client interface. CI policy selects runs and decides which results gate a merge -or release; it does not redefine the tested contracts. +| Passed | The behavioral assertions held. | +| Failed | Mandatory support is absent or an exercised contract was violated. | +| Unsupported | Optional support was not advertised. | +| Skipped | The test was deliberately excluded. | +| Error | Infrastructure or fixture failure prevented evaluation. | + +Advertised behavior must pass even when support is optional. A complete run +accounts for the entire identified suite without filtering; evaluated optional +unsupported cases count as accounted for. Filtered or interrupted runs are +partial. Completeness is not success: failures and errors cannot yield a complete +pass. Advisory CI policy must not relabel failed tests as passing. + +Initially, suite, CLI, and gateway use the same OpenShell revision; no +cross-version claim is made. Start reports with that SHA and the tmachine +configuration, or equivalent target identity. Record mock targets as mocks, +not evidence for production drivers. + +### 4. Separate target preparation from test execution + +A run combines a configured target with a selected suite. Platform, driver, +environment, gateway configuration, installation method, and client interface +may vary without changing the contract. Rootful and rootless Podman are two +environments of one driver, not two-driver evidence. ```mermaid flowchart LR @@ -230,240 +151,104 @@ flowchart LR client -->|exercises| target ``` -In the current tmachine configuration, a `Machine` identifies a base image. -An `Environment` references a machine and supplies setup playbooks. A separate -`Installer` supplies installation playbooks and artifact inputs, and a -`Testsuite` supplies test playbooks and inputs. Selecting an environment, -installer, and suite composes a run without making test ownership depend on -target preparation. +Nix pins build inputs and produces artifacts and test archives. Tmachine +composes a `Machine` and `Environment` for setup, an `Installer` for installation, +and a `Testsuite` for execution. Other provisioners may supply targets tmachine +cannot represent. General conformance must also run against externally prepared +gateways. -Runs vary along several dimensions; not every combination is meaningful: - -| Dimension | Examples | -| --- | --- | -| Platform | Host and workload OS and architecture | -| Driver | Docker, Podman, Kubernetes, VM, MXC | -| Environment | Local runtime, Kubernetes cluster, rootful/rootless Podman | -| Configuration | Gateway settings, in-process/external driver wiring | -| Installation | Candidate binaries/images, RPM, DEB, Snap, Helm, Homebrew | -| Client interface | CLI, language SDK | -| Suite | General conformance, feature-specific, driver-specific, disruption | - -For example, rootful and rootless Podman change the environment, not the driver -or expected contract. They do not supply two-driver admission evidence. Select -combinations for meaningful coverage rather than requiring a full cross-product. - -Nix pins source-check dependencies and builds candidate artifacts and test -archives. Tmachine provisions supported environments, applies installation and -configuration, and runs a selected suite. Tests should consume an already -configured target; general conformance must not require tmachine to have created -it. Native Windows/macOS, GPU, or other environments tmachine cannot represent -may use a suitable external provisioner with the same behavioral contracts. - -Keep environment definitions in tests/config.nix, artifact construction in -tests/artifacts.nix, and provisioning/installation in tests/ansible. Workflow -matrices select declared environment/suite combinations rather than duplicating -their setup logic. Existing mise commands remain usable migration entry points -until replacements provide equivalent coverage. - -After #3866 removes the standalone conformance executable, consolidate -crates/openshell-conformance under tests/suites/conformance. Preserve a shared -library for scenarios and execution helpers alongside the Cargo test entry -points that exercise the openshell CLI. Update workspace membership, lockfiles, -archive inputs, and CI references together. Extract utilities shared by feature -or disruption suites only when actual consumers require them. - -Source checks cover formatting, lint, unit tests, feature/platform compilation, -and existing dependency/security checks. The earlier target-state proposal used -Linux x86_64, Linux ARM64, and macOS ARM64 as the initial source matrix. SDK-local -changes run the relevant SDK checks; protobuf changes run all affected SDK -checks. Packaging selection includes transitive inputs and uses broader coverage -when impact cannot be determined safely. - -General conformance should run across available drivers, with expensive or -platform-constrained environments using appropriate dedicated or scheduled -workflows. Feature suites run where prerequisites exist; driver suites exercise -applicable environments; disruption suites require actuators. The exact required -merge checks, schedules, and release gates remain review decisions. The earlier -five-environment matrix is an initial candidate, not a permanent ceiling. - -Release validation installs exact candidate artifacts and then runs applicable -suites before publication. Representative RPM/Fedora, DEB/Ubuntu, Snap/Ubuntu, -and Helm/Kubernetes paths can reuse conformance. Native Homebrew qualification -uses its appropriate platform. Passing behavior after installation does not by -itself prove package ownership, service setup, upgrade, or uninstall contracts; -installation-specific assertions must cover those separately. Upgrade and -version-skew obligations remain explicit follow-up decisions. Initially, the -suite, CLI, and gateway use the same OpenShell revision. Version-skew testing is -deferred, and these results make no cross-version compatibility claim. - -Start report attribution with the OpenShell Git SHA and the tmachine -configuration used for a run. Preserve enough configuration identity to recover -the effective configuration. Other provisioners supply equivalent target -identification. This is a placeholder, not a complete report schema; richer -capability, scenario, and artifact reporting can follow a concrete need. +CI selects meaningful combinations, not a full cross-product, and reuses setup +definitions rather than duplicating them. Keep source checks and affected SDK +checks; protobuf changes require all affected SDK checks. Packaging selection +must include transitive inputs and broaden when impact is uncertain. +Release validation installs exact candidate artifacts before running applicable +suites. Installation, upgrade, and uninstall behavior need separate assertions; +successful conformance alone does not validate packaging. ## Implementation plan -### Build on the existing suite layout - -Retain the existing suite locations and extend them as contracts are migrated: +Build on the existing layout; only the marked documents are proposed additions: ```text tests/ -├── config.nix # Existing tmachine definitions -├── artifacts.nix # Existing artifact construction -├── ansible/ # Existing provisioning and execution -├── CONFORMANCE.md # Proposed agreed conformance policy +├── config.nix # tmachine definitions +├── artifacts.nix # Artifact construction +├── ansible/ # Provisioning and execution +├── CONFORMANCE.md # Proposed: agreed policy └── suites/ ├── conformance/ - │ ├── cli/ # Existing CLI test entry points - │ └── README.md # Proposed suite contributor guidance - ├── drivers/ - │ └── podman/ # Existing driver-specific tests - └── features/ - └── provider-refresh/ - └── keycloak/ # Existing external-integration tests + │ ├── cli/ # CLI test entry points + │ └── README.md # Proposed: contributor guidance + ├── drivers/podman/ + └── features/provider-refresh/keycloak/ ``` -Move the shared `crates/openshell-conformance` library under -`tests/suites/conformance` alongside its existing CLI entry points; choose its -precise internal layout in that relocation. Extend `drivers/` and `features/` -with focused suites rather than new catch-all binaries. Unit and component -integration tests remain alongside their components, and SDK-native tests may -remain in their SDK trees. Folder placement does not define the behavioral -contract. Disruption and load/scale locations remain open pending their concrete -infrastructure needs. - -### Publish the agreed model - -Documentation records the testing model and its implementation; reorganising -documentation is not a substitute for implementing reusable suites and target -preparation. Publish agreed policy as implementation lands: - -| Document | Responsibility | -| --- | --- | -| This RFC and its PR | Strategy discussion, rationale, alternatives, unresolved decisions | -| TESTING.md | Contributor entry point, concise routing rules, currently working local commands | -| Proposed tests/CONFORMANCE.md | Conformance contracts, admission, capabilities, result semantics, and agreed policy | -| tests/suites/conformance/README.md | Authoring and running the suite, fixtures, library extension points, selection, cleanup | -| Other suite READMEs | Their fixtures, prerequisites, commands, and specialized contracts | -| CI.md | Implemented workflow triggers, matrices, gates, and artifact investigation | -| AGENTS.md and contributor skills | Short routing instructions linking to canonical guidance | -| #3954 and focused issues | Strategy follow-ups, migration tasks, dependencies, and completion tracking | - -The suite README owns the relocated library's contributor guidance. A separate -crate README need not duplicate it. SDK specifications remain owned by the SDK -proposal while sharing terminology and provisioning boundaries. Accepted policy -is published in the living guides as implementation lands; current references -must not describe proposed gates as already enforced. - -### Adopt incrementally - -1. Discuss this RFC through existing PR #3460 and track follow-ups in #3954. - Resolve policy questions independently from per-test migration details. -2. Reconcile #3768, #3864, and #3866 for leaf selection, obsolete parity removal, - and Cargo execution. Relocate the shared library in a focused follow-up, - coordinating the same files with Windows work in #3769. -3. Add effective capability reporting and stable CLI discovery. Migrations can - be developed in parallel, but qualification under this proposal requires the - mandatory reporting contract. Add further capability/test pairs together. -4. Rescope #3945 around the documentation ownership above. Align SDK RFC #3238 - with the shared strategy. Assess #2182 and #2873 for supersession and preserve - unique assertions; this RFC does not automatically close those PRs. -5. Migrate coherent behavioral intent in parallel by destination family. One PR - can add destination coverage and remove covered source tests. Delete empty - binaries and rename or split residual binaries. Require two-driver evidence - for general conformance, one representative driver for feature tests, and - one working actuator for disruption. Keep implementation tests at the lowest - effective layer. -6. Use #3712 for the Podman migration inventory, including security portability - investigations and load/scale routing. Retire the e2e-podman feature, follow-up - exclusion list, and temporary tmachine suite when all source intent is covered - or explicitly retired. Do not introduce inventory-check machinery unless - continued source churn demonstrates a need. -7. Expand destination-suite CI and candidate-installation validation, documenting - actual gates and coverage gaps as they become operational. Preserve existing - coverage until its intent has a validated destination; no mandatory overlap - period is needed solely for migration. Add enforced offline validation as an - infrastructure follow-up after provisioning supplies images and dependencies. +1. Consolidate the conformance library under `tests/suites/conformance` alongside + Cargo test entry points after standalone CLI removal. Update build and CI + references together. Keep component tests and SDK-native tests in their + existing trees; extract shared helpers only when consumers need them. +2. Add effective capability discovery and the result semantics above. + Migrations may proceed in parallel, but conformance qualification requires + reporting. Keep test contracts alongside their implementations. +3. Migrate by behavioral intent, in parallel by destination family. One PR may + add coverage and remove covered source tests; delete empty binaries and + split or rename residual ones. Preserve coverage until validated replacement + or explicit retirement, without a mandatory overlap period. +4. Track dependencies in #3954 and the Podman inventory in #3712. Retire the + temporary e2e-podman feature, exclusion list, and tmachine suite when its + source intent is covered or explicitly retired. Add no inventory machinery + unless source churn warrants it. +5. Expand CI and candidate-installation validation, then demonstrate offline + runs once provisioning supplies their dependencies. Document actual gates + and remaining gaps, not proposed gates as if already enforced. + +As implementation lands, `TESTING.md` owns routing and working commands, +`tests/CONFORMANCE.md` owns shared policy, suite READMEs own authoring and +execution guidance, and `CI.md` owns implemented CI behavior. Agent guidance +links to these sources. Per-test contracts stay in tests; detailed migration +inventories and PR coordination belong in tracking issues. ## Risks -- Optional capability reporting can hide regressions if an implementation simply - stops advertising support. Mandatory contracts remain test-owned; review - removal of advertised support explicitly as a product-contract change, not - merely as a way to avoid failing tests. -- Two Linux container drivers can pass a scenario containing OS assumptions. - Admission evidence is a starting point; broader platform validation and probe - review remain necessary. -- Global API state mutation can affect other workloads or test processes. - Document side effects, coordinate conflicts within a run, and recommend - restoration. Cross-process coordination is not designed here. -- Nix, provisioning, and test-archive changes create migration cost and merge - conflicts. Keep scenario contracts independent of provisioners and sequence - shared library changes before many concurrent migrations. -- Declaring a contract mandatory can turn an existing product limitation into a - visible failure. Decide the product contract explicitly rather than weakening - a test or treating setup failure as unsupported. +- Two Linux drivers can share accidental OS assumptions. Review probe portability + and expand platform coverage; a pass applies only to the tested configuration. +- Capability withdrawal can hide regressions. Review it as a contract change, + and do not weaken mandatory requirements to accommodate product limitations. +- Global-state changes can affect other workloads. Disclose side effects and + coordinate within a run; cross-process coordination remains undesigned. +- Shared-library and provisioning changes can conflict with parallel migrations. + Sequence common infrastructure changes before dependent work. ## Alternatives -### Maintain independent driver-specific E2E suites - -Each driver could retain a complete E2E suite tailored to its runtime. This -minimises initial migration, but duplicates public behavioral checks and allows -expectations to diverge. Reuse portable contracts across targets and reserve -driver-specific coverage for implementation and integration requirements. - -### Couple behavioral tests to target provisioning - -Each suite could provision and configure its own gateway. This simplifies local -setup for that suite, but makes it harder to validate installed artifacts or an -externally prepared gateway with the same tests. Separate target preparation -from behavioral testing while allowing a harness to orchestrate both. - -### Introduce separate API and CLI conformance frameworks immediately - -The older #2182 and #2873 proposals offer direct API coverage, but add another -runner and overlapping scenarios before a CLI limitation requires them. Keep -general conformance CLI-based initially; SDK interface testing remains a -separate justified consumer under #3238. - -### Require every tested behavior on every configuration - -This gives a uniform baseline but excludes useful portable behavior that some -drivers do not implement. Mandatory scenarios plus precise optional capabilities -allow useful coverage while requiring advertised behavior to pass. Whether -optional portable contracts belong in general conformance or feature-specific -suites is deferred; this alternative concerns support requirements, not naming. +- **Independent driver E2E suites:** less migration now, but duplicated public + checks and diverging expectations. Retain only genuinely driver-specific work. +- **Tests provision their own gateways:** convenient locally, but harder to reuse + against installed artifacts or externally prepared targets. +- **Separate API and CLI frameworks immediately:** adds overlapping runners + before a CLI limitation requires them; retain SDK-specific coverage separately. +- **Require every behavior everywhere:** uniform, but excludes useful optional + behavior. Require advertised support to work without yet deciding its category. ## Prior art - [Kubernetes conformance testing](https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/conformance-tests.md) - separates stable behavioral requirements from execution infrastructure and - defines promotion, reliability, version compatibility, and normative test - descriptions. Those are useful policy questions for OpenShell; its mandatory - GA baseline and governance process are not adopted wholesale here. -- #2925, #3768, and #3866 establish reusable OpenShell CLI scenarios, independent - selection, and installed Cargo execution. They provide implementation pieces - without independently defining the entire strategy. -- #3107 introduced sandbox-continuity testing and #3342 reverted that - infrastructure. Separating portable assertions from disruption actuators - remains useful without committing to the former implementation. -- SDK RFC #3238 treats SDK-native behavior as its own compatibility surface. - Shared gateway provisioning does not require identical language runners. + informs public-contract testing and per-test specifications. OpenShell uses + normal PR review rather than its promotion process, soak period, or governance, + and defers version-skew testing. +- OpenShell's reusable CLI scenarios and Cargo execution provide the starting + implementation. Earlier sandbox-continuity work motivates separating portable + recovery assertions from environment-specific disruption actuators. +- SDK proposal #3238 retains interface-specific compatibility concerns without + making the client interface a separate behavioral test family. ## Open questions -- Should portable, capability-dependent tests belong in general conformance or - feature-specific suites? Resolve this using the first concrete migration - example that requires the distinction, without adding a category in advance. -- What fixture reachability assumptions are needed for tests against externally - provisioned gateways, without requiring administrative access for assertions? -- Which source and integration jobs are required for merges and release - promotion? How should Windows, GPU, disruption, and load/scale runs be scheduled? -- Where should disruption and load/scale suites live, and what is the smallest - actuator interface needed by the first migration? -- Which fields beyond SHA and configuration become necessary when the first - report consumer needs completeness or cross-version comparisons? +- Where do portable capability-dependent tests belong? Decide from a concrete + migration example, without adding a category in advance. +- How should test-owned fixtures be reachable from externally prepared gateways? +- Which jobs gate merges and releases, and which expensive/platform-specific runs + are scheduled? What upgrade coverage is required? +- Where should disruption and load/scale suites live, and what actuator interface + does the first disruption migration need? +- What report fields beyond SHA and configuration do actual consumers require? From 91273c458dfe191277d66ea1551e57e32b400bac Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Tue, 6 Oct 2026 15:27:29 +0200 Subject: [PATCH 7/9] docs(rfc): clarify testing strategy and execution policy Signed-off-by: Evan Lezar --- rfc/0016-testing-strategy/README.md | 367 +++++++++++++++------------- 1 file changed, 194 insertions(+), 173 deletions(-) diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md index fc88b99a97..16f2e6f268 100644 --- a/rfc/0016-testing-strategy/README.md +++ b/rfc/0016-testing-strategy/README.md @@ -11,244 +11,265 @@ links: ## Summary -Organise tests by the behavior they verify, independently of how OpenShell is -deployed. Reuse public-contract tests across drivers and environments, keep -implementation-specific coverage separate, and make each test's contract and -each run's coverage explicit. +Establish a shared testing strategy for OpenShell so required public behavior +is verified consistently across drivers and deployment environments. Separate +conformance end-to-end tests from feature-specific and driver-specific coverage, and +separate target preparation from test execution. -This proposal defines test categories, conformance rules, and an execution model -that builds on Nix and tmachine without requiring tests to depend on them. -Adopt it incrementally; it does not describe CI gates already in place. +Run end-to-end tests on request for PRs and during release qualification. Adopt the +strategy incrementally using the existing infrastructure. ## Motivation -Existing E2E binaries mix public behavior, runtime inspection, external +Existing end-to-end binaries mix public behavior, runtime inspection, external integrations, and performance measurements. This makes coverage difficult to reuse across drivers and obscures whether a failure reflects the product or its test infrastructure. -A shared model lets contributors place tests by purpose and reviewers understand -what passing establishes. The Podman migration in #3712 is an initial consumer, -not the definition of OpenShell conformance. +Clear test categories help contributors decide where tests belong and reviewers +understand what they verify. ## Non-goals -- Implement the framework, capability API, or CI matrices in this PR. +- Implement the framework or CI matrices in this PR. - Classify every existing test; retain that work in migration issues. -- Introduce certification, a review board, or version-skew testing. -- Replace the SDK compatibility design in #3238. +- Introduce certification or a review board. +- Define a cross-version compatibility matrix. +- Define SDK-specific compatibility testing. +- Define benchmark execution policy or performance thresholds. ## Proposal -### 1. Organise tests by contract +### 1. Organise tests by family -A behavioral contract specifies an operation's expected observable result under -stated conditions. A test case exercises it, an assertion checks an observation, -and a suite groups related cases. For example, deleting a sandbox must remove -it from the sandbox list; an assertion checks that its identifier is absent. +Test families describe the scope and purpose of a check. Use them to decide +where a test belongs and what a passing result establishes. -| Family | Purpose | +| Test Family | Description | | --- | --- | -| Unit and component integration | Internal logic and implementation mechanics, at the lowest effective layer. | -| General conformance | Public behavioral contracts across drivers and environments. | -| Feature-specific | Features requiring configured external integration or currently implemented on only one driver. | -| Driver-specific | Driver configuration, runtime and host integration, and implementation contracts. | -| Disruption conformance | Portable continuity or recovery behavior under deliberately induced disruption. | -| Load/scale | Performance and scaling measurements, separate from behavioral correctness. | - -The client interface is not a test category: CLI and SDK tests can exercise the -same public contract. SDK-specific semantics retain focused coverage. Security -tests span these families; introduce a separate family only if tests need shared -specialized infrastructure. Performance thresholds belong in load/scale unless -the deadline is itself a public contract. - -### 2. Define conformance through public behavior - -Each test owns its contract: keep its stable identity, preconditions, expected -behavior, mandatory/optional status, and significant side effects alongside the -test. Do not maintain a duplicate specification elsewhere. - -General conformance uses an already configured gateway, initially through the -CLI until direct API access is necessary. Assert observable behavior and -structured output, not incidental presentation or native runtime details. -Pass/fail requires only public CLI/API access and sandbox operations. -Administrative access belongs to provisioning, diagnostics, or disruption -actuators; unavailable diagnostics must not affect results. - -Tests must not change gateway startup configuration. They may mutate public -API-managed state, using unique names and cleanup and avoiding conflicting -global-setting changes within a run. Document global effects; restoring prior -global settings is recommended, not mandatory. - -Test-owned short-lived services are allowed. External services requiring gateway -configuration belong in feature-specific suites. Behavioral assertions must not -depend on public internet services; provisioning may download dependencies and -images. Use existing workload images and a small common toolset, supplied by the -harness. Missing prerequisites are setup errors. Dedicated fixture images and -enforced offline validation are follow-ups, not migration gates. - -General-conformance admission requires evidence from at least two drivers; -Docker and Podman suffice initially. Then run against every supported driver -with available infrastructure, recording gaps. Feature tests need one -representative driver. Disruption tests need one working actuator initially; -an absent actuator prevents selection, while a broken configured actuator is -an infrastructure error. - -Use normal PR review, without a soak period or separate promotion PR. Resolve -known flakiness rather than hiding it with retries. Changes or removals must -distinguish test corrections from changes to promised behavior, including -withdrawal of advertised support. - -### 3. Make applicability and coverage explicit - -The gateway must report effective capabilities for its running configuration -through the public API and machine-readable CLI output. Discovery failure aborts -conformance. Start with flat, namespaced booleans; defer hierarchy, parameters, -and profiles. Capabilities describe product behavior, not test selectors, -credentials, or external-service prerequisites. - -Portable, non-universal behavior is a reason to extend capability reporting; -prefer adding the capability and its test together. Whether those tests belong -in general conformance or feature-specific suites remains open until a concrete -migration example requires a decision. - -| Result | Meaning | +| Lint | Check formatting, style, and static rules. | +| Unit | Verify one component in isolation. Place tests inline in its Rust module or in an adjacent `test.rs` file under `src/`. | +| Integration | Verify interactions between components. Place Rust integration tests in the crate's top-level `tests/` directory. | +| End-to-end (e2e) | Verify a configured OpenShell target through a client interface. Place suites in `e2e/suites/`. | +| Benchmark | Measure performance or scale. Place crate benchmarks in crate-level `benchmarks/` directories and full-system benchmarks in a root `benchmarks/` directory. | + +Within the e2e family, group tests by the behavior they verify. This separates +reusable public behavior checks from checks tied to particular integrations +or drivers. + +| e2e type | What it verifies | Proposed folder | +| --- | --- | --- | +| Conformance | Required public behavior, including continuity across gateway restarts and upgrades. | `e2e/suites/conformance/` | +| Feature | Behavior requiring a named external service or special gateway configuration. | `e2e/suites/features/` | +| Driver | Behavior specific to a driver, its host integration, or its configuration. | `e2e/suites/drivers/` | + +Conformance may include disruption tests. Add these under +`e2e/suites/disruption/` initially, with their harness developed separately. + +### 2. Define conformance e2e requirements + +Conformance is the e2e test type that verifies required public behavior against +a configured gateway. Each test must pass; a failure indicates a conformance +gap. These tests provide common expectations across supported targets. + +Conformance tests can be grouped into broad behavioral areas. The following +section gives examples. + +#### Conformance areas + +These user stories illustrate each area from a user or administrator's +perspective. Tests requiring external services or special gateway configuration +belong in feature-specific suites. + +| Area | User stories | | --- | --- | -| Passed | The behavioral assertions held. | -| Failed | Mandatory support is absent or an exercised contract was violated. | -| Unsupported | Optional support was not advertised. | -| Skipped | The test was deliberately excluded. | -| Error | Infrastructure or fixture failure prevented evaluation. | - -Advertised behavior must pass even when support is optional. A complete run -accounts for the entire identified suite without filtering; evaluated optional -unsupported cases count as accounted for. Filtered or interrupted runs are -partial. Completeness is not success: failures and errors cannot yield a complete -pass. Advisory CI policy must not relabel failed tests as passing. - -Initially, suite, CLI, and gateway use the same OpenShell revision; no -cross-version claim is made. Start reports with that SHA and the tmachine -configuration, or equivalent target identity. Record mock targets as mocks, -not evidence for production drivers. - -### 4. Separate target preparation from test execution - -A run combines a configured target with a selected suite. Platform, driver, -environment, gateway configuration, installation method, and client interface -may vary without changing the contract. Rootful and rootless Podman are two -environments of one driver, not two-driver evidence. +| Sandbox lifecycle and continuity | I can create, inspect, start, stop, and delete a sandbox. I can observe workload status and exit codes. My sandbox retains its expected state across stops and gateway restarts, including upgrades. Ephemeral sandboxes are removed when their workloads finish. | +| Sandbox I/O | I can execute commands with input and receive complete output and exit status. I can connect interactively and reconnect as supported. I can upload and download files safely. I can forward a port to a sandbox service. | +| Workspaces and resource management | I can create, inspect, and delete workspaces. My resources remain scoped to their workspace. I can label and filter resources. Deletion guards prevent accidental removal of resources still in use. | +| Policy management | I can validate, apply, inspect, and update policy. I can see which policy is effective and when an update takes effect. Invalid policy is rejected without weakening existing protection. | +| Sandbox enforcement | My workload can access permitted files, processes, and network destinations. Prohibited access is blocked. Updating policy changes enforcement as promised. | +| Providers | I can create, attach, update, detach, and delete providers. Authorized workloads receive usable credentials without exposing secrets. Credential changes and revocation take effect as promised. | +| Identity and authorization | I can authenticate and access resources I am authorized to use. Unauthorized requests are rejected. Access through another client interface does not bypass authorization. | +| Settings | I can read, change, and remove settings through public APIs. I can inspect effective values. Global overrides take precedence and prevent conflicting sandbox changes. | +| Middleware | My requests use the selected middleware in the configured order. Transformations and failure handling follow the configured behavior. | +| Interceptors | My gateway requests are transformed or rejected as configured. Interception preserves authorization boundaries. | + +#### Test requirements + +Each conformance test documents its expected behavior, preconditions, required +permissions, and significant side effects alongside its implementation. Use a +descriptive, stable test name that identifies the behavior being verified and +allows results to be tracked across runs. + +Tests exercise a configured gateway through public client interfaces. Assertions +verify observable behavior and structured output. They must not depend on native +runtime details, incidental presentation, or diagnostic availability. + +Tests exercise public APIs within their declared permissions. Keep end-user and +administrative interactions in separate tests. Tests may change resources and +settings through public APIs, but must not change deployment configuration. + +Tests must clean up their changes. Behavioral checks must not depend on public +internet services. + +Run tests across supported targets. Changes or removals must distinguish +corrections to tests from changes to promised behavior. + +#### Failure handling + +Investigate each failure and track the underlying product, test, or +infrastructure issue. Notify the responsible maintainers of new failures. +Do not automatically retry failed tests; intermittent failures may expose races +that affect larger installations. + +Failures may be explicitly waived. Record each waiver's scope, rationale, and +tracking issue, and retain the failed result. + +Retain test results, failure logs, revision, target configuration, and run +duration for investigation. + +### 3. Separate target preparation from test execution + +Prepare the target separately from test execution so the same suite can run +against locally provisioned or externally managed gateways. + +Target preparation supplies deployment configuration, credentials, images, and +tools. Test runs accept the gateway endpoint and credentials. Host and runtime +access supports provisioning, diagnostics, and disruption mechanisms. + +Tests must not depend on a particular provisioning tool. Conformance requirements +remain the same across supported target configurations. ```mermaid flowchart LR - subgraph preparation[Target preparation] - machine[Machine / base image] --> setup[Environment setup] - setup --> install[Install and configure OpenShell] - end - artifacts[Build artifacts] --> install - install --> target[Configured OpenShell target] + preparation[Prepare target] --> target[Configured gateway] external[External provisioning] --> target suite[Test suite] --> client[CLI or SDK] client -->|exercises| target ``` +### 4. Select and trigger test runs + +Test execution depends on the purpose of the run: + +| Context | Tests to run | +| --- | --- | +| Pull requests | Automatically run lint, unit, and component integration checks. Run e2e tests when requested. | +| Release qualification | Run those checks and all applicable e2e tests for dev and stable releases. | +| Additional schedules | Add scheduled runs where further coverage is needed. | + +Allow conformance, feature-specific, and driver-specific tests to be requested +independently or together. Conformance includes disruption tests. Add separate +disruption, feature, or driver subcategory selection only when run durations +justify it. + +Release qualification validates installation and uninstall behavior using +candidate artifacts. Upgrade qualification verifies continuity across gateway +replacement. + +## Implementation plan + +### Execution infrastructure + Nix pins build inputs and produces artifacts and test archives. Tmachine composes a `Machine` and `Environment` for setup, an `Installer` for installation, and a `Testsuite` for execution. Other provisioners may supply targets tmachine -cannot represent. General conformance must also run against externally prepared -gateways. +cannot represent. -CI selects meaningful combinations, not a full cross-product, and reuses setup -definitions rather than duplicating them. Keep source checks and affected SDK -checks; protobuf changes require all affected SDK checks. Packaging selection -must include transitive inputs and broaden when impact is uncertain. -Release validation installs exact candidate artifacts before running applicable -suites. Installation, upgrade, and uninstall behavior need separate assertions; -successful conformance alone does not validate packaging. +CI reuses provisioning definitions across selected target configurations. -## Implementation plan +### Layout and migration + +Existing conformance tests use the CLI against a configured gateway. Planned +work will extend coverage to direct API usage. -Build on the existing layout; only the marked documents are proposed additions: +Move suites to `e2e/suites/`; keep provisioning under `tests/`: ```text tests/ -├── config.nix # tmachine definitions -├── artifacts.nix # Artifact construction -├── ansible/ # Provisioning and execution -├── CONFORMANCE.md # Proposed: agreed policy +├── config.nix +├── artifacts.nix +└── ansible/ +e2e/ └── suites/ ├── conformance/ - │ ├── cli/ # CLI test entry points - │ └── README.md # Proposed: contributor guidance + │ ├── cli/ + │ └── README.md # Requirements, authoring, and execution + ├── disruption/ # Added in a later phase ├── drivers/podman/ └── features/provider-refresh/keycloak/ ``` -1. Consolidate the conformance library under `tests/suites/conformance` alongside - Cargo test entry points after standalone CLI removal. Update build and CI - references together. Keep component tests and SDK-native tests in their - existing trees; extract shared helpers only when consumers need them. -2. Add effective capability discovery and the result semantics above. - Migrations may proceed in parallel, but conformance qualification requires - reporting. Keep test contracts alongside their implementations. -3. Migrate by behavioral intent, in parallel by destination family. One PR may - add coverage and remove covered source tests; delete empty binaries and - split or rename residual ones. Preserve coverage until validated replacement - or explicit retirement, without a mandatory overlap period. -4. Track dependencies in #3954 and the Podman inventory in #3712. Retire the - temporary e2e-podman feature, exclusion list, and tmachine suite when its - source intent is covered or explicitly retired. Add no inventory machinery - unless source churn warrants it. -5. Expand CI and candidate-installation validation, then demonstrate offline - runs once provisioning supplies their dependencies. Document actual gates - and remaining gaps, not proposed gates as if already enforced. - -As implementation lands, `TESTING.md` owns routing and working commands, -`tests/CONFORMANCE.md` owns shared policy, suite READMEs own authoring and -execution guidance, and `CI.md` owns implemented CI behavior. Agent guidance -links to these sources. Per-test contracts stay in tests; detailed migration -inventories and PR coordination belong in tracking issues. +1. Classify and migrate existing tests by behavioral intent, preserving coverage + and updating build and CI references together. Track migration work and + conformance gaps in implementation issues. +2. Assess conformance tests against the requirements and document their + contracts. Validate execution against externally prepared gateways and ensure + behavioral checks do not depend on public internet services. Resolve fixture + networking as part of this work. +3. Retain results and diagnostic evidence, notify maintainers of new failures, + and support issue tracking and explicit waivers. Preserve failed results + without automatic retries. +4. Implement independent and combined PR selection for the three e2e streams. + Run applicable suites during dev and stable release qualification, including + candidate installation and uninstall checks. +5. Update testing and CI documentation as each part is implemented. Record + actual gates and remaining gaps. + +### CI selection + +PR labels select e2e streams: `test:e2e/conformance` runs conformance tests, +including disruption tests when added, `test:e2e/features` runs feature-specific +tests, and +`test:e2e/drivers` runs driver-specific tests. `test:e2e` selects all three +streams. Migrate existing GPU and Kubernetes labels according to test intent. + +### Later phase: disruption tests + +Add restart and upgrade continuity tests in a later phase. Design their harness +separately from the migration of existing suites. Include these tests in +conformance PR runs and release qualification. Consolidate shared contracts and +helpers when practical. + +### Follow-up work + +Package the conformance suite to run without a source checkout as a follow-up. + +### Documentation + +`TESTING.md` defines test families and links to suite READMEs. +`e2e/suites/conformance/README.md` owns conformance requirements, authoring, and +execution guidance. `CI.md` records merge and release gates as qualification +workflows are implemented. Agent guidance links to these sources. ## Risks - Two Linux drivers can share accidental OS assumptions. Review probe portability and expand platform coverage; a pass applies only to the tested configuration. -- Capability withdrawal can hide regressions. Review it as a contract change, - and do not weaken mandatory requirements to accommodate product limitations. -- Global-state changes can affect other workloads. Disclose side effects and - coordinate within a run; cross-process coordination remains undesigned. - Shared-library and provisioning changes can conflict with parallel migrations. Sequence common infrastructure changes before dependent work. ## Alternatives -- **Independent driver E2E suites:** less migration now, but duplicated public +- **Independent driver e2e suites:** less migration now, but duplicated public checks and diverging expectations. Retain only genuinely driver-specific work. - **Tests provision their own gateways:** convenient locally, but harder to reuse against installed artifacts or externally prepared targets. - **Separate API and CLI frameworks immediately:** adds overlapping runners - before a CLI limitation requires them; retain SDK-specific coverage separately. -- **Require every behavior everywhere:** uniform, but excludes useful optional - behavior. Require advertised support to work without yet deciding its category. + before shared execution needs are understood; retain SDK-specific coverage + separately. ## Prior art - [Kubernetes conformance testing](https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/conformance-tests.md) informs public-contract testing and per-test specifications. OpenShell uses normal PR review rather than its promotion process, soak period, or governance, - and defers version-skew testing. + and defers a cross-version compatibility matrix. - OpenShell's reusable CLI scenarios and Cargo execution provide the starting implementation. Earlier sandbox-continuity work motivates separating portable recovery assertions from environment-specific disruption actuators. -- SDK proposal #3238 retains interface-specific compatibility concerns without - making the client interface a separate behavioral test family. ## Open questions -- Where do portable capability-dependent tests belong? Decide from a concrete - migration example, without adding a category in advance. -- How should test-owned fixtures be reachable from externally prepared gateways? -- Which jobs gate merges and releases, and which expensive/platform-specific runs - are scheduled? What upgrade coverage is required? -- Where should disruption and load/scale suites live, and what actuator interface - does the first disruption migration need? -- What report fields beyond SHA and configuration do actual consumers require? +None. Fixture networking and disruption harness design are deferred to +implementation. From fed2b98d7a787dd3a64801a1d97ce0185c3b1295 Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Tue, 13 Oct 2026 13:51:42 +0200 Subject: [PATCH 8/9] docs(rfc): clarify polling and test dependencies Signed-off-by: Evan Lezar --- rfc/0016-testing-strategy/README.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md index 16f2e6f268..45be6cb31f 100644 --- a/rfc/0016-testing-strategy/README.md +++ b/rfc/0016-testing-strategy/README.md @@ -122,6 +122,9 @@ infrastructure issue. Notify the responsible maintainers of new failures. Do not automatically retry failed tests; intermittent failures may expose races that affect larger installations. +Tests may poll for documented asynchronous transitions within a defined timeout. +They must not retry failed assertions merely to obtain a pass. + Failures may be explicitly waived. Record each waiver's scope, rationale, and tracking issue, and retain the failed result. @@ -134,8 +137,11 @@ Prepare the target separately from test execution so the same suite can run against locally provisioned or externally managed gateways. Target preparation supplies deployment configuration, credentials, images, and -tools. Test runs accept the gateway endpoint and credentials. Host and runtime -access supports provisioning, diagnostics, and disruption mechanisms. +tools. Supply images and tools without requiring public internet access during +test execution, for example through preloading or configured mirrors. + +Test runs accept the gateway endpoint and credentials. Host and runtime access +supports provisioning, diagnostics, and disruption mechanisms. Tests must not depend on a particular provisioning tool. Conformance requirements remain the same across supported target configurations. From b06ce382be4f0757f839b8161a1b637ffcf8217f Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Wed, 11 Nov 2026 16:13:11 +0100 Subject: [PATCH 9/9] docs(rfc): align testing strategy with testkit and tmachine Update RFC 0016 to the current tests/ layout, the testkit and runner execution model, the openshell-conformance program, and the in-progress migration tracked in #5281. Document the #[ignore] process as the only skip mechanism and record open questions. Signed-off-by: Evan Lezar --- rfc/0016-testing-strategy/README.md | 270 ++++++++++++++++++++-------- 1 file changed, 196 insertions(+), 74 deletions(-) diff --git a/rfc/0016-testing-strategy/README.md b/rfc/0016-testing-strategy/README.md index 45be6cb31f..6417f7d018 100644 --- a/rfc/0016-testing-strategy/README.md +++ b/rfc/0016-testing-strategy/README.md @@ -4,6 +4,7 @@ authors: state: draft links: - https://github.com/NVIDIA/OpenShell/issues/3954 + - https://github.com/NVIDIA/OpenShell/issues/5281 - https://github.com/NVIDIA/OpenShell/pull/3460 --- @@ -19,12 +20,21 @@ separate target preparation from test execution. Run end-to-end tests on request for PRs and during release qualification. Adopt the strategy incrementally using the existing infrastructure. +Much of the execution model already exists: tests written with testkit under +`tests/suites/`, run in tmachine environments by `openshell-test-runner`, or +against any gateway with the `openshell-conformance` program. This RFC records +the policy those pieces implement and the direction for what remains. The +migration of the legacy `e2e/rust` tests is tracked in +[#5281](https://github.com/NVIDIA/OpenShell/issues/5281). + ## Motivation -Existing end-to-end binaries mix public behavior, runtime inspection, external -integrations, and performance measurements. This makes coverage difficult to -reuse across drivers and obscures whether a failure reflects the product or -its test infrastructure. +The legacy end-to-end tests in `e2e/rust` (about 70 files) mix public behavior, +runtime inspection, external integrations, and performance measurements. This +makes coverage difficult to reuse across drivers and obscures whether a failure +reflects the product or its test infrastructure. The suites already moved to +testkit separate these concerns, but the rules for what belongs where are +written down only in `TESTING.md` and a suite README. Clear test categories help contributors decide where tests belong and reviewers understand what they verify. @@ -32,7 +42,9 @@ understand what they verify. ## Non-goals - Implement the framework or CI matrices in this PR. -- Classify every existing test; retain that work in migration issues. +- Classify every existing test or set the migration order; retain that work in + migration issues such as + [#5297](https://github.com/NVIDIA/OpenShell/issues/5297). - Introduce certification or a review board. - Define a cross-version compatibility matrix. - Define SDK-specific compatibility testing. @@ -48,23 +60,36 @@ where a test belongs and what a passing result establishes. | Test Family | Description | | --- | --- | | Lint | Check formatting, style, and static rules. | -| Unit | Verify one component in isolation. Place tests inline in its Rust module or in an adjacent `test.rs` file under `src/`. | +| Unit | Verify one component in isolation. Place tests inline in its Rust module or in an adjacent test module under `src/`. | | Integration | Verify interactions between components. Place Rust integration tests in the crate's top-level `tests/` directory. | -| End-to-end (e2e) | Verify a configured OpenShell target through a client interface. Place suites in `e2e/suites/`. | -| Benchmark | Measure performance or scale. Place crate benchmarks in crate-level `benchmarks/` directories and full-system benchmarks in a root `benchmarks/` directory. | +| End-to-end (e2e) | Verify a configured OpenShell target through a client interface. Place suites in `tests/suites/`. | +| Benchmark | Measure performance or scale. The location is decided when the first benchmark returns; the legacy live-Internet benchmarks were removed pending [#5472](https://github.com/NVIDIA/OpenShell/issues/5472). | + +A test that needs no running gateway or sandbox, such as a check of CLI +preflight output against a fake `docker` on `PATH`, belongs in a unit or +integration test of the crate that owns the behavior, not in e2e. Within the e2e family, group tests by the behavior they verify. This separates reusable public behavior checks from checks tied to particular integrations or drivers. -| e2e type | What it verifies | Proposed folder | +| e2e type | What it verifies | Folder | | --- | --- | --- | -| Conformance | Required public behavior, including continuity across gateway restarts and upgrades. | `e2e/suites/conformance/` | -| Feature | Behavior requiring a named external service or special gateway configuration. | `e2e/suites/features/` | -| Driver | Behavior specific to a driver, its host integration, or its configuration. | `e2e/suites/drivers/` | +| Conformance | Required public behavior, including continuity across gateway restarts and upgrades. | `tests/suites/conformance/`, one crate and one workspace | +| Feature | Behavior requiring a named external service or special gateway configuration. | `tests/suites/features/`, one crate per suite in one workspace | +| Driver | Behavior specific to a driver, its host integration, or its configuration. | `tests/suites/drivers//`, one workspace per driver | + +Shared test code sits beside the suites. `tests/testkit/` holds the support +library and the `#[testkit::test]` attribute, and helpers for OpenShell itself +(sandboxes, policies, providers, the CLI) belong there. `tests/support/` holds +fixtures for services outside OpenShell, such as Keycloak or a host-side +recording server, in crates named `-support` that any suite can use. Conformance may include disruption tests. Add these under -`e2e/suites/disruption/` initially, with their harness developed separately. +`tests/suites/disruption/` initially, with their harness developed separately. +Tests that restart or reconfigure the gateway cannot be conformance tests, +because conformance tests do not change deployment configuration; their family +is decided when the first such migration lands. ### 2. Define conformance e2e requirements @@ -97,23 +122,33 @@ belong in feature-specific suites. #### Test requirements Each conformance test documents its expected behavior, preconditions, required -permissions, and significant side effects alongside its implementation. Use a -descriptive, stable test name that identifies the behavior being verified and -allows results to be tracked across runs. +permissions, and significant side effects alongside its implementation; testkit +requires a doc comment on every test. Use a descriptive, stable test name that +identifies the behavior being verified and allows results to be tracked across +runs. Tests exercise a configured gateway through public client interfaces. Assertions verify observable behavior and structured output. They must not depend on native -runtime details, incidental presentation, or diagnostic availability. +runtime details, incidental presentation, or diagnostic availability. Assert +only what the proto comments and published documentation promise. Assert +workload `phase` and `exit_code`, not condition reasons or message text. Check +that something did not happen by reading the state that shows it, not by +sleeping. If a behavior is undocumented, document it before testing it. Tests exercise public APIs within their declared permissions. Keep end-user and administrative interactions in separate tests. Tests may change resources and settings through public APIs, but must not change deployment configuration. -Tests must clean up their changes. Behavioral checks must not depend on public -internet services. +Tests must clean up their changes, including after a failure or panic, and use +names unique to the run so that they could run in parallel against one gateway. +A test that changes a gateway-wide setting restores it on cleanup and relies on +serial execution until the runner supports parallel groups. Behavioral checks +must not depend on public internet services. -Run tests across supported targets. Changes or removals must distinguish -corrections to tests from changes to promised behavior. +A behavior that only some drivers or environments provide is not conformance; +place it in a feature or driver suite. Run tests across supported targets. +Changes or removals must distinguish corrections to tests from changes to +promised behavior. #### Failure handling @@ -125,11 +160,24 @@ that affect larger installations. Tests may poll for documented asynchronous transitions within a defined timeout. They must not retry failed assertions merely to obtain a pass. -Failures may be explicitly waived. Record each waiver's scope, rationale, and -tracking issue, and retain the failed result. - Retain test results, failure logs, revision, target configuration, and run -duration for investigation. +duration for investigation. CI uploads the JUnit report, the nextest log, and +the guest's diagnostics after a failure as an `e2e-results-*` artifact named by +suite, environment, and installer. + +#### Ignoring a test + +A broken or flaky test is ignored, not skipped conditionally. To ignore one: + +1. Open a tracking issue that describes the failure. +2. Mark the test `#[ignore = ""]`, so the reason is in + source. +3. Remove the marker in the change that fixes the issue. + +An ignored test does not run in any environment and shows as skipped in the +test report. The marker is global: a limitation specific to one environment is +a reason to move the test to a feature or driver suite, not to ignore it. This +is the only skip mechanism. ### 3. Separate target preparation from test execution @@ -140,17 +188,37 @@ Target preparation supplies deployment configuration, credentials, images, and tools. Supply images and tools without requiring public internet access during test execution, for example through preloading or configured mirrors. -Test runs accept the gateway endpoint and credentials. Host and runtime access -supports provisioning, diagnostics, and disruption mechanisms. +Tests find the gateway the way the CLI does, through `OPENSHELL_GATEWAY`, +`OPENSHELL_GATEWAY_ENDPOINT`, or the active gateway, and run the CLI named by +`OPENSHELL_BIN`. There is no separate target description. Host and runtime +access supports provisioning, diagnostics, and disruption mechanisms. Tests must not depend on a particular provisioning tool. Conformance requirements remain the same across supported target configurations. +A service outside OpenShell that a feature suite needs is started by the +environment, which publishes how to reach it as `OPENSHELL_TEST__*` +variables; the suite's fixture in `tests/support/` reads them, so the same suite +runs against any deployment that sets them. A short-lived server that serves +one test is a test-owned helper instead. A test whose upstream runs on the +machine executing the tests requires sandboxes to reach that machine as +`host.openshell.internal`, which a remote cluster may not allow; see the open +questions. + +To check a gateway without cargo, nextest, or a source checkout, the conformance +crate builds `openshell-conformance`, a program that finds the gateway as the +tests do and offers `list`, `ready`, and `run` with a JUnit report. CI builds it +once per architecture and passes it to every conformance row. + ```mermaid flowchart LR - preparation[Prepare target] --> target[Configured gateway] + subgraph preparation[Target preparation] + machine[Machine and environment] --> install[Installer] + end + artifacts[Build artifacts] --> install + install --> target[Configured gateway] external[External provisioning] --> target - suite[Test suite] --> client[CLI or SDK] + suite[Test suite or program] --> client[CLI or SDK] client -->|exercises| target ``` @@ -164,9 +232,14 @@ Test execution depends on the purpose of the run: | Release qualification | Run those checks and all applicable e2e tests for dev and stable releases. | | Additional schedules | Add scheduled runs where further coverage is needed. | -Allow conformance, feature-specific, and driver-specific tests to be requested -independently or together. Conformance includes disruption tests. Add separate -disruption, feature, or driver subcategory selection only when run durations +Each e2e type runs as its own CI job, so a failure names the type that failed. +Branch E2E runs `conformance-integration`, `feature-specific-integration`, and +`driver-specific-integration` when the `test:e2e` label is set; the conformance +and feature jobs also run in Release Dev and release qualification, while the +driver suites run only in Branch E2E. All three call the reusable +`e2e-suites.yml` workflow with a `family` input that selects rows from an +explicit matrix of environments and installers. Conformance includes disruption +tests. Add separate labels for requesting a single type only when run durations justify it. Release qualification validates installation and uninstall behavior using @@ -179,56 +252,77 @@ replacement. Nix pins build inputs and produces artifacts and test archives. Tmachine composes a `Machine` and `Environment` for setup, an `Installer` for installation, -and a `Testsuite` for execution. Other provisioners may supply targets tmachine -cannot represent. - -CI reuses provisioning definitions across selected target configurations. +and a `Testsuite` for execution. `openshell-test-runner` boots one environment +with OpenShell installed, runs a test family inside it with nextest, or runs a +prebuilt `openshell-conformance` program or test bundle, and writes a JUnit +report. A run passes if nextest succeeds, fails on a test failure, and is an +error when the environment cannot complete the run or no test ran. Other +provisioners may supply targets tmachine cannot represent. + +CI reuses provisioning definitions across selected target configurations. The +rows are an explicit matrix in `e2e-suites.yml`; generating the matrix from a +list of tests and environments, and checking that every expected test ran, are +planned after the legacy suites are retired. ### Layout and migration -Existing conformance tests use the CLI against a configured gateway. Planned -work will extend coverage to direct API usage. +Existing conformance tests use the CLI and the SDK against a configured gateway. +Planned work will extend coverage to direct API usage. -Move suites to `e2e/suites/`; keep provisioning under `tests/`: +Provisioning, runner, and suites share the `tests/` tree: ```text tests/ -├── config.nix -├── artifacts.nix -└── ansible/ -e2e/ +├── tmachine/ # Machines, environments, installers, Ansible +├── runner/ # openshell-test-runner +├── testkit/ # Test library and #[testkit::test] +├── support/ # Fixtures for services outside OpenShell └── suites/ - ├── conformance/ - │ ├── cli/ - │ └── README.md # Requirements, authoring, and execution - ├── disruption/ # Added in a later phase - ├── drivers/podman/ - └── features/provider-refresh/keycloak/ + ├── conformance/ # Requirements, authoring, execution: README.md + ├── features/ # keycloak-provider-refresh, host-gateway, oci-image, ... + ├── disruption/ # Added in a later phase + └── drivers/podman/ # One workspace per driver ``` -1. Classify and migrate existing tests by behavioral intent, preserving coverage - and updating build and CI references together. Track migration work and - conformance gaps in implementation issues. -2. Assess conformance tests against the requirements and document their - contracts. Validate execution against externally prepared gateways and ensure - behavioral checks do not depend on public internet services. Resolve fixture - networking as part of this work. -3. Retain results and diagnostic evidence, notify maintainers of new failures, - and support issue tracking and explicit waivers. Preserve failed results - without automatic retries. -4. Implement independent and combined PR selection for the three e2e streams. +The legacy `e2e/rust` tests and the tmachine `e2e-podman` testsuite remain until +their tests migrate. Progress: + +1. **Done.** Testkit and cleanup hooks, the runner, conformance on testkit and + required on Docker, Podman rootful and rootless, k3s, and HA k3s, host lanes + running `openshell-conformance`, the Keycloak, OCI image, host-gateway, and + Kubernetes update feature suites, the Podman user-namespace driver suite, and + per-run bundles so rows do not rebuild tests. +2. **In progress.** Migrate `e2e/rust` by behavioral intent in batches, each + deleting the files it ports, preserving coverage until a validated + replacement exists or the test is explicitly retired. The first batch ported + credential gating; open PRs move the driver-agnostic sandbox lifecycle and + API tests, and the CLI exec, attach, and SSH tests, into conformance and + features. Driver-agnostic files go to conformance, driver-specific behavior to + driver suites, and benchmarks and tests that check the test environment + rather than the product are dropped or moved to crate tests. Batch order and + per-file decisions live in + [#5297](https://github.com/NVIDIA/OpenShell/issues/5297) and + [#5281](https://github.com/NVIDIA/OpenShell/issues/5281), not here. +3. **Next.** Retire the interim `e2e-podman` testsuite and its exclusion list + when its intent is covered or explicitly retired + ([#3712](https://github.com/NVIDIA/OpenShell/issues/3712)), add driver + suites for Docker and Kubernetes where behavior is driver-specific, and + decide a home for gateway-reconfiguration and disruption tests. +4. Assess conformance tests against the requirements, and validate execution + against externally prepared gateways without depending on public internet + services. Resolve fixture reachability as part of this work. +5. Notify maintainers of new failures and track ignored tests through their issues. Run applicable suites during dev and stable release qualification, including candidate installation and uninstall checks. -5. Update testing and CI documentation as each part is implemented. Record +6. Update testing and CI documentation as each part is implemented. Record actual gates and remaining gaps. ### CI selection -PR labels select e2e streams: `test:e2e/conformance` runs conformance tests, -including disruption tests when added, `test:e2e/features` runs feature-specific -tests, and -`test:e2e/drivers` runs driver-specific tests. `test:e2e` selects all three -streams. Migrate existing GPU and Kubernetes labels according to test intent. +Today one `test:e2e` label runs all e2e types in Branch E2E, and each type is its +own job. The `test:e2e-gpu` and `test:e2e-kubernetes` labels select legacy lanes. +Migrate those labels according to test intent as `e2e/rust` retires. Introduce a +per-type label only when run durations justify it. ### Later phase: disruption tests @@ -239,13 +333,15 @@ helpers when practical. ### Follow-up work -Package the conformance suite to run without a source checkout as a follow-up. +Run conformance against a gateway without an image override. Today a gateway +outside the CI lanes must set `OPENSHELL_TEST_PYTHON_IMAGE`. ### Documentation `TESTING.md` defines test families and links to suite READMEs. -`e2e/suites/conformance/README.md` owns conformance requirements, authoring, and -execution guidance. `CI.md` records merge and release gates as qualification +`tests/suites/conformance/README.md` owns conformance requirements, authoring, and +execution guidance; `TESTING.md` should link to it rather than repeat the +assertion rules. `CI.md` records merge and release gates as qualification workflows are implemented. Agent guidance links to these sources. ## Risks @@ -254,6 +350,16 @@ workflows are implemented. Agent guidance links to these sources. and expand platform coverage; a pass applies only to the tested configuration. - Shared-library and provisioning changes can conflict with parallel migrations. Sequence common infrastructure changes before dependent work. +- Legacy and testkit suites run side by side during migration, and the legacy + host lanes differ from the tmachine rows. A migrated test must keep its + coverage on the lanes that ran it; a deleted file needs a recorded outcome. +- The runner runs one test at a time with one timeout profile. Families that + need longer timeouts, test groups, or parallelism, such as Kubernetes + high-availability tests, need per-family runner configuration first + ([#5295](https://github.com/NVIDIA/OpenShell/issues/5295)). +- An explicit matrix cannot show that a suite was left out of CI. Until matrices + are generated and expected tests are checked, reviewers confirm that a new + suite has a row. ## Alternatives @@ -264,6 +370,12 @@ workflows are implemented. Agent guidance links to these sources. - **Separate API and CLI frameworks immediately:** adds overlapping runners before shared execution needs are understood; retain SDK-specific coverage separately. +- **Declared requirements, capabilities, and per-environment waivers:** let one + test adapt to an environment, but add a declarative layer before any test + needs it. A test that cannot pass everywhere moves to a feature or driver + suite, and a broken test is ignored with a tracking issue. Revisit if a + portable test needs an environment condition + ([#5409](https://github.com/NVIDIA/OpenShell/issues/5409)). ## Prior art @@ -271,11 +383,21 @@ workflows are implemented. Agent guidance links to these sources. informs public-contract testing and per-test specifications. OpenShell uses normal PR review rather than its promotion process, soak period, or governance, and defers a cross-version compatibility matrix. -- OpenShell's reusable CLI scenarios and Cargo execution provide the starting - implementation. Earlier sandbox-continuity work motivates separating portable - recovery assertions from environment-specific disruption actuators. +- The first conformance suites came from reusable CLI scenarios and Cargo + execution; they now run as testkit tests. Earlier sandbox-continuity work + motivates separating portable recovery assertions from environment-specific + disruption actuators. ## Open questions -None. Fixture networking and disruption harness design are deferred to -implementation. +- How should conformance treat a gateway whose sandboxes cannot reach the test + machine, such as a remote Kubernetes cluster? Options are an opt-in declared + by the environment, a move of host-dependent tests to a feature suite, or an + up-front reachability check that names the missing requirement + ([#5409](https://github.com/NVIDIA/OpenShell/issues/5409)). +- Where do tests that restart or reconfigure the gateway live, and how do they + relate to disruption conformance? Migration notes propose a separate + `gateway-control` family; the first such migration decides. +- Do any benchmarks return, and where do they live + ([#5472](https://github.com/NVIDIA/OpenShell/issues/5472))? +- Disruption harness design is deferred to implementation.