Repository navigation
Open
Labels
topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Description
User Story
As a contributor porting or writing testkit tests, I want the code that runs a client in a sandbox and reads its result to live in shared crates, so a test in any suite (conformance, features or a driver suite) can reuse it instead of copying it.
Description
testkit's core helpers are already shared: Context, Sandbox, CliChild, policy, and the tests/support/host-server recorders. The layer above them isn't. Network, policy and credential tests need a long-running Python sandbox, a client script run inside it, its JSON result, and the test host's name. Today each suite builds those itself:
tests/suites/conformance/src/credential_gating.rsonmainhas its ownTEST_HOST,run_python,allowand inline Python clients.tests/suites/features/host-gatewayhas another inline client and typeshost.openshell.internalby hand.- The open ports add more copies:
- test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 (PR2) adds
conformance/src/support.rsand 22 client scripts inconformance/src/scripts/; features/routable-host-ipreads one of those scripts withinclude_str!("../../../conformance/src/scripts/status_client.py"), across workspaces;- test(e2e): port CLI exec, attach, and SSH host identity tests to testkit #5442 (PR4) adds two copies of a
ready_sandboxhelper; - test(e2e): migrate the Python E2E suites to Rust #5624 adds its own
python_sandboxin a secondconformance/src/support.rs.
- test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 (PR2) adds
- Suites outside conformance can't use the conformance helpers. PR5 (test(e2e): port Docker and Podman driver feature tests from e2e/rust #5676) and later ports put tests in
features/*anddrivers/*, which may depend only on testkit andtests/support.
This issue adds the shared layer (PR #5691) and converts the existing users on its base:
- testkit
- No long-running sandbox shorthand: it would only name two builder calls. Tests chain
.image(testkit::python_image()).command(["sleep", "infinity"]), and the README shows that as the standard pattern. SandboxBuilder::providers(..), so tests don't need.spec(..)to attach providers.Sandbox::run_python(script, args), which fails on a nonzero exit and returns the output, andExecOutput::last_json(), which returns the last JSON line of stdout. Some clients print a word instead of JSON, so the two are separate.cx.sandbox,cx.cli_sandboxandcx.trackfail at once on a sandbox name over the gateway's 19-character limit (5 characters of suffix), with a message that says so.policy::binaries_yaml(..)for thebinaries:list in hand-written policy YAML.Sandbox::policy_revisions()andSandbox::active_policy(), which read the policy revisions and the loaded policy, for tests of live policy updates.
- No long-running sandbox shorthand: it would only name two builder calls. Tests chain
tests/support/host-server: aTEST_HOSTconstant forhost.openshell.internal.- A new crate,
tests/support/sandbox-clients(sandbox_clients_support), for in-sandbox clients:- A shared
scriptsmodule, so any suite can use a script without reaching into another workspace. - Probes with typed Rust results, all run by one Python script (
scripts::TCP_PROBE):connect(&sandbox, host, port)reports whether it connected, the errno, or a timeout, and the elapsed time.http(&sandbox, host, port, &Request)sends an HTTP request withConnection: closeand reports the connect outcome, how the read ended, the bytes received and the parsedResponse(status, headers, body).http_raw(..)sends raw bytes and stops reading after 2 idle seconds.
- Protocol-specific clients (GraphQL, JSON-RPC, MCP) stay separate scripts, but in the same
scriptsmodule.
- A shared
- Conversions:
features/host-gatewayusesTEST_HOSTandhttp(..).- The two
ready_sandboxcopies from test(e2e): port CLI exec, attach, and SSH host identity tests to testkit #5442 and the one infeatures/connect-ssh-supervisionare removed; tests use the builder chain. - The
sh -c "exec sleep infinity"mains become["sleep", "infinity"]. conformance/src/cli.rsusesSandbox::get_json()instead of a local copy.- The CDI scenario (
features/cdi) creates its sandbox withcx.cli_sandbox(..). - The Keycloak refresh test (
features/keycloak-provider-refresh) is left to test(e2e): port provider tests from e2e/rust to testkit #5657, which converts it toinstall_provider(..)with its typed profile. credential_gating.rsis converted by test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665, which already rewrites it, to avoid conflicting edits. test(e2e): migrate the Python E2E suites to Rust #5624 adopts the helpers after this merges.
Context
- Part of test: retire the old e2e test system onto testkit #5281. Related to test(testkit): add the shared harness helpers e2e/rust tests need #5296 (inventory of the
e2e/rustharness helpers): this covers the in-sandbox client layer that PR2, PR5 and test(e2e): migrate the Python E2E suites to Rust #5624 each started building, so it lands ahead of those batches rather than with one of them. Lands under thee2e/rustports in test(e2e): decide which e2e/rust tests to port and in what order #5297. Stack: main ← test(e2e): port CLI exec, attach, and SSH host identity tests to testkit #5442 (PR4) ← test(testkit): share sandbox client helpers across testkit suites #5691 ← test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 (PR2) ← PR5 (test(e2e): port Docker and Podman driver feature tests from e2e/rust #5676). test(e2e): migrate the Python E2E suites to Rust #5624 is on main and adopts the helpers after this merges. - The provider profile builder (typed profiles instead of
format!YAML) is a separate change in test(e2e): port provider tests from e2e/rust to testkit #5657 (test(e2e): port provider tests from e2e/rust to testkit #5656). - Keep testkit independent of provisioning: the helpers take a sandbox and arguments, and know nothing about environments.
- Write the new crate's module docs so docs(testing): document how to write testkit tests and add test environments #5683 can link to them.
Definition of Done
- The shared helpers above exist in testkit and
tests/support/sandbox-clients, each with docs. The E2E suites exercise them; unit tests cover only parsing no E2E test reaches yet. - The conversions above use them; no suite on this branch defines its own long-running sandbox helper, and none but
credential_gating.rs(converted by test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665) defines its own test-host constant. - No suite reads a file from another workspace.
- The testkit README and the new crate's docs list the helpers.
-
cargo fmt,cargo clippy -D warningsand the lockfile alignment check pass for every touched workspace, and the converted tests pass on the E2E rows that ran them before.
Metadata
Metadata
Assignees
Labels
topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Activity