Skip to content

test(testkit): share sandbox client helpers across testkit suites - #5691

Open
matthewgrossman wants to merge 38 commits into
mainfrom
test/5689-share-sandbox-helpers/matthewgrossman
Open

matthewgrossman wants to merge 38 commits into
mainfrom
test/5689-share-sandbox-helpers/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds shared helpers for tests that run clients inside a sandbox, so later network, policy and credential suites don't each write their own. Converts the existing copies on this branch to use them. #5665 (PR2) and #5676 (PR5) stack on this. #5624 is on main and adopts the helpers after this merges.

Related Issue

Closes #5689
Part of #5281

Stack: main ← #5442 ← #5691 ← #5665 ← #5676. Review only the commits after #5442.

Changes

  • testkit: SandboxBuilder::providers(..), Sandbox::run_python(script, args) (fails on a nonzero exit), ExecOutput::last_json() (last JSON object or array line of stdout), Sandbox::policy_revisions() / active_policy(), and policy::binaries_yaml(..) for hand-written policy YAML.
  • Sandbox name check: cx.sandbox, cx.cli_sandbox and cx.track panic at the test's call when a name exceeds the gateway's 19 characters (testkit::MAX_SANDBOX_NAME_LEN), which leaves 5 for a suffix. test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 found that 21 of its 22 E2E failures came from long names. Context::name is unchanged, because provider and profile names have no limit, so test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 doesn't need to change it.
  • host-server: TEST_HOST (host.openshell.internal).
  • New tests/support/sandbox-clients crate (sandbox_clients_support): one Python probe script, plus connect, http and http_raw, which run it and return typed results (connect outcome and errno, how the read ended, received bytes, parsed HTTP response). HTTP is parsed in Rust. http sends Connection: close; http_raw stops reading after 2 idle seconds, so a keep-alive server can't stall a test.
  • No long-running sandbox shorthand. Tests write .image(testkit::python_image()).command(["sleep", "infinity"]); the testkit README shows this as the standard pattern.
  • Conversions:
    • features/host-gateway uses TEST_HOST and http(..) instead of its inline client.
    • The ready_sandbox helpers in conformance/src/exec_cli.rs, conformance/src/ssh.rs and features/connect-ssh-supervision are removed and inlined. connect-ssh-supervision keeps its sh -lc mains.
    • conformance/src/cli.rs reads sandbox get JSON through Sandbox::get_json(), the CDI scenario creates its sandbox with cli_sandbox. The Keycloak refresh test is left alone: test(e2e): port provider tests from e2e/rust to testkit #5657 converts it to install_provider with its typed profile.
    • sh -c "exec sleep infinity" mains (host-gateway, cli.rs, policy_advisor.rs) become sleep infinity, which leaves the same final process.
  • credential_gating.rs keeps its own TEST_HOST and clients until test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665, which rewrites that file.

Testing

  • cargo fmt --check and cargo clippy --locked --all-targets -- -D warnings for tests/testkit, tests/support, tests/suites/conformance, tests/suites/features and tests/suites/drivers/podman
  • cargo nextest run for tests/testkit and tests/support. Unit tests for the new helpers are limited to two HTTP parser cases no e2e test reaches yet; the e2e suites cover the rest
  • Cargo lockfile checks
  • Probe script checked locally against real sockets: clean close, keep-alive (idle and timeout), reset, refused and connect-only
  • Local Docker gateway: full conformance suite (43/43) and features/host-gateway pass. CDI and Keycloak need a GPU or Keycloak and run in CI
  • connect-ssh-supervision passes clippy in a Linux container
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

matthewgrossman and others added 30 commits November 10, 2026 09:39
Move the driver-agnostic sandbox lifecycle, labels, templates, settings,
image, process, Landlock and upload tests from e2e/rust to the conformance
suite, the Podman OCI identity test to the podman driver suite, and the
doctor, proxy-preflight and WebSocket tunnel checks to crate tests. Move the
live-Internet benchmark to mise run perf:live-internet. Delete the legacy
files and their references.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…d test image helpers

Add cx.cli_sandbox (with policy files), ExecOutput::success, cx.sandbox_details, cx.wait_absent and Engine::build_test_image, use them in the ported lifecycle tests, and replace the duplicate policy file, sandbox get and test image code in the host-gateway, oci-image and podman suites.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace hand-written exit-code checks in the smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor suites. Polls and checks that expect a failure keep reading the exit code.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…e helpers

Add SandboxBuilder::create_ready, ExecOutput::expect_stdout, Sandbox::poll_exec, Context::delete_and_expect_absent and testkit::with_deadline, and convert std::io::Error into testkit::Error so I/O results take .context directly. Use them across the conformance suite, which also gets shared template, global-setting and label-list helpers and a split on-failure restart poll. Make Engine::build_image private so callers go through build_test_image, which removes the image.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…proxy

Replace the hand-written WebSocket bridge with openshell_sdk::edge_tunnel::start_tunnel_proxy, so the test covers the client half of the tunnel and its edge-token headers as well as the router.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move expect_listed, expect_absent, wait_absent and the CLI's sandbox get JSON (now get_json) from Context onto Sandbox, and drop delete_and_expect_absent. Give ExecOutput failure() and CliOutput expect_stdout(), so both outputs offer the same checks. Put with_deadline's description first, like poll_until. Make CliSandboxBuilder::run consume the builder like create, and rename its policy setter to policy_yaml. Update the testkit README's list of what a test can do.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace CliSandboxBuilder's private PolicyFile with testkit::YamlFile, and drop YamlFile's map_err calls now that I/O errors convert into testkit::Error. CliOutput::success now includes stdout in its failure message, like ExecOutput::success, since many commands report errors there.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ting changes

Add SandboxBuilder::image and CliSandboxBuilder::create_ready, so both builders set an image and wait for Ready the same way. Convert tonic::Status into testkit::Error, keeping its code, so gRPC results take .context() directly, and drop Error::from_status. Create the host-gateway sandbox with cx.sandbox(..).image(..).policy(..) instead of a hand-written CLI create. In credential gating, use create_ready and poll_exec. Drop the remaining map_err calls on I/O results in host-server-support.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ble driver

The gateway feature matrix builds without Docker, Podman or the VM driver, where the test file has no drivers to check.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ure restart

Deleting a Docker sandbox that is still starting fails with a 409 because its supervisor volume is in use, so the labels test now waits for Ready before cleanup. The VM driver needs more than 45s to report an on-failure restart; the original e2e test allowed 60s.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…lders

Context, SandboxBuilder, Sandbox and CliSandboxBuilder now share one Handles value (clients, CLI, cleanup), so CliSandboxBuilder owns its state like SandboxBuilder instead of borrowing the context. Both builders are #[must_use] at the type level, and CliSandboxBuilder::image and policy_yaml take impl Into<String> like SandboxBuilder::image.

ExecOutput and CliOutput run success, failure and expect_stdout through one shared check, so their messages match and failure() also reports stderr. ExecOutput.command is renamed description, since it names the exec and its sandbox. CliOutput::message is renamed report_text.

serde_json and serde_yml errors convert into testkit::Error, which removes the last map_err(Error::from_error) calls. expect_listed uses let-else instead of ensure plus expect. kubernetes-updates uses create_ready.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Read errors in gateway.rs and the image test use .context(..) now that the error types convert. Error::from_error and from_display are crate-private. Document why the global-setting helpers take &Cli, why the template test names and adopts its sandbox by hand, SandboxBuilder::restart_policy, and that the --no-keep test checks only gateway-visible removal.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Restore edge_tunnel_e2e with the --enable-websocket-tunnel flag in the Docker and Podman gateway wrappers, and ephemeral_cleanup with its Podman and VM lane entries. Both check what conformance cannot: the tunnel flag's wiring through a real gateway, and that drivers remove their resources. They move in later batches.

Remove the in-process WebSocket tunnel server test and its openshell-sdk dev-dependency, which checked the listener but not the flag wiring. The --no-keep conformance test now says the driver-resource check stays in ephemeral_cleanup.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The two live-Internet benchmarks were manual, needed the live Internet, and nothing referenced them. Remove the Python port and its perf:live-internet task instead of keeping it. #5472 tracks whether to bring one back.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
proxy_settings_are_inserted_before_nested_driver_tables covers insert_docker_driver_settings, which the remaining corporate proxy tests still use. It was removed with the incoherent-configuration test by mistake.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
On the VM driver the replacement sandbox never starts: the preserved overlay disk fails e2fsck, and the sandbox stays in Starting. The test fails on every VM lane. Ignore it with a link to #5507, which also tracks removing the ignore.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…2e/rust

Restore `on_failure_policy_replaces_runtime_and_preserves_workspace` in
`sandbox_lifecycle.rs` and `settings_management.rs` as they were, and drop
their conformance counterparts.

The on-failure restart test ran on the Docker and Kubernetes lanes but never
on VM or Podman. Ported to conformance it fails on every VM row (#5507), and
an `#[ignore]` there removes the coverage it gave Docker and Kubernetes.

The global settings test changes a setting for the whole gateway, so it can't
share a conformance run with other tests until the runner has a test group
for it (#5295). `sandbox_setting_round_trip` stays in conformance.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
… trip

sandbox_setting_round_trip deleted the gateway-wide ocsf_json_enabled value and restored it on cleanup, which changes every sandbox on a shared gateway while it runs. Fail with a hint when a global value is set instead, as policy_advisor does.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add Cli::spawn and Cli::spawn_with_stdin (with Context wrappers) that
start the CLI and keep it running, for commands that stream output or
read input. Stdout and stderr are drained in the background so the pipe
never blocks. Also add Cli::apply_env so another program, such as ssh
with a ProxyCommand from sandbox ssh-config, reaches the same gateway.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move the CLI exec, attached create, no-keep, attach, and SSH host
identity tests from e2e/rust to the conformance suite, and the
procfs-based connect transport recovery and signal forwarding tests to
a new features/cli-attach suite. Delete the ported legacy tests,
ssh_host_identity.rs, the now-unused PTY harness helper, and the stale
Kubernetes nextest waiver.

sandbox_lifecycle.rs keeps only the on-failure restart test until #5507
is fixed.

The gateway-restart check of SSH host identity moves to a later port.

Part of #5281
Closes #5426

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…tach-exec/matthewgrossman

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

# Conflicts:
#	e2e/rust/Cargo.toml
#	e2e/rust/tests/sandbox_lifecycle.rs
#	tests/suites/conformance/README.md
#	tests/suites/conformance/src/cli.rs
#	tests/suites/conformance/src/lib.rs
#	tests/suites/features/Cargo.toml
#	tests/testkit/README.md
#	tests/testkit/testkit/src/cli.rs
#	tests/testkit/testkit/src/context.rs
#	tests/testkit/testkit/src/lib.rs
sandbox connect hands its stdin to an ssh child. Killing the CLI left
that pipe open, so the orphaned ssh kept the attachment's input and no
reconnect could take it over. Close stdin first, as tokio's Child::kill
does.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
On a multi-replica gateway, attaching to a sandbox whose main already
exited fails unless the request reaches the replica holding the
supervisor session (#4949). Delay the main's output and exit so the CLI
attaches while the sandbox is Ready.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rminal

The legacy tests ran under script, so the main got a PTY and the CLI
attached with ssh -tt. Run each case with --no-tty and with --tty.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Match only processes the CLI under test started.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…nd binaries_yaml

Shared helpers for suites that run client scripts in a sandbox: attach providers from the sandbox builder, run a Python script and read its JSON result, read a sandbox's policy revisions and active policy, and write a binaries list into hand-written policy YAML.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
A shared home for the clients tests run inside a sandbox. One probe script connects, optionally sends bytes, and reads until the connection ends; connect(), http() and http_raw() run it and return typed results. HTTP parsing happens in Rust and is unit-tested. Requests send Connection: close by default, and raw sends stop after two idle seconds, so a keep-alive server can't stall a read.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The test uses TEST_HOST and the sandbox-clients http probe in place of its own Python client. Its main changes from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ands

Tests create a long-running sandbox with the builder chain directly instead of a per-file ready_sandbox helper. The cli and policy-advisor mains change from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. connect-ssh-supervision keeps its sh -lc mains.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 18, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 70aaaf3

… keycloak tests

The cli tests read sandbox get JSON through Sandbox::get_json, the cdi scenario creates its sandbox with cli_sandbox, and the keycloak refresh test installs its provider with install_provider instead of importing the profile and creating the provider by hand.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test dcf9567

The e2e suites exercise the helpers. Keep only the HTTP parser tests for chunked bodies and non-HTTP bytes, which no e2e test reaches yet.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 70adbdc

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

#5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 4fa87f9

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 18, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 4fa87f9. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

The gateway limits sandbox names to 19 characters, which leaves 5 for a suffix after test-<run-id>-. cx.sandbox, cx.cli_sandbox and cx.track now panic at the test's call with that explanation, instead of the test failing later with the gateway's validation error. Context::name is unchanged, since provider and profile names have no such limit.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b76ff7b

@matthewgrossman
matthewgrossman marked this pull request as ready for review November 18, 2026 06:26
Base automatically changed from test/5426-port-cli-attach-exec/matthewgrossman to main November 18, 2026 09:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants