Repository navigation
test(testkit): share sandbox client helpers across testkit suites - #5691
Open
matthewgrossman wants to merge 38 commits into
Open
matthewgrossman wants to merge 38 commits into
matthewgrossman wants to merge 38 commits into
Conversation
Move the driver-agnostic sandbox lifecycle, labels, templates, settings, image, process, Landlock and upload tests from e2e/rust to the conformance suite, the Podman OCI identity test to the podman driver suite, and the doctor, proxy-preflight and WebSocket tunnel checks to crate tests. Move the live-Internet benchmark to mise run perf:live-internet. Delete the legacy files and their references. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…d test image helpers Add cx.cli_sandbox (with policy files), ExecOutput::success, cx.sandbox_details, cx.wait_absent and Engine::build_test_image, use them in the ported lifecycle tests, and replace the duplicate policy file, sandbox get and test image code in the host-gateway, oci-image and podman suites. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace hand-written exit-code checks in the smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor suites. Polls and checks that expect a failure keep reading the exit code. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…e helpers Add SandboxBuilder::create_ready, ExecOutput::expect_stdout, Sandbox::poll_exec, Context::delete_and_expect_absent and testkit::with_deadline, and convert std::io::Error into testkit::Error so I/O results take .context directly. Use them across the conformance suite, which also gets shared template, global-setting and label-list helpers and a split on-failure restart poll. Make Engine::build_image private so callers go through build_test_image, which removes the image. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…proxy Replace the hand-written WebSocket bridge with openshell_sdk::edge_tunnel::start_tunnel_proxy, so the test covers the client half of the tunnel and its edge-token headers as well as the router. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move expect_listed, expect_absent, wait_absent and the CLI's sandbox get JSON (now get_json) from Context onto Sandbox, and drop delete_and_expect_absent. Give ExecOutput failure() and CliOutput expect_stdout(), so both outputs offer the same checks. Put with_deadline's description first, like poll_until. Make CliSandboxBuilder::run consume the builder like create, and rename its policy setter to policy_yaml. Update the testkit README's list of what a test can do. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace CliSandboxBuilder's private PolicyFile with testkit::YamlFile, and drop YamlFile's map_err calls now that I/O errors convert into testkit::Error. CliOutput::success now includes stdout in its failure message, like ExecOutput::success, since many commands report errors there. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ting changes Add SandboxBuilder::image and CliSandboxBuilder::create_ready, so both builders set an image and wait for Ready the same way. Convert tonic::Status into testkit::Error, keeping its code, so gRPC results take .context() directly, and drop Error::from_status. Create the host-gateway sandbox with cx.sandbox(..).image(..).policy(..) instead of a hand-written CLI create. In credential gating, use create_ready and poll_exec. Drop the remaining map_err calls on I/O results in host-server-support. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ble driver The gateway feature matrix builds without Docker, Podman or the VM driver, where the test file has no drivers to check. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ure restart Deleting a Docker sandbox that is still starting fails with a 409 because its supervisor volume is in use, so the labels test now waits for Ready before cleanup. The VM driver needs more than 45s to report an on-failure restart; the original e2e test allowed 60s. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…lders Context, SandboxBuilder, Sandbox and CliSandboxBuilder now share one Handles value (clients, CLI, cleanup), so CliSandboxBuilder owns its state like SandboxBuilder instead of borrowing the context. Both builders are #[must_use] at the type level, and CliSandboxBuilder::image and policy_yaml take impl Into<String> like SandboxBuilder::image. ExecOutput and CliOutput run success, failure and expect_stdout through one shared check, so their messages match and failure() also reports stderr. ExecOutput.command is renamed description, since it names the exec and its sandbox. CliOutput::message is renamed report_text. serde_json and serde_yml errors convert into testkit::Error, which removes the last map_err(Error::from_error) calls. expect_listed uses let-else instead of ensure plus expect. kubernetes-updates uses create_ready. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Read errors in gateway.rs and the image test use .context(..) now that the error types convert. Error::from_error and from_display are crate-private. Document why the global-setting helpers take &Cli, why the template test names and adopts its sandbox by hand, SandboxBuilder::restart_policy, and that the --no-keep test checks only gateway-visible removal. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Restore edge_tunnel_e2e with the --enable-websocket-tunnel flag in the Docker and Podman gateway wrappers, and ephemeral_cleanup with its Podman and VM lane entries. Both check what conformance cannot: the tunnel flag's wiring through a real gateway, and that drivers remove their resources. They move in later batches. Remove the in-process WebSocket tunnel server test and its openshell-sdk dev-dependency, which checked the listener but not the flag wiring. The --no-keep conformance test now says the driver-resource check stays in ephemeral_cleanup. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The two live-Internet benchmarks were manual, needed the live Internet, and nothing referenced them. Remove the Python port and its perf:live-internet task instead of keeping it. #5472 tracks whether to bring one back. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
proxy_settings_are_inserted_before_nested_driver_tables covers insert_docker_driver_settings, which the remaining corporate proxy tests still use. It was removed with the incoherent-configuration test by mistake. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
On the VM driver the replacement sandbox never starts: the preserved overlay disk fails e2fsck, and the sandbox stays in Starting. The test fails on every VM lane. Ignore it with a link to #5507, which also tracks removing the ignore. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…2e/rust Restore `on_failure_policy_replaces_runtime_and_preserves_workspace` in `sandbox_lifecycle.rs` and `settings_management.rs` as they were, and drop their conformance counterparts. The on-failure restart test ran on the Docker and Kubernetes lanes but never on VM or Podman. Ported to conformance it fails on every VM row (#5507), and an `#[ignore]` there removes the coverage it gave Docker and Kubernetes. The global settings test changes a setting for the whole gateway, so it can't share a conformance run with other tests until the runner has a test group for it (#5295). `sandbox_setting_round_trip` stays in conformance. Signed-off-by: Evan Lezar <elezar@nvidia.com>
… trip sandbox_setting_round_trip deleted the gateway-wide ocsf_json_enabled value and restored it on cleanup, which changes every sandbox on a shared gateway while it runs. Fail with a hint when a global value is set instead, as policy_advisor does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add Cli::spawn and Cli::spawn_with_stdin (with Context wrappers) that start the CLI and keep it running, for commands that stream output or read input. Stdout and stderr are drained in the background so the pipe never blocks. Also add Cli::apply_env so another program, such as ssh with a ProxyCommand from sandbox ssh-config, reaches the same gateway. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move the CLI exec, attached create, no-keep, attach, and SSH host identity tests from e2e/rust to the conformance suite, and the procfs-based connect transport recovery and signal forwarding tests to a new features/cli-attach suite. Delete the ported legacy tests, ssh_host_identity.rs, the now-unused PTY harness helper, and the stale Kubernetes nextest waiver. sandbox_lifecycle.rs keeps only the on-failure restart test until #5507 is fixed. The gateway-restart check of SSH host identity moves to a later port. Part of #5281 Closes #5426 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…tach-exec/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> # Conflicts: # e2e/rust/Cargo.toml # e2e/rust/tests/sandbox_lifecycle.rs # tests/suites/conformance/README.md # tests/suites/conformance/src/cli.rs # tests/suites/conformance/src/lib.rs # tests/suites/features/Cargo.toml # tests/testkit/README.md # tests/testkit/testkit/src/cli.rs # tests/testkit/testkit/src/context.rs # tests/testkit/testkit/src/lib.rs
sandbox connect hands its stdin to an ssh child. Killing the CLI left that pipe open, so the orphaned ssh kept the attachment's input and no reconnect could take it over. Close stdin first, as tokio's Child::kill does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
On a multi-replica gateway, attaching to a sandbox whose main already exited fails unless the request reaches the replica holding the supervisor session (#4949). Delay the main's output and exit so the CLI attaches while the sandbox is Ready. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rminal The legacy tests ran under script, so the main got a PTY and the CLI attached with ssh -tt. Run each case with --no-tty and with --tty. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Match only processes the CLI under test started. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…nd binaries_yaml Shared helpers for suites that run client scripts in a sandbox: attach providers from the sandbox builder, run a Python script and read its JSON result, read a sandbox's policy revisions and active policy, and write a binaries list into hand-written policy YAML. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
A shared home for the clients tests run inside a sandbox. One probe script connects, optionally sends bytes, and reads until the connection ends; connect(), http() and http_raw() run it and return typed results. HTTP parsing happens in Rust and is unit-tested. Requests send Connection: close by default, and raw sends stop after two idle seconds, so a keep-alive server can't stall a read. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The test uses TEST_HOST and the sandbox-clients http probe in place of its own Python client. Its main changes from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ands Tests create a long-running sandbox with the builder chain directly instead of a per-file ready_sandbox helper. The cli and policy-advisor mains change from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. connect-ssh-supervision keeps its sh -lc mains. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Member
Author
|
/ok to test 70aaaf3 |
… keycloak tests The cli tests read sandbox get JSON through Sandbox::get_json, the cdi scenario creates its sandbox with cli_sandbox, and the keycloak refresh test installs its provider with install_provider instead of importing the profile and creating the provider by hand. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test dcf9567 |
The e2e suites exercise the helpers. Keep only the HTTP parser tests for chunked bodies and non-HTTP bytes, which no e2e test reaches yet. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test 70adbdc |
5 tasks
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> #5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
Member
Author
|
/ok to test 4fa87f9 |
|
Label |
The gateway limits sandbox names to 19 characters, which leaves 5 for a suffix after test-<run-id>-. cx.sandbox, cx.cli_sandbox and cx.track now panic at the test's call with that explanation, instead of the test failing later with the gateway's validation error. Context::name is unchanged, since provider and profile names have no such limit. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test b76ff7b |
matthewgrossman
marked this pull request as ready for review
November 18, 2026 06:26
This was referenced Nov 18, 2026
Base automatically changed from
test/5426-port-cli-attach-exec/matthewgrossman
to
main
November 18, 2026 09:50
matthewgrossman
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
November 18, 2026 09:50
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds shared helpers for tests that run clients inside a sandbox, so later network, policy and credential suites don't each write their own. Converts the existing copies on this branch to use them. #5665 (PR2) and #5676 (PR5) stack on this. #5624 is on main and adopts the helpers after this merges.
Related Issue
Closes #5689
Part of #5281
Stack: main ← #5442 ← #5691 ← #5665 ← #5676. Review only the commits after #5442.
Changes
SandboxBuilder::providers(..),Sandbox::run_python(script, args)(fails on a nonzero exit),ExecOutput::last_json()(last JSON object or array line of stdout),Sandbox::policy_revisions()/active_policy(), andpolicy::binaries_yaml(..)for hand-written policy YAML.cx.sandbox,cx.cli_sandboxandcx.trackpanic at the test's call when a name exceeds the gateway's 19 characters (testkit::MAX_SANDBOX_NAME_LEN), which leaves 5 for a suffix. test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 found that 21 of its 22 E2E failures came from long names.Context::nameis unchanged, because provider and profile names have no limit, so test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 doesn't need to change it.TEST_HOST(host.openshell.internal).tests/support/sandbox-clientscrate (sandbox_clients_support): one Python probe script, plusconnect,httpandhttp_raw, which run it and return typed results (connect outcome and errno, how the read ended, received bytes, parsed HTTP response). HTTP is parsed in Rust.httpsendsConnection: close;http_rawstops reading after 2 idle seconds, so a keep-alive server can't stall a test..image(testkit::python_image()).command(["sleep", "infinity"]); the testkit README shows this as the standard pattern.features/host-gatewayusesTEST_HOSTandhttp(..)instead of its inline client.ready_sandboxhelpers inconformance/src/exec_cli.rs,conformance/src/ssh.rsandfeatures/connect-ssh-supervisionare removed and inlined. connect-ssh-supervision keeps itssh -lcmains.conformance/src/cli.rsreadssandbox getJSON throughSandbox::get_json(), the CDI scenario creates its sandbox withcli_sandbox. The Keycloak refresh test is left alone: test(e2e): port provider tests from e2e/rust to testkit #5657 converts it toinstall_providerwith its typed profile.sh -c "exec sleep infinity"mains (host-gateway,cli.rs,policy_advisor.rs) becomesleep infinity, which leaves the same final process.credential_gating.rskeeps its ownTEST_HOSTand clients until test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665, which rewrites that file.Testing
cargo fmt --checkandcargo clippy --locked --all-targets -- -D warningsfortests/testkit,tests/support,tests/suites/conformance,tests/suites/featuresandtests/suites/drivers/podmancargo nextest runfortests/testkitandtests/support. Unit tests for the new helpers are limited to two HTTP parser cases no e2e test reaches yet; the e2e suites cover the restfeatures/host-gatewaypass. CDI and Keycloak need a GPU or Keycloak and run in CIconnect-ssh-supervisionpasses clippy in a Linux containerChecklist