Repository navigation
test(e2e): port CLI exec, attach, and SSH host identity tests to testkit - #5442
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Label |
|
/ok to test bbbe043 |
1ce77a5 to
f8b1ced
Compare
Move the driver-agnostic sandbox lifecycle, labels, templates, settings, image, process, Landlock and upload tests from e2e/rust to the conformance suite, the Podman OCI identity test to the podman driver suite, and the doctor, proxy-preflight and WebSocket tunnel checks to crate tests. Move the live-Internet benchmark to mise run perf:live-internet. Delete the legacy files and their references. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…d test image helpers Add cx.cli_sandbox (with policy files), ExecOutput::success, cx.sandbox_details, cx.wait_absent and Engine::build_test_image, use them in the ported lifecycle tests, and replace the duplicate policy file, sandbox get and test image code in the host-gateway, oci-image and podman suites. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace hand-written exit-code checks in the smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor suites. Polls and checks that expect a failure keep reading the exit code. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…e helpers Add SandboxBuilder::create_ready, ExecOutput::expect_stdout, Sandbox::poll_exec, Context::delete_and_expect_absent and testkit::with_deadline, and convert std::io::Error into testkit::Error so I/O results take .context directly. Use them across the conformance suite, which also gets shared template, global-setting and label-list helpers and a split on-failure restart poll. Make Engine::build_image private so callers go through build_test_image, which removes the image. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…proxy Replace the hand-written WebSocket bridge with openshell_sdk::edge_tunnel::start_tunnel_proxy, so the test covers the client half of the tunnel and its edge-token headers as well as the router. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move expect_listed, expect_absent, wait_absent and the CLI's sandbox get JSON (now get_json) from Context onto Sandbox, and drop delete_and_expect_absent. Give ExecOutput failure() and CliOutput expect_stdout(), so both outputs offer the same checks. Put with_deadline's description first, like poll_until. Make CliSandboxBuilder::run consume the builder like create, and rename its policy setter to policy_yaml. Update the testkit README's list of what a test can do. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace CliSandboxBuilder's private PolicyFile with testkit::YamlFile, and drop YamlFile's map_err calls now that I/O errors convert into testkit::Error. CliOutput::success now includes stdout in its failure message, like ExecOutput::success, since many commands report errors there. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ting changes Add SandboxBuilder::image and CliSandboxBuilder::create_ready, so both builders set an image and wait for Ready the same way. Convert tonic::Status into testkit::Error, keeping its code, so gRPC results take .context() directly, and drop Error::from_status. Create the host-gateway sandbox with cx.sandbox(..).image(..).policy(..) instead of a hand-written CLI create. In credential gating, use create_ready and poll_exec. Drop the remaining map_err calls on I/O results in host-server-support. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
f8b1ced to
a96df3a
Compare
bbbe043 to
1b45c0b
Compare
|
/ok to test 1b45c0b |
…ble driver The gateway feature matrix builds without Docker, Podman or the VM driver, where the test file has no drivers to check. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ure restart Deleting a Docker sandbox that is still starting fails with a 409 because its supervisor volume is in use, so the labels test now waits for Ready before cleanup. The VM driver needs more than 45s to report an on-failure restart; the original e2e test allowed 60s. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…lders Context, SandboxBuilder, Sandbox and CliSandboxBuilder now share one Handles value (clients, CLI, cleanup), so CliSandboxBuilder owns its state like SandboxBuilder instead of borrowing the context. Both builders are #[must_use] at the type level, and CliSandboxBuilder::image and policy_yaml take impl Into<String> like SandboxBuilder::image. ExecOutput and CliOutput run success, failure and expect_stdout through one shared check, so their messages match and failure() also reports stderr. ExecOutput.command is renamed description, since it names the exec and its sandbox. CliOutput::message is renamed report_text. serde_json and serde_yml errors convert into testkit::Error, which removes the last map_err(Error::from_error) calls. expect_listed uses let-else instead of ensure plus expect. kubernetes-updates uses create_ready. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test d20313c |
1b45c0b to
d20313c
Compare
Read errors in gateway.rs and the image test use .context(..) now that the error types convert. Error::from_error and from_display are crate-private. Document why the global-setting helpers take &Cli, why the template test names and adopts its sandbox by hand, SandboxBuilder::restart_policy, and that the --no-keep test checks only gateway-visible removal. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Restore edge_tunnel_e2e with the --enable-websocket-tunnel flag in the Docker and Podman gateway wrappers, and ephemeral_cleanup with its Podman and VM lane entries. Both check what conformance cannot: the tunnel flag's wiring through a real gateway, and that drivers remove their resources. They move in later batches. Remove the in-process WebSocket tunnel server test and its openshell-sdk dev-dependency, which checked the listener but not the flag wiring. The --no-keep conformance test now says the driver-resource check stays in ephemeral_cleanup. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The two live-Internet benchmarks were manual, needed the live Internet, and nothing referenced them. Remove the Python port and its perf:live-internet task instead of keeping it. #5472 tracks whether to bring one back. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
proxy_settings_are_inserted_before_nested_driver_tables covers insert_docker_driver_settings, which the remaining corporate proxy tests still use. It was removed with the incoherent-configuration test by mistake. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
On the VM driver the replacement sandbox never starts: the preserved overlay disk fails e2fsck, and the sandbox stays in Starting. The test fails on every VM lane. Ignore it with a link to #5507, which also tracks removing the ignore. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…2e/rust Restore `on_failure_policy_replaces_runtime_and_preserves_workspace` in `sandbox_lifecycle.rs` and `settings_management.rs` as they were, and drop their conformance counterparts. The on-failure restart test ran on the Docker and Kubernetes lanes but never on VM or Podman. Ported to conformance it fails on every VM row (#5507), and an `#[ignore]` there removes the coverage it gave Docker and Kubernetes. The global settings test changes a setting for the whole gateway, so it can't share a conformance run with other tests until the runner has a test group for it (#5295). `sandbox_setting_round_trip` stays in conformance. Signed-off-by: Evan Lezar <elezar@nvidia.com>
d20313c to
77794c7
Compare
|
/ok to test 77794c7 |
…tach-exec/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> # Conflicts: # e2e/rust/Cargo.toml # e2e/rust/tests/sandbox_lifecycle.rs # tests/suites/conformance/README.md # tests/suites/conformance/src/cli.rs # tests/suites/conformance/src/lib.rs # tests/suites/features/Cargo.toml # tests/testkit/README.md # tests/testkit/testkit/src/cli.rs # tests/testkit/testkit/src/context.rs # tests/testkit/testkit/src/lib.rs
|
/ok to test 40b8c5a |
sandbox connect hands its stdin to an ssh child. Killing the CLI left that pipe open, so the orphaned ssh kept the attachment's input and no reconnect could take it over. Close stdin first, as tokio's Child::kill does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test eedd41f |
On a multi-replica gateway, attaching to a sandbox whose main already exited fails unless the request reaches the replica holding the supervisor session (#4949). Delay the main's output and exit so the CLI attaches while the sandbox is Ready. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test f394ed8 |
elezar
left a comment
There was a problem hiding this comment.
Thanks for the port. The conformance/feature split makes sense, and CliChild.kill() closing stdin first is a good catch (the unit test for it is helpful). I read the testkit, CI and attach changes closely and skimmed the rest; I haven't run the tests, so I'm relying on Branch E2E.
One thing I'd like resolved before merge, and one small coverage gap:
- Track the gateway-restart host-identity check. The legacy
ssh_host_identity_survives_restarts…also checked that the identity survives a gateway restart. That part is deferred to "PR6", but I couldn't find an issue for it. Could you link or open one and add it to thegateway-controlrow of #5281? (inline comment onssh.rs) --no-keepunder a PTY. The legacy tests ran under a PTY, and the new one only covers the pipe path. (inline comment oncli.rs)
Also noted, not blocking:
- Transport recovery and SIGTERM forwarding now run on
ubuntu-docker-rootfulonly. I think that's fine. Both are client-side, the gateway-to-sandbox hop is covered by conformance, and the one driver-specific gap is #4445. - Suite naming (inline comment on
Cargo.toml).
No issues with crypto boundaries or docs/skills drift.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…rminal The legacy tests ran under script, so the main got a PTY and the CLI attached with ssh -tt. Run each case with --no-tty and with --tty. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Match only processes the CLI under test started. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test f90a4cc |
elezar
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. The tracking issue (#5652), the --tty/--no-tty modes in conformance, the rename and the descendant-scoped process lookup all address my comments. The --tty approach is better than the openpty stdin I suggested. Branch E2E passes on f90a4cc. I haven't run the tests locally.
Summary
Port the e2e/rust tests that drive a long-running
openshellCLI process (exec streaming, attached create,--no-keep, attach/reconnect, and SSH host identity) to testkit, and delete the legacy tests.Related Issue
Closes #5426
Part of #5281.
Changes
CliChild(Cli::spawn/spawn_with_stdin,cx.cli_spawn/cx.cli_spawn_with_stdin): a long-running CLI process with stdin, a line reader, its pid, signal/kill and wait.killcloses stdin first, as tokio'sChild::killdoes, so thesshchild ofsandbox connectexits and releases the attachment's input. AlsoCli::gateway()andCli::apply_env(), which export the CLI environment for a plainsshclient.exec_cli.rs:cli_exec_large_output_is_complete,cli_exec_piped_stdin_crosses_grpc_message_limitcli.rs:cli_attached_create_streams_main_and_reports_exit,cli_no_keep_removes_sandbox_after_main_exitattach.rs:attach_reconnect_replays_history_for_the_same_main_process(it also stops and starts the sandbox while a connect session is open),attach_main_exit_255_is_not_retried_as_transport_failuressh.rs:ssh_host_identity_stable_across_stop_start_and_new_on_recreatetests/suites/features/connect-ssh-supervisionwith one Linux row (ubuntu-docker-rootful, deb) in.github/workflows/e2e-suites.yml:connect_recovers_its_ssh_transport,connect_forwards_pid_targeted_termination_and_reaps_ssh. It reads host procfs directly, matching only processes started by the CLI under test, and fails with a clear message off Linux. The row isn't on k3s, because of Investigate delayed SSH input-owner release through kubectl port-forward #4445.e2e/rust/tests/ssh_host_identity.rs, and the tests this PR ports frome2e/rust/tests/sandbox_lifecycle.rs. That file stays, holding onlyon_failure_policy_replaces_runtime_and_preserves_workspaceuntil bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507 (a VM driver bug) is fixed, so its follow-up entry intests/tmachine/artifacts.nixstays too, with an updated comment. Also removed:[profile.e2e-kubernetes]waiver forcanonical_main_connect_recovers_its_ssh_transportin.config/nextest.toml;openshell_tty_cmd,shell_escapeandnixdependency in e2e/rust.tests/testkit/README.md,tests/suites/conformance/README.md, andTESTING.md.Legacy test → new test
sandbox_lifecycle::sandbox_exec_large_output_is_completeexec_cli::cli_exec_large_output_is_completesandbox_lifecycle::piped_exec_stdin_crosses_grpc_message_limitexec_cli::cli_exec_piped_stdin_crosses_grpc_message_limitcanonical_main_exit_zero_completes_persistent_sandbox+canonical_main_nonzero_exit_preserves_statuscli::cli_attached_create_streams_main_and_reports_exitsandbox_create_with_no_keep_cleans_up_after_tty_command+…_preserves_failure_then_cleans_upcli::cli_no_keep_removes_sandbox_after_main_exitcanonical_main_disconnect_reconnect_replays_history_for_same_processattach::attach_reconnect_replays_history_for_the_same_main_processcanonical_main_exit_255_is_not_retried_as_transport_failureattach::attach_main_exit_255_is_not_retried_as_transport_failurecanonical_main_connect_recovers_its_ssh_transportfeatures/connect-ssh-supervision::connect_recovers_its_ssh_transportcanonical_main_connect_forwards_pid_targeted_termination_and_reaps_sshfeatures/connect-ssh-supervision::connect_forwards_pid_targeted_termination_and_reaps_sshssh_host_identity::ssh_host_identity_survives_restarts_and_changes_after_recreationssh::ssh_host_identity_stable_across_stop_start_and_new_on_recreate; the gateway-restart part is tracked in #5652Defaults applied
cli_pty. The legacy attached-create and--no-keeptests ran underscriptto get a terminal. The ports run each case with--no-ttyand with--tty, which gives the main a PTY and makes the CLI attach withssh -tt, through plainCli::run. Onlyconnect_forwards_pid_targeted_termination_and_reaps_sshneeds one; it passes anopenptyfollower as theCliChild's stdin.features/connect-ssh-supervisionon one Linux row.sshhost identity test needs one, becausesandbox ssh-configwrites it into itsProxyCommand. Without one, the test fails with a clear message. It reads the fingerprint through the SDK and keeps the "auth.jsonunreadable" check.Testing
cargo fmtandcargo clippy --all-targetsare clean for testkit, conformance and features, on macOS and in Linux (Docker,rust:1.95-bookworm) for the Linux-only connect-ssh-supervision codecargo test -p openshell-testkit --libpasses, including the newCliChildunit testscargo check --features e2e,e2e-host-gateway --testspasses in e2e/rustChecklist