Skip to content

test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance - #5665

Draft
matthewgrossman wants to merge 19 commits into
test/5689-share-sandbox-helpers/matthewgrossmanfrom
test/5655-port-network-l7-policy/matthewgrossman
Draft

matthewgrossman wants to merge 19 commits into
test/5689-share-sandbox-helpers/matthewgrossmanfrom
test/5655-port-network-l7-policy/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Port the network, L7, policy-update, and port-forward tests from e2e/rust to testkit, and delete the legacy files. Most tests move into the conformance suite. The IP-literal egress test moves to a new routable-host-ip feature package.

Stack: main ← #5442 ← #5691 ← this PR ← #5676. The base is test/5689-share-sandbox-helpers/matthewgrossman (#5691), merged in with a normal merge. I'll retarget as the PRs below this one merge.

Related Issue

Closes #5655
Part of #5281

GitHub links #5655 only after this PR is retargeted to main, because closing keywords apply only to PRs into the default branch.

Changes

  • testkit: Context::compute_driver, SandboxBuilder::driver_config, Sandbox::logs / logs_until, policy::set, policy::from_yaml, and policy::base_with.
  • host-server support: handler responses, chunked bodies, a raw TcpServer, free_port, and a loopback exchange client.
  • Shared helpers from test(testkit): share sandbox client helpers across testkit suites #5691: the suites use testkit's Sandbox::run_python, ExecOutput::last_json, Sandbox::policy_revisions / active_policy, SandboxBuilder::providers and policy::binaries_yaml, plus host_server_support::TEST_HOST. Conformance no longer has a support module.
  • Probes instead of one-off scripts: the bypass, reserved-destination, status, REST, REST upgrade, echo, uninspectable, redaction, pipeline and credential rewrite clients are now sandbox_clients_support::{connect, http, http_raw} calls. Credential tests read the provider placeholder with printenv and send it with http.
  • Scripts moved to tests/support/sandbox-clients/scripts: the GraphQL, JSON-RPC, JSON-RPC audit, MCP, localhost, listener lifetime, preconnect owner, socket inheritance, reload keep-alive, echo server, WebSocket text and binary, and bound credential (bash) clients, exposed through sandbox_clients_support::scripts. routable-host-ip no longer includes a conformance script from another workspace.
  • New conformance suites:
    • network: bypass denial, reserved destinations, localhost without proxy env, listener lifetime.
    • network_l7 + network_l7/mcp: REST, GraphQL, JSON-RPC, and MCP.
    • network_egress: reload, ambiguous update, tls: skip, middleware, pipelining.
    • policy_update and port_forward.
  • credential_gating additions: header and body rewrite, WebSocket text rewrite, and static credentials bound to profile endpoints.
  • New feature package tests/suites/features/routable-host-ip: the IP-literal endpoint test (explicit allowed_ips and implicit). CI rows: Docker rootful (deb), Podman rootful (rpm), k3s, and k3s-ha.
  • Push mode: live_policy_update is removed from e2e:rust:push. That lane already runs openshell-conformance against a push-mode gateway, so it runs the policy_update tests instead. The tmachine config-push environment is left to test(conformance): run the conformance suite against a gateway that pushes configuration #5622.
  • Removed:
    • The 11 legacy files, plus static_provider_credentials_are_bound_to_profile_endpoints and its helpers from host_gateway_alias.rs. The other two tests in that file stay until PR5.
    • The matching [[test]] entries, PODMAN_CI_TESTS entries, and artifacts.nix follow-up binary.
    • The e2e:websocket-conformance task and the now-unused SandboxGuard::spawn_forward.
  • Docs: conformance README suites table, TESTING.md (routable-host-ip), and a note on Context::name that sandbox suffixes can have at most 5 characters (test(testkit): share sandbox client helpers across testkit suites #5691's MAX_SANDBOX_NAME_LEN check enforces it).

Defaults applied

  • Upstream fixtures are in-process Rust servers on the test host, reached at host.openshell.internal. The legacy private-range allowed_ips entries are gone.
  • Client scripts run through Sandbox::exec in sleep infinity sandboxes, using pipes. The legacy tests were also non-TTY, so PTY/pipe parity is unchanged.
  • There is no exec_until helper; Sandbox::poll_exec already covers it.
  • Network tests use policy::base(), python_image(), and the shared Python binaries list.
  • Merges keep every assertion:
    • REST allow and deny are merged.
    • The JSON-RPC audit-mode case is a second endpoint in the JSON-RPC test.
    • MCP is split into three tests, one per protocol version set.
    • The live_policy_update round-trip and from-empty tests are merged into policy_set_versions_and_ignores_repeats.
  • Policy-update tests wait until the active revision is loaded and none is pending or failed. This covers the supervisor's enrichment of the base policy.

Deviations from the plan

  • port_forward is its own module, not part of cli.rs, to avoid conflicts with test(e2e): port CLI exec, attach, and SSH host identity tests to testkit #5442. It starts forward start -d with cli_spawn and waits for the exit code, because the backgrounded ssh keeps the CLI's pipes open. A cleanup hook runs forward stop.
  • The static credential binding test lives in credential_gating.rs. The unbound endpoint is a second port on the test host instead of host.docker.internal, so the 403 holds on every driver. The legacy Podman connect-denied special case is gone.

Rows that kept a container fixture

None. The IP-literal test needs a host address that the supervisor allows as an IP literal. Rootless Podman's address is link-local, which is always blocked. So instead of keeping a container fixture, that test is a feature package that does not run on rootless Podman.

Driver coverage changes

  • Docker and Kubernetes ran all of these tests before and still do, through the conformance rows.
  • Podman previously ran only bypass_detection, forward_proxy_*, host_gateway_alias, live_policy_update, and no_proxy. It now runs all of the ported conformance tests, on rootful and rootless.
  • The VM host lane previously ran only host_gateway_alias from this set. It now runs all of them through openshell-conformance in e2e-vm.sh. That is new coverage. If a newly covered row fails, I'll raise it here before skipping anything.

Testing

  • cargo fmt --check and cargo clippy --locked --all-targets -D warnings for tests/testkit, tests/support, tests/suites/conformance, tests/suites/features
  • cargo test for tests/testkit and tests/support (new host-server unit tests included)
  • tasks/scripts/check-conformance-program.sh (71 tests)
  • cargo check --tests for e2e/rust with docker, podman, kubernetes, vm, and config-push features. cargo clippy on host_gateway_alias is clean.
  • E2E suites (CI)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…olicy set helpers

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…vers to host-server

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ewrite tests

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 858dbc6

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 858dbc6. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

… tests

These tests now run in the conformance suite and the routable-host-ip
feature package.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…suites

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…er limit

The gateway rejects sandbox names longer than 19 characters, which leaves
5 for a suffix after test-<run-id>-. The static credential binding test
also needs the Python workload image, which has the sandbox user that the
base policy runs as.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 45fd589

…matthewgrossman' into test/5655-port-network-l7-policy/matthewgrossman

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

# Conflicts:
#	tests/testkit/README.md
#	tests/testkit/testkit/src/policy.rs
@matthewgrossman
matthewgrossman changed the base branch from test/5426-port-cli-attach-exec/matthewgrossman to test/5689-share-sandbox-helpers/matthewgrossman November 18, 2026 06:23
Drops the include of a conformance client script from another workspace.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace conformance's support module and generic client scripts with
testkit's run_python, last_json, policy readers, providers and
binaries_yaml, and the connect, http and http_raw probes. Protocol and
specialized clients move to the sandbox-clients scripts module.

The pipelined request tests now count every response in the reply; the
403 follows the first response's body with no line break, so the old
line-based count missed it. The policy update tests use the Python
image, which has the sandbox user the base policy runs as.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 251835b

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant