Skip to content

test(testkit): share sandbox client helpers across testkit suites #5689

Open

Description

@matthewgrossman

User Story

As a contributor porting or writing testkit tests, I want the code that runs a client in a sandbox and reads its result to live in shared crates, so a test in any suite (conformance, features or a driver suite) can reuse it instead of copying it.

Description

testkit's core helpers are already shared: Context, Sandbox, CliChild, policy, and the tests/support/host-server recorders. The layer above them isn't. Network, policy and credential tests need a long-running Python sandbox, a client script run inside it, its JSON result, and the test host's name. Today each suite builds those itself:

This issue adds the shared layer (PR #5691) and converts the existing users on its base:

  • testkit
    • No long-running sandbox shorthand: it would only name two builder calls. Tests chain .image(testkit::python_image()).command(["sleep", "infinity"]), and the README shows that as the standard pattern.
    • SandboxBuilder::providers(..), so tests don't need .spec(..) to attach providers.
    • Sandbox::run_python(script, args), which fails on a nonzero exit and returns the output, and ExecOutput::last_json(), which returns the last JSON line of stdout. Some clients print a word instead of JSON, so the two are separate.
    • cx.sandbox, cx.cli_sandbox and cx.track fail at once on a sandbox name over the gateway's 19-character limit (5 characters of suffix), with a message that says so.
    • policy::binaries_yaml(..) for the binaries: list in hand-written policy YAML.
    • Sandbox::policy_revisions() and Sandbox::active_policy(), which read the policy revisions and the loaded policy, for tests of live policy updates.
  • tests/support/host-server: a TEST_HOST constant for host.openshell.internal.
  • A new crate, tests/support/sandbox-clients (sandbox_clients_support), for in-sandbox clients:
    • A shared scripts module, so any suite can use a script without reaching into another workspace.
    • Probes with typed Rust results, all run by one Python script (scripts::TCP_PROBE):
      • connect(&sandbox, host, port) reports whether it connected, the errno, or a timeout, and the elapsed time.
      • http(&sandbox, host, port, &Request) sends an HTTP request with Connection: close and reports the connect outcome, how the read ended, the bytes received and the parsed Response (status, headers, body).
      • http_raw(..) sends raw bytes and stops reading after 2 idle seconds.
    • Protocol-specific clients (GraphQL, JSON-RPC, MCP) stay separate scripts, but in the same scripts module.
  • Conversions:

Context

Definition of Done

  • The shared helpers above exist in testkit and tests/support/sandbox-clients, each with docs. The E2E suites exercise them; unit tests cover only parsing no E2E test reaches yet.
  • The conversions above use them; no suite on this branch defines its own long-running sandbox helper, and none but credential_gating.rs (converted by test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665) defines its own test-host constant.
  • No suite reads a file from another workspace.
  • The testkit README and the new crate's docs list the helpers.
  • cargo fmt, cargo clippy -D warnings and the lockfile alignment check pass for every touched workspace, and the converted tests pass on the E2E rows that ran them before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions