Skip to content

test(e2e): migrate the Python E2E suites to Rust - #5624

Open
pimlock wants to merge 5 commits into
mainfrom
5435-e2e-python-to-rust/pimlock
Open

pimlock wants to merge 5 commits into
mainfrom
5435-e2e-python-to-rust/pimlock

Conversation

@pimlock

@pimlock pimlock commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The Python SDK has moved to ai-openshell/sdk-python, and core is removing its copy in python/ (#5287). Core's Python E2E suites still import the SDK from python/, so python/ can't be removed yet.

This PR ports the coverage of those suites to Rust, so core keeps it without depending on Python. It then deletes e2e/python/. The suites that test SDK client behavior (exec_admission, sandbox_api, sandbox_exec_python and workspace_api) aren't ported. sdk-python already runs copies of them against core gateways.

The ported tests cover:

  • Policy validation: unsafe and ambiguous policies are refused at create, and the default MCP version is filled in.
  • Policy enforcement: Landlock filesystem allow and deny, the run-as user, and denial of TCP connections no rule allows.
  • Provider credentials:
    • credentials are refused in request locations their profile doesn't allow;
    • the Google metadata server returns a placeholder token;
    • unknown providers are refused at create;
    • credentials stay out of the stored sandbox spec.
  • Gateway mTLS:
    • a client certificate authenticates a user;
    • a client with only the CA certificate is refused;
    • a certificate from another CA or a plaintext connection is refused.
  • OIDC and workspace authorization:
    • token scopes and client credentials;
    • the non-member matrix across every workspace-scoped RPC;
    • Platform Admin bypass;
    • workspace roles;
    • platform-scoped RPCs.

Related Issue

Closes #5435. Unblocks #5287.

Changes

  • New conformance tests in tests/suites/conformance cover policy validation, policy enforcement and provider credentials. They run on every driver lane.
  • A new tests/suites/features/gateway-mtls suite covers mTLS.
  • New e2e/rust/tests/oidc_authz.rs covers OIDC and workspace authorization. It runs in the existing Keycloak lanes.
  • testkit gains Gateway::mtls_dir() and Sandbox::spec().
  • The PR removes e2e/python/, its mise tasks and its CI rows. The workload image's Dockerfile moves to e2e/workload/Dockerfile; the Rust tests still use it.
Case-by-case mapping
Python suite New Existing coverage Dropped
test_policy_validation.py root, traversal and broad-path rejection; default MCP version Updates that remove immutable fields: handler test in grpc/policy.rs Conformance port of the immutable-field update: it duplicated the handler test and cost a sandbox start on every lane
test_sandbox_policy.py exec user and writes, folded into the Landlock test; transparent TCP denial for 3 cases; ambiguous destination rejected at create e2e/rust/tests/transparent_tcp.rs none
test_sandbox_landlock.py the whole allow/deny matrix, including a user-owned path outside the policy conformance filesystem_policy.rs covers device paths and path replacement none
test_sandbox_venv.py Landlock refer right (the EXDEV failure behind uv), tested offline Image ENV propagation: driver unit tests venv on PATH (asserts the fixture image, not product behavior); pip / uv installs (need live PyPI)
test_security_tls.py all 4 cases none none
test_sandbox_providers.py placement refusal for header and query credentials; Google metadata token and its 503; unknown provider; spec contains no credentials Env placeholders: credential_gating.rs, provider_auto_create.rs. Attach/detach: provider_readiness.rs. Imported profiles: host_gateway_alias.rs, stable_external_placeholders.rs. Update/merge semantics, the other placement styles, and profile scoping: server and core unit tests GitHub git clone (needs github.com); real google-auth SDK compatibility (needs google-auth in every lane's image)
oidc/oidc_auth_test.py no token, health, scopes, client credentials Admin/user provider and list cases: oidc_pkce.rs none
oidc/workspace_authz_test.py the full non-member matrix plus UpdateConfig; Platform Admin bypass; member and workspace-admin roles; all_workspaces; platform profile and global policy RPCs none GetSandboxLogs with a spoofed workspace (the same scenario is unit-tested in grpc/policy.rs)
exec_admission, sandbox_api, sandbox_exec_python, workspace_api none Copies in sdk-python tests/e2e not ported: SDK client behavior (per #5435)

Testing

  • mise run pre-commit passes (the pre-commit hook ran rust:format and rust:lint on each commit; I didn't run the full task)
  • Unit tests added/updated
  • E2E tests added/updated

Local runs against Docker gateways, before rebasing onto current main:

  • full conformance suite: 33/33
  • gateway-mtls: 4/4
  • oidc_authz against Keycloak: 7/7

After the rebase, I re-ran clippy with -D warnings on the conformance, gateway-mtls and e2e/rust (--features e2e-oidc-pkce) workspaces, plus rust:lockfiles:check. The Podman, Kubernetes, VM and tmachine lanes haven't run locally, so CI covers them.

Known limits, unchanged from the Python tests:

  • The mTLS refusal tests pass on any connect failure.
  • CI's e2e/rust clippy doesn't enable e2e-oidc-pkce, so it doesn't lint oidc_pkce.rs or oidc_authz.rs.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…ored spec

Suites that check the gateway's TLS boundary need the CA and client
certificate testkit already found for the gateway, and suites that check
what the gateway persists need the stored sandbox spec.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
… suites

Ports the cases from e2e/python that had no Rust coverage into the
conformance suite, so they run on every compute driver instead of only
the Docker Python lane:

- policy_validation: create refuses root, path traversal, a writable /
  and ambiguous endpoint pinning; MCP endpoints get the default version.
- policy_enforcement: Landlock read/write/deny matrix, the policy user,
  cross-directory renames (Landlock refer), and EPERM/EACCES for
  connections no rule allows.
- provider_credentials: credentials refused outside their profile
  placement, the Google Cloud metadata server's placeholder token,
  unknown providers, and credentials kept out of the stored spec.

Refs #5435

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Ports e2e/python/test_security_tls.py: a client certificate from the
gateway's CA authenticates a user, a CA-only client reaches only Health,
and a foreign certificate or plaintext connection is refused.

Refs #5435

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Ports e2e/python/oidc to e2e/rust/tests/oidc_authz.rs in the Keycloak
lane: requests without a token, token scopes, client credentials, the
non-member matrix over every workspace-scoped RPC, Platform Admin bypass,
workspace user and admin roles, and platform-scoped RPCs. Tokens come
from password and client-credentials grants; calls use the Rust SDK's
generated client.

Refs #5435

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Every case in e2e/python now maps to Rust coverage or an explicit drop
(#5435). The SDK-client suites (exec admission, sandbox
API, exec_python, workspace API) run in ai-openshell/sdk-python against
published core gateways.

- Delete e2e/python and the e2e:python and e2e:oidc-python:docker tasks;
  e2e no longer depends on python:proto or PYTHONPATH=python.
- Move the workload image's Dockerfile to e2e/workload/Dockerfile. The
  image tag and contents are unchanged; testkit, the Rust harness and
  tmachine keep using openshell/e2e-python:dev.
- Docker E2E: replace the python and oidc-python rows with gateway-mtls,
  drop the python:proto setup step, and run oidc_authz in the
  oidc-pkce lanes. Add a gateway-mtls row on the deb tmachine gateway.
- Update TESTING.md and the testkit docs.

Refs #5435; python/ itself is removed by
#5287.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

@pimlock pimlock added the test:e2e Requires end-to-end coverage label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 9123110. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@elezar elezar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough mapping table. The ported coverage doesn't duplicate what's already in conformance (filesystem_policy.rs, credential_gating.rs) or transparent_tcp.rs, and the testkit additions are small. CI is red, though, and I have a few structural questions.

CI failures (k3s and k3s-ha; I haven't checked the other lanes)

  1. policy_enforcement::landlock_enforces_filesystem_policy: the probe calls pwd.getpwuid(os.getuid()). On k3s the exec runs as uid 10001, which has no passwd entry (KeyError). The Kubernetes driver's fallback uid is DEFAULT_SANDBOX_UID = 10001, so the exec user may not be the policy's run_as_user: sandbox there. Is "exec runs as the policy user" meant to be a conformance contract? If so, the Kubernetes driver may need a look. If not, resolve the name tolerantly and assert only the Landlock behaviour.
  2. provider_credentials::google_metadata_serves_placeholder_token and provider_credentials_are_not_stored_in_sandbox_spec: provider create --type google-cloud and --type claude-code fail with provider profile '…' not found; import a matching profile. They rely on profiles already being on the gateway. The two placement tests in the same file import an inline profile with install_provider, which works on every lane. Please do the same here.
  3. gateway-mtls::foreign_client_certificate_is_rejected fails on ubuntu-docker-rootful. I haven't looked at the cause.

Structure and direction

  1. e2e/rust/tests/oidc_authz.rs adds about 1.2k lines to the legacy e2e/rust tree, on raw openshell-sdk, behind e2e-oidc-pkce. The direction elsewhere (#4891, #5137) is testkit-based tests/suites/features/* suites on Keycloak, and keycloak-provider-refresh already exists there. Is the legacy location deliberate? Also, e2e/rust clippy doesn't enable e2e-oidc-pkce, so this file isn't linted in CI.
  2. The mTLS refusal tests pass on any connect failure. A wrong port or a down gateway would also pass them. Please assert the specific TLS or gRPC error.
  3. e2e/workload/ is next to e2e/gpu/ (task names e2e:workload:build and e2e:workloads:build), which is easy to confuse. A README line, or a clearer name, would help. The Dockerfile also still installs cloudpickle, google-auth, requests and a uv venv, which served the removed SDK tests. Trimming it would shrink an image that is loaded into every environment.
  4. #5442 edits testkit/src/context.rs, conformance/src/lib.rs, e2e-suites.yml, TESTING.md and the testkit README. Whichever merges second will need a rebase.

On lane cost: only about four of the new conformance tests start sandboxes (about 16–19s each on k3s), and the rest take milliseconds. I don't see a problem there.

@matthewgrossman: could you take a look at points 1 and 4? They concern where the OIDC tests belong (legacy e2e/rust or a testkit features suite) and whether "exec runs as the policy user" is a conformance contract.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(e2e): migrate the Python SDK E2E suites in core to Rust

2 participants