Skip to content

tests: keep three tests off host state they don't own - #278

Merged
ac-pplx merged 1 commit into
mainfrom
ac/test-safety-gaps
Oct 12, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/test-safety-gaps

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Tests only. CI's "Rust tests" job runs the whole workspace, ignored tests included (ci/scripts/run-rust-checks.sh passes --run-ignored all), as root in a --privileged --network host container. That means a test's assumptions about the host also run on any reused development host. Reviews of #275 found three tests that touch host state they don't own.

Gaps and changes

1. node-mgr loop_devices/tests.rs: ensure_capacity_creates_attachable_devices_past_max_loop

  • Gap: The only guard was #[ignore]. The test needs root, removes and re-creates /dev/loop63, and provisions up to 64 kernel loop devices that it never removes. Since CI runs ignored tests, it ran there. On a reused host the devices leak, and the test fails instead of skipping when max_loop >= 64 or loop63 is busy.
  • Change: The test now returns early unless SPACE_NODEMGR_TEST_LOOP_DEVICES is set. This follows the crate's existing SPACE_NODEMGR_TEST_BTRFS_DIR tests: the same // CI includes ignored tests; host mutation still requires explicit opt-in. comment and the same not opted in: … message. The doc comment's invocation and the ignore reason now name the opt-in.
  • CI coverage: No CI job or script sets SPACE_NODEMGR_TEST_BTRFS_DIR or any similar node-mgr opt-in. I searched .github/, ci/, the Kind scripts and the rest of the tree outside crates/, and the variable appears only in the tests that read it. So this test now runs only where someone opts in, and CI no longer exercises it. No CI job was added. The fake-provisioner tests in the same file still cover the reconcile logic.

2. api routes/e2b/tests.rs: test_state()

  • Gap: connect_lazy("postgresql://postgres:postgres@localhost/space") used the default port 5432, under a comment saying nothing listens there. A dev box or host-network runner may serve 5432.
  • Change: The pool now points at postgresql://127.0.0.1:1/unused, a closed port, like cluster_admission/tests.rs and the kubernetes substrate_with_unreachable_store. The existing 100 ms acquire timeout stays, and the comment now says why the port is safe.

3. sbx-mgr service/tests.rs: runtime_liveness_options

  • Gap: It hard-coded the production SANDBOX_HOST_CONTROL_SOCKET and CREDMGR_SOCKET paths and backs service_with_runtime_liveness (about 61 callers). Only the _and_absent_sockets variant overrode them.
  • Change: The base options now put both sockets in a per-service directory under the temp dir that nothing creates (space-sbx-mgr-test-<uuidv7>/absent-*.sock). Any path that reaches either socket fails to connect instead of reaching the host's services. The helpers were not folded: _and_absent_sockets takes a caller-owned directory, so its signature still differs, and its callers are in grpc_lifecycle/tests.rs, which open PRs are editing. Only its doc comment changed.

Overlap with open PRs

Validation

Check Result
cargo fmt --all -- --check pass
cargo clippy --locked -p pplx-space-node-mgr -p pplx-space-api -p pplx-space-sbx-mgr --all-targets --all-features -- -D warnings pass
cargo nextest run -p pplx-space-api --all-targets --all-features --run-ignored all, SPACE_CI_REQUIRE_TEST_DEPS=1, Postgres and Redis from the nextest setup scripts, ×2 1670/1670 pass, both runs
same, -p pplx-space-sbx-mgr, ×2 1523/1524 pass, both runs; the one failure is below
same, -p pplx-space-node-mgr, ×2 456/457 pass, both runs; the one failure is below
Gated test without the opt-in, as non-root and as root skips with not opted in: set SPACE_NODEMGR_TEST_LOOP_DEVICES=1 …, and /dev/loop63 is untouched
Gated test with SPACE_NODEMGR_TEST_LOOP_DEVICES=1, as root pass

The local nextest runs were done as a non-root user. Each run had exactly one failure, in a root-only ignored test whose file this PR doesn't touch, and each fails on its own root assertion:

  • clone_tree::tests::private_clone_blocks_foreign_uid_grafts_and_shared_clone_is_held: "run in a disposable root container"
  • transfer_directory::tests::distinct_service_uids_can_exchange_downloads: "run this test with a root runner"

I didn't re-run those two as root on a shared development host. CI runs them as root.

The opted-in root run used a host whose kernel already had more than 64 loop devices and whose /dev is a container tmpfs. So it re-created only the /dev/loop63 node and added no kernel devices, and loop63 was unbound afterwards. The test still passed ensure_capacity, attached and detached loop63, and checked the node's mode and group.

Not changed

The same class of gap remains elsewhere in sbx-mgr's tests. It needs either a crate-wide test-constructor change or a node-manager client seam, so it is left for follow-ups:

  • service_for_teardown_test (about 121 callers across 11 test files), service_with_runtime_probe, and SandboxManagerService::new still build options with Default. That uses the production host-control and credential-manager socket paths, and service_with_runtime_probe also reads the host's node identity. A shared absent-socket test-options base would close this. Much of the file is also edited by open PRs.
  • The node-manager client (pplx_space_node_mgr_client::{connect, lazy}) always dials the fixed /space/run/node-manager socket. Tests that reach it, such as fatal_kill_signal_failure_still_schedules_admitted_cleanup, would contact a live node-manager on a SPACE node, and their Internal assertion would also accept that server's rejection. Fixing this needs an injectable node-manager transport.
  • With the loop-device test opted out, no automated test covers the real LOOP_CTL_ADD path or the post-mknod chmod/chown. Kind runs the real ensure_capacity at node-manager startup, but it only logs failures. An opted-in job on a disposable runner, or a small test for the permission step, would restore coverage.

Other test pools still name a default-port localhost Postgres: routes/e2b/template_build/translate/tests.rs and internal/api/server/src/run/tests.rs. They are outside this PR's scope and could get the same treatment in a follow-up.

🤖 Generated with Claude Code

CI runs the whole workspace, ignored tests included, as root in a
privileged host-network container, so a test's host assumptions run there
and on any reused development host.

- node-mgr: ensure_capacity_creates_attachable_devices_past_max_loop
  removes and re-creates /dev/loop63 and leaves up to 64 kernel loop
  devices behind. It now returns early unless
  SPACE_NODEMGR_TEST_LOOP_DEVICES is set, the same skip the crate's
  SPACE_NODEMGR_TEST_BTRFS_DIR tests use. No CI job sets either opt-in.
- api: the e2b routes' test_state() lazy pool pointed at localhost:5432,
  which a dev box or host-network runner may serve. It now points at the
  closed 127.0.0.1:1, like the cluster_admission tests.
- sbx-mgr: runtime_liveness_options, behind service_with_runtime_liveness,
  used the production host-control and credential-manager socket paths.
  It now uses absent paths in a per-service directory nothing creates.
  The _and_absent_sockets variant keeps its caller-owned directory, so
  the two helpers stay separate.
@ac-pplx
ac-pplx marked this pull request as ready for review October 11, 2026 06:57
@the-gigi-pplx

Copy link
Copy Markdown
Collaborator

Review — 2 findings

Low

  1. crates/libs/sbx/mgr/server/src/service/tests.rs:12301-12303 — the new doc comment overstates the mechanism for one of the two sockets.
    It says a path that "reaches either one fails to connect". sbx-mgr never dials host_control_socket_path. It only passes the path through as --host-control-socket in a node-manager Launch (service/spaced_bridge.rs:13-34). The credential-manager path is the one that gets dialed (service/mod.rs:1641). The change itself is right. Suggest "sit in a directory nothing creates, so neither names a live host service".
  2. crates/libs/node/mgr/server/src/loop_devices/tests.rs:148 — the opted-out run passes silently, so the real LOOP_CTL_ADD + chmod/chown path no longer has automated coverage anywhere.
    With no opt-in, --ignored reports ok in 0.00s. The body already says so under "Not changed", so the trade is priced. A nit: var_os(...).is_none() means =0 also opts in. The SPACE_NODEMGR_TEST_BTRFS_DIR precedent takes a path naming the disposable target, which makes an accidental opt-in less likely.

For each of the three tests I checked what it touched before and that it still can fail. The loop test ran in CI under --run-ignored all in a privileged, host-network container, and now returns before any /dev or /sys access, with the opted-in assertions unchanged. api test_state() moved from the default local Postgres port to 127.0.0.1:1, which matches cluster_admission/tests.rs. sbx-mgr runtime_liveness_options moved from the production socket paths to a per-call directory nothing creates. No test asserts the production paths through these options.

Verified against 33f11fdfcf42a785814fdc6d707e5475dcf2ec66. Reproduced: loop_devices -- --ignored skip-passes, routes::e2b::tests 126/126, and sbx-mgr --lib 1522 passed / 1 ignored. I didn't run the opted-in loop test, because the box I ran on isn't a disposable host.

@the-gigi-pplx the-gigi-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. 2 non-blocking findings in the review comment above.

@ac-pplx

ac-pplx commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Both noted as follow-ups:

  1. Doc wording. Agreed: sbx-mgr only passes host_control_socket_path through to node-manager's Launch and never dials it. Only the credential-manager path is dialed. I'll take your suggested wording in the next sbx-mgr test-constructor change.
  2. Loop-device opt-in. The coverage trade-off is the one listed under "Not changed". Switching the opt-in to name a disposable target, like SPACE_NODEMGR_TEST_BTRFS_DIR, is a good tightening. It's tracked with the unprivileged-CI work, where the privileged tests get an explicit job.

@ac-pplx
ac-pplx added this pull request to the merge queue Oct 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 12, 2026
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit 1cb3bdc Oct 12, 2026
13 checks passed
@ac-pplx
ac-pplx deleted the ac/test-safety-gaps branch October 12, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants