[ac-pplx]

Stacked: this PR is based on #322 (ac/sbxmgr-explicit-test-service), which is based on main. It will be retargeted to main once #322 merges. Tests only: the release space-sbxmgr-svc binary's machine code is identical to main's.

What merging does

Adds tests for sandbox-manager paths that earlier characterization deferred because they reached fixed template-manager or node-manager sockets. They now run against FakeTemplateResolver, FakeNodeManager and #322's test options.

Every test that builds a service runs in a child process (enter_route_test_process) that clears EPPO_SDK_KEY and SPACE_FLAGD_ENABLED, so FeatureFlagClient::from_env in service construction cannot contact a provider.

Create after template lookup — service/grpc_core/create_after_lookup_tests.rs

Host fence. Once create inserts a new sandbox row, create_with_id calls ensure_sandbox_cache_root. That is create_dir_all under the fixed /space/cache/firecracker/sandboxes, which would create host directories when the tests run as root. Every service in this module carries a temporary SQLite trigger that refuses each new sandboxes row (BEFORE INSERT … RAISE(ABORT, <the row as JSON>)). A create that reaches the insert stops there; this was verified for the cases and the mutations listed below. The refusal carries the row create tried to write, which the cases assert on.

The template resolver answers with a build in a test-owned directory, built for this node with a supported spaced guest. The template's defaults are an access-idle suspend policy at 600 s, auto-lifecycle authority Local and network-policy authority Local, so a hard-coded Cluster cannot pass for a template default. Seeded rows carry their own environment (ROW_ENV=from-row), distinct from every request's (CREATE_ENV=from-request). Each create is bounded by CREATE_WAIT (30 s, real time).

Test	Pins
create_reaches_the_row_insert_with_the_templates_build_and_defaults	Four cases. The row create inserts is creating, with the resolved build hash, the request's owner and environment, and the template's network authority local. It takes the template's default policy and local auto-lifecycle authority unless the request carries a policy. A requested id is used as the row id. BootMode::Pool with an empty pool falls back to a cold create. Each case sends one lookup by name and makes no node-manager call.
autostart_create_checks_ready_network_slots_before_the_row_insert	An autostart create calls node-manager's GetNetworkPoolStatus before inserting. Zero ready slots → ResourceExhausted "no ready network slots". A refused status → its code with the "network pool status failed:" prefix. One ready slot → the insert.
pool_create_adopts_a_donor_of_the_resolved_build_with_the_templates_defaults	BootMode::Pool with a paused donor of the resolved build returns the donor's id with boot_mode_used = Pool and inserts no row. The donor takes the request's owner and environment, replacing its own, and the template's default policy and authority. No node-manager call.
pool_create_never_adopts_a_donor_of_another_build	With only a donor of another build, create fails, and the donor keeps its state, owner, environment and policy. Whether create should have fallen through instead is a finding (below), not pinned here.
requested_id_create_reuses_only_a_compatible_settled_row	A requested id naming an existing compatible row, swept over SandboxState::all() (18 states), with the expected outcome written out per state in the test. The request's policy differs from the row's. created, started, paused, suspended and stopped rows are reused (reused_existing) and take the request's policy. cleanup_required and cleanup_failed refuse with retryable Unavailable "…still completing activation cleanup; retry". Every other state, quarantined included, refuses AlreadyExists "…already exists in state " and leaves the policy. Controls: a compatible started row with the same policy is reused. A started row of another build refuses AlreadyExists "…incompatible create parameters: template_hash", both with the same policy and with a changed one; the build is checked once up front and again under the wake lock when the policy changes. Every case keeps the row's own environment.

Finding: a pool create with only stale donors does not fall through. manager.proto:366 documents boot_mode as "fallthrough: pool → base_snapshot → cold", and so does the space-sbxmgr CLI help (args.rs:245). Create checks pool occupancy per template (reconcile.rs:995-1003, find_pool_sandbox(template, None, None)) but adopts per resolved build (adopt_pool_sandbox with the resolved content_hash). To reproduce:

1. Insert a paused pool donor (pool_template = 'seed') at build A.
2. Have template-manager resolve seed at build B.
3. Send Create { template: "seed", boot_mode: POOL } with no requested id.

Create returns ResourceExhausted "no available pool sandboxes for template 'seed'" instead of falling through to a base-snapshot or cold create. Reachability is low: in this repository only the space-sbxmgr CLI sends BootMode::Pool to sandbox-manager. The API's direct substrate refuses pool itself (direct.rs:173), and template-manager sends BaseSnapshot or Cold.

Unpersisted stop and delete — service/grpc_core/unpersisted_cleanup_tests.rs

This is #271's deferred "unpersisted stop/delete" path. A trigger refuses the stopping/deleting claim write, so Unfenced admission returns Unpersisted. Stop runs with immediate_kill, which skips the guest flush, because that flush dials the guest's fixed spaced-host socket.

unpersisted_stop_and_delete_clean_up_through_node_manager_and_report_the_claim_failure covers stop and delete × blocking and non-blocking × node-manager accepting or refusing:

• node-manager receives exactly one CleanupSandbox with force, the row's execution fence, and stop's options (keep filesystem, return after slot release, immediate kill) or delete's (none of them);
• when node-manager accepts, the RPC returns Internal "runtime cleanup completed but {stopping|deleting} transition could not be persisted for sandbox : claim cleanup lifecycle failed: …";
• when node-manager refuses, the RPC returns node-manager's code with the "cleanup after unpersisted {stop|delete}:" prefix;
• the row stays Started, with no cleanup intent, episode, active cleanup job or dirty ref.

Stop's target selection reads the test's run directory through #322's builder.

Endpoint capture — service/tests/endpoint_capture_tests.rs

gated_spaced_calls_capture_the_endpoint_proof_from_node_manager covers three calls:

• node-ctl opening a loopback bridge on a restoring sandbox;
• node-ctl closing a loopback bridge on a restoring sandbox;
• a trusted node-api readiness wait on a started sandbox.

Each runs twice:

• node-manager up: each passes its state and caller gates and sends exactly one CaptureSandboxEndpointProof { sandbox_id }. node-manager's FailedPrecondition surfaces as PermissionDenied "spaced endpoint attestation failed".
• node-manager down (the default unbound connector): each returns retryable Unavailable "spaced endpoint proof is temporarily unavailable", before any spaced-host connection.

This replaces five service/tests.rs tests that asserted only the node-manager-down outcome. Two pairs of them had identical bodies:

• spaced_open_loopback_bridge_passes_state_gate_while_restoring and …_exempts_node_ctl_without_route_token;
• spaced_close_loopback_bridge_passes_state_gate_while_restoring and …_exempts_node_ctl_without_route_token;
• spaced_wait_ready_trusted_node_api_reaches_spaced_host_without_route_token.

Their rationale comment, on why the bridge must not gate on Started, moves with them. Three helpers they shared become pub(crate).

Snapshot GC — snapshot_gc/tests.rs

reaps_idle_uploaded_unpinned_artifact now serves node-manager and asserts that the reaped artifact's <path>.gc-pending-<id> staging directory is handed to DeleteCacheArtifact. Before, the test passed whether or not that call happened. The test's clock is paused, so the call is bounded by NODE_MANAGER_WAIT (30 s) on real time, through a watchdog thread rather than a tokio timeout. #322 moves the module's other tests to an unbound node-manager.

Mutation proofs

Each mutation was a scratch edit, reverted with git checkout after the run, against only the tests listed. Runs were strictly sequential, each under strace -f -e trace=%file,connect. None wrote under /space or connected to a /space socket. The only /space syscalls were stats of fresh-UUID pid files (see Host safety).

Mutation	Failing tests
A1 grpc_core.rs: ignore the template's default auto-lifecycle authority (always Cluster)	fresh create, pool adoption
A1b create.rs: ignore the template's default network authority (always Cluster)	fresh create
A2 grpc_core.rs: a requested id with no row gets a fresh id	fresh create
A3 create.rs: skip the autostart ready-slot check	autostart
A4 grpc_core.rs: pool adoption ignores the resolved build	stale donor: it is adopted
A5 grpc_core.rs: skip the up-front requested-id mismatch check	requested-id sweep (other build, same policy)
A6 grpc_core.rs: only CleanupRequired gets the retryable refusal	requested-id sweep (cleanup_failed)
A7 grpc_core.rs: skip the reused row's policy update	requested-id sweep
A8 create_support.rs: a Quarantined row is reusable	requested-id sweep (quarantined)
A9 grpc_core.rs: a refused reuse writes the requested policy first	requested-id sweep (refusal keeps the policy)
A10 grpc_core.rs: reuse clears the row's environment	requested-id sweep
B1 grpc_core.rs: the unpersisted stop wipes the filesystem	unpersisted stop/delete
B2 grpc_core.rs: the unpersisted delete drops the persistence error	unpersisted stop/delete
B3 grpc_core.rs: the unpersisted stop drops the persistence error	unpersisted stop/delete
C1 spaced_bridge.rs: a refused proof reports "temporarily unavailable"	endpoint capture
C2 state.rs: the bridge gate refuses Restoring	endpoint capture
C3 spaced_bridge.rs: a down node-manager reports Internal	endpoint capture (down column)
D1 snapshot_gc.rs: the reaper skips the staging cleanup	reaps_idle_uploaded_unpinned_artifact
Control: no edit	all tests of these modules and snapshot_gc::tests pass

The seams and bounds are real:

Removed or misconfigured	Failing tests
The create fixture's template resolver (falls back to the unbound default)	all 5 create tests
The create fixture's served node-manager	autostart
CREATE_WAIT set to 1 ns	autostart (the one create that waits on a served RPC)
The unpersisted test's served node-manager	unpersisted stop/delete ("cannot connect to node-manager")
The endpoint test's served node-manager in the up column	endpoint capture
The GC test's served node-manager	reaps_idle_uploaded_unpinned_artifact
NODE_MANAGER_WAIT set to 1 ns	reaps_idle_uploaded_unpinned_artifact (the real-time bound fires under the paused clock)

An earlier draft let A5 survive: with a changed request policy, the second build check under the wake lock still refused. The same-policy control was added for that.

Still unreachable

• Clone-disposal races (sbx-mgr: characterize stop, delete and create core RPC paths (tests only) #271): the template recheck under the fence and the Aborted "clone disposal hold changed" result. Both need a concurrent writer between the clone-disposal read and the fenced re-arm, and there is no await point between them. An interleaving hook is needed.
• Create past the row insert (ensure_sandbox_cache_root, base-snapshot staging, launch) writes under the fixed cache root. It needs a cache-directory seam.
• Stop with a guest flush (immediate_kill = false) dials the guest's fixed spaced-host socket. It needs a spaced-host seam.
• suspend_proceeds_when_wake_predates_request in service/tests.rs still asserts only "not Aborted". With a served node-manager, suspend's next step dials the guest's fixed spaced-host socket (verified with strace).
• checkpoint_retry_keeps_mark_suspended_for_restoring_row_with_registered_artifact in service/tests.rs still applies the cleanup completion by hand. Letting node-manager complete it would take the restore on into fixed cache-dir staging.
• restart_drain_allows_process_reconnect_for_started_sandbox in service/tests.rs reaches the same endpoint capture through a route-token helper. It stays on the unbound default, because its helpers sit beside lines sbx-mgr: keep quarantined rows through inventory and bound recovery that can never succeed #270 edits.
• The delete_defers_* tests in service/tests.rs and the cleanup_executor lifecycle_commit_and_recovery_* tests reach node-manager only incidentally, and pass on the unbound default.

Host safety

• The create tests and the GC reaping test issue no syscall naming a /space path.
• The unpersisted stop/delete and endpoint tests only stat fresh-UUID fc.pid and space-netgtw-svc.pid paths under /space/run/firecracker/sandboxes. These come from fixed run-directory probes outside the service options (watchdog removal). No writes, no socket connects.
• The control run also includes snapshot_gc::tests::run_loop_startup_does_not_reset_live_restore_lease, which read_dirs the fixed /space/cache through sweep_orphans (read-only; ENOENT here).
• The switch to an unbound node-manager closed a latent host-deletion path in the GC tests. On main, a regressed in-loop cancel check in a GC tick would have sent DeleteCacheArtifact for any real host .gc-pending-* directory to the host node-manager.
• Triggers are TEMP and live only on each test's in-memory database.

Overlap with open PRs

Every open PR's diff that touches crates/libs/sbx/mgr/server applies to this head exactly as it does to #322 (see #322 for #270's and #331's merge-order notes):

• service/tests.rs: one mod line, three helpers made pub(crate), and the five superseded tests and their comment removed.
• service/grpc_core.rs: two #[cfg(test)] mod lines.

Validation

Check	Result
cargo fmt --all --check	pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings	pass
Gitleaks pre-commit hook, public-safety review of the staged diff	pass, no findings
cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0, SPACE_CI_REQUIRE_TEST_DEPS=1, 3 consecutive runs	3/3: 1585 passed, 1 skipped (an existing #[ignore] test), 0 dependency skips; the Postgres setup script ran in each run
Release space-sbxmgr-svc binary vs main	.text identical (tests only)
GitHub CI at 8d7088c	13/13 checks pass (Pull CI run 38349144589, including Rust tests)
Kind	not run: tests only

AI assistance: the tests and this description were written with Claude Code.

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 4 findings

Medium

1. crates/libs/sbx/mgr/server/src/service/grpc_core/create_after_lookup_tests.rs:597 — the requested-id sweep doesn't cover the build recheck under the wake lock, though the comment and the body's table read as if the changed-policy case reaches it.
   With a template-hash mismatch, create refuses at the up-front check (grpc_core.rs:331) before it takes the wake lock, so neither the same-policy nor the changed-policy case reaches the recheck at grpc_core.rs:368. That recheck only fires when a concurrent writer changes the row between the two reads. I deleted it, and all five create_after_lookup_tests still pass. I'd reword the comment, and the body's "the build is checked once up front and again under the wake lock" line, to say neither case reaches the recheck, and list it under "Still unreachable" next to the clone-disposal races, since it needs the same kind of interleaving hook.

Low

2. crates/libs/sbx/mgr/server/src/service/grpc_core/create_after_lookup_tests.rs:537 — the stale-donor fall-through bug is written up here and in the body, but there's no issue for it, and the test can't notice a fix.
   I confirmed the bug by reading. resolve_boot_mode checks pool occupancy per template (reconcile.rs:997, find_pool_sandbox(template, None, None)), and adopt_pool_sandbox filters on the resolved build (grpc_pool_wait_clone.rs:219), so a pool holding only stale donors refuses ResourceExhausted instead of falling through as manager.proto documents. The test asserts only is_err(), and a fixed fall-through would also be an error here because the insert fence refuses the cold create. I'd open an issue and reference it from the comment. Once it's fixed, the assertion can become "reaches the row insert" (fenced_row).
3. crates/libs/sbx/mgr/server/src/service/tests/endpoint_capture_tests.rs:60 — expect_err("node-manager refuses the endpoint proof") also covers the node-manager-down column, where nothing refuses anything, and that column serves a FakeNodeManager it never connects.
   Only the panic text on an unexpected Ok is affected. Something like "the endpoint proof is not granted" fits both columns, and building the fake only when node_manager_up is set drops the idle server.
4. crates/libs/sbx/mgr/server/src/service/grpc_core/unpersisted_cleanup_tests.rs:110, crates/libs/sbx/mgr/server/src/service/tests/endpoint_capture_tests.rs:67 — both copy the EPPO_SDK_KEY / SPACE_FLAGD_ENABLED child-process setup that create_after_lookup_tests.rs:30 and grpc_core/tests.rs:26 already wrap as enter_isolated_test_process.
   The list now appears six times in the crate. One shared helper in service::tests would keep a future provider variable from being missed in one copy.

Verified against 56897e8aa29c6449305e473bcb0bca42b45d8abc, diffed against PR #322's head 7e6b46a3. Reproduced locally: the full pplx-space-sbx-mgr nextest (1575 passed, 1 skipped), plus the wake-lock-recheck deletion above run against create_after_lookup_tests. By reading, I checked the unpersisted stop/delete path (grpc_core.rs:639 and :851, cleanup.rs:2554) against the test's expected CleanupSandboxRequest and messages, and the endpoint-capture rewrite against the five tests it replaces. I also checked that the GC test's thread-based bound is needed under its paused clock. I didn't rerun the A–D mutation table or the strace host-path checks.

[the-gigi-pplx]

Approved. 4 non-blocking findings in the review comment above.