[ac-pplx]

Stacked on #296 (ac/sbxmgr-seam-node-manager-transport). This PR's base is that branch, and its diff is only the template-manager seam.

Summary

Every template-manager GetTemplate that sandbox-manager issues now goes through one injected TemplateResolver. Production keeps today's exact client construction against the fixed template-manager socket. Tests can script the answer and observe each request.

The seam

crates/libs/sbx/mgr/server/src/template_resolver.rs adds:

• TemplateResolver, an async_trait trait with two methods. Both take a tonic::Request<GetTemplateRequest> and return a tonic::Response<GetTemplateResponse>, so request and response metadata cross the seam; callers call into_inner() as before.
  • get_template: GetTemplate on a client dialed for this call under the default connect-retry policy, as pplx_space_template_mgr_client::connect() does.
  • get_template_on_shared_channel: GetTemplate on the client crate's process-wide lazy() channel.
• TemplateLookupError, which keeps connect failures and RPC failures apart (Connect(ConnectError), Rpc(Status)), so each caller maps them to exactly the codes and messages it produced before.
• HostTemplateManager, the default. It calls pplx_space_template_mgr_client::connect() and lazy(), which reach TEMPLATE_MANAGER_SOCKET.

The resolver is a new field, SandboxManagerServiceOptions::template_resolver: Arc<dyn TemplateResolver>, defaulting to HostTemplateManager. The service hands it to the template network-policy cache. Handlers pass it to:

• resolve_template_paths, and through resolve_template_paths_for_start, template_source_for_sandbox_prepare and template_runtime_defaults_for_sandbox_start (create_support), which serve create, start, wake (auth_lookup), and pool adoption and donor cloning (grpc_pool_wait_clone);
• restore staging's parent-template lookup, snapshot_restore::ensure_local_template, through restore_pull_rootfs and prepare_restore_dir*.

Production wiring is unchanged

• Same client construction.
  • resolve_template_paths and ensure_local_template still dial a fresh client with connect() for each call.
  • The network-policy cache's fetch still uses the shared lazy() channel inside its existing timeout and admission bounds.
• Same error mapping.
  • Connect failures still produce cannot connect to template-manager: … and tmplmgr connect for GetTemplate(…): ….
  • RPC failures still produce GetTemplate failed: … with the server's code, template_pull_error's mapping, and template-manager GetTemplate(…) failed: ….
• Same requests. Request fields are built exactly as before. Each call site wraps its message in tonic::Request::new, which is what the generated client's IntoRequest did for a bare message: no metadata, no extensions.
• No bypass. TemplateNetworkPolicyCache no longer has a Default constructor that hard-wired HostTemplateManager. Its only constructor takes the injected resolver.
• No wire, schema, metric or configuration change.

Call-site inventory, production code. Base is #296's head, 56a295d (the inventory was taken at its pre-rebase head a314315; the rebase changed no production code):

	before	after
pplx_space_template_mgr_client::connect()	2: create_support::resolve_template_paths, snapshot_restore::ensure_local_template	1, inside HostTemplateManager::get_template
pplx_space_template_mgr_client::lazy() for GetTemplate	1: network_policy::fetch_template_default_network_policy	1, inside HostTemplateManager::get_template_on_shared_channel
callers of the lookups above	create (grpc_core), start ×3 (start), restore prepare (create), wake (auth_lookup), pool adoption and donor clone ×2 (grpc_pool_wait_clone), create_support ×2, the network-policy cache, restore staging	the same callers, now passing self.template_resolver

Call sites left unrouted

• grpc_core::list_templates keeps pplx_space_template_mgr_client::lazy() for ListTemplates. It is a catalog passthrough for sandbox-manager's own ListTemplates RPC, not a template resolution. Routing it would widen the resolver to a second RPC that no create, start or restore path uses.
• Existing tests that build options with ..Default::default() still get HostTemplateManager, exactly as before. The network-policy cache's own tests now build the cache with a refusing fake (see below).

Tests

FakeTemplateResolver (template_resolver/fake.rs, test-only) records each lookup and returns a scripted answer:

• Recorded: the method used (Dialed for get_template, Shared for get_template_on_shared_channel), the request metadata and the request message.
• Scripted: a closure produces each answer, so a test can return a response with scripted response metadata, a refusal, or a ConnectError. A connect error reaches a shared-channel lookup as Unavailable, as a lazy channel reports it.

New module service/grpc_core/create_template_lookup_tests.rs. It is a separate file because #271, open when it was written and since merged, edited service/grpc_core/tests.rs. It adds create_checks_auto_lifecycle_authority_before_template_lookup, covering the create cases #271 deferred because a regression would have reached the fixed template-manager socket. Each case gets its own FakeTemplateResolver and a FakeNodeManager that refuses everything. The test runs in an isolated child process, like #271's create tests. Every accepted case must send exactly one lookup, on the dialed channel, carrying no request metadata.

Case	Expected outcome	Template lookups
Local auto-lifecycle authority from a forwarded peer	PermissionDenied (authority=local requires a local operator peer)	none
Unspecified auto-lifecycle authority	InvalidArgument (policy authority proto value is unspecified)	none
Unknown auto-lifecycle authority (99)	InvalidArgument (policy authority proto value 99 is unknown)	none
Cluster auto-lifecycle authority from a forwarded peer (accepting control)	NotFound, message starting GetTemplate failed: and carrying the scripted refusal	exactly one dialed lookup, for seed with no hash
Local auto-lifecycle authority from a local peer (accepting control)	NotFound, message starting GetTemplate failed:	exactly one dialed lookup, for seed with no hash
Cluster authority with template-manager unreachable (scripted ConnectError)	Internal, message starting cannot connect to template-manager:	exactly one dialed lookup

In every case no sandbox row is created and node-manager receives no call.

These cases cover the auto-lifecycle checks #271 deferred and stop at template lookup. Everything create does after lookup is still not covered by fake-backed tests: intent resolution, pool adoption, requested-id reuse and create_with_id.

Two existing tests call the start-path free functions directly, with no service. Each now passes a fake resolver and asserts that no lookup was sent. Previously they relied on template-manager being absent:

• existing_sandbox_cannot_resolve_a_missing_artifact_by_latest_alias (create_support/tests.rs, calls resolve_template_paths_for_start)
• orphaned_sandbox_without_rootfs_start_prepare_does_not_fetch_template (service/tests.rs, calls template_source_for_sandbox_prepare)

The network-policy cache's tests (service/network_policy/tests.rs) now build their cache with a refusing fake instead of the removed Default. Two of them reach the owner lookup through require_hint when the cache misses or the admission lock is not honored, and both now assert that no lookup is sent:

• template_policy_hints_are_admission_requirements_not_cache_authority
• custom_policy_override_does_not_require_template_owner_progress

service/tests/node_manager_transport_tests.rs (from #296) gains isolated_service, which takes both fakes. Its service_with_node_manager now refuses every template lookup.

Mutation proof

Each mutation was applied, the named test was run with --retries 0, and the file was restored. All lookups went to the fake.

#	Mutation	Test	Result
C1	create skips the auto-lifecycle authority checks	create authority	killed
C2	create lets a forwarded peer set a Local auto-lifecycle authority	create authority	killed
C3	resolve_template_paths drops the requested template name	create authority	killed
C4	resolve_template_paths turns a lookup refusal into Internal	create authority	killed
C7	resolve_template_paths uses the shared lazy channel instead of a dialed client	create authority	killed (Shared ≠ Dialed)
C8	resolve_template_paths drops the GetTemplate failed: prefix	create authority	killed
C9	resolve_template_paths maps a connect failure as an RPC failure	create authority	killed
C5	start sends a lookup without the saved template hash	latest alias	killed
C6	an orphaned sandbox without rootfs looks up its template	orphaned prepare	killed
S1	Seam misconfigured: the service holds a different resolver than the one the test inspects	create authority	killed
P1	The network-policy owner lookup bypasses the cache and its admission lock	both cache tests above	killed

The other required seam check, a service built with HostTemplateManager in place of the fake, was not run. It would dial the fixed host socket, which the host-safety rule forbids even on a mutation's regression path. S1 proves the same dependence without contacting the host.

Host safety

• Every new or converted test builds its service with explicit options: a fake resolver, a refusing fake node-manager, socket paths nothing binds inside a test-owned temporary directory, and no live runtime. The two start-path tests build no service; they pass the fake resolver to the free function.
• On any regression path, a template lookup stops at the scripted answer and a node-manager call stops at Unimplemented.
• The create test isolates the feature-flag and route environment in a child process (enter_route_test_process) instead of reading the host's.

Overlaps with open PRs

Rebased onto #296's rebased head 56a295d, on main a8b5df6, which merged #269, #271, #273, #275, #277, #278 and #302. Open PRs were re-checked against this head with git merge-tree, plus a scan of each diff's added lines for the APIs this PR changes and for SandboxManagerServiceOptions literals:

• sbx-mgr: characterize stop, delete and create core RPC paths (tests only) #271 (merged). Its isolated_service in service/grpc_core/tests.rs lists every SandboxManagerServiceOptions field. sbx-mgr: inject the node-manager client transport and template-manager resolver #296 added node_manager there, an unbound connector on an absent socket. This PR adds template_resolver: Arc::new(FakeTemplateResolver::refusing(Status::unavailable("template-manager is outside this test"))), the same refusal service_with_node_manager uses, so no sbx-mgr: characterize stop, delete and create core RPC paths (tests only) #271 test can reach the fixed template-manager socket. sbx-mgr: characterize stop, delete and create core RPC paths (tests only) #271's tests are otherwise unchanged. Its deferred auto-lifecycle create cases are covered by this PR's create_template_lookup_tests.rs.
• tests: keep three tests off host state they don't own #278 (merged). The rebase was clean. This PR adds one template_resolver: line to runtime_liveness_options and one fake to orphaned_sandbox_without_rootfs_start_prepare_does_not_fetch_template in service/tests.rs, away from tests: keep three tests off host state they don't own #278's hunks.
• docs: localized AGENTS.md for sbx-mgr, node-mgr and the api crate #275 (merged). No file overlap.
• node-mgr: validate LoadSnapshot requests before staging host state #269, api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273, api: release a Kubernetes create's claimed execution on every early exit #277, Stamp effective TLS interception mode on sandbox create events #302 (merged). No overlap.
• sbx-mgr: keep quarantined rows through inventory and bound recovery that can never succeed #270. Its only conflict, in cache_cleanup/periodic/tests.rs, is also present against main.
• Preserve authenticated scope in cached template lookups #217. It merges cleanly with main and with sbx-mgr: inject the node-manager client transport and template-manager resolver #296. Its added lines rewrite every GetTemplate call site this PR routes, which gives 14 conflict regions against this PR's head: auth_lookup 2, create_support 6, create_support/tests.rs 2, grpc_core, grpc_pool_wait_clone, network_policy and snapshot_restore. Preserve authenticated scope in cached template lookups #217 attaches a caller scope to the tonic::Request (scope.attach(&mut request)) and checks an acknowledgement on the tonic::Response (scope.require_acknowledgement(&resp)) before into_inner(). Because the resolver takes and returns the tonic wrappers, Preserve authenticated scope in cached template lookups #217 can rebase by doing that around each templates.get_template(…) call without changing the trait.
• Quarantine sandboxes after sustained guest-agent probe failure #110. Same as sbx-mgr: inject the node-manager client transport and template-manager resolver #296: one added conflict hunk in watchdog.rs, from sbx-mgr: inject the node-manager client transport and template-manager resolver #296's change. This PR adds none.
• Other open PRs touching sbx-mgr. Recover teardown of restored snapshots with read-only metadata #281, Approve bounded publication of validated retained suspends #283, Pace failed teardown drivers without holding worker capacity #285 and Defer btrfs-send capture by default in sandbox-manager #287 merge cleanly. sbx-mgr: keep teardown polls pending while a delete is in flight #201's conflict in grpc_lifecycle.rs is also present against main. fix(sandbox): wait before retrying failed activations #3, [api] restore full snapshots only onto CPU-compatible hosts, behind a default-off gate #176, [sbx-mgr] record the capture host CPU on snapshots and publish it as a node label #203, Add an opt-in Firecracker balloon device per template #212 and [api] stop retrying restore pods whose snapshot the host CPU cannot load #221 already conflict with main; this PR adds no hunk to any of them.

Validation

Check	Command / evidence	Result
fmt	cargo fmt --all -- --check, at 66508e3	pass
clippy	cargo clippy --workspace --locked --all-targets --all-features -- -D warnings, at 66508e3	pass
repo guards	check-rust-error-policy.sh, generate-env-contract.sh --check, source_guards/env_test_hook_boundary.py, at 66508e3	pass
repo guards (pre-existing failures)	check_no_inline_rust_test_modules.py and source_guards/thin_service_mains.py, at 66508e3	output identical to main a8b5df6; files not touched here
pre-commit	gitleaks hook on the commit	pass
unit tests ×2 after rebase	SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0, two consecutive runs at 66508e3 (nextest's Postgres setup script passed)	1569/1569 passed each run (1 #[ignore] test not selected); 0 dependency-skip markers
unit tests ×3 before rebase	same command, three consecutive runs at the pre-rebase head b196857	1563/1563 passed each run
mutation proof	see the table above, run at the pre-rebase head b196857; not re-run after the rebase	11/11 killed
Kind cluster	one new space-local cluster per attempt on the current harness, built from #299's head b196857	up
deployed binaries	space-sbxmgr-svc sha256 8e9ce23a…7090 in both worker space-sandbox-manager pods, before and after the suites, equals the staged artifact built from this checkout; the synced build source for sbx-mgr is identical to b196857; the binary contains the node_manager_transport and template_resolver module paths	proven
Tilt quiet	tilt get uiresources: 42/42 ok, none pending or in error	quiet
smoke-kind-e2e.sh	BOOTSTRAP_TIMEOUT=1800 DATA_PLANE_URL=http://127.0.0.1:8090	PASS on both clusters
smoke-kind-typed.sh all, attempt 1	SKIP_BOOTSTRAP=1, SPACE_SMOKE_KIND_ADMIN_TOKEN_FILE and SPACE_SMOKE_KIND_COLD_RECEIPTS set as in kind-ci.yml, unsharded	ENVIRONMENTAL. 58 scenarios passed, then local.template_cache_lru_eviction_reclaims failed and the following scenarios were inconclusive. Docker's data disk (49 GB) reached 100%, so the Kind nodes' overlays could not write: os error 5 under /space/cache and Postgres could not write init file. A rerun on that cluster could not start sandbox pods (pod_never_ready), so the cluster was torn down
disk	docker builder prune -af and docker image prune -f only	freed 20,668,022,784 bytes after attempt 1 and 6,930,202,624 bytes before the final typed run
smoke-kind-typed.sh all, rerun on a new cluster	same environment	ECW 76 passed, native 23 passed; 0 warned, 0 failed, 0 inconclusive (includes ecw.cold_create_environment)

The Kind rows above come from the pre-rebase head b196857 and were not re-run. Since then, the only changes to this PR are the new template_resolver field in #271's test isolated_service, and #296's test-only changes; main's merged PRs change no sbx-mgr production code.

Follow-ups

• sbx-mgr: characterize stop, delete and create core RPC paths (tests only) #271's comments in service/grpc_core/tests.rs (for example on BARRIER_AUTHORITY) still say a lookup past the barrier would contact the fixed template-manager socket. With the refusing fake it would stop at the fake. They were left as they are, by design, because this change only adds the fields.
• service/tests/node_manager_transport_tests.rs's module doc should also list the template resolver among the host dependencies that isolated_service redirects.

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 1 finding

Reviewed as the diff on top of PR #296's head e1e992711. No blocking findings. The only pplx_space_template_mgr_client::{connect, lazy} calls left in production code are the two inside HostTemplateManager and grpc_core.rs:1085, the ListTemplates passthrough the description deliberately leaves unrouted. Each converted call site keeps its connect and RPC error text, with TemplateLookupError::Connect and ::Rpc mapped separately.

Low

1. service/grpc_core/tests.rs:72-74: BARRIER_AUTHORITY's doc still says a lookup past the barrier "would contact the fixed template-manager socket". Eleven lines above it (:63), this PR gives the same isolated_service a FakeTemplateResolver::refusing(...), so the sentence is now false. A later reader who trusts it will over-guard, or will assume that removing the barrier is host-unsafe. The description leaves this alone by design, but the edit costs one line: say a lookup past the barrier stops at the refusing fake.

Verified against 17387b762114314f200156e7347ee7223d5ab3bf. Reproduced locally: create_checks_auto_lifecycle_authority_before_template_lookup, the two start-path tests, the full service::network_policy::tests module and node_manager_transport_tests all pass (22 tests). I tried to re-run mutation C8 (dropping the GetTemplate failed: prefix), but my build host ran out of disk mid-build, so the mutation table is the author's evidence, not mine. I read the rest of the production diff without running it. I didn't run a Kind smoke.

[the-gigi-pplx]

Approved. 1 non-blocking finding in the review comment above.