[ac-pplx]

Based on main: #368 merged as 723bbcc, and this PR is now its one commit rebased onto that. The rebase had one conflict, with #431's leader-election change (see Rebase onto main).

What merging does

After this PR, shared code outside the substrate module no longer matches on Substrate's variants anywhere outside tests. The four sites this phase covers fall into two kinds, and each kind gets one seam:

• Kubernetes-only state comes from Substrate::as_kubernetes() -> Option<&KubernetesSubstrate>, behind the kubernetes feature. Two cluster-controller sites use it: the Kubernetes reconciler set in reconcilers.rs (still typed on KubernetesSubstrate) and the leader-election arm in lib.rs, which contends for the Lease only on Kubernetes. Since Observability wave 4a: drop background trace noise, fix the gateway hydration race, add shim teardown spans #431 that arm builds the Lease store's own failure-traced client, so it binds nothing (Some(_)).
• Execution routability becomes substrate behaviour: SubstrateAdapter::ensure_routable_execution. The contract is "started promises a routable execution row that may need hydrating from a live pod". The api's cancelled-wake completion redrive and the cluster controller's transient-lifecycle sweeper both ask it.

So shared code can't reach around the new method through as_kubernetes(), KubernetesSubstrate::hydrate_restored_execution and ExecutionHydrationOutcome narrow to the Kubernetes substrate module. The outcome type's unused as_str and Display are removed. Substrate's Direct arm of observe_pod_identities now forwards to DirectSubstrate instead of repeating the default inline; it was the only defaulted method that did, and the answer is the same.

Production behaviour, wire formats, schema, metrics, logs and configuration are unchanged. Two behavioural tests cover Kubernetes paths that had no test, using existing fixtures. Test construction (Substrate::Kubernetes(Box::new(...))) is the only place outside the substrate module that still names a variant; making the variants private is left for the phase's last PR.

The accessor

impl Substrate {
    #[cfg(feature = "kubernetes")]
    pub fn as_kubernetes(&self) -> Option<&kubernetes::KubernetesSubstrate>;
}

It returns Some exactly for the Kubernetes variant, borrowing the same boxed value the old match arms bound. Its doc comment states the split the repo owner chose: Kubernetes-only state goes through the accessor, behaviour goes on SubstrateAdapter, and static facts go in SubstrateCapabilities.

The routability method

pub enum ExecutionRoutabilityCheck { Unconditional, Hydrated, AlreadyRoutable, NotRoutable }
impl ExecutionRoutabilityCheck { pub const fn is_routable(self) -> bool } // false only for NotRoutable

trait SubstrateAdapter {
    async fn ensure_routable_execution(&self, sandbox_id: SandboxId, budget: Duration)
        -> Result<ExecutionRoutabilityCheck, SubstrateError> {
        Ok(ExecutionRoutabilityCheck::Unconditional) // Direct's answer
    }
}

• Default. The default is Direct's answer: Direct's started makes no pod-routability promise, so nothing is checked or written. DirectSubstrate does not override it.
• Kubernetes override. KubernetesSubstrate overrides it with today's hydrate_restored_execution(sandbox_id, budget). Its three outcomes map one to one, and its error passes through unchanged. budget bounds the apiserver read. NotRoutable can still have recorded a matching pod's identity on an execution row that is not running, such as a recovering one.
• Forwarding. impl SubstrateAdapter for Substrate forwards both arms: Direct to DirectSubstrate (and so to the default), Kubernetes to the override. The new tests call the method through Substrate, so a missing Kubernetes arm fails them (mutation M3).

How the two sites differ, and why this is one method with a parameter and a four-way answer:

	wake-completion redrive (api)	transient-lifecycle sweeper (cluster controller)
input	sandbox_id, REDRIVE_HYDRATE_BUDGET (per attempt)	sandbox_id, the sweeper's configured probe_timeout
error mapping	err.to_string()	emits the execution_hydration Error outcome event, then err.to_string()
outcome telemetry	none	Hydrated → info HydratedExecutionFromLivePod; AlreadyRoutable → debug Skipped/already_routable; NotRoutable → debug Deferred/no_routable_pod; Direct → no event
result	bool	the sweeper's own ExecutionRoutability::{Routable, NotRoutable}

The inputs differ only in the budget value, which becomes the parameter. Each site keeps its own error mapping and telemetry. The sweeper emits nothing for Direct but a Skipped event for an already-routable Kubernetes row, so the answer has to separate Unconditional from AlreadyRoutable; a bool or a two-way enum would have changed telemetry. The sweeper's two-variant ExecutionRoutability stays as its SweepSink contract, and the new type is named …Check to avoid colliding with it. The sweeper's SweepSink::ensure_routable_execution calls SubstrateAdapter::ensure_routable_execution(self.substrate.as_ref(), …) explicitly, so the call doesn't read as recursion.

Site table

site	before	after	Direct	Kubernetes	why identical
crates/libs/cluster/controller/server/src/reconcilers.rs run_reconcilers	match substrate.as_ref() { Substrate::Direct(_) => log skip, Substrate::Kubernetes(kube) => { let kube = Arc::new((**kube).clone()); … } }	match substrate.as_kubernetes() { None => log skip, Some(kube) => { let kube = Arc::new(kube.clone()); … } }	None: same "skipping Kubernetes-only reconcilers on direct substrate" log	Some: same KubernetesSubstrate::clone, same reconcilers in the same order	same arm per substrate; arm bodies unchanged except the deref
crates/libs/cluster/controller/server/src/lib.rs run (leader election)	match &substrate { Substrate::Kubernetes(kube) if le_config.enabled => campaign, _ => election disabled }	match substrate.as_kubernetes() { Some(_) if le_config.enabled => campaign, _ => election disabled }	None → _: election disabled, same log	enabled → campaign on the Lease store's own failure-traced client, built exactly as #431 builds it; kill switch → _	same guard, same arms, same arm body as main
crates/libs/api/src/controllers/sandbox/wake_completion.rs wake_completion_redrive	Direct(_) => Ok(true); Kubernetes(kube) => hydrate_restored_execution(id, REDRIVE_HYDRATE_BUDGET): Hydrated | AlreadyRoutable → true, NotRoutable → false, Err → err.to_string()	substrate.ensure_routable_execution(id, REDRIVE_HYDRATE_BUDGET): Ok(check) → check.is_routable(), Err → err.to_string()	Unconditional → true (a ready future, no await point)	same hydration, same budget, same bool, same error string	same value per substrate and outcome
crates/libs/cluster/controller/server/src/transient_lifecycle_sweeper/store.rs SubstrateSweepSink::ensure_routable_execution	Direct(_) => Ok(Routable); Kubernetes(kube) => hydrate_restored_execution(id, probe_timeout) with the events above, then Routable/NotRoutable	SubstrateAdapter::ensure_routable_execution(self.substrate.as_ref(), id, probe_timeout): same Error event and string, Unconditional → no event, the other three → the same events, then is_routable() → Routable/NotRoutable	Unconditional → Routable, no event	same hydration, budget, events and result	same events and result per substrate and outcome

The #[cfg(feature = "kubernetes")] arms at the two routability sites go away. Without the feature, Direct is the only variant and its answer is the same. The comments that named these sites are updated: the hydrate_restored_execution doc (which named only the sweeper), the redrive's docs, and the reconcilers' sweeper comment (the sink needs SubstrateAdapter methods, not just delete_execution). The NotRoutable doc on ExecutionHydrationOutcome no longer claims nothing was written, because a recovering execution records the pod identity.

Coverage by mutation

These runs were on the pre-rebase head 05b795c and were not repeated after the rebase, which changed only the leader-election arm's binding (Some(kube) → Some(_)). Each mutation was applied on top of the commit. For each, cargo nextest run --locked -p <pkg> --all-features --retries 0 --run-ignored all ran for both pplx-space-api and pplx-space-cluster-controller-server with SPACE_CI_REQUIRE_TEST_DEPS=1, and the mutation was then reverted. The tree was clean afterwards. Below, redrive_on_kubernetes is controllers::sandbox::lifecycle::wake_completion_postgres_tests::wake_completion_redrive_on_kubernetes_hydrates_before_completing, and routability_check is …::hydration_and_suspend::routability_check_answers_with_the_hydration_verdict.

#	mutation	breaks	pplx-space-api	cluster controller
M1	trait default answers NotRoutable	Direct, both routability sites	FAIL controllers::sandbox::lifecycle::characterization_tests::redrive_completes_only_a_restoring_row	pass
M2	Kubernetes override maps AlreadyRoutable → NotRoutable	Kubernetes, both routability sites	FAIL routability_check case 2	pass
M12	Kubernetes override maps Hydrated → NotRoutable	Kubernetes, both routability sites	FAIL redrive_on_kubernetes, routability_check case 1	pass
M3	Substrate forwarding: Kubernetes arm answers the default	Kubernetes, both routability sites	FAIL redrive_on_kubernetes, routability_check cases 1–4	pass
M4	redrive site: routable only when Unconditional	Kubernetes at the redrive	FAIL redrive_on_kubernetes	pass
M11	redrive closure hard-coded to Ok(true) (never asks the substrate)	the redrive's hydration	FAIL redrive_on_kubernetes	pass
M5	redrive site: Unconditional not routable	Direct at the redrive	FAIL redrive_completes_only_a_restoring_row	pass
M6	sweeper site: routable only when Unconditional	Kubernetes at the sweeper	pass	pass: uncovered
M7	sweeper site: Unconditional not routable	Direct at the sweeper	pass	pass: uncovered
M8	reconcilers.rs: as_kubernetes() → None	Kubernetes reconciler set	pass	pass: uncovered
M9	lib.rs: as_kubernetes() → None	Kubernetes leader election	pass	pass: uncovered
M10	as_kubernetes() returns None for Kubernetes	both cluster-controller accessor sites	pass	pass: uncovered

Uncovered sites:

• The sweeper's routability site (M6, M7). The cluster controller has no KubernetesSubstrate fixture (the api crate's kubernetes::tests fixtures are pub(crate)), and its sweeper tests drive a fake SweepSink. The method underneath is covered at the api layer (M1–M3, M12). The site's own logic is the event selection and the two-way mapping. Its Direct path is unreachable in practice: the candidate select treats managed_by <> 'k8s' rows as routable before this call.
• reconcilers.rs and lib.rs (M8–M10). These are process-startup wiring, and no test runs run or run_reconcilers. The Kind run below is their evidence: on the deployed binary, a cluster-controller replica contended for leadership, was elected, and started the Kubernetes reconciler set. Direct's answer at these sites (None) can't be broken into Some without a KubernetesSubstrate, so only the Kubernetes direction was mutated.

New tests, both behavioural, using existing fixtures:

• routability_check_answers_with_the_hydration_verdict: an rstest table asking ensure_routable_execution through Substrate on Kubernetes. The cases are a matching pod (Hydrated), an already-routable row with a broken apiserver (AlreadyRoutable), an absent pod (NotRoutable), and an apiserver error (Err).
• wake_completion_redrive_on_kubernetes_hydrates_before_completing: the redrive on Kubernetes, with a scripted apiserver, over a restoring row whose execution is creating with no pod IP. The database reports that row unroutable, so the completion CAS alone would refuse it.
  • When the apiserver serves the execution's matching, annotated Running pod, the hydration makes the row routable and the redrive completes it to started.
  • When the pod is gone, the row stays restoring; the CAS's routability predicate also holds that case.

The redrive test reaches scripted_test_doubles::{scripted_kube_client, kube_status, pod_reply} (and the KubeRequest, KubeReply and KubeLog types in their signatures) and adapter_characterization_tests::sandbox_pod through pub(crate) visibility. All of these are #[cfg(test)]. No existing test was changed. The characterization suites pass unchanged: #323 (cluster controller) and #273/#290/#326/#330 (api).

Decisions made unattended

1. One method, a budget parameter, a four-way answer. The two sites differ only in budget value, error mapping and telemetry. The last two stay at the call sites, and the answer keeps the distinction the sweeper's events need (table above).
2. Substrate forwards Direct to DirectSubstrate rather than answering inline. This is the pattern retire_fence_mismatch uses, so a future Direct override can't be silently bypassed. observe_pod_identities, the one defaulted method that answered inline, now follows the same pattern.
3. The repo owner's decision over the phase 0 inventory. The inventory proposed a lease-store accessor for lib.rs and a capability field for reconcilers.rs; I followed the repo owner's decision (as_kubernetes()) for both.
4. Visibility narrowed to the kubernetes module. hydrate_restored_execution and ExecutionHydrationOutcome become pub(in crate::controllers::substrate::kubernetes); the re-exports become pub(super) in pod_ops and private in kubernetes. That is the narrowest visibility their users (the override and the module's own tests) need. Removing as_str/Display compiles and lints cleanly on every target, so nothing used them.
5. Archcheck growth not split. cargo arch reports advisory file_size.listed_no_growth growth for substrate/mod.rs and kubernetes/adapter/substrate_adapter.rs, which api: ask the substrate for capabilities instead of matching on it #368 also grows. The trait, its forwarding impl and the single Kubernetes impl live there, so I didn't split files in a behaviour-preserving PR. There are 0 required findings.
6. Tests. The two tests above close the Kubernetes gaps at the lowest owning layer and at the redrive. I added no cluster-controller test (no fixture) and no test that pins Direct's default by itself; the Direct redrive characterization test covers it.
7. Kind run shape. The typed smoke ran unsharded, so all four CI shards' scenarios ran serially on one stack. BOOTSTRAP_TIMEOUT was 3600 instead of CI's 1800, because this box builds the Rust artifact image itself instead of downloading CI's.
8. Kind build and storage layout. The box's home volume was full, so the run kept its state elsewhere. The Rust artifact image was built once by build-rust-artifacts.sh in its docker mode from this commit's tree. The final bring-up then used the harness's prebuilt mode with that image's ID and this commit's SHA, the path CI's smoke shards take. The Kind nodes' containerd storage (SPACE_KIND_DATA_ROOT) was a task-owned directory on the Docker host's tmpfs, and TMPDIR pointed off the home volume. The Kind containerd image cache under the home volume and the Docker volumes were not touched.
9. Some(_) in the leader-election arm. Observability wave 4a: drop background trace noise, fix the gateway hydration race, add shim teardown spans #431's body no longer uses the substrate, so the arm binds nothing. The other way to write the same guard is if substrate.as_kubernetes().is_some() && le_config.enabled. I kept the match, as the rebase plan asked, so the diff against main stays two lines. After Observability wave 4a: drop background trace noise, fix the gateway hydration race, add shim teardown spans #431 this site asks only "is this Kubernetes?", a capability-shaped question; turning it into a SubstrateCapabilities flag is left to the phase's later PRs.

Overlaps

These trials were run before #368 merged, on the previous base, and were not repeated for this rebase. All nine PRs are still open. For each overlapping PR, I cherry-picked #368's commit and then this commit onto its head as a trial:

• cluster-controller: bind the substrate value instead of writing it in SQL #373 (transient_lifecycle_sweeper/store.rs SQL literals) edits that file's first import line and SQL constants and queries; this PR edits the pplx_space_api import block and the ensure_routable_execution body. The trial applies cleanly, so I expect the rebase to be clean.
• api: document the Direct substrate's contract for agents #357 (controllers/substrate/AGENTS.md) is not merged. Its bullet "Matching Substrate::Kubernetes(k) to reach Kubernetes-only state is fine" overlaps with this PR's accessor: once both land, it should name as_kubernetes(). Trial: clean.
• Bound reconnect recovery on unpublished local suspends #350, PoC: claim BYOK warm sandboxes by switching Firecracker drives #336, Fence pod materializations and record worker incarnation #319 touch substrate_adapter.rs, substrate/mod.rs, kubernetes/mod.rs, pod_ops.rs or pod_ops/restore.rs in other hunks. Trials: clean.
• api: decide admission pool claims for the create's owner; collect refused claims promptly #168, [api] restore full snapshots only onto CPU-compatible hosts, behind a default-off gate #176, credmgr: scope volume DEKs to the node running the sandbox #128, [space] order data-plane scale-in by open connections via pod-deletion-cost #28 touch substrate/mod.rs, kubernetes/mod.rs, pod_ops.rs, or the cluster controller's lib.rs/reconcilers.rs. All four are on older bases that already conflict with api: ask the substrate for capabilities instead of matching on it #368. Their own hunks don't touch this PR's lines; [space] order data-plane scale-in by open connections via pod-deletion-cost #28's new reconciler inside the Kubernetes arm uses kube.clone(), which still compiles under Some(kube).
• None of these PRs uses hydrate_restored_execution, ExecutionHydrationOutcome, as_str/Display on it, or the widened test doubles from outside the kubernetes module.
• No open PR adds a new Substrate::Kubernetes(…)/Substrate::Direct(…) match outside tests.

Rebase onto main

This PR's commit was rebased with git rebase --onto 723bbcc 251e4c7 (dropping #368's original commit). There was one conflict, in crates/libs/cluster/controller/server/src/lib.rs's leader-election arm. #431 had replaced kube.k8s.client.clone() with a Lease client of its own, failure_traced_kube_client_from_env(shutdown_token.clone()), mapped to RunClusterControllerError::LeaseClient. This PR had changed the arm's pattern. The resolution keeps both: this PR's as_kubernetes() match and #431's body exactly as main has it. Because that body doesn't use the substrate, the arm binds nothing. The whole diff against main in this file:

-        match &substrate {
-            Substrate::Kubernetes(kube) if le_config.enabled => {
+        match substrate.as_kubernetes() {
+            Some(_) if le_config.enabled => {
                 // Renewals and standby polls are the idle steady state, so the
                 // lease gets a client that records spans only for failures.

git range-diff against the pre-rebase commit shows only that hunk. Apart from the commit message, nothing else changed. git diff 723bbcc HEAD touches the same 13 files with the same counts (+358/−137).

Commits on main since the previous base (64afd0b) that touch this PR's files, and how each interacts:

• api: ask the substrate for capabilities instead of matching on it #368 (723bbcc). This is the squash of the old parent 251e4c7; git range-diff shows only the commit title changed. No interaction.
• Observability wave 4a: drop background trace noise, fix the gateway hydration race, add shim teardown spans #431 (Observability wave 4a), cluster/controller/server/src/lib.rs. This is the conflict above. Its other hunk in that file (the LeaseClient error variant) is untouched and still used.
• perf(api): keep worker channels to sandbox pods across requests #405 (keep worker channels to sandbox pods), substrate/mod.rs, kubernetes/mod.rs and kubernetes/adapter/substrate_adapter.rs. It adds kept worker channels: RuntimeClient::kept/retire_kept_connection, ResolvedPodInstance, pod_worker_channels, and resolve_pod_instance/kept_worker_client in the runtime-client methods. Those are different hunks and methods from this PR's ensure_routable_execution, as_kubernetes(), observe_pod_identities forwarding and visibility changes. It adds no Substrate variant match outside the substrate module, and no defaulted trait method that Substrate answers inline.
• Treat TCP connect-phase failures as never sent #408 (TCP connect-phase failures are never sent), substrate/mod.rs. In this file it removes the node_api_transport helper and rewords node_api_connect_endpoint's doc. That is node-api transport code this PR doesn't touch, so there's no interaction.
• After the rebase, Substrate::Direct/Substrate::Kubernetes is named outside controllers/substrate/ only in test code.

Validation

All runs below were on head 0eaa61f: this commit on main at 723bbcc (#368). Its tree is identical to 77ca814, the same rebase before a message-only amend. Some runs started on 77ca814.

check	result
cargo fmt --all -- --check	pass
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings	pass, 0 warnings
cargo check --locked -p pplx-space-api (no features)	pass, 0 warnings
cargo nextest run --locked --retries 0 -p pplx-space-api --all-features, SPACE_CI_REQUIRE_TEST_DEPS=1 (nextest's Postgres and Redis setup scripts), 3 consecutive runs	1746/1746 passed each run; 1 skipped, the #[ignore]d benchmark_single_node_occupancy
same for pplx-space-cluster-controller-server (--all-features), 3 consecutive runs	960/960 passed, 0 skipped, each run
pre-commit hook (gitleaks)	passed on commit; pre-commit run --from-ref 723bbcc --to-ref HEAD passed
public review (space-public-review skill) of the staged change, the commit message and this text	no findings: fixtures use TEST-NET addresses, the repo's existing zero-UUID IDs and synthetic pod names and UIDs

Not repeated after the rebase: the mutation runs M1–M12 and cargo arch, which ran on the pre-rebase head 05b795c (see above). The rebase changes only the leader-election arm's binding, which no unit test reaches (M9). The Kind run below exercises that arm.

Full Kind validation

These are this rebase's results, on a fresh space-local cluster at 0eaa61f. The cluster was created after the artifact image was built (decision 8), by smoke-kind-e2e.sh's bring-up.

Build provenance. The Rust artifact image was built by build-rust-artifacts.sh (docker mode) from a clean checkout of this tree. The bring-up then used it in prebuilt mode, which checks the checkout's SHA and the image ID. Before and after the suites, the sha256 of space-api-svc, space-internal-api-svc and space-cluster-controller was taken in every running api (2), internal-api (1) and cluster-controller (2) pod. Every hash equalled the artifact image's. The controllers report git_sha=0eaa61f9c98e. Tilt reported 50/50 resources ok, none pending or in error, after the bring-up and again after the suites.

The rebased leader-election arm, at runtime. Both cluster-controller replicas logged "contending for leadership", and one logged "elected leader; starting reconcilers". The leader then started the Kubernetes reconciler set that as_kubernetes() gates in reconcilers.rs (warm pool, kube-node-sync, the transient-lifecycle sweeper, node drain, …), with no Direct "skipping" log.

Suite	Result
smoke-kind-e2e.sh (bring-up, DATA_PLANE_URL set as CI does)	PASS
smoke-kind-typed.sh all (unsharded; SKIP_BOOTSTRAP=1, SPACE_SMOKE_KIND_ADMIN_TOKEN_FILE and SPACE_SMOKE_KIND_COLD_RECEIPTS set, and the data-plane wait, as CI does; host dm targets loaded)	ECW surface: 77 passed, 0 failed, 1 infra error. Native surface: 23 passed, 0 failed, 0 inconclusive. Exit 2, from the infra error.
the infra error: local.ecw_template_build_from_oci_boots	INFRA_ERROR: storage admission requires 55834574848 available bytes but found 18519506944. Builder storage admission (build/mgr/src/storage.rs, a statvfs of the builder's storage root) needs 52 GiB free. On this box that root was the task-owned tmpfs (decision 8), with 17.2 GiB free and the cluster in it. Rerun once on the same cluster and binaries: the same error, with the same free bytes as the tmpfs at that moment. Classified ENVIRONMENTAL, since the failure is explained by host storage, so I made no main comparison. The scenario never reached code this PR changes; it passed on #386's previous head.
smoke-kind-leader-election-brownout.sh, SPACE_SMOKE_LEADER_ELECTION=1	PASS, all three scenarios. (1) One leader and one reconciler runner. (2) A no-leases-RBAC contender reported the refusal on sight, ran no reconcilers and did not restart over 2× its budget. (3) Apiserver outage: the readiness marker cleared, the holder resigned 37 s into the outage (TTL 60 s), and one replica spent its liveness budget and was restarted. After the apiserver came back, there was one leader, one reconciler runner, and both replicas were Ready.
smoke-kind-sandbox-fork.py	PASS
smoke-kind-e2e.sh again (SKIP_BOOTSTRAP=1)	PASS

🤖 Generated with Claude Code

[nickdujay-pplx]

Approved at 0eaa61f, unchanged from the detailed review. No blocking findings: the accessor and routable-execution abstraction preserve the Direct and Kubernetes behavior, including the rebased leader-election path.

The earlier local run passed 150 targeted tests with real PostgreSQL and mock Kubernetes coverage, and CI is green for this exact commit.