[ac-pplx]

Gives KubernetesSubstrate a SubstrateBudgets value for the wall-clock budgets of create_execution, suspend_execution, started warm-pool adoption and teardown's worker connect. Production takes the default, which is exactly today's constants. Tests can shorten the budgets, so the readiness-failure, joined-create and suspend-timeout arms that #273 deferred now run through the real entry points, against real Postgres and #273's scripted apiserver and worker. The production diff is only this seam.

The seam

kubernetes/budgets.rs defines SubstrateBudgets, a Copy struct of six Durations. KubernetesSubstrate holds one in a private budgets field, which the production constructor sets the way it sets the neighbouring recovery_join_backoff policy. Tests in the kubernetes module tree assign the field directly; there is no setter.

KubernetesSubstrate::try_from_env, the only production constructor, sets budgets: SubstrateBudgets::default(). Default returns SubstrateBudgets::DEFAULT, a const built from the existing constants, so each field is its constant by construction:

Field	Default	Constant
create_ready	135 s	CREATE_READY_BUDGET (= NODE_RPC_BLOCKING_BUDGET)
create_ready_phase_margin	5 s	READY_WAIT_PHASE_MARGIN
preserve_decision	5 s	PRESERVE_DECISION_BUDGET
suspend_durability	300 s	TEARDOWN_SUSPEND_DURABILITY_BUDGET (now pub(super) so the default can name it)
pooled_split_receipt_wait	5 s	WARM_POOL_SETUP_RPC_TIMEOUT
teardown_worker_connect	3 s	POD_WORKER_CONNECT_TIMEOUT

The constants, their docs and their static asserts are unchanged.

SubstrateBudgets::ready_wait_bounds is the existing support::ready_wait_bounds arithmetic over the struct's two create fields:

• The inner window is clamped to create_ready − create_ready_phase_margin, and the outer backstop is one margin later.
• So the backstop never exceeds create_ready while the margin fits inside it. A debug_assert! checks that, and the subtraction saturates rather than panics in release builds.
• The free ready_wait_bounds, still called by wake, start, restore and access, delegates to SubstrateBudgets::DEFAULT.ready_wait_bounds. For the defaults that is the same 130 s inner window and 135 s backstop.
• The existing ready_wait_bounds unit tests in kubernetes/tests.rs still pass unchanged against the free function.

Production deadlines, error messages and log lines are identical. The create and suspend timeout messages print their budget with Duration's Debug ({:?}), which renders the 135 s and 300 s defaults as 135s and 300s, exactly the earlier {}s of as_secs(), and a test's sub-second budget as, say, 600ms instead of 0s. Two unit tests in adapter/tests.rs (create_ready_timeout_message_names_the_budget, generation_suspend_timeout_message_names_the_budget) compare each message at the default budget with the old format's output byte for byte, and pin the sub-second rendering. These two are the only messages derived from a SubstrateBudgets field; no test asserted their text before. Nothing changes on the wire, in the schema, in metrics or in configuration.

Call-site inventory, before → after

Constant	Use site	Before	After
CREATE_READY_BUDGET	create_execution outer readiness timeout (adapter/substrate_adapter.rs)	constant	self.budgets.ready_wait_bounds(self.budgets.create_ready)
	create_execution outer-timeout message	constant	self.budgets.create_ready, through the new create_ready_timeout_error
	joined-create (ActivationInFlight) phase deadline	constant	self.budgets.ready_wait_bounds(self.budgets.create_ready), of which the arm keeps only the phase deadline; it has no outer timeout (Finding 3)
	free ready_wait_bounds (9 call sites: 7 in execution.rs, 2 in adapter/substrate_adapter.rs; wake, start, restore, access)	constant	SubstrateBudgets::DEFAULT (unrouted, same value)
	static asserts in mod.rs, docs	constant	unchanged
READY_WAIT_PHASE_MARGIN	the two create call sites above	constant	create_ready_phase_margin
	free ready_wait_bounds	constant	SubstrateBudgets::DEFAULT (unrouted)
PRESERVE_DECISION_BUDGET	create_execution preserve decision	constant	self.budgets.preserve_decision
TEARDOWN_SUSPEND_DURABILITY_BUDGET	suspend_execution deadline	constant	self.budgets.suspend_durability
	poll_generation_suspend timeout message (adapter/suspend.rs)	constant	new durability_budget parameter, passed self.budgets.suspend_durability; the two lower-layer tests in adapter/tests.rs pass their own budget
WARM_POOL_SETUP_RPC_TIMEOUT	started adoption's split-receipt deadline	constant	self.budgets.pooled_split_receipt_wait
	5 other sites: non-started receipt health check, metadata update, network policy, auto-lifecycle, route-adoption Wake	constant	unrouted
POD_WORKER_CONNECT_TIMEOUT	teardown_execution worker client (adapter/teardown.rs)	constant	self.budgets.teardown_worker_connect
	endpoint_for_pod_ip, builder_node_client_at, the suspend node-api endpoint	constant	unrouted

Budgets these paths use that stay as they are

• Already injectable through KubernetesState: create_data_plane_ready_timeout, create_command_ready_timeout and k8s_blocking_delete_timeout.
• Sub-second and shared with wake, start and access: START_WAIT_CONNECT_TIMEOUT (500 ms proxy probe), START_WAIT_COMMAND_PROBE_TIMEOUT (1 s), the start-wait poll intervals and window, COMMAND_READY_MAX_POLL_INTERVAL, REFLECTOR_WAIT_BUDGET (250 ms), and the 250 ms suspend poll pacing.
• Bounded by an injected budget on these paths: NODE_API_ENDPOINT_RESOLVE_TIMEOUT and its retry interval, which run inside timeout_at(suspend_deadline) on suspend.
• Not a wait the tests depend on:
  • POD_IP_WATCH_TIMEOUT_SECS (the server-side watch timeout);
  • K8S_BLOCKING_DELETE_FALLBACK_GET_TIMEOUT (only after a full blocking-delete budget);
  • SUSPEND_PENDING_REASON_LOG_INTERVAL (log pacing only);
  • NODE_RPC_TIMEOUT (the channel timeout shared with every node RPC).

No clock is injected. Short real budgets are enough, and Postgres stays real.

Call sites left unrouted, and why

• The 9 free ready_wait_bounds callers (wake, start, restore, access). They are outside the deferred paths, no test here needs them, and execution.rs, which holds 7 of them, is edited by open PR [api] stop retrying restore pods whose snapshot the host CPU cannot load #221. They keep today's values through SubstrateBudgets::DEFAULT.
• The 5 other WARM_POOL_SETUP_RPC_TIMEOUT RPC timeouts. No test here waits them out. One of them, in apply_pooled_execution_auto_lifecycle_with_client, is an associated function without self, so routing it means a new parameter.
• The 3 other POD_WORKER_CONNECT_TIMEOUT sites, all endpoint connect timeouts. endpoint_for_pod_ip is an associated function without self used across the substrate. The builder client and the suspend node-api endpoint are not on a path a test here waits out.

Tests

The tests live in #[cfg(test)] modules under kubernetes/adapter_characterization_tests/. They call create_execution, suspend_execution_tracked (the TaskTracker wrapper around suspend_execution), prepare_pooled_execution or retire_execution, each against its own Postgres database from pg_test_db(). Every new or converted test that starts the scripted worker enters its child process with enter_sim_worker_process_with_owned_token_files(), so bearer loading reads test-owned token files and never the host's /var/run/space-auth/*.token.

New module create_readiness.rs

Wired by one mod create_readiness; line.

create_readiness_failure_keeps_or_releases_only_what_its_arm_owns has 7 cases.

• The pod create's response carries the pod's UID and IP. Recording them moves the execution out of creating, so readiness reads the pod rather than watching it; resolve_pod_ip skips that read only for creating and recovering executions.
• Outer-budget cases: the scripted apiserver never answers readiness's first pod read. No readiness phase bounds that read, so only the outer budget ends the wait, whatever the runner's load. The budgets are create_ready 1 s with a 500 ms margin and preserve_decision 1 s. Readiness reaches that read after the 250 ms reflector wait, well inside the budget.
• Cases that end on their own phase keep the production create budget and a 1 s preserve decision.
• Every case asserts that the substrate's task tracker is empty, so no cleanup is left in the background.

Case	Readiness failure	Apiserver afterwards	Asserted outcome
outer_budget_preserves_a_reachable_pod	outer budget cuts off the hung read	pod scheduled on the cached node, proxy reachable	Ok: started / on_node on the worker's node id, command_ready: false, no binding; no DELETE; the execution is still active and running with the pod's UID and IP; no cancel counted
outer_budget_tears_down_a_terminating_pod	same	pod has deletionTimestamp	Err(Timeout { phase: None }); exactly one DELETE, of the pod's path; execution retired; no cancel counted
slow_preserve_decision_tears_down	same	pod scheduled on an uncached node whose GET never answers	as above, and the create returns before the 5 s production preserve budget
outer_budget_defers_cleanup_of_a_vanished_pod	same	pod gone (404)	Err(Timeout { phase: None }); no DELETE; the claim stays for the orphaned-sandbox reconciler; one cancel counted toward backpressure
unreachable_node_mgr_defers_cleanup_of_a_vanished_pod	every guest probe returns Unavailable until the 500 ms command-ready phase ends	pod gone	Err(Timeout { phase: GuestNotReady }); no DELETE; claim kept; one cancel counted
contention_with_accepted_cleanup_stays_retryable	first execution read hits a held ACCESS EXCLUSIVE table lock under lock_timeout	DELETE accepted	Err(LockContention); one DELETE; execution retired
contention_with_rejected_cleanup_is_demoted	same	DELETE returns 500	Err(Failed), demoted; one DELETE; execution retired

The cleanup decision is err_defers_cleanup = is_nodemgr_unreachable(&err) || matches!(err, Timeout { .. }). Case 4's outer-budget timeout names neither phase nor node-mgr, so only the Timeout half defers it; removing that half fails it (D1). The is_nodemgr_unreachable half has no genuine input on this path and is not pinned (Finding 4).

joined_create_waits_on_the_owners_pod has 2 cases. Another attempt owns the activation, claimed through K8sStore::begin_activation_execution_row.

• returns_the_owners_reachable_pod: Ok with no lease_completion, command_ready: true, the owner's pod UID as the binding's execution object, and the worker's node id.
• times_out_without_disturbing_the_owner: Err(Timeout { phase: PodNeverReady }) within the create budget, although the data-plane timeout is 60 s.
• Both cases: no pod POST or DELETE; the owner keeps its activation attempt; the execution stays active; the task tracker is empty.

Suspend: hydration_and_suspend.rs

suspend_execution_polls_the_node_within_its_durability_budget has 3 cases. It adds only what the entry point owns: the substrate's budget reaches the poll, and the claim is kept on timeout.

• The commit and the fenced pod retirement after a durable reply are pinned by suspend_execution_commits_the_nodes_artifact_before_retiring_the_pod.
• The poll's per-reply handling is pinned by adapter/tests.rs.

Cases:

• pending_then_durable: the node answers Unavailable, then Pending, then Durable (2 s budget). The suspend returns the node's artifact.
• pending_past_the_budget (600 ms budget): the node stays Pending.
• node_api_never_resolves (600 ms budget): there is no node-api pod for the node.
• For both timeout cases: Err(Timeout) before the 5 s bound; no apiserver request; the sandbox stays suspending and the execution active, so a retry joins the claim.

Converted slow tests

• started_pooled_adoption_refuses_a_worker_without_split_receipts (pooled_adoption.rs) sets pooled_split_receipt_wait to 300 ms and asserts the refusal arrives before the 5 s production budget. Wall time: 5.0 s → 0.32 s.
• teardown_keeps_execution_tracked_when_the_node_cannot_finish::retire_worker_unreachable (create_and_teardown.rs) keeps api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273's dial to TEST-NET-1 192.0.2.1 and its outcome assertions: the retire fails, no pod is deleted, and the execution stays tracked. It sets teardown_worker_connect to 300 ms and asserts the retire returns within POD_WORKER_CONNECT_TIMEOUT (3 s).
  • A TEST-NET connect never succeeds, so the bound cannot false-fail. Either the host answers ENETUNREACH and the dial fails at once, or it drops the SYN and the 300 ms budget ends the dial.
  • Where the host drops the SYN, the test proves the budget: reverting teardown's connect to the constant fails it (S6). This host drops it (a direct connect to 192.0.2.1:80 was still pending after 3 s), and the case takes 0.39 s instead of 3.2 s.
  • Where the host answers ENETUNREACH, the assertion passes trivially and proves nothing about the budget.

Shared doubles

scripted_test_doubles.rs: ScriptedReply gains Hang (never answer). The suspend test uses the Sequence reply that #326 added.

Mutation proofs

Each mutation was a temporary edit to one file, followed by the whole adapter_characterization_tests:: set (55 tests). The listed tests were confirmed to fail, then the file was restored and touched. 24 mutations, all killed:

• S1–S6 show the seam is real: each restores a production constant, and the converted test fails. S6 needs a host that drops the TEST-NET SYN; see "Converted slow tests" above.
• T1–T3 show the test fakes are real.
• D1 removes the Timeout half of err_defers_cleanup.
• The rest break the behavior the tests pin.

Where each one ran:

• S6 ran on this head: 54 passed, and only retire_worker_unreachable failed, at 3.07 s, on the connect-budget bound.
• The 15 that touch create_readiness (S1, S2, B1–B9, D1, T1–T3) ran on 4fb236f. Since then the only change to their paths is the create timeout message's formatting.
• The other 8 (S3–S5, J1, J2, P1–P3) ran on 1c25d9d. Their production code and tests are the same here apart from the suspend test's move to api: pin the pooled-setup error arms #273 called unreachable #326's Sequence API.

"readiness N", "joined N" and "suspend N" refer to the cases in the order listed above.

#	Mutation	Failing tests
S1	create's outer wait uses ready_wait_bounds(CREATE_READY_BUDGET) again	readiness 1, 2, 3, 4 (the create no longer ends within 10 s)
S2	preserve decision uses PRESERVE_DECISION_BUDGET again	readiness 3 (took longer than 5 s)
S3	joined create uses the constant budget again	joined 2
S4	suspend deadline uses TEARDOWN_SUSPEND_DURABILITY_BUDGET again	suspend 2, 3
S5	split-receipt deadline uses WARM_POOL_SETUP_RPC_TIMEOUT again	split-receipt test (5.0 s)
S6	teardown's worker connect uses POD_WORKER_CONNECT_TIMEOUT again	retire_worker_unreachable (3.07 s; on a host that drops the TEST-NET SYN)
B1	never preserve a slow create	readiness 1
B2	preserve a pod that is deleting	readiness 2
B3	unbounded preserve decision	readiness 3
B4	never defer cleanup of an authoritatively absent pod	readiness 4, 5
B5	defer without mark_cause(NodeMgrUnreachable) (the guard then spawns cleanup)	readiness 4, 5
B6	never demote after a rejected cleanup delete	readiness 7
B7	demote after an accepted cleanup delete too	readiness 6
B8	cleanup forgets tracking for the wrong generation	readiness 2, 3, 6, 7
B9	outer-budget timeout claims a phase	readiness 2, 3, 4
D1	err_defers_cleanup keeps only is_nodemgr_unreachable(&err)	readiness 4
J1	an in-flight activation for the same target is not joined	joined 1, 2
J2	joined create reports command_ready: false	joined 1
P1	a Pending observation is terminal	suspend 1, 2
P2	a transient node-api failure is terminal	suspend 1
P3	the node-api resolve timeout is reported as Unavailable	suspend 3
T1	(test fake) the contention pool has no lock_timeout	readiness 6, 7 (the read waits on the lock instead of aborting)
T2	(test fake) a vanished pod is still there	readiness 4, 5
T3	(test fake) readiness's first pod read answers 404 instead of hanging	readiness 4. Cases 2 and 3 still pass: readiness then reaches the hung guest probe, and the outer budget fires there instead

Findings (reported, not changed)

1. The create-cleanup delete is not UID-fenced (defense-in-depth inconsistency). When readiness fails and the pod is not preserved, create_execution deletes it with delete_sandbox_pod (DeleteParams::default()). CancelGuard's cleanup deletes the same pod with delete_sandbox_pod_if_uid. No production path to deleting the wrong pod was found: pod names are per generation, and the create holds the execution claim throughout. The tests do not pin the missing precondition.
2. A preserved slow create keeps its activation attempt until its lease expires.
   • In the preserve arm, release_activation_attempt_best_effort → complete_activation_attempt matches zero rows, because it requires the sandbox lifecycle to equal the attempt's target (started) and the sandbox is still creating.
   • The outcome also carries lease_completion: None, so the caller does not complete the attempt either.
   • A one-off probe in the preserved case (not committed) read activation_attempt_id still set after create_execution returned.
   • The attempt therefore lives for its 300 s lease (ACTIVATION_ATTEMPT_LEASE_TTL_SECS). Meanwhile, activations for other targets get ActivationInFlight.
3. Some apiserver reads after the create budget, or outside it, have no bound.
   • After the budget expires, the defer decision's read_sandbox_pod falls back to fetch_sandbox_pod_uncached, an apiserver GET with no timeout, so a hung apiserver extends a failed create past its budget.
   • The success-path placement GET after readiness and the joined path's pod GET after readiness are likewise unbounded.
   • The joined path has no outer timeout: it keeps only the phase deadline from ready_wait_bounds. Its readiness wait starts with a resolve_pod_ip read in wait_for_pod_ip_assignment (wait_ready.rs), which runs before that phase deadline's timeout_at. That read can fall through to an uncached apiserver GET, so a hung apiserver there hangs the joined create. The create_ready doc says only that its phase deadline bounds the joined create's readiness phases.
   • The tests keep the apiserver answering there.
4. On create's path, the is_nodemgr_unreachable(&err) half of err_defers_cleanup has no genuine input; its only match is incidental.
   • Every readiness error that half matches is a Timeout, which the other half already defers.
   • The one exception reaches it by accident: resolve_pod_ip returns Unavailable("sandbox pod is unschedulable: {scheduler message}"), and the heuristic matches when the scheduler's message happens to contain "unavailable", for example a node.kubernetes.io/network-unavailable taint.
   • Such a match would defer cleanup of a never-scheduled pod to the reconciler, without a delete, and count a cancel under NodeMgrUnreachable.
   • No test pins this.

Observations, not changed here:

• Warm-pool adoption's network-policy annotation PATCH runs on a detached tokio::spawn, untracked and unbounded, since kube-client has no read timeout. Routing it through the substrate's TaskTracker would let graceful shutdown wait on a hung PATCH for the whole drain grace, which is a production change and outside this seam.
• CancelGuard's background pod DELETE is on the tracker and likewise unbounded.
• Most readiness-failure arms were already reachable without this seam: they end on a phase timeout, which the KubernetesState per-phase timeouts bound. The budgets are what make the outer-budget timeout, the preserve-decision budget, the joined-create timeout and the suspend timeouts testable in about a second.

Host safety

• Listeners and sockets.
  • Every listener is a test-owned scripted worker on 127.0.0.1:0, closed on drop.
  • The apiserver is an in-process tower service.
  • No real Kubernetes, fixed path, socket, firewall rule or network namespace is touched.
• Which addresses get dialed.
  • Created pods carry 127.0.0.1, with the substrate's proxy port set to the test's own worker port.
  • Execution rows no code path dials carry TEST-NET-1 192.0.2.1, so a regression that does dial one fails to connect rather than reaching a host listener. The unreachable-worker teardown case dials it on purpose, as api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273's did: the connect fails or gives up within 300 ms.
  • The suspend tests reach a node only through the advertised node-api port, with the fallback port set to the closed port 1. The no-node-api case has no node-api pod to resolve.
• Credentials. Worker tests enter their child process with owned token files.
• Databases. The split-receipt test keeps api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273's substrate_with_unreachable_store (127.0.0.1:1, fast failure). The contention cases build a second pool to their own test database, with lock_timeout set at connect, and hold and release their own table lock.
• Background tasks. No test leaves a detached task: the create and joined-create tests assert an empty task tracker.
• Bounds on hangs. The create, joined-create and suspend tests bound their entry call at 10 s, so a regression that hangs fails instead of stalling. The converted split-receipt and teardown tests have no test-owned bound; under a regression they are bounded by the production budgets they exercise (5 s and 3 s).
• Privileges. Every pplx-space-api test runs in CI's unprivileged half; none is in the privileged nextest group.

Overlaps with open PRs

The named PRs #269, #271, #273, #275, #277 and #278 are merged. #270 (open) touches only sbx-mgr files, so there is no overlap.

Open PRs that touch this directory predate the split of kubernetes/adapter.rs into adapter/:

• [api] stop retrying restore pods whose snapshot the host CPU cannot load #221 (reported CONFLICTING). Its adapter.rs hunks add abandon_when_retired to the two PodProxyWaitTarget literals in create_execution. Those now live in adapter/substrate_adapter.rs, about 8 lines below this PR's two ready_wait_bounds edits. It also edits execution.rs and wait_ready.rs, which this PR does not touch, and appends tests at the end of tests.rs; this PR adds one budgets: line to each of the three substrate literals there.
• api: decide admission pool claims for the create's owner; collect refused claims promptly #168. Its adapter.rs hunk now corresponds to code about 40 lines from this PR's split-receipt edit in adapter/substrate_adapter.rs. This PR does not touch pool.rs, which api: decide admission pool claims for the create's owner; collect refused claims promptly #168 also edits.
• [api] restore full snapshots only onto CPU-compatible hosts, behind a default-off gate #176. It edits mod.rs, where this PR adds one mod, one use and one field, and tests.rs near the restore-shortlist tests.

Neither #221 nor #168 touches the TEARDOWN_SUSPEND_DURABILITY_BUDGET line in adapter.rs. The new tests are in new or #273-owned test files that no open PR edits.

Validation

At head 1a95969f39987efd4c6d381359bff80a89471113, one commit on main at 9d89c26. main has moved to c52a6c3; GitHub reports the PR mergeable, so it was not rebased.

This revision fixes three review findings: the create_ready doc and Finding 3 now describe the joined create's bounds accurately, retire_worker_unreachable asserts its connect bound again (S6), and the two budget-derived timeout messages print the Duration with {:?}.

Check	Result
bash ci/scripts/run-rust-checks.sh fmt (env-contract check, Rust error policy, cargo fmt --all --check)	pass
bash ci/scripts/run-rust-checks.sh clippy (workspace --all-targets --all-features -D warnings, plus the observability, common-svc and feature-flags isolation lints)	pass
pre-commit (gitleaks) on the staged change	pass
cargo nextest run --locked --all-features -p pplx-space-api --retries 0 ×3 in a row, Postgres and Redis through the nextest setup scripts, SPACE_CI_REQUIRE_TEST_DEPS=1, SPACE_TEST_DEP_SKIP_LOG set	3/3: 1731 passed, 1 skipped (the ignored benchmark) each; no dependency-skip log written, no SPACE-TEST-DEP-SKIP marker
S6 mutation	killed (see the mutation table); adapter/teardown.rs restored and touched
GitHub Pull CI on 1a95969 (run 38345438698) (Rust tests, clippy, format, dependency checks, guide, docs, migration guards, shell, Python, Go shim, gateway traffic, space-pull-ci)	all green

Carried from 4fb236f, not rerun because this revision changes production code only in two messages' formatting:

Check	Result
scripts/source_guards/{env_test_hook_boundary,node_selector_extras_smoke,observability_vendor_boundary}.py	pass
scripts/check_no_inline_rust_test_modules.py, scripts/source_guards/thin_service_mains.py	fail only on untouched files (crates/libs/observability/src/{otlp_endpoint,selection}.rs, crates/apps/vol/mgr/svc/src/main.rs); pre-existing, unrelated
the ignored api test (--run-ignored only)	1/1 passed
create_readiness:: module ×50 under load (48 busy-loop processes on the 48-core host, load average ~25)	50/50 runs passed, 10/10 tests each, on 1c25d9d and again on 6c18406
Mutation proofs S1–S5, B1–B9, D1, J1, J2, P1–P3, T1–T3	23/23 killed (see "Where each one ran")
Kind (fresh space-local cluster; deployed space-api-svc sha256 equal to the build's artifact)	run on an earlier revision whose production diff was this seam plus a since-reverted TaskTracker change: smoke-kind-e2e.sh passed twice; smoke-kind-typed.sh all passed 23/23 native and 76/77 ECW, the 77th being local.ecw_template_build_from_oci_boots, which is ENVIRONMENTAL (builder storage admission needs 55.8 GB free and found 26.7 GB on the host's shared disk; failed again on rerun). Not rerun: production code has since changed only by the revert to main's adapter/pooled.rs, a debug_assert!, and the timeout messages' formatting, which is byte-identical at the production defaults.

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 3 findings

Medium

1. crates/libs/api/src/controllers/substrate/kubernetes/budgets.rs:20 — the create_ready doc says it also bounds a create that joins an in-flight activation, but the joined arm has no outer bound.
   At adapter/substrate_adapter.rs:138 the joined arm keeps only phase_deadline from ready_wait_bounds and drops the backstop, and nothing wraps it in a tokio::time::timeout. The readiness wait starts with resolve_pod_ip (wait_ready.rs:173), whose pod read falls back to an uncached apiserver GET with no phase bound. That's the read your outer-budget cases rely on staying unbounded. I ran a probe on this head: a joined create whose owner's pod has a recorded IP, with every apiserver request hanging, create_ready at 1 s and a 60 s data-plane timeout. It was still pending when an 8 s test timeout fired, while the non-joined arm ends at 1 s under the same hang. This is pre-existing and the commit message words it correctly ("the phase deadline of a create that joins"), but the new doc says the opposite. I'd narrow the doc to "its phase deadline also bounds the readiness phases of a joined create", and add this pre-readiness read to the body's Finding 3, which currently names only the joined path's post-readiness GET.

Low

2. crates/libs/api/src/controllers/substrate/kubernetes/adapter_characterization_tests/create_and_teardown.rs:247 — the teardown connect budget can be pinned without a connect that portably stays pending.
   assert!(elapsed < POD_WORKER_CONNECT_TIMEOUT) around retire_worker_unreachable's retire can't false-fail anywhere: a host that answers ENETUNREACH fails the dial at once under either budget. Where the SYN is dropped, it catches the revert. I tried it on this head: it passed at 0.49 s. With adapter/teardown.rs:361 reverted to POD_WORKER_CONNECT_TIMEOUT, it failed at 3.00 s. It doesn't discriminate on ENETUNREACH hosts, but it costs nothing there and turns the "routed but not proven" note into a proof wherever the host drops the SYN.
3. crates/libs/api/src/controllers/substrate/kubernetes/adapter/suspend.rs:114 (and adapter/substrate_adapter.rs:506) — the timeout messages print as_secs(), so a sub-second budget reads as "0s".
   Production is unaffected because the defaults are whole seconds. The seam exists for sub-second budgets, though, and with the suspend tests' 600 ms budget the error says "did not become durable within 0s". {:?} on the Duration would keep production's text close and stay accurate for short budgets.

Verified against 4fb236fcd24f1e004fbb3717bf71c354d2c6ac64. Reproduced locally: the joined-create hung-read probe, plus the teardown elapsed assertion on this head and with the teardown connect reverted to the constant (cargo nextest -p pplx-space-api, two targeted tests per run). I also checked by reading that every routed site reads its budget field, that try_from_env is the only production constructor, and the body's activation-lease and is_nodemgr_unreachable observations against source. I didn't re-run the full suite or the body's 23 mutations.

[the-gigi-pplx]

Approved. 3 non-blocking findings in the review comment above.

[ac-pplx]

Thanks. All three are addressed in 1a95969f:

1. create_ready doc: narrowed. Its phase deadline bounds the readiness phases of a joined create, and the joined wait has no outer bound. The body's Finding 3 now also names the joined path's pre-readiness resolve_pod_ip read. Behavior is unchanged.
2. Teardown connect budget: retire_worker_unreachable asserts elapsed < POD_WORKER_CONNECT_TIMEOUT again, around the TEST-NET retire. S6 (teardown connect reverted to the constant) is killed again, failing at about 3 s, and is back in the mutation table.
3. Timeout messages: the create and suspend messages format the budget with {:?}. New unit tests check that the production defaults render byte-identical to the old {}s text (135s, 300s), and that sub-second budgets read 500ms and 600ms.