[ac-pplx]

Pins the two error arms of KubernetesSubstrate::apply_adopted_per_sandbox_setup (kubernetes/adapter/pooled.rs) that #273's characterization called effectively unreachable through the retrying node-api client. Both arms are reachable. Their cases now script the statuses production delivers. Tests only: no production code changes, and every status mapping and the reclaim classification stay as they are.

The two arms and how they are reached

Network-policy step, Unavailable → 503.

• GrpcClient::call replays a rate-limited ResourceExhausted that carries a server retry hint after exactly that hint, with no lower bound (common/grpc/src/retry.rs, retry_delay_for_status). That replay spends the same retry count as Unavailable.
• sbx-mgr's main Unix-socket listener runs conservative_uds_grpc_rate_limiter_layer, a per-peer GCRA limit of 20,000 requests per second. Its request-rate rejection is rate_limited_status(…, Some(retry_after)) with retry_after at most the 50 µs interval, so the hint reads 0 ms. The rejection is not marked refused-before-start.
• Node-api forwards SetSandboxNetworkPolicy with RpcReplaySafety::RequiresProof and relays statuses unchanged.
• After DEFAULT_TCP_GRPC_CONNECT_RETRY_POLICY.max_retries (12) such rejections, sbx-mgr's fresh Unavailable ("synchronous network policy update requires a live net-gateway") meets backoff_delay(12) == None. It comes back raw, well inside the 5 s WARM_POOL_SETUP_RPC_TIMEOUT, and the arm maps it to 503.

Metadata update, Cancelled → 408.

• Node-api forwards UpdateSandbox over its Unix-socket tonic channel to sandbox-manager. When that connection closes while the forward is still queued in hyper's HTTP/2 dispatch channel, hyper fails it as canceled ("connection closed"). Tonic reports Cancelled("operation was canceled"), with a local transport error source.
• RequiresProof returns that status unchanged.
• The API decodes it from trailers without the local source. The temporary-TCP classifier (is_temporary_tcp_failure, shared by GrpcClient::call and retry_grpc_status_with_context) matches none of its fragments, so the client returns it without a replay, and the arm maps it to 408.

Test changes

• pooled_adoption_stops_at_a_failed_setup_step::update_cancelled now scripts Cancelled("operation was canceled"), the status node-api relays. It previously scripted Cancelled("Timeout expired"), which tonic produces only from its 150 s channel timer and 120 s server timer, both preempted by the 5 s budget. The expectation is unchanged: 408, not reclaimable.
• New case policy_unavailable_after_rate_limits. The scripted node-api answers the network-policy update with 12 rate-limit rejections carrying sbx-mgr's message and a 0 ms hint, then sbx-mgr's live-gateway Unavailable. The case expects 503, reclaimable by pooled_prepare_error_is_reclaimable, and exactly 13 calls, so every rejection was replayed and the final status was not. The count comes from DEFAULT_TCP_GRPC_CONNECT_RETRY_POLICY.max_retries, the policy NodeApiClient uses.
• The scripted node-api (scripted_test_doubles.rs) gains ScriptedReply::Sequence, which answers the n-th call to a method with the n-th reply and later calls with the last one. Existing replies behave as before.

Mutation proofs

Each edit was temporary and was reverted afterwards.

Edit	Result
Delete the network-policy Unavailable arm	policy_unavailable_after_rate_limits fails: left: 500, right: 503
Delete the metadata-update Cancelled arm	update_cancelled fails: left: 500, right: 408
Script one rate limit too few (11)	the case fails on the call count (left: 13, right: 12): one real backoff replays the Unavailable once more
Script no rate limits	the case fails after ~4.6 s with left: 500, right: 503: the client replays the Unavailable until the budget ends and reports DeadlineExceeded

Observation, not fixed here

A 0 ms rate-limit hint uses up the node-api client's 12-retry count with no backoff. Under node-api → sbx-mgr overload, a setup call therefore ends after 13 back-to-back attempts within milliseconds, instead of spreading its replays over the 5 s budget.

Host safety

The tests run in their own re-executed process against a scripted node-api on an ephemeral 127.0.0.1 port. They reach no fixed host path, socket, firewall or netns.

Overlap with open PRs

No open PR touches adapter_characterization_tests/pooled_adoption.rs or scripted_test_doubles.rs.

Validation

Check	Result
cargo fmt --all -- --check	pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings	pass
pre-commit (gitleaks)	pass
cargo nextest run --locked -p pplx-space-api --all-features --retries 0, three consecutive runs with SPACE_CI_REQUIRE_TEST_DEPS=1 (Postgres and Redis via the nextest setup scripts)	3/3 green: 1717 passed, 1 skipped each (the #[ignore] 400k-sandbox occupancy benchmark), 0 dependency skips
Kind	not run: tests-only change

A Kind run of an earlier revision of this branch saw local.deferred_btrfs_send_capture_restores_cross_node fail once and pass on rerun. It is unrelated to this diff and was not reproduced on the base commit. The failure was node-manager's runtime-artifact cleanup (node/mgr/server/src/service/cleanup_helpers.rs, inspect …/instance: Invalid argument (os error 22)), reached from the suspend cleanup. It left the prior-generation pod Terminating, so the restore deferred with 503.

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 2 findings

Medium

1. crates/libs/api/src/controllers/substrate/kubernetes/adapter_characterization_tests/pooled_adoption.rs:311: PR api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273's Finding 1 failure is still there, and no case pins it or issue tracks it.
   That finding was about a persistently unavailable gateway during the pooled network-policy sync. The client retries Unavailable until the 5 s setup budget runs out, and the resulting DeadlineExceeded lands in the internal arm (adapter/pooled.rs:323-326). The create path then can't retry in place or re-claim. This PR shows that the Unavailable arm can be reached through one narrow sequence (12 zero-hint rate limits first). The ordinary case is unchanged: I set RATE_LIMITS to 0 at this head and the case returned 500 after 4.7 s. The title and the "Both arms are reachable" framing read as if Finding 1 is settled, but the test matrix has no case for the common path, and I couldn't find an issue for it.
   Add a policy_unavailable_persistent case that pins today's 500, false, with a comment marking it as the known gap, so a fix has a test to flip. Also open an issue for the mapping. The case costs about 4.7 s of wall clock until the budget becomes injectable. If that's too slow, an issue alone keeps the gap tracked.

Low

2. PR description, "Observation, not fixed here": the zero-hint retry burn is recorded only in the PR body.
   retry_delay_for_status (common/grpc/src/retry.rs:225-231) replays a ResourceExhausted after exactly its hint and spends a retry index each time. sbx-mgr's per-peer request-rate rejection carries retry_after.as_millis() (common/svc/src/rate_limiter.rs:873), which reads 0 for sub-millisecond intervals. So under node-api → sbx-mgr overload a setup call uses up all 12 retries within milliseconds instead of spreading them over its budget. Once this merges, nothing points back to that observation. Open an issue for it.

Verified

• Both new or rewritten cases reach the arms they name. With the network-policy Unavailable arm disabled, policy_unavailable_after_rate_limits fails (500 vs 503). With the metadata Cancelled arm disabled, update_cancelled fails (500 vs 408).
• The update_cancelled comment's mechanism holds. Adding "canceled" to TEMPORARY_TCP_FAILURE_FRAGMENTS makes the client replay the status until the budget runs out, and the case fails with 503 vs 408. So the classifier not matching the relayed message is what makes the arm reachable.
• The message is production-shaped. Tonic maps a canceled hyper::Error to Status::cancelled(err.to_string()), and hyper's Display prints only "operation was canceled" (the "connection closed" cause is in source(), which doesn't cross the wire). Node-api forwards UpdateSandbox and SetSandboxNetworkPolicy through fwd_sbx! with RpcReplaySafety::RequiresProof, which returns any non-proof failure unchanged (node/api/server/src/restart_handoff.rs:91-92, :332).
• The scripted rate-limit message matches the real rejection text (rate_limiter.rs:873), and the trailing Unavailable matches sandbox_network_policy.rs:165.
• ScriptedReply::Sequence: the per-method attempt index is taken under the call-log lock, and an empty sequence falls through to the unscripted path, as its doc says.
• pooled_adoption_stops_at_a_failed_setup_step: all 9 cases pass.
• Not checked: the full pplx-space-api suite and a Kind run. CI is green.

Verified against b4b5f401c4036698be5673f0ade3feb7fe8b2e6d. Reproduced locally: the case matrix, three mutations (the two arms and the classifier fragment, all reverted), and the persistent-Unavailable variant.

[the-gigi-pplx]

Approved. 2 non-blocking findings in the review comment above.