[ac-pplx]

Stack: Part of the Kind fidelity stack, bottom to top: #485 → #480 → #472 → #484 → #507. Based on #484 (ac/kind-fidelity-internal-auth).
This PR is one commit, and that commit is the only thing to review.

What Kind now does like deployments

Kind's control plane used to stop abruptly and left its backstop reconcilers off, so the code that drains work on shutdown, survives voluntary disruption and repairs stuck rows never ran in Kind. This PR changes only Kind (charts, both bring-up paths, README, and one comment in the smoke harness). No production Rust changes.

Graceful shutdown and rollouts

• space-api and space-admin-api (same binary) now get --shutdown-grace-secs, rendered explicitly at the binary's default DEFAULT_SHUTDOWN_GRACE_SECS = 150 (crates/libs/api/src/run.rs). They also get a 5 s preStop sleep and a 180 s termination grace; the template refuses a grace that doesn't exceed preStop + shutdown grace. On SIGTERM the process reports draining on /readyz and stops accepting in the same instant, so the preStop delay is what takes the pod out of the entry Services first. The drain then waits for HTTP requests, SSE streams, the ingress-activation listener and the substrate work its TaskTracker holds (restore leaders, suspend continuations).
• API startupProbe on /readyz. This is the boot budget the Pod reflector leaves to Kubernetes (pod_reflector.rs: a replica whose initial LIST never completes stays NotReady until restarted). It is 18 × 5 s, covering the 30 s STORE_READY_TIMEOUT on the node caches plus the LIST. minReadySeconds: 10 is applied to every Deployment in this list.
• space-data-plane has no drain budget of its own. On SIGTERM it stops accepting, keeps answering /readyz ok, and axum's graceful shutdown waits with no deadline only for plain HTTP streaming responses: native session SSE and ECW process and file streams (run, crates/libs/data/plane/server/src/lib.rs). Upgraded connections do not drain. Service-port proxies and ingress, upgraded HTTP proxies and raw service sessions watch child tokens of the shutdown token and close at SIGTERM (proxy_upgraded_connection, bridge_raw_service_session). The other WebSocket session attaches run outside the server's connection tracking and end with the process. The data plane now gets the same 5 s preStop and a 180 s grace as the bound on the HTTP streams. Its startupProbe covers its wait for ClusterApi before it binds.
• space-internal-api goes from one replica with Recreate to two replicas with RollingUpdate, a 5 s preStop and a 50 s grace. Raising the count is safe in code: all state is in Postgres under row locks and claim tokens (FOR UPDATE / SKIP LOCKED), every RPC is unary, nothing is a singleton or a background job, and its per-replica caches (template visibility, TokenReview) are bounded and documented as per replica. The grace is sized from the code:
  • tonic's graceful shutdown has no deadline of its own, and the server bounds no call (PublishNodeState runs several database operations with no overall deadline). A call ends when its client times out: 30 s for every ClusterApi client (CLUSTER_API_RPC_TIMEOUT in the shared client, CLUSTER_RPC_TIMEOUT in node-ctl's).
  • Telemetry shutdown then flushes metrics and traces in turn, each capped at 5 s.
  • 5 s preStop + 30 s + 10 s, plus 5 s to exit, is 50 s. A call admitted at the end of the preStop runs to its client's own timeout before SIGKILL.
• cluster-controller is unchanged: it already bounds its reconciler drain at 10 s inside a 30 s grace.

Disruption and priority

• PodDisruptionBudgets with maxUnavailable: 1 for space-api, space-admin-api, space-data-plane, space-cluster-controller, space-internal-api and space-credmgr-authority. All of them run on the infra node; the node-drain scenarios drain sandbox workers, which carry none of these pods.
• manifests/priorityclasses.yaml is applied by bootstrap.sh and by Tilt (in label-kind-infra, which every workload naming a SPACE class waits on) before any workload, because admission rejects a pod whose class doesn't exist yet. It defines two classes:
  • space-control-plane: the control-plane Deployments plus the in-cluster Postgres, object store, Redis and key broker.
  • space-replaceable: the UI and the one-shot Jobs. It has preemptionPolicy: Never.
• system-node-critical on every node-service DaemonSet, through a new space-node-service.priorityClass helper.
• UI is pinned to the infra node, which the README already claimed.

Backstop reconcilers (space-cluster-controller-local/values.yaml; each setting cites its code). These now run as deployments run them:

• Transient-lifecycle sweeper: on. restoring rows are failed after 600 s. The other transient lifecycles are effectively off: suspending and checkpointing are set to 0, and creating, starting, pausing, resuming and stopping, which have no per-lifecycle knob, inherit a one-year default age. Failed-sandbox cleanup stays on, and the unrestorable-restoring phase runs under the same switch.
• Stuck-snapshot reaper: on, with its fixed 24 h age and 5 min tick.
• Started-sandbox repair and failed-restorable repair: on and live (neither has a dry-run mode).
• Reconcile-queue GC (SPACE_RECONCILE_QUEUE_GC_*, the EC2 instance): stays off, rendered explicitly as false. A Kind bench must not call AWS (see the differences below).
• Gone-node queue GC (SPACE_GONE_NODE_QUEUE_GC_*): on, dry-run. These are the names the code uses, confirmed from env parsing and log lines.
• Orphaned-restore-pod GC: on, dry-run, at the binary's 60 s interval, 45 s budget and 300 s threshold. The old Kind 5 s / 4 s overrides are dropped; the GC scenario scopes its own live config.

Postgres. On top of #485, this PR keeps #485's 8 GiB memory limit and its reasoning, and adds only the control-plane priority class to that chart. The second internal-api replica and its rollout surge are counted in #485's max_connections budget (1328 of 1400). The standalone round below ran on main with a 1Gi limit: there, the concurrent-restore scenario opened 79 client backends and brought Postgres within 4% of the old 512Mi, and one run had a backend OOM-killed.

What stays different (README "Differences from the EKS deployment modules")

• preStop delays are 5 s. kube-proxy removes a terminating pod from the NodePort entry and ClusterIP Services in about a second; a delay sized for a cloud load balancer would only slow every rollout.
• No anti-affinity or topology spread. Every control-plane pod runs on the one infra node, where a spread constraint has nothing to spread over and a required anti-affinity would leave second replicas Pending.
• space-admin-api is one replica under maxUnavailable: 1. It never holds a drain, and it rolls by Recreate, so a rollout leaves the admin surface down while the old pod drains.
• Reconcile-queue GC stays off. Deployments run it, but for a non-ready node with live refs older than 24 h its EC2 detector (Ec2Detector) signs DescribeInstances with the controller's AWS_REGION and the object store's static keys to the public EC2 API.
  • A reply with no live instance reads as Gone (Ec2InstanceLookup::lookup_by_private_dns). Kind's refs would be kept only because that call fails (Unknown), not because there is no instance.
  • No switch disables the EC2 detector alone.
  • The controller's EC2 client (pplx_space_aws_client::ec2) takes only the global AWS_ENDPOINT_URL, which would also redirect the controller's S3 clients. Pointing it at the object store, or at a dead address, would be a fake EC2 endpoint, not an honest configuration.
  • Its Postgres-only dead-letter pass runs only in that same instance (run_dead_prune), so it stays off too.
  • The gone-node queue GC (Kubernetes presence, dry-run) still runs.

Validation (one Devbox, CI's settings: SPACE_KIND_TELEMETRY_MODE=otel, DATA_PLANE_URL=http://127.0.0.1:8090, admin token file and cold-receipts dir as in kind-ci.yml)

This round validated ac138f6 (base 0286c67, then the latest main) on a fresh cluster. The pushed head 6e022d7 is the same patch rebased onto three more main commits (#489, #502, #500). They touch node-manager's btrfs send path and the docs site, none of the files this PR changes, and git range-diff shows the patch unchanged.

Every suite ran after the health gate (Tilt quiet, no ReplicaSet younger than 180 s, every node Ready and published, 120 s settle). Provenance on the live pods:

• Binaries: every service's logs carry version = ac138f627c44.
• Image IDs: the same set before and after the run (57 containers).
• Pod specs: internal-api has terminationGracePeriodSeconds: 50 on the Deployment and on both pods, with a 5 s preStop. The controller has SPACE_RECONCILE_QUEUE_GC_ENABLED=false.

Run	Result
Fresh bring-up + smoke-kind-e2e.sh	PASS, 553 s (previous round: 569 s)
native_recovery.* (native surface)	3/3 PASS, 55 s
local.concurrent_restore_converges_to_one_execution	PASS, 91 s; Postgres 0 restarts at the 1Gi limit
local.orphaned_restore_pod_gc_reaps_only_superseded	PASS, 144 s
local.forced_drain_evacuates_sandbox	PASS, 24 s
ecw.command_round_trip (ECW)	PASS
ecw.pause_connect (ECW and native)	PASS on both
ecw.filesystem_round_trip (ECW)	PASS
native.resident_pause	PASS
native_concurrency.lifecycle_contention	PASS
smoke-kind-leader-election-brownout.sh (SPACE_SMOKE_LEADER_ELECTION=1)	PASS, 2270 s (previous round: 2253 s)

Every scenario passed on its first attempt, so no reruns were needed. The full typed suite was not rerun this round. The previous round's full-suite results (below) stand for the parts this round did not change: everything except internal-api's grace, the reconcile-queue GC switch, and comments.

internal-api rolled while node-ctl publishes. kubectl rollout restart deploy/space-internal-api ran three times. node-ctl publishes on a 60 s tick, so two of the rolls were timed so that an old pod's terminating window covered a tick.

• Every old pod got SIGTERM 5 s after its deletion (the preStop) and exited with code 0 (Completed) in under a second.
• In the second roll, the pod already being deleted served PublishNodeState for two nodes at 01:35:14.1, during its preStop, then took SIGTERM at 01:35:15 and exited 0.
• Across the three rolls (01:30:30–01:39:30), Tempo holds 216 node-ctl client spans for ClusterApi calls and 436 server spans, none with error status. The same query does find error spans elsewhere in the run.
• The node-ctl, data-plane, template-manager, build-manager, sandbox-manager and node-api pods logged no WARN or ERROR in any roll window, apart from node-ctl's standing read-only orphan-inventory notice on builder nodes, which is not a ClusterApi call.

No call to public AWS (reconcile-queue GC off). Every leader start in the pods' logs says reconcile-queue GC (ec2 detector) is disabled; skipping its AWS SDK init and starts only the kube_presence detector. A monitor sampled conntrack on every Kind node every 10 s from the first suite to the end of the brownout (01:20–02:20). It recorded each destination contacted from a cluster-controller pod IP: 26 distinct destinations, all in-cluster (Postgres, the kube-apiserver, CoreDNS, the OTLP collectors, node and pod gRPC peers on 9800), and none outside the pod, Service and Kind node networks. The logs contain no EC2 lookup failed line and no amazonaws host.

Observed, not change-induced: the four credential-manager DaemonSet pods restarted twice each during bring-up (01:14, before the first gate), on a DNS lookup for the authority before it resolved. Node services are documented to restart per pod until their peers are ready, and nothing restarted after the gate.

Previous round (full suite; stands for the unchanged parts)

Heads validated then: 1bd4173 (baseline), ac39ba7, 31c512f and 809901e. On 809901e the results were: bring-up + e2e 569 s PASS, typed all 78/78 ECW + 24/24 native, brownout 2253 s PASS. That round also showed:

• Graceful api shutdown. Three ECW restores, a native :suspend and its operation SSE finished inside the drain, and the container exited 0 sixteen seconds after SIGTERM.
• Disruption budgets. Six PDBs; a second eviction was refused with disruptionsAllowed: 0.
• Newly enabled reconcilers. None acted on a scenario's row. That round still ran the reconcile-queue GC's EC2 instance (6–7 ticks, candidates_scanned: 0); this round turns it off.

The one change-induced failure then was the Kind Postgres sizing, fixed in that round by a 1Gi limit and now covered by #485's 8 GiB. No product failure was exposed in either round.

Review follow-up

Restack onto #484 (content changed):

• Postgres: kind: run Postgres like the deployed database #485's 8 GiB limit and its reasoning are kept. This PR adds only priorityClassName to that chart, and its own 1Gi comment is gone. kind: run Postgres like the deployed database #485's max_connections budget comment now counts internal-api's second replica and its rollout surge (33 × 3 = 99), 1328 of 1400, so the setting is unchanged. The new reconcilers share the cluster-controller's two pools.
• Conflicts with kind: run node and runtime configuration like a deployed node #472 in the cluster-controller values: kind: run node and runtime configuration like a deployed node #472's removal of the dispatcher's claim-staleness hook and its comment on the retention sweeper's minimum age are kept, beside this PR's reconciler settings.
• The space-cluster installer from Add cluster deployment overlays for existing SPACE charts #391 now checks the space-control-plane and space-replaceable PriorityClasses before installing (read-only, as it checks RuntimeClass space), and its README asks for manifests/priorityclasses.yaml once per cluster. The control plane and the migrate Job name those classes, and a pod naming a missing class is refused.

Earlier rounds:

• internal-api grace (fixed): sized from the code at 50 s = 5 s preStop + the 30 s ClusterApi client timeout + two 5 s telemetry flushes + 5 s to exit. The comment, README and this description no longer cite ingress preparation's 15 s cap, which covers only that call.
• Data-plane drain wording (fixed): the chart, README and this description now say that only plain HTTP streaming responses drain. Upgraded connections close at SIGTERM or end with the process. No production change.
• Reconcile-queue GC (fixed): turned off in Kind, because its EC2 detector would sign calls to the public EC2 API. No switch disables that detector alone, and the only endpoint override would also redirect the controller's S3 clients. Both texts now describe the real mechanism (no live instance reads as Gone; only a failed call keeps refs), and the differences table records the reason.
• Comments on the wrong key (fixed): the priorityClassName blocks in the object-store and Redis values now sit above the infra-node comments, which again describe nodeSelector.
• native_recovery rollout wait (comment added): ROLLOUT_TIMEOUT now explains why 90 s is enough beside the 180 s grace: each restart holds no open stream.
• Postgres limit and kind: run Postgres like the deployed database #485: resolved in the restack above; kind: run Postgres like the deployed database #485's limit and reasoning win.

Rollback

Reverting the chart settings needs no schema rollback. Repairs the newly enabled reconcilers have already applied to sandbox state stay applied. admin-api still runs one replica with Recreate, so its surface is briefly down during a rollout, as the README says.

This change was written and validated with AI assistance (Claude Code); a person should review it before merge.

🤖 Generated with Claude Code

[nickdujay-pplx]

Reviewed 02ebd18 against #484. No additional blocker in this layer.

Checked shutdown budgets against the server/client code, startup/readiness probes, internal-api's concurrent-replica behavior, PDB selectors, PriorityClass creation before workloads, and the umbrella installer's prerequisite check. Reviewed each newly enabled repair/GC setting: the EC2-backed queue GC stays disabled, while the Kubernetes-presence and orphaned-pod GCs remain dry-run. No schema, RPC or production Rust change is introduced here.

Exact-head chart renders passed; Kubernetes schema validation accepted all 119 rendered resources. The existing harness/chart run passed 648 tests; its one macOS wc-formatting failure also passed when rerun with GNU tools. All four Kind CI shards passed. I did not independently rerun the Linux/KVM shutdown/leader-election drills.

Rollback can revert these chart settings without a schema rollback, but cannot undo repairs already applied to sandbox state. The single admin replica still has rollout downtime, as documented. Approval covers this delta; #480's separate installer finding remains outstanding.