[ac-pplx]

Stack: Part of the Kind fidelity stack, bottom to top: #485 → #480 → #472 → #484 → #507. Based on #480 (ac/kind-fidelity-redis).
This PR is one commit, and that commit is the only thing to review.

Kind already runs the production node layout. This change makes its node and runtime configuration match what deployed nodes run, so a green Kind run covers the code paths those settings select.

What Kind now does like deployments

Change	Code path it now exercises
The cluster-controller chart no longer sets the public-operation dispatcher's tick, claim limit or claim staleness, or the snapshot-retention sweeper's minimum age. These are test hooks no scenario needs; the retention scenarios still shorten the minimum age for their own run.	The dispatcher claims queued operations on its release 25 ms tick instead of every second (public_operation_dispatcher.rs). Create pickup in the suite fell from a 566 ms median to 14 ms (below).
node-manager runs with SPACE_NODEMGR_NETWORK_POOL_TARGET, _RETAIN_READY and _MIN_READY at 0, as deployed nodes do.	Each slot is its own network namespace, bridged either to a sandbox pod's namespace or to the node's. claim_allocation serves only node-bridged allocations from the pool, so sandbox slots were already set up on allocation and torn down on release. But the binary defaults filled the pool to max(target, retained), which is every slot the node has (300), all bridged to the node (the fill loop in slot_pool/worker.rs). So each sandbox claim took one of those slots and rebuilt it for the pod (handoff_network_namespace), and the pool rebuilt it after release. Builders, whose build VMs allocate without a pod namespace (build-manager's netns_path: None), leased pooled slots. Now builders set up slots on demand too, nothing is pre-built or rebuilt, and the pool reports healthy while empty (pool_status). Measured: 300 pool namespaces per worker before; after a full suite, none on any SPACE node.
The static space.pplx.net/network-slots: 100 seeding in bootstrap.sh and the Tiltfile is removed.	On the baseline, node-ctl had already replaced the seed with node-manager's network_pool_max_slots (300), and its server-side-apply field manager owned the field. Node-ctl is now the only source, as on a deployed node, including after the drain scenarios delete and re-register a Node.
The SPACE Kind node image bakes in a kubelet --config-dir drop-in with eventRecordQPS: 0 and failSwapOn: false.	Kubelet on sandbox workers and builders has no client-side event rate limit (verified through configz). The image tag changes, so check-kind-node-image.sh refuses clusters created without it.
The image registers the space runtime with the deployed CniConfDir option through a drop-in that the image's containerd config imports. The drop-in is written in the version 2 format of kind's own config.toml, so containerd migrates both into io.containerd.cri.v1.runtime, where a deployed node has the handler. install-shim.sh no longer edits containerd's config or restarts it; it installs the binary and checks its SHA-256 on the node.	containerd 2.1 loads the deployed runtime shape on first start and keeps kind's runc settings, so kubelet takes the systemd cgroup driver from containerd and the shim's joinPodCgroup joins the pod's systemd slice, as on a deployed node. The image build checks the loaded config (containerd config dump) and fails unless it has the space runtime, its annotations and CniConfDir, runc's SystemdCgroup = true and discard_unpacked_layers = true. containerd is never restarted (NRestarts=0).
space-cache.service reformats the cache image on every boot of the node container, as deployed nodes reformat their instance-store cache. It stays idempotent within a boot (a /run marker, as in 11-nvme-cache.sh) and mounts nosuid,noatime.	After a node restart, template-manager pulls templates again, node-manager's init container stages the guest kernel again (staged=1 unchanged=0), and node-ctl republishes. The opt-in bootstrap-prod-seed.sh stages templates by hand onto that cache, so it now says to re-run it after a node restart. On a re-run it skips a worker only if the build is in that worker's catalog and also on its cache.

The disk-admission comment no longer calls the allocation reservation a production setting; it is a test hook.

What stays different (README rows)

• Shim installation. Deployed nodes install a checksum-verified shim before containerd starts. Kind registers the same runtime in the image but copies the binary onto running workers after the cluster exists, now checksum-checked. Installing it at boot would need the shim built and staged on a host path before kind create in every bring-up path, which other open work is changing.
• Infra-node kubelet. Every deployed node group sets the kubelet event settings. Kind's infra node runs the unmodified base image and keeps kind's event rate limit. No sandbox pod runs there.
• Test-hooks build. Kept, because five scenarios need their overrides. The row now names the hooks the charts set on every run: sandbox-manager's snapshot reaper (target 0, interval 15 s, grace 60 s, idle 120 s, against 0.40, 300 s, 1800 s and 86400 s), template-manager's cache GC (interval 15 s, grace 60 s, against 300 s and 1800 s), prefetch latest-N (1, against 5), the disk-admission reservation, and the TokenReview namespace. The charts leave the dispatcher and retention-sweeper hooks at release values.
• Disk-admission reservation. 1 GiB through the test hook, against 8 GiB in release builds, so allocations fit the 40 GiB cache.
• Updated rows and sections: the /space/cache row (it outlives pod restarts but not node restarts), the node-layout section (capacity comes from node-ctl, no warm pool), and the node-layer section (format on every boot, re-running bootstrap-prod-seed.sh, the kubelet drop-in, and the containerd drop-in with its version-2 format, plus docker restart to start a cache clean).

Validation

One fresh cluster from this PR's merge commit (cc6c4ba: head 023ea6d on main at 0f1d3be; same tree as the head), on a Linux/KVM host with CI's settings (SPACE_KIND_TELEMETRY_MODE=otel, DATA_PLANE_URL, admin token file and cold-receipts directory as in kind-ci.yml). It ran smoke-kind-e2e.sh, then smoke-kind-typed.sh all unsharded with SKIP_BOOTSTRAP=1, behind a health gate: Tilt quiet, no ReplicaSet younger than 180 s, every node published, then a 120 s settle. A new baseline run was not needed: the comparison columns are the previous round's baseline on the starting commit.

• Rust binaries. CI's verified Kind Rust artifact for the merge commit (SPACE_KIND_RUST_ARTIFACTS=prebuilt, image sha256:ff4898a8…), as the previous round used.
• Provenance. Pod image IDs were recorded before and after the suite, and they were identical. The SPACE nodes ran node image space-kind-node:abcf737f2e7c8172. The pod-spec environment of cluster-controller and both node-manager DaemonSets showed the chart change.
• containerd and cgroups, live. On every sandbox worker and builder: containerd config dump has the space runtime with CniConfDir, runc's SystemdCgroup = true and base_runtime_spec, and discard_unpacked_layers = true. kubelet logged Using cgroup driver setting received from the CRI runtime cgroupDriver="systemd", crictl runtime-config reports SYSTEMD, and configz shows cgroupDriver: systemd, eventRecordQPS: 0, failSwapOn: false. The shim logged joinPodCgroup: … → /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/…/cri-containerd-<id>.scope/space-runtime, which is the systemd-slice path. containerd NRestarts=0.
• Before and after for that defect. A throwaway kind cluster ran one worker per node image: main's, the previous head's, and this head's. kubelet took cgroupDriver="systemd" on main's and this head's image, and cgroupDriver="cgroupfs" on the previous head's. A pod's OCI cgroupsPath was the slice form (kubelet-kubepods-besteffort-pod<uid>.slice:cri-containerd:<id>) on the first two and absolute (/kubelet/kubepods/besteffort/pod<uid>/<id>) on the previous head's. That absolute form takes joinPodCgroup's absolute-path branch.
• Host disk. After the first fresh bring-up the Docker disk had 6 GB free. That bring-up and its e2e passed (257 s). The cluster was then deleted, the BuildKit cache pruned (16 GiB freed), and a second fresh cluster brought up for the suite below.
• PR CI. Pull CI and all four Kind shards passed on 023ea6d.
• Final head. 80a2144 is 023ea6d rebased onto newer main (9254f09). Two files this PR edits (the Kind README and bootstrap.sh) also changed on main, in neighbouring lines; the rebase merged them without conflict, and the node image tag is unchanged. The force-push starts PR CI on it.
• Not run. bootstrap-prod-seed.sh needs an externally configured S3 source. Its new skip check was run on its own against the restarted worker: the seed's build directory there (templates/seed/sha256:<hash>) matches the glob.

Run	Bring-up + e2e	Typed suite (all)	ECW	Native
Previous round, baseline (starting commit)	305 s	2881 s	77 pass, 1 infra error	23 pass
This head	250 s	2863 s	78 pass	24 pass

Every scenario passed. local.ecw_template_build_from_oci_boots passed for the first time on this host: its INFRA_ERROR in every earlier run was the builder's disk admission, and the harness now prunes superseded images from the node stores (#476). native.encrypted_double_restore_retires_prior_pods is new on main.

Every scenario (102)

Surface	Scenario	Previous baseline	This head
ecw	ecw.access_idle_reconnect_preserves_workspace	PASS	PASS
ecw	ecw.authenticated_access_with_public_traffic_disabled	PASS	PASS
ecw	ecw.background_process_control	PASS	PASS
ecw	ecw.binary_file_round_trip	PASS	PASS
ecw	ecw.cold_create_environment	PASS	PASS
ecw	ecw.command_round_trip	PASS	PASS
ecw	ecw.domain_access_survives_default_deny	PASS	PASS
ecw	ecw.filesystem_mutations	PASS	PASS
ecw	ecw.filesystem_ownership	PASS	PASS
ecw	ecw.filesystem_private_boundary	PASS	PASS
ecw	ecw.filesystem_read_identity	PASS	PASS
ecw	ecw.filesystem_round_trip	PASS	PASS
ecw	ecw.filesystem_watcher	PASS	PASS
ecw	ecw.filesystem_write_identity	PASS	PASS
ecw	ecw.guest_vlan_frames_dropped	PASS	PASS
ecw	ecw.immediate_egress	PASS	PASS
ecw	ecw.internet_access_allowed	PASS	PASS
ecw	ecw.internet_access_denied	PASS	PASS
ecw	ecw.large_stdout_integrity	PASS	PASS
ecw	ecw.loopback_route_stays_open	PASS	PASS
ecw	ecw.pause_connect	PASS	PASS
ecw	ecw.pause_connect_race	PASS	PASS
ecw	ecw.pause_timeout_requires_connect	PASS	PASS
ecw	ecw.process_identity	PASS	PASS
ecw	ecw.process_reattach	PASS	PASS
ecw	ecw.public_port_to_all_interfaces	PASS	PASS
ecw	ecw.public_port_to_loopback	PASS	PASS
ecw	ecw.refresh_keepalive	PASS	PASS
ecw	ecw.sandbox_delete	PASS	PASS
ecw	ecw.sandbox_info	PASS	PASS
ecw	ecw.sandbox_list_pagination	PASS	PASS
ecw	ecw.timeout_rearm	PASS	PASS
ecw	ecw.timeout_update	PASS	PASS
ecw	ecw.warm_pool_environment	PASS	PASS
ecw	ecw.watcher_identity	PASS	PASS
ecw	local.btrfs_send_restore_fails_closed_without_parent_source	PASS	PASS
ecw	local.btrfs_send_restore_pulls_parent_template	PASS	PASS
ecw	local.btrfs_send_restore_rejects_corrupted_stream	PASS	PASS
ecw	local.cold_sandbox_eviction_wake_round_trip	PASS	PASS
ecw	local.concurrent_restore_converges_to_one_execution	PASS	PASS
ecw	local.deferred_btrfs_send_capture_restores_cross_node	PASS	PASS
ecw	local.ecw_template_build_from_oci_boots	INFRA_ERROR	PASS
ecw	local.encrypted_resuspend_preserves_cow	PASS	PASS
ecw	local.forced_drain_evacuates_sandbox	PASS	PASS
ecw	local.gentle_drain_leaves_sandbox_resident	PASS	PASS
ecw	local.node_ctl_fence_refuses_and_reconciles	PASS	PASS
ecw	local.node_group_boundary_schedules	PASS	PASS
ecw	local.node_selector_extras_render_and_enforce	PASS	PASS
ecw	local.orphaned_restore_pod_gc_reaps_only_superseded	PASS	PASS
ecw	local.pending_backup_census_reports_blocked_upload	PASS	PASS
ecw	local.private_template_build_isolation	PASS	PASS
ecw	local.quarantine_recovery_backs_off	PASS	PASS
ecw	local.rate_limiters_enforce_and_fail_open	PASS	PASS
ecw	local.retention_claim_finish_forward	PASS	PASS
ecw	local.retention_restore_source_guard	PASS	PASS
ecw	local.sandbox_network_isolation_holds	PASS	PASS
ecw	local.snapshot_reaper_reclaims_backed_fork_snapshot	PASS	PASS
ecw	local.suspend_restore_cross_node_from_object_storage	PASS	PASS
ecw	local.suspend_snapshot_rows_are_per_attempt	PASS	PASS
ecw	local.telemetry_reaches_local_backends	PASS	PASS
ecw	local.template_cache_lru_eviction_reclaims	PASS	PASS
ecw	local.template_startup_recovery_decision_table	PASS	PASS
ecw	network_policy.v1_cidr_allow	PASS	PASS
ecw	network_policy.v1_cidr_deny	PASS	PASS
ecw	network_policy.v1_deny_precedes_allow	PASS	PASS
ecw	network_policy.v1_domain_binding	PASS	PASS
ecw	network_policy.v1_exact_ip_allow	PASS	PASS
ecw	network_policy.v1_exact_ip_deny	PASS	PASS
ecw	network_policy.v1_fastly_shared_edge_bypass	PASS	PASS
ecw	network_policy.v1_platform_guardrail_precedes_allow	PASS	PASS
ecw	network_policy.v1_rules_survive_restore	PASS	PASS
ecw	network_policy.v1_tcp_dns_transport_respects_ip_allow	PASS	PASS
ecw	network_policy.v1_udp_deny_precedes_allow	PASS	PASS
ecw	network_policy.v1_udp_respects_cidr_allow	PASS	PASS
ecw	network_policy.v1_udp_respects_cidr_deny	PASS	PASS
ecw	network_policy.v1_udp_respects_ip_allow	PASS	PASS
ecw	network_policy.v1_udp_respects_ip_deny	PASS	PASS
ecw	network_policy.v1_wildcard_domain_allow	PASS	PASS
native	ecw.background_process_control	PASS	PASS
native	ecw.command_round_trip	PASS	PASS
native	ecw.pause_connect	PASS	PASS
native	ecw.sandbox_delete	PASS	PASS
native	ecw.sandbox_info	PASS	PASS
native	ecw.sandbox_list_pagination	PASS	PASS
native	native.encrypted_double_restore_retires_prior_pods	— (new on main)	PASS
native	native.resident_pause	PASS	PASS
native	native.stopped_restart_preserves_rootfs	PASS	PASS
native	native_concurrency.command_idempotency	PASS	PASS
native	native_concurrency.create_idempotency	PASS	PASS
native	native_concurrency.independent_commands	PASS	PASS
native	native_concurrency.lifecycle_contention	PASS	PASS
native	native_concurrency.slow_reader	PASS	PASS
native	native_contract.command_validation	PASS	PASS
native	native_contract.completed_delete_replay	PASS	PASS
native	native_contract.idempotency_conflict	PASS	PASS
native	native_contract.permanent_command_error	PASS	PASS
native	native_contract.tenant_isolation	PASS	PASS
native	native_contract.token_scope	PASS	PASS
native	native_contract.zero_vcpu_admission	PASS	PASS
native	native_recovery.command_survives_data_plane_restart	PASS	PASS
native	native_recovery.queued_create_survives_api_restart	PASS	PASS
native	native_recovery.slow_operation_does_not_block_later_work	PASS	PASS

Node restart recovery. After the suite, a native sandbox was created on space-local-worker, written to and suspended. Then docker restart was run on space-local-worker2:

• the node never went NotReady; every node-service pod on it was running again within 6 s and ready within 16 s;
• space-cache logged a fresh format at boot, and template-manager pulled seed again 23 s after the restart;
• node-manager's init container staged the guest kernel again (staged=1 unchanged=0), and node-ctl republished the node as ready (0 ready slots, min_ready 0) with capacity 300;
• with the other worker cordoned, a new sandbox ran on the restarted worker, and the suspended sandbox's cross-node restore landed there with its file intact.

Event throttling. No throttling or dropped-event lines in either sandbox worker's kubelet journal after the suite and three bursts of 20 concurrent native creates.

Create latency (local Kind, medians). The previous round's branch column was measured while kind's runc settings were lost, with nodes on the cgroupfs driver. Those numbers are superseded by this round's:

	Previous baseline	Previous branch (superseded)	This head
Native create, queued → claimed (public_operations, suite)	566 ms (n=28)	15 ms	14 ms (n=27)
Native create, queued → succeeded (same)	2232 ms	1040 ms	1161 ms
ECW create, synchronous API latency (space_sandbox_api_latency_milliseconds, suite)	2141 ms	1727 ms	1740 ms (n≈80)
20 concurrent native creates, first request to last create complete	5.89 s (rerun 5.75 s)	3.77 s	4.09 s, 4.33 s, 4.13 s

The gain over the baseline is consistent with the dispatcher tick, and it holds with the systemd cgroup driver. Concurrent bursts finished about 0.3–0.5 s slower than the superseded measurement.

No scenario failed because of this change, so there is no product fix to depend on.

Review follow-up

Restack onto #480: conflicts in the README bring-up paragraph and bootstrap.sh, where #241's profile wording and unconditional install-observability.sh call are kept beside this PR's shim text. install-shim.sh merged without conflict: it keeps #241's shim wrapper and telemetry environment, and nothing in Kind edits containerd's config or restarts containerd. The commit's content is otherwise unchanged.

Earlier rounds:

• containerd drop-in hid kind's config (fixed). A version 3 drop-in raised the merged version to 3, so containerd skipped migrating kind's version 2 CRI settings, and nodes ran the cgroupfs driver. The drop-in is now version 2. The image build checks the loaded config for the systemd driver and discard_unpacked_layers as well as the runtime. Verified live and against the previous image.
• Test-hooks row (fixed). It now names the reaper, template GC, prefetch, disk-admission and TokenReview-namespace hooks the charts set on every run, with their release values.
• Stale shim-install text (fixed). runtimeclass.yaml and the README now say the node image registers the handler and install-shim.sh only installs the binary, never restarting containerd.
• Opt-in seed staging assumed the cache survives (fixed). The script header and the README cache section say to re-run bootstrap-prod-seed.sh after a node restart. Its skip check now also requires the build directory on the cache, so a re-run soon after a restart stages again.
• Slot-pool description (fixed in part, rejected in part). The chart comment and the row above now say that sandbox slots were already allocated on demand, that each slot is its own namespace, and that builders change from pooled to on-demand slots. Rejected: the claim that the default pre-creates min(max, 100) slots. The fill loop fills to max(target, retained), and the retained default is max_slots, so it is 300, which matches the 300 namespaces measured on the baseline.
• Superseded latencies (re-measured). See above.

Rollback

The node image tag changes, so an existing Kind cluster must be recreated to pick this up, and also to revert it: the image check refuses nodes from another image. A node-container restart now deliberately starts with an empty cache. Reverting cannot recover a cache that was already reset, and templates staged by hand with bootstrap-prod-seed.sh must be staged again after a restart, as documented.

🤖 Generated with Claude Code. The change and its validation were produced by an AI agent; a person reviews it before merge.

[nickdujay-pplx]

Reviewed cd605e3 against #480. No additional blocker in this layer.

Checked the zero-sized warm pool against the allocation/readiness code, node-ctl's ownership of advertised capacity, the containerd v2 import and systemd-cgroup preservation checks, atomic checksum-verified shim installation, and the cache-format/loop-device safeguards. Exact-head chart renders, shell syntax and the existing node-image input/hash/compatibility check passed. All four Kind CI shards passed; Rust CodeQL is still running.

The image compatibility guard correctly requires recreating an older Kind cluster. A node-container restart now deliberately loses its local cache; reverting the script cannot recover that state, and manually staged seed artifacts must be staged again as documented. I did not independently repeat the Linux/KVM restart drill. Approval covers this PR's delta; #480's separate installer finding still needs resolving before the stack lands.