[ac-pplx]

Summary

Adds a Direct-substrate lane to the Kind harness: the same cluster and images, with the API and cluster controller on the Direct substrate (crates/libs/api/src/controllers/substrate/direct.rs) instead of Kubernetes. Until now nothing exercised Direct end to end.

• SPACE_KIND_SUBSTRATE=kubernetes|direct (infra/space/kind/lib/substrate.sh) selects it. The Tiltfile, up.sh and scripts/tests/lib/bootstrap-kind-stack.sh read it, and the typed smoke runner reads the bench's substrate the same way.
  • A cluster can't switch: the bring-up, up.sh and prepare-cluster.sh (CI's prepare step) refuse one running the other substrate, the bring-up under SKIP_BOOTSTRAP=1 too.
  • They refuse an invalid value, or direct without static storage credentials, before a running Tilt is stopped or a cluster created.
  • The deployed substrate is read from the API Deployment's SPACE_K8S_CONTROL_PLANE: true is Kubernetes, false is Direct, and any other value fails. kubectl's stderr is kept out of the value.
• space-api-local and space-cluster-controller-local take substrate: kubernetes|direct.
  • On direct the controller runs one replica, and the chart refuses any other count. Leader election currently follows the substrate and doesn't contend for the Lease there.
  • cluster-controller: contend for the leader Lease in a pod on any substrate #1157 changes that. Whichever of this PR and cluster-controller: contend for the leader Lease in a pod on any substrate #1157 merges second removes the guard and values-direct.yaml's replicaCount: 1.
• The node-service charts take node.substrate. On direct they leave SPACE_NODE_ID unset, because ClusterApi resolves a kube node name only to a Kubernetes registration.
  • node-api and node-ctl then read the UUID in /space/etc/node-id (mount.nodeId), as on a Direct host.
  • sandbox-manager reads it through its own /space/etc mount, for its activation identity only (see Not covered).
• infra/space/kind/scripts/register-direct-nodes.sh seeds that UUID before the node services start, and fails if it can't list the nodes.
  • It then registers each sandbox and builder node through POST /admin/api/v1/nodes with verify: true. The address is the node's InternalIP, which ClusterApi checks its calls against.
  • Both steps are parallel Tilt resources, on the Direct lane only.
• The lane requires SPACE_KIND_STORAGE_CREDENTIALS=static. In exchange mode the credential-manager authority accepts a node service only through its Kubernetes node registration (node_k8s). A Direct node has none, so every template pull is refused.
• wait-node-publication.sh matches a Direct registration by its name.
• Smoke scenarios get a substrates attribute, derived from the fixture type.
  • Fixtures that need no Kind-only Kubernetes wiring (Ecw, CommandSandboxFixture, NativeHardeningFixture) serve both substrates. KindEcw and KindNativeHardeningFixture serve Kubernetes only.
  • On Kind the suite reads the bench's substrate right after its bootstrap, before the other bench-mode filters. It uses a bench reader built whatever the first scenario's fixture, which also serves the telemetry, storage and Rust-build reads. It prints each scenario it skips.
  • A run fails before anything runs if its SPACE_SMOKE_SUBSTRATE or SPACE_KIND_SUBSTRATE names the substrate the bench does not run, or names one when the bench can't be read. That holds for a shard the other filters would empty too.
  • On the deployed backend, SPACE_SMOKE_SUBSTRATE is the explicit selector; unset, it filters nothing.
  • Three scenarios narrow to Kubernetes for what Direct doesn't support by contract: the two fork scenarios (Direct answers Unsupported), and native.encrypted_double_restore_retires_prior_pods, which asserts sandbox pods.
• The space-cluster umbrella (infra/space/cluster), whose subcharts are these charts, refuses any substrate or node.substrate other than kubernetes: the Direct substrate lane is Kind-only.

No production code changes. No CI job is added.

Validation

On this head's code (a final amend changed only an up.sh comment and one README sentence):

• bash ci/scripts/run-shell-tests.sh: all 41 tests passed (exit 0).
  • This box lacks tools some unrelated tests need, so the run used stand-ins on PATH:
    • cargo was the host's.
    • AWS CLI v2 was installed user-locally.
    • mksquashfs and unsquashfs ran from an Alpine container with -no-xattrs, with TMPDIR under the home directory, because the Docker daemon doesn't share the host's /tmp.
    • gpgv was mapped to gpg --verify.
  • test-kind-up-restart.sh has new cases. A reused cluster whose API runs Direct is refused before Tilt is stopped.
  • With no Tilt running, a misspelt substrate (Direct) and direct with exchange credentials are refused before the bring-up. The misspelt case failed before the fix: the run went on to the bring-up.
  • Its kubectl stub also warns on stderr for the substrate read. That failed vendor-copy-matches before the stderr fix: the Kubernetes cluster was refused as Direct.
• uv run --frozen --python 3.11 --group test python -m pytest space_smoke/tests scripts/tests/test_kind_*.py scripts/tests/test_cluster_chart_validate.py: 1076 passed, 1 skipped.
  • The new BenchSubstrateTests cover these cases:
    • A Kind run filters by the substrate read from the bench.
    • A request for the other substrate fails.
    • An unreadable bench fails only a request.
    • The deployed backend filters only by SPACE_SMOKE_SUBSTRATE.
    • A native category whose first scenario is a NativeHardeningFixture still filters by the bench's substrate on Kind. Before the fix it exited 2: 'the substrate is only readable on the kind backend'.
    • A shard emptied by the telemetry filter still fails a mismatched SPACE_SMOKE_SUBSTRATE. Before the fix it exited 0.
  • Each test failed before the runner change: _apply_substrate didn't exist.
• kind_stack_prepare_cluster, in a disposable stubbed reproduction (no shell test covers it): before the fix Direct, and direct with exchange, created a cluster, and an existing cluster on the other substrate passed. After it, all three are refused and matching requests pass.
• shellcheck 0.11.0 (shellcheck -x) on every changed script (lib/substrate.sh, register-direct-nodes.sh, up.sh, wait-node-publication.sh, the bootstrap library and the three tests): no new findings.
  • The bootstrap library has one more SC1091 (info: not following a source), for the substrate.sh line, like its eight sibling source lines.
• helm template (helm v3), compared against a git archive of the merge base:
  • Releases rendered:
    • the umbrella's space, with values.example.yaml;
    • the umbrella's space-builder, with chart/values-builder.yaml then values.example.yaml;
    • both umbrella releases with --kube-version 1.31.0;
    • space-api-local with values-kind.yaml, space-cluster-controller-local, and every node-service release, including the builder ones.
  • Defaults: every rendered field matches the merge base. The only differences are two rendered YAML comments in the controller Deployment, which this PR corrects. They show in the controller and umbrella space renders.
  • The umbrella refuses space-api-local.substrate=direct, space-node-api-local.node.substrate=direct, space-node-ctl-local.node.substrate=direct and (on space-builder) space-build-manager-local.node.substrate=direct. The message is "the Direct substrate lane is Kind-only; see infra/space/kind/README.md".
  • space-cluster-controller-local.substrate=direct fails on the controller's replica guard, whose message now says the same. With replicaCount=1 it fails on the umbrella's check.
  • The Kind charts with substrate=direct render SPACE_K8S_CONTROL_PLANE: "false", drop SPACE_NODE_ID, add the node-id mounts and run one controller replica.
• Smoke inventory (REGISTRY.select, quarantined and ignored excluded):
  • Kind: 118 unfiltered and on kubernetes (89 ECW, 35 native), unchanged.
  • Kind on direct: 70 (49 ECW, 27 native). This was 71 before the pod-asserting scenario narrowed to Kubernetes.
  • Deployed: 73, and 70 with SPACE_SMOKE_SUBSTRATE=direct.
  • The sharding test runs the default lane's four shards with the bench read as kubernetes. Every eligible scenario runs once.
• Direct lane, run before the last fixes to the smoke runner and prepare-cluster.sh (not rerun since), on a new cluster:
  • SPACE_KIND_SUBSTRATE=direct SPACE_KIND_STORAGE_CREDENTIALS=static bash infra/space/kind/scripts/up.sh: exit 0.
    • seed-direct-node-ids seeded all four sandbox and builder nodes.
    • register-direct-nodes registered each node on its first attempt.
    • The API ran SPACE_K8S_CONTROL_PLANE=false with one controller replica.
    • bash infra/space/kind/lib/substrate.sh check with the default request refused the cluster and printed the remedy.
  • python3 examples/first_sandbox.py printed Hello from SPACE, and no pod was created in the sandboxes namespace. The script then stopped at its interactive delete prompt, and first_sandbox.py delete removed the sandbox.
  • SKIP_BOOTSTRAP=1 SPACE_KIND_SUBSTRATE=direct SPACE_KIND_STORAGE_CREDENTIALS=static bash scripts/tests/smoke-kind-typed.sh all, with SPACE_SMOKE_SUBSTRATE unset:
    • The runner read Direct from the bench and printed 44 substrate skips, including the newly narrowed scenario.
    • ECW: 49 run, 47 passed, 2 failed.
    • Native: 27 run, 22 passed, 5 failed.
    • The failures are the ones listed under Not covered, and no sandbox pod was created.
  • The same command with SPACE_SMOKE_SUBSTRATE=kubernetes exited 2 before running any scenario: SPACE_SMOKE_SUBSTRATE=kubernetes, but the bench runs the direct substrate.
• Not rerun this round: the Direct lane (no live --category native_contract run), the default lane's bring-up and typed suite, and cargo arch. The identical renders, the unchanged inventory, and the shell and Python tests above cover the default lane.

Severity and risk

Severity: does not apply (test harness only).

Risk: Kind, smoke-harness and chart paths only.

• These charts are also the space-cluster umbrella's subcharts. Its default renders keep every field (checked above), and it refuses the Direct values.
• The default Kubernetes lane's scenario inventory is unchanged.
• Every Kind suite run now also reads the API Deployment's env once. If that read fails and the run names no substrate, the suite warns and keeps the inventory.

To roll back, revert the commits.

Not covered

Direct failures from the typed suite, none of them fixed here:

• ECW:
  • ecw.access_idle_reconnect_preserves_workspace fails on the idle wake (api: wake and delete a Direct sandbox the node suspended for idleness #1169).
  • ecw.pause_timeout_requires_connect shows the same symptom ("explicit connect did not resume the paused sandbox"). It has been reported as intermittent, and it failed in both Direct runs so far.
• Native:
  • The four native_contract.* idle-wake scenarios fail on the idle wake (api: wake and delete a Direct sandbox the node suspended for idleness #1169).
  • native.stopped_restart_preserves_rootfs looks up the worker for its cleanup check through the sandbox pod, which Direct doesn't have ("cannot locate the fixture sandbox worker"). This is a harness gap; its rootfs check applies to Direct.
  • In the first Direct run, before it was narrowed to Kubernetes, native.encrypted_double_restore_retires_prior_pods was inconclusive: the volume key was refused for lack of node_k8s.

Known gaps:

• Waking or deleting a sandbox that Direct suspended for idleness: api: wake and delete a Direct sandbox the node suspended for idleness #1169 and internal-api: wake an unplaced Direct sandbox on reattach instead of routing it to a stale route identity #1170.
• The credential-manager authority accepts a node service only through its Kubernetes node registration. So storage-credential exchange, volume keys and network credentials refuse Direct nodes; this is why the lane requires static.
• Template catalog reads resolve node identity through the Kubernetes registration too, so private templates are unavailable to Direct nodes.
• sandbox-manager's restore placement (crates/libs/sbx/mgr/server/src/service/start.rs) and quarantine recovery (crates/libs/sbx/mgr/server/src/snapshot_restore.rs) read only SPACE_NODE_ID, which is unset on this lane ([sbx-mgr] read the node id from SPACE_NODE_ID, then the node-id file, everywhere #1160). Neither explains a failure listed above.
• The node services still get SPACE_K8S_NODE_NAME. So node-ctl starts its Kubernetes node-local controller (crates/libs/node/ctl/server/src/kubernetes_controller/mod.rs), and node-manager sees a Kubernetes signal for ingress isolation. The lane is Direct for node identity and placement, not for every node-service behaviour.
• A refused registration surfaces only as the node-publication timeout.
• register checks only a node's id, so it keeps a stale address after a Kind node's IP changes. It retries 403, 409 and 422 until its deadline.
• No CI job runs the lane yet.
  • The proposal is one nightly shard plus path-filtered PRs, not required at first. The shard needs only SPACE_KIND_SUBSTRATE=direct SPACE_KIND_STORAGE_CREDENTIALS=static.
  • Holding back the Direct failures above needs a substrate-scoped exclusion. quarantined= withholds a scenario on every substrate, the required Kubernetes lane included.

AI assistance: Claude Code wrote the change and its tests, ran the validation and wrote this description.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[the-gigi-pplx]

Review — 2 findings

Low

1. scripts/tests/lib/bootstrap-kind-stack.sh:1618-1619 — no test covers the bring-up's substrate refusal.
   The description says the bring-up refuses a cluster on the other substrate, under SKIP_BOOTSTRAP=1 too. Only the kind_stack_assert_existing_cluster copy (line 1328, reached from up.sh) has a test. In a scratch copy of this head I deleted it, and test-kind-up-restart.sh failed its three other-substrate cases. I put it back and deleted lines 1618-1619 in bootstrap_kind_stack__bring_up instead. test-kind-up-restart.sh, test-kind-stale-tilt-recovery.sh and test-kind-localhost-ports.sh all still passed. The typed runner's _apply_substrate backstops the typed suites when SPACE_KIND_SUBSTRATE is exported, but not the Bash suites or a run with it unset. A stubbed SKIP_BOOTSTRAP=1 case beside the existing other-substrate ones would pin it.

2. Tiltfile:106 — the substrate check also runs under tilt down.
   The other live-cluster reads at Tiltfile load are guarded with if config.tilt_subcommand != "down": (lines 87 and 142), because teardown evaluates this file too. This one isn't, so on a Direct cluster, tilt down without SPACE_KIND_SUBSTRATE=direct exported fails at load. I stubbed kind and kubectl to show a Direct cluster: substrate.sh check space-local with the variable unset exits 1 with the refusal. README.md:2664 still documents tilt down && kind delete cluster --name space-local as a teardown path. teardown.sh doesn't run tilt down, so the scripted path is unaffected. Guarding the check the same way, or naming the variable in that README line, would fix it.

Checked and found sound:

• Default renders match the merge base (helm 4) for space-node-api-local, space-sandbox-manager-local, space-api-local and space-cluster-controller-local, apart from the two rewritten controller comments.
• The Direct renders set SPACE_K8S_CONTROL_PLANE: "false" on both API Deployments, drop SPACE_NODE_ID and add the node-id mount.
• The controller guard refuses substrate=direct at two replicas.
• The umbrella's $nodeCharts covers every chart that renders env.identity.
• contends_for_leader_lease is false on Direct (crates/libs/api/src/controllers/substrate/mod.rs:1170), so the one-replica guard is needed.
• On Kind both the ecw and native service paths build the bench reader whatever the fixture, so _apply_substrate can always read it there.
• Only the shared API Deployment carries app.kubernetes.io/name=space-api (the admin one is space-admin-api), so the substrate read gets a single value.

Verified against f5c7ecfc63c63ce7dac04570506506da1178b2b8. Reproduced locally:

• The three changed Kind shell tests on a Linux box: all pass, plus the two mutations above.
• pytest space_smoke/tests: 615 passed, 1 skipped, and 1 failure that fails identically on the merge base (test_fixed_response_script_emits_valid_http, a macOS-only failure).
• helm template of the default and Direct variants.

Not run: the Direct lane or any Kind smoke.

[the-gigi-pplx]

Approved. 2 non-blocking findings in the review comment above.

[zibi-pplx]

One follow-up on the README for the Direct lane.

• [low] The README doesn't say which node-service behaviour on the lane still uses Kubernetes
  infra/space/kind/README.md ("Direct substrate lane"); infra/space/kind/charts/space-node-service/templates/_helpers.tpl:133
  The node-service charts still set SPACE_K8S_NODE_NAME (and KUBE_NODE_NAME) on node.substrate=direct. So node-ctl starts its Kubernetes node-local controller, and node-manager sees a Kubernetes signal for ingress isolation. The PR description's "Known gaps" says so ("Direct for node identity and placement, not for every node-service behaviour"), but the README section, which is the lasting reference, does not. Neither does it mention the template-catalog limit (private templates are unavailable to Direct nodes). A green run on this lane, for example from a CI job, is evidence for the Direct control plane and node registration, not for node services as they run on a Direct host.
  Suggestion: add a short "What the lane does not cover" bullet to that README section listing these gaps, so readers of lane results see the scope.