[ac-pplx]

Summary

The cluster controller decided whether to contend for its leader Lease from the substrate's contends_for_leader_lease capability, which Direct set to false. A Direct controller in a multi-replica Kubernetes Deployment therefore ran every reconciler in every replica: two public-operation dispatchers, two trigger fan-outs, and two copies of each sweeper and retention loop driving the same compare-and-set writes.

After this change the controller contends when the kill switch leaves election on and either:

• it runs in a Kubernetes pod (KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT both set, the kube client's in-cluster rule), or
• its substrate uses the Kubernetes API.

The new uses_kubernetes_api capability replaces contends_for_leader_lease and is true only for Kubernetes, so the Kubernetes-substrate decision is unchanged, kubeconfig included. A Direct controller outside a pod (for example under systemd) still runs the reconcilers without election. The no-election log line now names its reason, and docs/configuration.md says when the controller contends.

The Kind Direct lane (#1168) held the controller to one replica because of the old behaviour. This PR removes that guard, values-direct.yaml, and the comments that tied the replica count to the substrate. The Direct lane now runs the chart's two replicas with one leader, like the Kubernetes lane. The default Kubernetes render's objects are unchanged; only the rewritten leader-election comment in the controller Deployment differs in text.

Validation

At 60d4f598:

• cargo nextest run -p pplx-space-cluster-controller-server --all-features: 1176 passed, 0 skipped, against the Postgres the nextest setup script started (no dependency skips). This includes the 48 leader_election::tests, among them the full table of kill switch × pod × substrate.
• bash ci/scripts/run-rust-checks.sh clippy: passed.
• cargo arch with origin/main's architecture-baseline.toml, ARCHITECTURE.toml and renames, as CI runs it: 0 required findings. The only finding in a touched file is the existing advisory that substrate/mod.rs is below its cap.
• bash ci/scripts/run-shell-tests.sh (each test run individually): 40 of 41 pass. scripts/tests/test-seed-download-verification.sh fails on the test host because unsquashfs cannot write a security.selinux xattr as non-root; this change doesn't touch it.
• pytest scripts/tests/test_kind_*.py scripts/tests/test_cluster_chart_validate.py scripts/tests/test_space_cluster_smoke.py: 571 passed.
• helm template, before and after, for space-cluster-controller-local and the infra/space/cluster/chart umbrella with values.example.yaml (both releases): parsed objects identical.
  • Only text difference: the 5 rewritten comment lines in the controller Deployment.
  • substrate=direct with replicaCount=2 now renders (on main the guard refused it).
• Kind, Kubernetes lane (SPACE_KIND_TELEMETRY_MODE=otel bash infra/space/kind/scripts/up.sh): both replicas contended and one won. local.leader_election_brownout_keeps_one_leader passed.
• Kind, Direct lane (SPACE_KIND_SUBSTRATE=direct SPACE_KIND_STORAGE_CREDENTIALS=static bash infra/space/kind/scripts/up.sh), 2/2 controller replicas, SPACE_K8S_CONTROL_PLANE=false on the API and controller:
  • Election: both replicas logged contending for leadership. Exactly one logged elected leader; starting reconcilers and started the dispatcher, trigger fan-out and sweepers ("skipping Kubernetes-only reconcilers on direct substrate"). The other stayed Ready with no reconciler output. The cluster-controller Lease named the leader.
  • examples/first_sandbox.py: create, command ("Hello from SPACE"), list, inspect and delete all succeeded.
  • ecw.pause_connect: passed (SPACE_SMOKE_BACKEND=kind SPACE_SMOKE_SUBSTRATE=direct SPACE_SMOKE_API_SURFACE=ecw).
  • native.resident_pause: passed.
  • native.stopped_restart_preserves_rootfs: every stop/restart phase passed; the final node-cleanup check failed with "cannot locate the fixture sandbox worker". That check finds the worker through the sandbox pod, a Direct gap kind: run the Direct substrate on the Kind cluster (SPACE_KIND_SUBSTRATE=direct) #1168 already lists.
  • Reconcilers ran once: across the run, all delete phases, trigger fan-outs and worker selections appeared only in the leader's log.
  • Failover: deleting the leader pod moved the Lease to one other replica within about a second, and only that replica started reconcilers.

Severity and risk

Severity: moderate. Duplicate reconcilers on Direct-in-a-pod deployments double-drive dispatch and sweeps, and only compare-and-set fencing keeps that safe.

Risk:

• Direct in a pod: the controller now needs Lease RBAC in its namespace (the Kind chart already grants it). A Direct pod that cannot reach the apiserver now goes NotReady and exits on its liveness budget, like the Kubernetes controller, instead of running unelected.
• Kubernetes substrate, and Direct outside a pod: unchanged.
• Rollback: revert the commit. That also restores the Kind Direct lane's one-replica guard.

Rollout

During a rolling update of a multi-replica Direct controller in a pod, old pods run reconcilers unelected (as they do today) alongside the new elected leader until they terminate. That is no worse than the current state. Kubernetes-substrate deployments see no behaviour change.

AI assistance: Claude Code wrote the code change, tests, the rebase and conflict resolution, the guard removal, and this description.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[the-gigi-pplx]

Review — 2 findings

Low

1. crates/libs/api/src/controllers/substrate/AGENTS.md:72: the new "add a capability instead" example cites the one capability that PR substrate: make Kubernetes and Direct first-class in the substrate contract #1153 rewrites as debt.
   This sentence now reads "as the admin drain actions read supports_node_drain". That's accurate on main (routes/admin/nodes.rs:977, :1365). But PR substrate: make Kubernetes and Direct first-class in the substrate contract #1153 replaces the whole bullet, and its new contract lists supports_node_drain as "a capability for a missing feature" that is "debt, not precedent". Whichever PR lands second, the example ends up pointing readers at a field the contract tells them not to copy.
   uses_kubernetes_api, the field this PR adds, is a better example: it states an infrastructure fact, which fits the capability rule both on main and at PR substrate: make Kubernetes and Direct first-class in the substrate contract #1153's head. Or drop the example and let the rebase take PR substrate: make Kubernetes and Direct first-class in the substrate contract #1153's wording.

2. crates/libs/cluster/controller/server/src/leader_election/mod.rs:818-820: the rewritten ElectionDisabled doc keeps "which is what the chart enforces for this case". Neither case it now names involves a chart.
   The Kind chart pins SPACE_CLUSTER_CONTROLLER_LEADER_ELECTION_ENABLED to "true" "whatever the replica count" (infra/space/kind/charts/space-cluster-controller-local/templates/deployment.yaml:180-185), so it never takes the kill-switch path. The other case, a Direct process outside a pod, has no chart at all. Suggest ending at "Only safe at one replica", or saying that nothing enforces it.

Checked and fine:

• election() matches the description's four-way rule, and the 8-row table in leader_election/tests/mod.rs asserts each cell directly.
• ProcessHost::from_environment requires both service variables, and the test covers each one missing.
• The only reader of uses_kubernetes_api is election(). Nothing else is left reading contends_for_leader_lease or contends(.
• The lease Role and RoleBinding in space-cluster-controller-local/templates/rbac.yaml aren't gated on the substrate, so a Direct controller in a pod from this chart has the lease access it now needs.
• The two Kind chart comment fixes match Substrate::from_env_with_lifecycle, which selects Direct without a truthy SPACE_K8S_CONTROL_PLANE.

Verified against a9f278ce968892dea7f324d035a9b9f5922682ae. Reproduced locally: nothing executed. Both findings are from reading the source and the chart, plus a comparison with PR #1153's head. I relied on this head's green CI Rust tests run for the new unit tests, and I didn't run a Kind or Direct-in-pod deployment.

[the-gigi-pplx]

Approved. 2 non-blocking findings in the review comment above.