Skip to content

api: document the Direct substrate's contract for agents - #357

Merged
ac-pplx merged 1 commit into
mainfrom
ac/substrate-direct-contract-doc
Oct 14, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/substrate-direct-contract-doc

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What merging does

Documents the Direct substrate's contract for agents, in two places:

  • Rules go in a Direct Substrate Contract section in crates/libs/api/src/controllers/substrate/AGENTS.md, plus a paragraph under its ## Tests:
    • when the api selects Direct;
    • implement new lifecycle operations for Kubernetes. Direct returns Unsupported unless the operation is one of the calls examples/first_sandbox.py makes (create, command session, list/inspect, delete);
    • new SubstrateAdapter operations are required methods, so the compiler demands each implementation and the forwarding arm. Direct returns Unsupported explicitly. Only "nothing to do" hooks called on every path keep a default, and they still need their arm;
    • where shared code needs a static fact about the substrate, add a SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read it through capabilities(); where the substrate must act, add a required method;
    • in new SQL, scope to a substrate's rows by binding the value instead of a 'k8s'/'space' literal: .bind(substrate.managed_by()) for the running substrate's rows, or a fixed SubstrateKind::Kubernetes/::Space where code targets one substrate's rows; return early in Rust when a fixed-scope statement may run under another substrate, as the sweeper's failed-active-execution cleanup does. Existing literals, such as the shared routable_execution_or_non_k8s fragment, are debt;
    • ownership of the Direct-only tables, enforced by archcheck's pattern_ratchet.direct_tables rule (advisory today);
    • don't cite a cluster-controller reconciler as the retry path for a failed Direct execution retire or repair, and don't change a known gap as a side effect of other work;
    • existing substrate matches, Direct-table joins in shared code and Unsupported trait defaults are debt, not precedent;
    • which test fixtures to use.
  • What Direct does today goes in a //! module doc on direct.rs, so it is reviewed with the code:
    • what is supported, what is Unsupported, what is a no-op and what is Direct-only;
    • how admin drain/cordon actions fail on a Direct node;
    • which cluster-controller loops are Kubernetes-only, and that the public-operation dispatcher runs for both substrates;
    • two gaps that succeed silently today.

It also corrects two comments:

  • Direct retire_execution named the cluster controller's failed-cleanup pass as its retry path. That pass does nothing unless the running substrate is Kubernetes, so a failed Direct retire is not retried.
  • The resolves_cold_create_shape field doc said that without the capability the requested shape goes to the node unchanged. Direct's one-shot create cannot carry vCPU or memory, so the doc now says so and points at the direct module doc.

It makes the rules reachable from the code they govern:

  • adds crates/libs/api/src/controllers/substrate/CLAUDE.md as a symlink to AGENTS.md, following the root convention;
  • widens the pointer in crates/libs/api/AGENTS.md to shared code that branches on the substrate or names a Direct-only table;
  • adds to crates/libs/cluster/controller/server/AGENTS.md the scoping rule (managed_by = $n, never a 'k8s'/'space' literal, bound from substrate.managed_by() for the running substrate's rows, or a fixed SubstrateKind for one substrate's rows: the Kubernetes arm of run_reconcilers, and tombstone_retention's Direct tombstones) and a link to the section;
  • adds a pointer in crates/libs/internal/api/server/AGENTS.md, since internal-api is the largest reader of the Direct-only tables outside the api crate.

Documentation and comments only. No code, schema, wire format, metric or configuration changes.

Dependencies. #368, #373 and #358 are on main. The capability rule uses #368's names (SubstrateCapabilities, ::DIRECT, ::KUBERNETES, required capabilities()). The binding rule uses #373's SubstrateKind (pub type SubstrateKind = NodeManagedBy; in common-core): the cluster-controller SQL binds managed_by = $n with .bind(SubstrateKind::Kubernetes), tombstone_retention binds SubstrateKind::Space, the transient-lifecycle sweeper's failed-active-execution cleanup returns early on any other substrate and binds Kubernetes otherwise, and docs/rust/space-conventions.md and the space-rust-development skill say to bind in new code and treat existing literals, including routable_execution_or_non_k8s, as debt. The Direct-table rule names #358's pattern_ratchet.direct_tables rule. The intra-doc links in this change name only items in the api crate; SubstrateKind appears only as plain code text, because nothing in the rustdoc changed here mentions it.

Review follow-up

Suggestion Change
1. An Unsupported default for new operations hides a forgotten forwarding arm New operations are required methods; Direct returns SubstrateError::Unsupported explicitly; defaults only for "nothing to do" hooks called on every path (retire_fence_mismatch), which still need their arm
2a. Capability wording will go stale with #368 Add a SubstrateCapabilities field, set it in SubstrateCapabilities::DIRECT and ::KUBERNETES, read it via capabilities()
2b. "may use 'k8s'" will go stale with #373 Both AGENTS.md files now say to bind: managed_by = $n with .bind(substrate.managed_by()), or .bind(SubstrateKind::Kubernetes) in a Kubernetes-only loop
3. "Selected Direct behavior" and "Known gaps" describe current code and will drift Moved into a //! module doc on direct.rs; AGENTS.md keeps the rules and a pointer. No issues filed; the gaps are described as current behavior
Finding #3: stale comment in Direct retire_execution Fixed: the comment now says a failed Direct retire is not retried, because the failed-cleanup pass skips every substrate but Kubernetes
Nit: "unless single-host or no-Kubernetes builds need it" is circular Replaced by a concrete test: the create, command, inspect and delete flow in docs/first-sandbox.md must keep working on Direct
Nit: direct/tests.rs Now "tests under direct/", which covers direct/cluster_admission_tests.rs
Nit: point the Direct-table rule at pattern_ratchet.direct_tables once #358 lands Done in the rebase pass below: the Direct-table rule names pattern_ratchet.direct_tables

Final fix pass (this head)

Finding Change
MAJOR 1: "Don't cite a reconciler as the retry path for a failed Direct operation" is false; the public-operation dispatcher runs on Direct and retries transient failures AGENTS.md: "Don't cite a cluster-controller reconciler as the retry path for a failed Direct execution retire or repair; the direct.rs module doc lists the Kubernetes-only loops." Module-doc heading is now # Kubernetes-only reconcilers and says the dispatcher runs for both substrates and retries transient operation failures
MAJOR 2: binding rules and merge gate described a #373 that wasn't pushed Rechecked against #373's head 14e5b7b, which binds. Rechecked against #373 as merged: its failed-cleanup SQL binds Kubernetes and returns early on other substrates (reworded in the rebase pass below)
MINOR 3: binding rule named only 'k8s' and was silent on existing literals api: "In new SQL, bind the substrate value instead of a 'k8s'/'space' literal: .bind(substrate.managed_by()) for the running substrate's rows, or a fixed SubstrateKind::Kubernetes/::Space where code targets one substrate's rows. Existing literals are debt." Cluster controller: bound from substrate.managed_by() for the running substrate's rows, or a fixed SubstrateKind for one substrate (Kubernetes arm of run_reconcilers; tombstone_retention's Direct tombstones)
MINOR 4: capability rule dropped #368's "acting stays a method" boundary "Where shared code needs a static fact about the substrate, add a SubstrateCapabilities field…; where the substrate must act, add a required method (above)."
MINOR 5: docs/first-sandbox.md is a Kind walkthrough, so "the flow" was unclear Now "one of the calls examples/first_sandbox.py makes (create, command session, list/inspect, delete)"; dropped "which must keep working on Direct"
MINOR 6: one-shot gap overstated (disk is forwarded; fallback is node-side) "One-shot create does not forward requested vCPU or memory overrides: sbx-mgr's CreateRequest has no fields for them, so autostart uses node-side defaults and memory policy while public.sandboxes records the requested values. Disk size is forwarded."
NIT 7: AGENTS.md repeated the module doc's node-api sentence Removed from AGENTS.md; kept only in the direct.rs module doc
NIT 8: resolves_cold_create_shape doc says the shape reaches the node unchanged Field doc corrected (doc-only): "…except that Direct's one-shot create cannot carry vCPU or memory (see the [direct] module doc)"
NIT 9: existing Unsupported trait defaults contradict the required-method rule Debt bullet now lists Unsupported trait defaults (list_pool_configs, upsert_pool_config, disable_pool_config)

Rebase pass (this head, after #358 and #373 merged)

Item Change
Direct-table rule had no enforcement pointer Names archcheck's pattern_ratchet.direct_tables rule, advisory today, in one clause; its config is not restated
#373 names checked against main SubstrateKind, .bind(SubstrateKind::Kubernetes)/::Space, tombstone_retention's ::Space, the Kubernetes arm of run_reconcilers, and the conventions doc and skill all match. The sweeper's list/rotate/repair failed-active-execution statements now return early on a non-Kubernetes substrate instead of selecting 'k8s' rows in SQL, so the direct.rs module doc and the retire_execution comment now say the failed-cleanup retry does nothing unless the running substrate is Kubernetes. The AGENTS.md rule now also states the early-return convention and names routable_execution_or_non_k8s as remaining debt, as the skill does
Intra-doc links No rustdoc touched by this change mentions SubstrateKind, so none was added
"Merges after #373" note Removed; both dependencies are on main

Evidence

Rows added or changed in the final fix pass cite current main (1bd4173). Other line references are at c565c8e, the base the facts were first checked against. The cited Direct, reconciler and sweeper code is unchanged on current main (02e0656) apart from line shifts and the new direct/cluster_admission_tests.rs module. On this head, the module doc moves later direct.rs lines down by 46. Unless a full path is given, direct.rs, mod.rs and kubernetes/… are under crates/libs/api/src/controllers/substrate/; lifecycle.rs and queries.rs are under crates/libs/api/src/controllers/sandbox/; nodes.rs, node_store.rs and node_maintenance.rs are under crates/libs/api/src/routes/admin/; CC/ is crates/libs/cluster/controller/server/src/.

Statement Evidence
DirectSubstrate dials each worker's node-api at node_direct.node_api_endpoint, with no sandbox pods direct.rs:1934-1946 (worker_client_for_node uses node.node_api_endpoint); mod.rs:3915-3927 (connect_worker opens a worker-node-api channel); direct.rs:1463-1468 (node_api_endpoint from node_direct); docs/architecture.md:31
Direct exists for single-host deployments without Kubernetes and for builds without the kubernetes feature mod.rs:2722-2738 (Kubernetes is tried only under #[cfg(feature = "kubernetes")]; otherwise Self::Direct); mod.rs:12-27 (readiness stub for builds without the feature: "the direct substrate (local dev)"); the Direct node registry node_maintenance.rs:998, :1137
Kubernetes is returned only with that feature and a truthy SPACE_K8S_CONTROL_PLANE; unset, false or unrecognized selects Direct kubernetes/mod.rs:410-411 (env_flag_var("SPACE_K8S_CONTROL_PLANE", false) → Ok(None)); crates/libs/common/core/src/env.rs:47-55 (only 1/true/yes/on/enabled are true; anything unrecognized falls back to the default false), :59-61; mod.rs:2728 (Ok(None) => {} falls through to Direct)
A Kubernetes init failure is an error, not a fallback mod.rs:2729 (Err(err) => return Err(err))
Shipped images and Kind enable the feature; a plain cargo build -p pplx-space-api-svc does not docker/Dockerfile.rust-build-common:252-259 (build-api stage: -F pplx-space-api-svc/kubernetes), :343-348 (copied into space-rust-build-api, which docker/Dockerfile.api:1-2 consumes); docker/Dockerfile.rust-build-kind:110; crates/apps/api/svc/Cargo.toml:18 (default = []), :31
Create/delete, start, suspend/restore and wake go through SubstrateAdapter direct.rs:3436, :3727, :3784, :3855, :3840, :3886
Stop, kill, pause/resume and checkpoint go through the shared controller over Direct's runtime clients, which never recreate an execution lifecycle.rs:2005-2035 (Stop/Kill → runtime_client), :2047-2091 (Pause, Resume, Checkpoint → reachable_runtime_client_or_unavailable / runtime_client), :2236-2290; Direct's clients only dial the placed worker: direct.rs:4047-4069 → node_client :3038-3044
Warm-pool claims adopt node-side; custom vcpu/memory or encrypted claims go cold direct.rs:3177, :3120, :3126
Admission-time pool slot claims always miss mod.rs:2525-2537 (default Ok(None), "substrates without a warm pool always miss"); direct.rs does not override claim_admission_pool_slot
Unsupported: node-pool placement direct.rs:3440-3446
Unsupported: forks crates/libs/api/src/fork_operation_executor.rs:1050 (the executor's target create, the only caller of restore_fork_disk_as_new); queries.rs:1097-1115 (restore_from: Some(restore.snapshot)); direct.rs:3447, :4210-4216
Unsupported: cluster pool configs trait defaults mod.rs:2588-2613, not overridden in direct.rs; dispatch mod.rs:3801-3828
No-ops: sandbox_ip, access_sandbox_ip, list_inventory_nodes (Postgres is the inventory) direct.rs:4119-4131, :4199-4207
No-ops: retire_fence_mismatch and observe_pod_identities trait defaults mod.rs:2214-2220 (Ok(None)), :2232-2237 (empty map); neither is overridden in direct.rs
retire_execution ignores its fence but still stops the sandbox and clears its routing direct.rs:3746-3761 (fence: _fence), :3769 (stop_sandbox), :3779 (clear_direct_route_identity)
Direct-only: node registry and template pull and delete routes (501 on Kubernetes) node_maintenance.rs:431-439 (require_direct_substrate → NOT_IMPLEMENTED), called by create_node :998, update_node :1192, delete_node :1329, pull_node_template :1581, delete_node_template :1712; list_node_templates (:1872) is not gated
After drain-state validation, admin :drain, :undrain, :cordon and :uncordon return 502 on a Direct node validation first: nodes.rs:794-800 (validate_drain_transition; a repeated :drain :185-195, :undrain on an undrained node :216-219 and :uncordon while draining :236-241 are conflicts → 409); then :1006-1011 (all four have a desired schedulability), :1027-1031 ("node has no associated substrate node id"; substrate_node_id is node_k8s.k8s_node_name, node_store.rs:292-295, NULL for a Direct node), :984-996 (BAD_GATEWAY)
:drain still commits drain_phase='draining', which nothing advances nodes.rs:185-190 (Move to draining), :902, :924 (commit before converge at :936); drained is written only by the node-drain reconciler (nodes.rs:195-197), which is not started on Direct (CC/reconcilers.rs:390-393)
run_reconcilers starts drain, orphan, failed-row repair and node-sync loops only in its Substrate::Kubernetes arm CC/reconcilers.rs:189; :390-393 (Substrate::Direct(_) => "skipping Kubernetes-only reconcilers"); Kubernetes arm spawns: orphaned_sandboxes :420, started_sandbox_repair :449, failed_restorable_repair :466, kube_node_sync :516, node_drain :645-668, drained_node_terminator :819
The public-operation dispatcher runs for both substrates and retries transient failures of the operations it executes on by default: CC/public_operation_dispatcher.rs:71 (enabled: true); spawned before the substrate match: CC/reconcilers.rs:285 (match at :390); requeues transient failures within a retry budget: CC/public_operation_dispatcher.rs:705-726, :838. Trigger fanout also runs for both: CC/reconcilers.rs:222
Direct reaches retire_execution only through the sweeper's best-effort cleanup_after_sweep non-test callers of .retire_execution( in the control plane: CC/transient_lifecycle_sweeper/store.rs:1151 (cleanup_after_sweep) and :1231 (cleanup_failed_execution), plus dispatch at mod.rs:3290; CC/transient_lifecycle_sweeper/mod.rs:87-95 ("The call is best-effort: if it fails, the row keeps its resting lifecycle")
The failed-cleanup retry is 'k8s'-only, so nothing retries a failed Direct retire (also the corrected retire_execution comment) CC/transient_lifecycle_sweeper/store.rs:1166-1167 (s.managed_by = $1 AND s.managed_by = 'k8s'), :1213-1214; CC/transient_lifecycle_sweeper/mod.rs:107-108 ("retries active Kubernetes executions for authoritative failed rows"). The narrowing matters: Direct does run trigger fanout (CC/reconcilers.rs:221-224), which records failed deliveries for retry (crates/libs/cluster/reconciler/src/trigger_fanout.rs:416, crates/libs/cluster/trigger-queue/src/lib.rs:315-325)
Gap: one-shot create does not forward requested vCPU or memory; autostart uses node-side defaults and memory policy; disk size is forwarded (at 1bd4173) proto/sandbox/manager/v1/manager.proto:359-394 (no vcpu/memory; rootfs_size_mb at :374); direct.rs:2990 on this head (rootfs_size_mb: request.resources.disk_mib); crates/libs/sbx/mgr/server/src/service/start.rs:1806-1812 (node-read template default, else 1024 MiB, then node memory policy), :1914 (else 2 vCPU)
Earlier wording of the same gap (at c565c8e) direct.rs:180-186, :3448-3449, :3519-3544 (returns without a StartRequest); direct.rs:2926-2957 (CreateRequest without vcpu/memory); proto/sandbox/manager/v1/manager.proto:359-394 (no such field; StartRequest has them at :411-412); sbx-mgr autostart falls back to template defaults (crates/libs/sbx/mgr/server/src/service/create.rs:1358, service/start.rs:1807, :1909)
…yet the public.sandboxes row records the requested shape queries.rs:606 (non-k8s substrates keep the request unchanged); queries.rs:1369-1370 (INSERT INTO sandboxes … vcpu, memory_mib bound from resources)
Gap: retire_execution never enqueues the stopped-disk backup the trait documents; the Kubernetes teardown does trait mod.rs:2175; direct.rs:3755-3782 (no enqueue_artifact_backup/BackupSource anywhere in direct.rs); kubernetes/adapter/teardown.rs:384-409; sbx-mgr Stop doesn't enqueue one either: crates/libs/sbx/mgr/server/src/service/grpc_core.rs:579-692
A new operation is a required method, so the compiler demands each implementation and the forwarding arm; a default would let a forgotten arm give both substrates the default mod.rs:3229 (the enum implements the trait by per-method match forwarding, e.g. :3238-3244); shared code holds the enum: controllers/sandbox/mod.rs:1030 (substrate: Substrate). Rust rejects an impl that omits a required method and silently uses the default for an omitted provided one. #368 makes capabilities() required for this reason.
Only hooks shared code calls on every path keep a "nothing to do" default, as retire_fence_mismatch does Existing hook defaults: complete_activation_lease mod.rs:2159-2164 (no-op; called on every create, queries.rs:1793, and start, lifecycle.rs:1641), persist_pooled_execution_claim mod.rs:2543-2550 (Ok(()); called at queries.rs:2929), retire_fence_mismatch :2214-2220, observe_pod_identities :2232-2237. The existing Unsupported operation defaults (mod.rs:2588-2613, :2627-2635) predate the rule.
Static facts are capabilities; acting stays a method mod.rs:2169-2172 at 1bd4173 (#368's SubstrateCapabilities doc: "behavior that needs the substrate to act stays a [SubstrateAdapter] method")
Unsupported trait defaults are debt mod.rs:2674-2698 at 1bd4173 (list_pool_configs, upsert_pool_config, disable_pool_config default to Unsupported)
resolves_cold_create_shape doc correction mod.rs:2193-2199 on this head; see the one-shot gap row
Direct keeps the calls examples/first_sandbox.py makes examples/first_sandbox.py:172 (create), :178 (process list), :230 (command session), :268 (inspect), :286 (delete)
Capabilities: add a SubstrateCapabilities field, set it in DIRECT and KUBERNETES, read it through capabilities() #368's diff of mod.rs (struct, both constants, required fn capabilities(&self) -> SubstrateCapabilities) and of routes/admin/sandboxes.rs (reads .supports_node_pool_placement through it)
Don't branch behavior on managed_by(); never choose a runtime endpoint from a row's label mod.rs:2137-2141 ("This is descriptive metadata. Runtime routing should use the active execution binding, not this label.")
Bind in new SQL: substrate.managed_by() for the running substrate's rows, a fixed SubstrateKind for one substrate's rows; existing literals are debt (at 1bd4173) tombstone_retention runs for both substrates (CC/reconcilers.rs:253) and targets Direct tombstones with a fixed n.managed_by = 'space' (CC/tombstone_retention.rs:91, :191), which #373 binds as SubstrateKind::Space. Existing api literals #373 doesn't touch: n.managed_by = 'space' in direct.rs (12 sites), 'k8s' in kubernetes/store/node_inventory.rs, 'space' in routes/admin/node_maintenance.rs. #373's conventions doc and skill: bind in new code, raw literals are existing debt
Scoping a query to the rows a substrate owns is required; bind the value rather than writing 'k8s' (at c565c8e) the sweeper runs for both substrates (CC/reconcilers.rs:381-388) and binds self.substrate.managed_by() into its queries (CC/transient_lifecycle_sweeper/store.rs:826-829); its doc comment (store.rs:785-793) says this scope stops a Kubernetes controller from claiming Direct rows "and vice versa". NodeManagedBy already binds as a SQL parameter (crates/libs/cluster/reconciler/src/trigger_fanout/tests.rs:74); #373 names it SubstrateKind (crates/libs/common/core/src/types/node.rs in its diff). Kubernetes-arm loops that still use the literal: CC/node_drain/store.rs:536, CC/drained_node_terminator/store.rs:192
Direct-only tables: outside tests, write them only from direct.rs and the Direct node registry these are the only non-test writers at this commit: direct.rs:468, :499, :747, :1117, :1185, :1289, :1379, :1423; node_maintenance.rs:1066, :1137, :1269. 17 files under crates/ contain INSERT INTO node_direct, mostly test fixtures. All three tables cascade from nodes/sandboxes (db/postgres/20261002070625_baseline.up.sql:294, :834, :845)
Shared tests reach Kubernetes through a Substrate::Kubernetes built with the kubernetes::tests helpers, as admission_test_fixture.rs does kubernetes/mod.rs:1198-1199 (#[cfg(test)] pub(crate) mod tests); kubernetes/tests.rs:1850-1869 (pub(crate) fn substrate_with_store, substrate_with_store_and_client, substrate_with_store_client_proxy_port_and_pod); crates/libs/api/src/routes/public/sandboxes/admission_test_fixture.rs:156-166; also crates/libs/api/src/routes/e2b/tests.rs:338, :382, crates/libs/api/src/public_sandbox_operations/tests.rs:126
Adapter tests inside kubernetes/ use scripted_test_doubles.rs, as adapter_characterization_tests/ does (it is private to kubernetes) kubernetes/mod.rs:1195-1196 (#[cfg(test)] mod scripted_test_doubles;, private, with pub(super) items, e.g. kubernetes/scripted_test_doubles.rs:32, :35); kubernetes/scripted_test_doubles.rs:1-9; kubernetes/adapter_characterization_tests.rs:1-10 (added by #273)
Both compile only with --features kubernetes; CI's --workspace --all-features includes them, but cargo test -p pplx-space-api skips them mod.rs:2-3 (#[cfg(feature = "kubernetes")] pub mod kubernetes;); crates/libs/api/src/routes/public/sandboxes.rs:1356-1357 (#[cfg(all(test, feature = "kubernetes"))]); crates/libs/api/Cargo.toml:14 (default = []); ci/scripts/run-rust-checks.sh:553-560, :573-579

The remaining rules are guidance, not facts, so they have no row: Kubernetes first, Unsupported on Direct outside the examples/first_sandbox.py calls, capabilities over matches, reading through existing joins, "other existing matches and joins are debt, not precedent" (which excludes managed_by scoping), and "don't change a known gap as a side effect". Each is grounded in the rows above.

Findings outside this change (not fixed)

  1. Admin drain actions on a Direct node leave a stuck phase and promise a retry that never comes.

    • nodes.rs:902-924 commits drain_phase='draining' before converging.
    • The converge fails for every Direct node (:1027-1031), and the 502 body reports "reconciler_will_retry": true (:927, :993). The node-drain reconciler is never started on Direct (CC/reconcilers.rs:390-393) and only scans managed_by = 'k8s' (CC/node_drain/store.rs:536).
    • :undrain clears the phase (nodes.rs:210-215) but still returns 502 with the same retry promise.
    • Nothing moves a Direct node from draining to drained, so :terminate, which requires drained (nodes.rs:199-208), returns 409.
    • Under mode: forced, the shared drain gates return 409 for pause, checkpoint and access on that node's sandboxes (crates/libs/common/core/src/types/node.rs:930-934, :1021-1022; lifecycle.rs:1136-1148, :2048-2049).
    • Direct placement doesn't read nodes.drain_phase (it filters on node-reported node_status_snapshots.scheduling_status, direct.rs:1463-1477, :1513, :1621), so new creates can still land on the node.
  2. The Direct-only tables already have many readers outside direct.rs (non-test, at c565c8e):

    • crates/apps/admin/cli/src/cli.rs:181
    • crates/libs/api/src/controllers/sandbox/responses.rs:74,78,440,528
    • controllers/sandbox/spaced.rs:795
    • crates/libs/api/src/public_template_operations.rs:200
    • node_maintenance.rs:658,687,1032,1066,1137,1269,1349 (the registry, which also writes)
    • node_store.rs:294
    • crates/libs/api/src/routes/admin/templates.rs:509,525
    • crates/libs/api/src/routes/public/sandboxes.rs:943
    • CC/node_drain/store.rs:2065
    • CC/tombstone_retention.rs:90,190 (deletes Direct nodes rows, cascading to node_direct)
    • crates/libs/cluster/trigger-queue/src/lib.rs:240
    • crates/libs/common/core/src/types/admin.rs:510
    • crates/libs/internal/api/server/src/data_plane_sessions.rs:872,877,917,922,1010,1217,1222
    • crates/libs/internal/api/server/src/lib.rs:1215
    • crates/libs/internal/api/server/src/published_state.rs:2244,2306,2349,2423
    • crates/libs/sandbox/portability/src/lib.rs:490 (doc comment only)
    • crates/libs/sandbox/store/src/lib.rs:36,296,508
    • outside Rust: scripts/admin/prune_cluster_artifacts.py:2377

    Shared code also reads the row's managed_by label for behavior, which the trait doc says not to do: controllers/sandbox/spaced.rs:75 takes the wake fallback only when sandbox.managed_by == "k8s".

  3. A comment in Direct retire_execution names the wrong retry loop. Fixed in this PR (see above).

  4. After a Direct one-shot create, the shape in Postgres can differ from the shape the VM booted with. With wait_until_ready=true and boot_mode cold or snapshot, Direct sends a single CreateRequest, which has no vcpu/memory fields. sbx-mgr's autostart StartRequest uses ..Default::default(), so the VM boots with the template's defaults, else 2 vCPU / 1024 MiB (crates/libs/sbx/mgr/server/src/service/start.rs:1807, :1909; crates/libs/sbx/mgr/server/src/service/create.rs:1358). The public.sandboxes row stores the requested vcpu/memory_mib (queries.rs:1369-1370). disk_mib is unaffected; it is sent as rootfs_size_mb. The non-one-shot cold path does send vcpu/memory (direct.rs:3667-3668).

Validation

At head 74fbd44, rebased on main (fc47b31):

  • cargo fmt --all -- --check: clean.
  • cargo clippy -p pplx-space-api --all-targets --all-features --locked -- -D warnings: clean.
  • cargo doc -p pplx-space-api --no-deps: exits 0. Its 8 warnings are the private intra-doc links and HTML tag already on main; none is on a changed line. The [direct] link in the resolves_cold_create_shape doc still resolves.
  • The pre-commit hook (gitleaks) passed on the amended commit.
  • Public-review skill on the staged change and on the commit and PR text: no findings in the reviewed scope.
  • Markdown links: ../../../api/src/controllers/substrate/AGENTS.md#direct-substrate-contract (cluster-controller and internal-api AGENTS.md) resolves to an existing file, and ## Direct Substrate Contract produces that anchor. examples/first_sandbox.py exists.
  • No tests run: the change is documentation, doc comments and a code comment.
  • Instruction chain sizes (wc -c):
File main Previous head This head
AGENTS.md 5948 5948 5948
crates/libs/api/AGENTS.md 1551 1665 1665
crates/libs/api/src/controllers/substrate/AGENTS.md 1792 5531 5876
api substrate chain 9291 13144 13489 (budget 32768)
crates/libs/cluster/controller/server/AGENTS.md 2012 2478 2478
cluster-controller chain 7960 8426 8426
crates/libs/internal/api/server/AGENTS.md 2270 2509 2509
internal-api chain 8218 8457 8457

The "previous head" column is 10e0d52.

🤖 Generated with Claude Code

@zibi-pplx zibi-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I checked the facts against this head and current main, and they hold. The branch merges cleanly with main. Checked:

  • substrate selection;
  • the Unsupported and no-op list;
  • Direct drain behavior;
  • Kubernetes-only reconcilers;
  • the retire-backup gap;
  • the Direct-table writers;
  • the test-fixture paragraph.

None of the suggestions below blocks merging.

Suggestions

  1. crates/libs/api/src/controllers/substrate/AGENTS.md:82. The rule "a new operation gets an Unsupported default" leads to the failure it warns about.
    • If the forwarding arm in impl SubstrateAdapter for Substrate is forgotten, both substrates silently get the default, and a Kubernetes override is skipped.
    • A required method makes the compiler demand the arm. That's why #368 makes capabilities() required.
    • Suggested rule: new operations are required methods, and Direct returns Unsupported explicitly. Keep defaults only for "nothing to do" hooks, which still need their arm.
  2. :87-88, :92-93, and crates/libs/cluster/controller/server/AGENTS.md:26-27 will go out of date with #368 and #373.
    • After #368, a capability is a SubstrateCapabilities field: add the field, set it in DIRECT and KUBERNETES, and read it via capabilities().
    • #373 is moving to binding the value (managed_by = $n with .bind(SubstrateKind::Kubernetes)) instead of a raw 'k8s', so "may use 'k8s'" should say to bind.
    • Simplest: land this PR after those two, or use their wording now.
  3. "Selected Direct behavior" (:40) and "Known gaps" (:66) describe what the code does today rather than giving instructions, so they will drift.
    • Consider moving them into a //! module doc on direct.rs, where they get reviewed with the code, and keeping the rules plus a pointer here.
    • Filing the two gaps as issues and linking them would give those bullets a removal condition.

Nits

  • :79-80: "return Unsupported unless single-host or no-Kubernetes builds need the operation" is circular, since those are Direct's only users. A concrete test would be easier to act on, for example the flows in docs/first-sandbox.md.
  • :112: main now also has direct/cluster_admission_tests.rs, so "tests under direct/" would age better.
  • Once #358 lands, point the Direct-table rule at pattern_ratchet.direct_tables as what enforces it.

On the findings listed outside this change

  • #1, drain on a Direct node, is real and worth an issue.
    • :drain commits draining, then returns 502 with "reconciler_will_retry".
    • Nothing retries it: the node-drain reconciler runs only on Kubernetes and only scans 'k8s' rows.
    • :terminate then returns 409, and Direct placement filters on node-reported scheduling_status, so new creates still land on the node.
  • #3 is a one-line fix. The comment at direct.rs:3765-3767 names the cluster-controller failed-cleanup pass, but that pass can never select a Direct row.
  • #4, the one-shot create shape, is also real and worth its own issue.
    • With wait_until_ready=true and boot_mode cold or snapshot, Direct sends a single CreateRequest, which has no vcpu/memory fields.
    • sbx-mgr's autostart StartRequest uses ..Default::default(), so the VM boots with the template's defaults (or 2 vCPU / 1024 MiB).
    • Meanwhile the sandboxes row records the requested shape.
    • disk_mib is unaffected; it is sent as rootfs_size_mb.

@ac-pplx
ac-pplx force-pushed the ac/substrate-direct-contract-doc branch 2 times, most recently from 0fd7e20 to 10e0d52 Compare October 14, 2026 17:19
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and
a module doc on direct.rs, so agents changing the substrate, or shared
code that branches on it, know what Direct is for and where its contract
ends.

The AGENTS.md section holds the rules:

- when the api selects Direct (no `kubernetes` feature, or
  SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure
  is an error, not a fallback;
- implement new lifecycle operations for Kubernetes; Direct returns
  Unsupported unless the operation is one of the calls
  examples/first_sandbox.py makes (create, command session, list/inspect,
  delete);
- make new SubstrateAdapter operations required methods, so the compiler
  demands each substrate's implementation and the forwarding arm in
  `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that
  shared code calls on every path keeps a default, and it still needs
  its arm;
- where shared code needs a static fact about the substrate, add a
  SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read
  it through capabilities() instead of matching the substrate; where the
  substrate must act, add a required method (#368);
- in new SQL, scope to a substrate's rows by binding the value
  (`substrate.managed_by()` for the running substrate's rows, or a fixed
  `SubstrateKind::Kubernetes`/`::Space` for one substrate's rows) instead
  of a 'k8s'/'space' literal; existing literals are debt (#373);
- keep writes to the Direct-only tables in direct.rs and the Direct node
  registry;
- don't cite a cluster-controller reconciler as the retry path for a
  failed Direct execution retire or repair, and don't change a known gap
  as a side effect of other work;
- existing substrate matches, Direct-table joins in shared code and
  Unsupported trait defaults are debt, not precedent;
- which test fixtures to use: Direct only for tests under `direct/` or
  substrate-independent behavior, `kubernetes::tests` helpers for shared
  tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both
  compile only with `--features kubernetes`.

The direct.rs module doc records what Direct does today, so it is
reviewed with the code: how it reaches workers, which lifecycle
operations it implements, which it rejects as Unsupported (node-pool
placement, forks, cluster pool configs), its no-op trait methods, its
Direct-only admin routes, how admin drain/cordon actions fail on a
Direct node, and which cluster-controller loops are Kubernetes-only: the
drain, orphan, failed-row repair and node-sync loops never run for
Direct, while the public-operation dispatcher runs for both substrates
and retries transient operation failures. Nothing retries a failed
Direct retire. It lists two gaps that succeed silently: one-shot create
does not forward requested vCPU or memory (disk size is forwarded), and
retire_execution does not enqueue the stopped-disk backup the trait
documents.

Correct the comment in Direct retire_execution that named the
failed-cleanup pass as its retry path: that pass selects only 'k8s'
rows, so a failed Direct retire is not retried. Correct the
`resolves_cold_create_shape` field doc, which said that without the
capability the requested shape reaches the node unchanged: Direct's
one-shot create cannot carry vCPU or memory.

Make the rules reachable from the code they govern: add the
`controllers/substrate/CLAUDE.md` symlink, widen the pointer in
crates/libs/api/AGENTS.md, and point the cluster-controller and
internal-api AGENTS.md files at the section, with the cluster
controller's binding rule for `managed_by` scoping stated there,
including `tombstone_retention`'s fixed scope to Direct tombstones.

The Direct-table rule names archcheck's `pattern_ratchet.direct_tables`
rule (#358) as its enforcement, advisory today. The SQL-scoping rule also
covers returning early in Rust when a fixed-scope statement may run under
another substrate (#373), and names the shared
`routable_execution_or_non_k8s` fragment as remaining debt. The Direct
retire wording now says the sweeper's failed-cleanup retry does nothing
unless the running substrate is Kubernetes, matching the code after #373.

Documentation and comments only; no code, schema, wire or configuration
change.
@ac-pplx
ac-pplx force-pushed the ac/substrate-direct-contract-doc branch from 10e0d52 to 74fbd44 Compare October 14, 2026 19:19
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit e9ebff2 Oct 14, 2026
20 checks passed
@ac-pplx
ac-pplx deleted the ac/substrate-direct-contract-doc branch October 14, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants