Repository navigation
api: document the Direct substrate's contract for agents - #357
Merged
Merged
Conversation
ac-pplx
force-pushed
the
ac/substrate-direct-contract-doc
branch
2 times, most recently
from
October 13, 2026 04:28
4d9c89c to
682d5b7
Compare
This was referenced Oct 13, 2026
ac-pplx
marked this pull request as ready for review
October 13, 2026 15:42
This was referenced Oct 13, 2026
Merged
zibi-pplx
approved these changes
Oct 14, 2026
zibi-pplx
left a comment
Collaborator
There was a problem hiding this comment.
Approving. I checked the facts against this head and current main, and they hold. The branch merges cleanly with main. Checked:
- substrate selection;
- the
Unsupportedand no-op list; - Direct drain behavior;
- Kubernetes-only reconcilers;
- the retire-backup gap;
- the Direct-table writers;
- the test-fixture paragraph.
None of the suggestions below blocks merging.
Suggestions
crates/libs/api/src/controllers/substrate/AGENTS.md:82. The rule "a new operation gets anUnsupporteddefault" leads to the failure it warns about.- If the forwarding arm in
impl SubstrateAdapter for Substrateis forgotten, both substrates silently get the default, and a Kubernetes override is skipped. - A required method makes the compiler demand the arm. That's why #368 makes
capabilities()required. - Suggested rule: new operations are required methods, and Direct returns
Unsupportedexplicitly. Keep defaults only for "nothing to do" hooks, which still need their arm.
- If the forwarding arm in
:87-88,:92-93, andcrates/libs/cluster/controller/server/AGENTS.md:26-27will go out of date with #368 and #373.- After #368, a capability is a
SubstrateCapabilitiesfield: add the field, set it inDIRECTandKUBERNETES, and read it viacapabilities(). - #373 is moving to binding the value (
managed_by = $nwith.bind(SubstrateKind::Kubernetes)) instead of a raw'k8s', so "may use'k8s'" should say to bind. - Simplest: land this PR after those two, or use their wording now.
- After #368, a capability is a
- "Selected Direct behavior" (
:40) and "Known gaps" (:66) describe what the code does today rather than giving instructions, so they will drift.- Consider moving them into a
//!module doc ondirect.rs, where they get reviewed with the code, and keeping the rules plus a pointer here. - Filing the two gaps as issues and linking them would give those bullets a removal condition.
- Consider moving them into a
Nits
:79-80: "returnUnsupportedunless single-host or no-Kubernetes builds need the operation" is circular, since those are Direct's only users. A concrete test would be easier to act on, for example the flows indocs/first-sandbox.md.:112:mainnow also hasdirect/cluster_admission_tests.rs, so "tests underdirect/" would age better.- Once #358 lands, point the Direct-table rule at
pattern_ratchet.direct_tablesas what enforces it.
On the findings listed outside this change
- #1, drain on a Direct node, is real and worth an issue.
:draincommitsdraining, then returns 502 with"reconciler_will_retry".- Nothing retries it: the node-drain reconciler runs only on Kubernetes and only scans
'k8s'rows. :terminatethen returns 409, and Direct placement filters on node-reportedscheduling_status, so new creates still land on the node.
- #3 is a one-line fix. The comment at
direct.rs:3765-3767names the cluster-controller failed-cleanup pass, but that pass can never select a Direct row. - #4, the one-shot create shape, is also real and worth its own issue.
- With
wait_until_ready=trueandboot_modecoldorsnapshot, Direct sends a singleCreateRequest, which has no vcpu/memory fields. - sbx-mgr's autostart
StartRequestuses..Default::default(), so the VM boots with the template's defaults (or 2 vCPU / 1024 MiB). - Meanwhile the
sandboxesrow records the requested shape. disk_mibis unaffected; it is sent asrootfs_size_mb.
- With
ac-pplx
force-pushed
the
ac/substrate-direct-contract-doc
branch
2 times, most recently
from
October 14, 2026 17:19
0fd7e20 to
10e0d52
Compare
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and a module doc on direct.rs, so agents changing the substrate, or shared code that branches on it, know what Direct is for and where its contract ends. The AGENTS.md section holds the rules: - when the api selects Direct (no `kubernetes` feature, or SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure is an error, not a fallback; - implement new lifecycle operations for Kubernetes; Direct returns Unsupported unless the operation is one of the calls examples/first_sandbox.py makes (create, command session, list/inspect, delete); - make new SubstrateAdapter operations required methods, so the compiler demands each substrate's implementation and the forwarding arm in `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that shared code calls on every path keeps a default, and it still needs its arm; - where shared code needs a static fact about the substrate, add a SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read it through capabilities() instead of matching the substrate; where the substrate must act, add a required method (#368); - in new SQL, scope to a substrate's rows by binding the value (`substrate.managed_by()` for the running substrate's rows, or a fixed `SubstrateKind::Kubernetes`/`::Space` for one substrate's rows) instead of a 'k8s'/'space' literal; existing literals are debt (#373); - keep writes to the Direct-only tables in direct.rs and the Direct node registry; - don't cite a cluster-controller reconciler as the retry path for a failed Direct execution retire or repair, and don't change a known gap as a side effect of other work; - existing substrate matches, Direct-table joins in shared code and Unsupported trait defaults are debt, not precedent; - which test fixtures to use: Direct only for tests under `direct/` or substrate-independent behavior, `kubernetes::tests` helpers for shared tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both compile only with `--features kubernetes`. The direct.rs module doc records what Direct does today, so it is reviewed with the code: how it reaches workers, which lifecycle operations it implements, which it rejects as Unsupported (node-pool placement, forks, cluster pool configs), its no-op trait methods, its Direct-only admin routes, how admin drain/cordon actions fail on a Direct node, and which cluster-controller loops are Kubernetes-only: the drain, orphan, failed-row repair and node-sync loops never run for Direct, while the public-operation dispatcher runs for both substrates and retries transient operation failures. Nothing retries a failed Direct retire. It lists two gaps that succeed silently: one-shot create does not forward requested vCPU or memory (disk size is forwarded), and retire_execution does not enqueue the stopped-disk backup the trait documents. Correct the comment in Direct retire_execution that named the failed-cleanup pass as its retry path: that pass selects only 'k8s' rows, so a failed Direct retire is not retried. Correct the `resolves_cold_create_shape` field doc, which said that without the capability the requested shape reaches the node unchanged: Direct's one-shot create cannot carry vCPU or memory. Make the rules reachable from the code they govern: add the `controllers/substrate/CLAUDE.md` symlink, widen the pointer in crates/libs/api/AGENTS.md, and point the cluster-controller and internal-api AGENTS.md files at the section, with the cluster controller's binding rule for `managed_by` scoping stated there, including `tombstone_retention`'s fixed scope to Direct tombstones. The Direct-table rule names archcheck's `pattern_ratchet.direct_tables` rule (#358) as its enforcement, advisory today. The SQL-scoping rule also covers returning early in Rust when a fixed-scope statement may run under another substrate (#373), and names the shared `routable_execution_or_non_k8s` fragment as remaining debt. The Direct retire wording now says the sweeper's failed-cleanup retry does nothing unless the running substrate is Kubernetes, matching the code after #373. Documentation and comments only; no code, schema, wire or configuration change.
ac-pplx
force-pushed
the
ac/substrate-direct-contract-doc
branch
from
October 14, 2026 19:19
10e0d52 to
74fbd44
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What merging does
Documents the Direct substrate's contract for agents, in two places:
crates/libs/api/src/controllers/substrate/AGENTS.md, plus a paragraph under its## Tests:Unsupportedunless the operation is one of the callsexamples/first_sandbox.pymakes (create, command session, list/inspect, delete);SubstrateAdapteroperations are required methods, so the compiler demands each implementation and the forwarding arm. Direct returnsUnsupportedexplicitly. Only "nothing to do" hooks called on every path keep a default, and they still need their arm;SubstrateCapabilitiesfield, set it inDIRECTandKUBERNETES, and read it throughcapabilities(); where the substrate must act, add a required method;'k8s'/'space'literal:.bind(substrate.managed_by())for the running substrate's rows, or a fixedSubstrateKind::Kubernetes/::Spacewhere code targets one substrate's rows; return early in Rust when a fixed-scope statement may run under another substrate, as the sweeper's failed-active-execution cleanup does. Existing literals, such as the sharedroutable_execution_or_non_k8sfragment, are debt;pattern_ratchet.direct_tablesrule (advisory today);Unsupportedtrait defaults are debt, not precedent;//!module doc ondirect.rs, so it is reviewed with the code:Unsupported, what is a no-op and what is Direct-only;It also corrects two comments:
retire_executionnamed the cluster controller's failed-cleanup pass as its retry path. That pass does nothing unless the running substrate is Kubernetes, so a failed Direct retire is not retried.resolves_cold_create_shapefield doc said that without the capability the requested shape goes to the node unchanged. Direct's one-shot create cannot carry vCPU or memory, so the doc now says so and points at thedirectmodule doc.It makes the rules reachable from the code they govern:
crates/libs/api/src/controllers/substrate/CLAUDE.mdas a symlink toAGENTS.md, following the root convention;crates/libs/api/AGENTS.mdto shared code that branches on the substrate or names a Direct-only table;crates/libs/cluster/controller/server/AGENTS.mdthe scoping rule (managed_by = $n, never a'k8s'/'space'literal, bound fromsubstrate.managed_by()for the running substrate's rows, or a fixedSubstrateKindfor one substrate's rows: the Kubernetes arm ofrun_reconcilers, andtombstone_retention's Direct tombstones) and a link to the section;crates/libs/internal/api/server/AGENTS.md, since internal-api is the largest reader of the Direct-only tables outside the api crate.Documentation and comments only. No code, schema, wire format, metric or configuration changes.
Dependencies. #368, #373 and #358 are on
main. The capability rule uses #368's names (SubstrateCapabilities,::DIRECT,::KUBERNETES, requiredcapabilities()). The binding rule uses #373'sSubstrateKind(pub type SubstrateKind = NodeManagedBy;in common-core): the cluster-controller SQL bindsmanaged_by = $nwith.bind(SubstrateKind::Kubernetes),tombstone_retentionbindsSubstrateKind::Space, the transient-lifecycle sweeper's failed-active-execution cleanup returns early on any other substrate and binds Kubernetes otherwise, anddocs/rust/space-conventions.mdand thespace-rust-developmentskill say to bind in new code and treat existing literals, includingroutable_execution_or_non_k8s, as debt. The Direct-table rule names #358'spattern_ratchet.direct_tablesrule. The intra-doc links in this change name only items in the api crate;SubstrateKindappears only as plain code text, because nothing in the rustdoc changed here mentions it.Review follow-up
Unsupporteddefault for new operations hides a forgotten forwarding armSubstrateError::Unsupportedexplicitly; defaults only for "nothing to do" hooks called on every path (retire_fence_mismatch), which still need their armSubstrateCapabilitiesfield, set it inSubstrateCapabilities::DIRECTand::KUBERNETES, read it viacapabilities()'k8s'" will go stale with #373managed_by = $nwith.bind(substrate.managed_by()), or.bind(SubstrateKind::Kubernetes)in a Kubernetes-only loop//!module doc ondirect.rs; AGENTS.md keeps the rules and a pointer. No issues filed; the gaps are described as current behaviorretire_executiondocs/first-sandbox.mdmust keep working on Directdirect/tests.rsdirect/", which coversdirect/cluster_admission_tests.rspattern_ratchet.direct_tablesonce #358 landspattern_ratchet.direct_tablesFinal fix pass (this head)
direct.rsmodule doc lists the Kubernetes-only loops." Module-doc heading is now# Kubernetes-only reconcilersand says the dispatcher runs for both substrates and retries transient operation failures14e5b7b, which binds. Rechecked against #373 as merged: its failed-cleanup SQL binds Kubernetes and returns early on other substrates (reworded in the rebase pass below)'k8s'and was silent on existing literals'k8s'/'space'literal:.bind(substrate.managed_by())for the running substrate's rows, or a fixedSubstrateKind::Kubernetes/::Spacewhere code targets one substrate's rows. Existing literals are debt." Cluster controller: bound fromsubstrate.managed_by()for the running substrate's rows, or a fixedSubstrateKindfor one substrate (Kubernetes arm ofrun_reconcilers;tombstone_retention's Direct tombstones)SubstrateCapabilitiesfield…; where the substrate must act, add a required method (above)."docs/first-sandbox.mdis a Kind walkthrough, so "the flow" was unclearexamples/first_sandbox.pymakes (create, command session, list/inspect, delete)"; dropped "which must keep working on Direct"CreateRequesthas no fields for them, so autostart uses node-side defaults and memory policy whilepublic.sandboxesrecords the requested values. Disk size is forwarded."direct.rsmodule docresolves_cold_create_shapedoc says the shape reaches the node unchangeddirect] module doc)"Unsupportedtrait defaults contradict the required-method ruleUnsupportedtrait defaults (list_pool_configs,upsert_pool_config,disable_pool_config)Rebase pass (this head, after #358 and #373 merged)
pattern_ratchet.direct_tablesrule, advisory today, in one clause; its config is not restatedmainSubstrateKind,.bind(SubstrateKind::Kubernetes)/::Space,tombstone_retention's::Space, the Kubernetes arm ofrun_reconcilers, and the conventions doc and skill all match. The sweeper'slist/rotate/repairfailed-active-execution statements now return early on a non-Kubernetes substrate instead of selecting'k8s'rows in SQL, so thedirect.rsmodule doc and theretire_executioncomment now say the failed-cleanup retry does nothing unless the running substrate is Kubernetes. The AGENTS.md rule now also states the early-return convention and namesroutable_execution_or_non_k8sas remaining debt, as the skill doesSubstrateKind, so none was addedmainEvidence
Rows added or changed in the final fix pass cite current
main(1bd4173). Other line references are atc565c8e, the base the facts were first checked against. The cited Direct, reconciler and sweeper code is unchanged on currentmain(02e0656) apart from line shifts and the newdirect/cluster_admission_tests.rsmodule. On this head, the module doc moves laterdirect.rslines down by 46. Unless a full path is given,direct.rs,mod.rsandkubernetes/…are undercrates/libs/api/src/controllers/substrate/;lifecycle.rsandqueries.rsare undercrates/libs/api/src/controllers/sandbox/;nodes.rs,node_store.rsandnode_maintenance.rsare undercrates/libs/api/src/routes/admin/;CC/iscrates/libs/cluster/controller/server/src/.DirectSubstratedials each worker's node-api atnode_direct.node_api_endpoint, with no sandbox podsdirect.rs:1934-1946(worker_client_for_nodeusesnode.node_api_endpoint);mod.rs:3915-3927(connect_workeropens aworker-node-apichannel);direct.rs:1463-1468(node_api_endpointfromnode_direct);docs/architecture.md:31kubernetesfeaturemod.rs:2722-2738(Kubernetes is tried only under#[cfg(feature = "kubernetes")]; otherwiseSelf::Direct);mod.rs:12-27(readiness stub for builds without the feature: "the direct substrate (local dev)"); the Direct node registrynode_maintenance.rs:998,:1137SPACE_K8S_CONTROL_PLANE; unset, false or unrecognized selects Directkubernetes/mod.rs:410-411(env_flag_var("SPACE_K8S_CONTROL_PLANE", false)→Ok(None));crates/libs/common/core/src/env.rs:47-55(only1/true/yes/on/enabledare true; anything unrecognized falls back to the defaultfalse),:59-61;mod.rs:2728(Ok(None) => {}falls through to Direct)mod.rs:2729(Err(err) => return Err(err))cargo build -p pplx-space-api-svcdoes notdocker/Dockerfile.rust-build-common:252-259(build-apistage:-F pplx-space-api-svc/kubernetes),:343-348(copied intospace-rust-build-api, whichdocker/Dockerfile.api:1-2consumes);docker/Dockerfile.rust-build-kind:110;crates/apps/api/svc/Cargo.toml:18(default = []),:31SubstrateAdapterdirect.rs:3436,:3727,:3784,:3855,:3840,:3886lifecycle.rs:2005-2035(Stop/Kill →runtime_client),:2047-2091(Pause, Resume, Checkpoint →reachable_runtime_client_or_unavailable/runtime_client),:2236-2290; Direct's clients only dial the placed worker:direct.rs:4047-4069→node_client:3038-3044direct.rs:3177,:3120,:3126mod.rs:2525-2537(defaultOk(None), "substrates without a warm pool always miss");direct.rsdoes not overrideclaim_admission_pool_slotUnsupported: node-pool placementdirect.rs:3440-3446Unsupported: forkscrates/libs/api/src/fork_operation_executor.rs:1050(the executor's target create, the only caller ofrestore_fork_disk_as_new);queries.rs:1097-1115(restore_from: Some(restore.snapshot));direct.rs:3447,:4210-4216Unsupported: cluster pool configsmod.rs:2588-2613, not overridden indirect.rs; dispatchmod.rs:3801-3828sandbox_ip,access_sandbox_ip,list_inventory_nodes(Postgres is the inventory)direct.rs:4119-4131,:4199-4207retire_fence_mismatchandobserve_pod_identitiesmod.rs:2214-2220(Ok(None)),:2232-2237(empty map); neither is overridden indirect.rsretire_executionignores its fence but still stops the sandbox and clears its routingdirect.rs:3746-3761(fence: _fence),:3769(stop_sandbox),:3779(clear_direct_route_identity)node_maintenance.rs:431-439(require_direct_substrate→NOT_IMPLEMENTED), called bycreate_node:998,update_node:1192,delete_node:1329,pull_node_template:1581,delete_node_template:1712;list_node_templates(:1872) is not gated:drain,:undrain,:cordonand:uncordonreturn 502 on a Direct nodenodes.rs:794-800(validate_drain_transition; a repeated:drain:185-195,:undrainon an undrained node:216-219and:uncordonwhile draining:236-241are conflicts → 409); then:1006-1011(all four have a desired schedulability),:1027-1031("node has no associated substrate node id";substrate_node_idisnode_k8s.k8s_node_name,node_store.rs:292-295, NULL for a Direct node),:984-996(BAD_GATEWAY):drainstill commitsdrain_phase='draining', which nothing advancesnodes.rs:185-190(Move to draining),:902,:924(commit before converge at:936);drainedis written only by the node-drain reconciler (nodes.rs:195-197), which is not started on Direct (CC/reconcilers.rs:390-393)run_reconcilersstarts drain, orphan, failed-row repair and node-sync loops only in itsSubstrate::KubernetesarmCC/reconcilers.rs:189;:390-393(Substrate::Direct(_) => "skipping Kubernetes-only reconcilers"); Kubernetes arm spawns:orphaned_sandboxes:420,started_sandbox_repair:449,failed_restorable_repair:466,kube_node_sync:516,node_drain:645-668,drained_node_terminator:819CC/public_operation_dispatcher.rs:71(enabled: true); spawned before the substrate match:CC/reconcilers.rs:285(match at:390); requeues transient failures within a retry budget:CC/public_operation_dispatcher.rs:705-726,:838. Trigger fanout also runs for both:CC/reconcilers.rs:222retire_executiononly through the sweeper's best-effortcleanup_after_sweep.retire_execution(in the control plane:CC/transient_lifecycle_sweeper/store.rs:1151(cleanup_after_sweep) and:1231(cleanup_failed_execution), plus dispatch atmod.rs:3290;CC/transient_lifecycle_sweeper/mod.rs:87-95("The call is best-effort: if it fails, the row keeps its resting lifecycle")'k8s'-only, so nothing retries a failed Direct retire (also the correctedretire_executioncomment)CC/transient_lifecycle_sweeper/store.rs:1166-1167(s.managed_by = $1 AND s.managed_by = 'k8s'),:1213-1214;CC/transient_lifecycle_sweeper/mod.rs:107-108("retries active Kubernetes executions for authoritativefailedrows"). The narrowing matters: Direct does run trigger fanout (CC/reconcilers.rs:221-224), which records failed deliveries for retry (crates/libs/cluster/reconciler/src/trigger_fanout.rs:416,crates/libs/cluster/trigger-queue/src/lib.rs:315-325)1bd4173)proto/sandbox/manager/v1/manager.proto:359-394(no vcpu/memory;rootfs_size_mbat:374);direct.rs:2990on this head (rootfs_size_mb: request.resources.disk_mib);crates/libs/sbx/mgr/server/src/service/start.rs:1806-1812(node-read template default, else 1024 MiB, then node memory policy),:1914(else 2 vCPU)c565c8e)direct.rs:180-186,:3448-3449,:3519-3544(returns without aStartRequest);direct.rs:2926-2957(CreateRequestwithout vcpu/memory);proto/sandbox/manager/v1/manager.proto:359-394(no such field;StartRequesthas them at:411-412); sbx-mgr autostart falls back to template defaults (crates/libs/sbx/mgr/server/src/service/create.rs:1358,service/start.rs:1807,:1909)public.sandboxesrow records the requested shapequeries.rs:606(non-k8ssubstrates keep the request unchanged);queries.rs:1369-1370(INSERT INTO sandboxes … vcpu, memory_mibbound fromresources)retire_executionnever enqueues the stopped-disk backup the trait documents; the Kubernetes teardown doesmod.rs:2175;direct.rs:3755-3782(noenqueue_artifact_backup/BackupSourceanywhere indirect.rs);kubernetes/adapter/teardown.rs:384-409; sbx-mgr Stop doesn't enqueue one either:crates/libs/sbx/mgr/server/src/service/grpc_core.rs:579-692mod.rs:3229(the enum implements the trait by per-methodmatchforwarding, e.g.:3238-3244); shared code holds the enum:controllers/sandbox/mod.rs:1030(substrate: Substrate). Rust rejects an impl that omits a required method and silently uses the default for an omitted provided one. #368 makescapabilities()required for this reason.retire_fence_mismatchdoescomplete_activation_leasemod.rs:2159-2164(no-op; called on every create,queries.rs:1793, and start,lifecycle.rs:1641),persist_pooled_execution_claimmod.rs:2543-2550(Ok(()); called atqueries.rs:2929),retire_fence_mismatch:2214-2220,observe_pod_identities:2232-2237. The existingUnsupportedoperation defaults (mod.rs:2588-2613,:2627-2635) predate the rule.mod.rs:2169-2172at1bd4173(#368'sSubstrateCapabilitiesdoc: "behavior that needs the substrate to act stays a [SubstrateAdapter] method")Unsupportedtrait defaults are debtmod.rs:2674-2698at1bd4173(list_pool_configs,upsert_pool_config,disable_pool_configdefault toUnsupported)resolves_cold_create_shapedoc correctionmod.rs:2193-2199on this head; see the one-shot gap rowexamples/first_sandbox.pymakesexamples/first_sandbox.py:172(create),:178(process list),:230(command session),:268(inspect),:286(delete)SubstrateCapabilitiesfield, set it inDIRECTandKUBERNETES, read it throughcapabilities()mod.rs(struct, both constants, requiredfn capabilities(&self) -> SubstrateCapabilities) and ofroutes/admin/sandboxes.rs(reads.supports_node_pool_placementthrough it)managed_by(); never choose a runtime endpoint from a row's labelmod.rs:2137-2141("This is descriptive metadata. Runtime routing should use the active execution binding, not this label.")substrate.managed_by()for the running substrate's rows, a fixedSubstrateKindfor one substrate's rows; existing literals are debt (at1bd4173)tombstone_retentionruns for both substrates (CC/reconcilers.rs:253) and targets Direct tombstones with a fixedn.managed_by = 'space'(CC/tombstone_retention.rs:91,:191), which #373 binds asSubstrateKind::Space. Existing api literals #373 doesn't touch:n.managed_by = 'space'indirect.rs(12 sites),'k8s'inkubernetes/store/node_inventory.rs,'space'inroutes/admin/node_maintenance.rs. #373's conventions doc and skill: bind in new code, raw literals are existing debt'k8s'(atc565c8e)CC/reconcilers.rs:381-388) and bindsself.substrate.managed_by()into its queries (CC/transient_lifecycle_sweeper/store.rs:826-829); its doc comment (store.rs:785-793) says this scope stops a Kubernetes controller from claiming Direct rows "and vice versa".NodeManagedByalready binds as a SQL parameter (crates/libs/cluster/reconciler/src/trigger_fanout/tests.rs:74); #373 names itSubstrateKind(crates/libs/common/core/src/types/node.rsin its diff). Kubernetes-arm loops that still use the literal:CC/node_drain/store.rs:536,CC/drained_node_terminator/store.rs:192direct.rsand the Direct node registrydirect.rs:468,:499,:747,:1117,:1185,:1289,:1379,:1423;node_maintenance.rs:1066,:1137,:1269. 17 files undercrates/containINSERT INTO node_direct, mostly test fixtures. All three tables cascade fromnodes/sandboxes(db/postgres/20261002070625_baseline.up.sql:294,:834,:845)Substrate::Kubernetesbuilt with thekubernetes::testshelpers, asadmission_test_fixture.rsdoeskubernetes/mod.rs:1198-1199(#[cfg(test)] pub(crate) mod tests);kubernetes/tests.rs:1850-1869(pub(crate) fn substrate_with_store,substrate_with_store_and_client,substrate_with_store_client_proxy_port_and_pod);crates/libs/api/src/routes/public/sandboxes/admission_test_fixture.rs:156-166; alsocrates/libs/api/src/routes/e2b/tests.rs:338,:382,crates/libs/api/src/public_sandbox_operations/tests.rs:126kubernetes/usescripted_test_doubles.rs, asadapter_characterization_tests/does (it is private tokubernetes)kubernetes/mod.rs:1195-1196(#[cfg(test)] mod scripted_test_doubles;, private, withpub(super)items, e.g.kubernetes/scripted_test_doubles.rs:32,:35);kubernetes/scripted_test_doubles.rs:1-9;kubernetes/adapter_characterization_tests.rs:1-10(added by #273)--features kubernetes; CI's--workspace --all-featuresincludes them, butcargo test -p pplx-space-apiskips themmod.rs:2-3(#[cfg(feature = "kubernetes")] pub mod kubernetes;);crates/libs/api/src/routes/public/sandboxes.rs:1356-1357(#[cfg(all(test, feature = "kubernetes"))]);crates/libs/api/Cargo.toml:14(default = []);ci/scripts/run-rust-checks.sh:553-560,:573-579The remaining rules are guidance, not facts, so they have no row: Kubernetes first,
Unsupportedon Direct outside theexamples/first_sandbox.pycalls, capabilities over matches, reading through existing joins, "other existing matches and joins are debt, not precedent" (which excludesmanaged_byscoping), and "don't change a known gap as a side effect". Each is grounded in the rows above.Findings outside this change (not fixed)
Admin drain actions on a Direct node leave a stuck phase and promise a retry that never comes.
nodes.rs:902-924commitsdrain_phase='draining'before converging.:1027-1031), and the 502 body reports"reconciler_will_retry": true(:927,:993). The node-drain reconciler is never started on Direct (CC/reconcilers.rs:390-393) and only scansmanaged_by = 'k8s'(CC/node_drain/store.rs:536).:undrainclears the phase (nodes.rs:210-215) but still returns 502 with the same retry promise.drainingtodrained, so:terminate, which requiresdrained(nodes.rs:199-208), returns 409.mode: forced, the shared drain gates return 409 for pause, checkpoint and access on that node's sandboxes (crates/libs/common/core/src/types/node.rs:930-934,:1021-1022;lifecycle.rs:1136-1148,:2048-2049).nodes.drain_phase(it filters on node-reportednode_status_snapshots.scheduling_status,direct.rs:1463-1477,:1513,:1621), so new creates can still land on the node.The Direct-only tables already have many readers outside
direct.rs(non-test, atc565c8e):crates/apps/admin/cli/src/cli.rs:181crates/libs/api/src/controllers/sandbox/responses.rs:74,78,440,528controllers/sandbox/spaced.rs:795crates/libs/api/src/public_template_operations.rs:200node_maintenance.rs:658,687,1032,1066,1137,1269,1349(the registry, which also writes)node_store.rs:294crates/libs/api/src/routes/admin/templates.rs:509,525crates/libs/api/src/routes/public/sandboxes.rs:943CC/node_drain/store.rs:2065CC/tombstone_retention.rs:90,190(deletes Directnodesrows, cascading tonode_direct)crates/libs/cluster/trigger-queue/src/lib.rs:240crates/libs/common/core/src/types/admin.rs:510crates/libs/internal/api/server/src/data_plane_sessions.rs:872,877,917,922,1010,1217,1222crates/libs/internal/api/server/src/lib.rs:1215crates/libs/internal/api/server/src/published_state.rs:2244,2306,2349,2423crates/libs/sandbox/portability/src/lib.rs:490(doc comment only)crates/libs/sandbox/store/src/lib.rs:36,296,508scripts/admin/prune_cluster_artifacts.py:2377Shared code also reads the row's
managed_bylabel for behavior, which the trait doc says not to do:controllers/sandbox/spaced.rs:75takes the wake fallback only whensandbox.managed_by == "k8s".A comment in DirectFixed in this PR (see above).retire_executionnames the wrong retry loop.After a Direct one-shot create, the shape in Postgres can differ from the shape the VM booted with. With
wait_until_ready=trueandboot_modecoldorsnapshot, Direct sends a singleCreateRequest, which has no vcpu/memory fields. sbx-mgr's autostartStartRequestuses..Default::default(), so the VM boots with the template's defaults, else 2 vCPU / 1024 MiB (crates/libs/sbx/mgr/server/src/service/start.rs:1807,:1909;crates/libs/sbx/mgr/server/src/service/create.rs:1358). Thepublic.sandboxesrow stores the requestedvcpu/memory_mib(queries.rs:1369-1370).disk_mibis unaffected; it is sent asrootfs_size_mb. The non-one-shot cold path does send vcpu/memory (direct.rs:3667-3668).Validation
At head
74fbd44, rebased onmain(fc47b31):cargo fmt --all -- --check: clean.cargo clippy -p pplx-space-api --all-targets --all-features --locked -- -D warnings: clean.cargo doc -p pplx-space-api --no-deps: exits 0. Its 8 warnings are the private intra-doc links and HTML tag already onmain; none is on a changed line. The [direct] link in theresolves_cold_create_shapedoc still resolves.../../../api/src/controllers/substrate/AGENTS.md#direct-substrate-contract(cluster-controller and internal-api AGENTS.md) resolves to an existing file, and## Direct Substrate Contractproduces that anchor.examples/first_sandbox.pyexists.wc -c):mainAGENTS.mdcrates/libs/api/AGENTS.mdcrates/libs/api/src/controllers/substrate/AGENTS.mdcrates/libs/cluster/controller/server/AGENTS.mdcrates/libs/internal/api/server/AGENTS.mdThe "previous head" column is
10e0d52.🤖 Generated with Claude Code