Repository navigation
cluster-controller: bind the substrate value instead of writing it in SQL - #373
Merged
Merged
Conversation
ac-pplx
force-pushed
the
ac/substrate-kind-sql-cluster-controller
branch
2 times, most recently
from
October 13, 2026 07:37
3e7126a to
5bb4620
Compare
ac-pplx
marked this pull request as ready for review
October 13, 2026 15:42
ac-pplx
force-pushed
the
ac/substrate-kind-sql-cluster-controller
branch
from
October 13, 2026 17:27
5bb4620 to
9d0cdc2
Compare
zibi-pplx
requested changes
Oct 14, 2026
zibi-pplx
left a comment
Collaborator
There was a problem hiding this comment.
Requesting changes for a design reason, not a correctness problem. As written, the conversion is careful and complete:
- all 27 statements and the Rust comparison are converted;
- nothing is left in the crate's production code apart from the namespace defaults;
- the byte-identical check is convincing;
- the branch merges cleanly with
main.
Request: bind the substrate value instead of splicing it into the SQL text
NodeManagedBy/SubstrateKindalready implements sqlxType/Encode/Decode. Somanaged_by = $nwith.bind(SubstrateKind::Kubernetes)gives a single source of truth without the two macros, the const assertion, or the newsql!(concat!)arm.- It also follows
space-data-store-development(SKILL.md:35-42): "Keep every SQL statement syntactically whole … values bound as parameters". - The
concat!split cuts statements mid-predicate, so as written the Rust conventions doc and the store skill disagree.
What to keep:
- the
SubstrateKindalias: it names the bound value, and #368's console-label follow-up wants it; - the docs and skill update, reworded for binding;
- the plugin version bump.
Specifics for the conversion
row_state_guardsinstarted_sandbox_repair/store.rs:43andfailed_restorable_repair/store.rs:45is spliced into a scan and an apply that number their parameters differently ("$1"and"$2"). It needs a second placeholder argument.- The sweeper statements already scope by
s.managed_by = $ownand then add= 'k8s'(transient_lifecycle_sweeper/store.rs:87-96,:1162-1172,:1218). Binding a second value there reads oddly. Returning early in Rust when the substrate isn't Kubernetes, then using a single bound predicate, gives the same result and skips a query that matches nothing on Direct. - Test callers. Some test callers of shared query builders will need the extra
.bind, innode_drain/store/gentle_escalation_tests.rs, the twodrained_node_terminator/store/*_gate_tests.rsfiles, andtransient_lifecycle_sweeper/tests.rs. orphaned_restore_pod_gc/store.rs:23is still a string comparison. Please parse each row and match the variant,!matches!(managed_by.parse::<SubstrateKind>(), Ok(SubstrateKind::Kubernetes)). That keeps the "one bad row can't fail the batch" property this module already has forlifecycle.- Convention wording. Phrase it as applying to new code, with existing sites as debt. A few dozen production lines outside the cluster controller still compare
managed_bywith a raw'k8s'/'space'; for example,api/.../substrate/direct.rshas about a dozen.
Evidence to replace the byte-identical proof
- For each converted statement, a Postgres-backed test, or a pointer to an existing one, showing that a
'space'row is excluded. EXPLAIN (GENERIC_PLAN)forlist_candidates_after(orphaned_sandboxes.rs) andevaluate_gone_node(reconcile_queue_gc/store.rs).- Postgres may switch a prepared statement to a generic plan, which estimates
managed_by = $nat 1/ndistinct instead of the real frequency. - There are no partial indexes on
managed_by, so I don't expect a change, but it's worth confirming for these busy scans.
- Postgres may switch a prepared statement to a generic plan, which estimates
ac-pplx
added a commit
that referenced
this pull request
Oct 14, 2026
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and a module doc on direct.rs, so agents changing the substrate, or shared code that branches on it, know what Direct is for and where its contract ends. The AGENTS.md section holds the rules: - when the api selects Direct (no `kubernetes` feature, or SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure is an error, not a fallback; - implement new lifecycle operations for Kubernetes; Direct returns Unsupported unless the operation is part of the create, command, inspect and delete flow in docs/first-sandbox.md; - make new SubstrateAdapter operations required methods, so the compiler demands each substrate's implementation and the forwarding arm in `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that shared code calls on every path keeps a default, and it still needs its arm; - where shared code must differ, add a SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read it through capabilities() instead of matching the substrate (#368); - scope queries to a substrate's own rows by binding the value (`substrate.managed_by()`, or `SubstrateKind::Kubernetes` in a Kubernetes-only loop) rather than a 'k8s' literal (#373); - keep writes to the Direct-only tables in direct.rs and the Direct node registry; - don't cite a reconciler as the retry path for a failed Direct operation, and don't change a known gap as a side effect of other work; - which test fixtures to use: Direct only for tests under `direct/` or substrate-independent behavior, `kubernetes::tests` helpers for shared tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both compile only with `--features kubernetes`. The direct.rs module doc records what Direct does today, so it is reviewed with the code: which lifecycle operations it implements, which it rejects as Unsupported (node-pool placement, forks, cluster pool configs), its no-op trait methods, its Direct-only admin routes, how admin drain/cordon actions fail on a Direct node, that the cluster controller starts its drain, orphan, failed-row repair and node-sync loops only for Kubernetes so nothing retries a failed Direct retire, and two gaps that succeed silently: one-shot creates drop custom vcpu/memory, and retire_execution does not enqueue the stopped-disk backup the trait documents. Correct the comment in Direct retire_execution that named the failed-cleanup pass as its retry path: that pass selects only 'k8s' rows, so a failed Direct retire is not retried. Make the rules reachable from the code they govern: add the `controllers/substrate/CLAUDE.md` symlink, widen the pointer in crates/libs/api/AGENTS.md, and point the cluster-controller and internal-api AGENTS.md files at the section, with the cluster controller's binding rule for `managed_by` scoping stated there. Documentation and comments only; no code, schema, wire or configuration change.
ac-pplx
force-pushed
the
ac/substrate-kind-sql-cluster-controller
branch
from
October 14, 2026 15:54
9d0cdc2 to
14e5b7b
Compare
ac-pplx
added a commit
that referenced
this pull request
Oct 14, 2026
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and a module doc on direct.rs, so agents changing the substrate, or shared code that branches on it, know what Direct is for and where its contract ends. The AGENTS.md section holds the rules: - when the api selects Direct (no `kubernetes` feature, or SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure is an error, not a fallback; - implement new lifecycle operations for Kubernetes; Direct returns Unsupported unless the operation is one of the calls examples/first_sandbox.py makes (create, command session, list/inspect, delete); - make new SubstrateAdapter operations required methods, so the compiler demands each substrate's implementation and the forwarding arm in `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that shared code calls on every path keeps a default, and it still needs its arm; - where shared code needs a static fact about the substrate, add a SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read it through capabilities() instead of matching the substrate; where the substrate must act, add a required method (#368); - in new SQL, scope to a substrate's rows by binding the value (`substrate.managed_by()` for the running substrate's rows, or a fixed `SubstrateKind::Kubernetes`/`::Space` for one substrate's rows) instead of a 'k8s'/'space' literal; existing literals are debt (#373); - keep writes to the Direct-only tables in direct.rs and the Direct node registry; - don't cite a cluster-controller reconciler as the retry path for a failed Direct execution retire or repair, and don't change a known gap as a side effect of other work; - existing substrate matches, Direct-table joins in shared code and Unsupported trait defaults are debt, not precedent; - which test fixtures to use: Direct only for tests under `direct/` or substrate-independent behavior, `kubernetes::tests` helpers for shared tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both compile only with `--features kubernetes`. The direct.rs module doc records what Direct does today, so it is reviewed with the code: how it reaches workers, which lifecycle operations it implements, which it rejects as Unsupported (node-pool placement, forks, cluster pool configs), its no-op trait methods, its Direct-only admin routes, how admin drain/cordon actions fail on a Direct node, and which cluster-controller loops are Kubernetes-only: the drain, orphan, failed-row repair and node-sync loops never run for Direct, while the public-operation dispatcher runs for both substrates and retries transient operation failures. Nothing retries a failed Direct retire. It lists two gaps that succeed silently: one-shot create does not forward requested vCPU or memory (disk size is forwarded), and retire_execution does not enqueue the stopped-disk backup the trait documents. Correct the comment in Direct retire_execution that named the failed-cleanup pass as its retry path: that pass selects only 'k8s' rows, so a failed Direct retire is not retried. Correct the `resolves_cold_create_shape` field doc, which said that without the capability the requested shape reaches the node unchanged: Direct's one-shot create cannot carry vCPU or memory. Make the rules reachable from the code they govern: add the `controllers/substrate/CLAUDE.md` symlink, widen the pointer in crates/libs/api/AGENTS.md, and point the cluster-controller and internal-api AGENTS.md files at the section, with the cluster controller's binding rule for `managed_by` scoping stated there, including `tombstone_retention`'s fixed scope to Direct tombstones. Documentation and comments only; no code, schema, wire or configuration change. Names from #373 (`SubstrateKind`) exist on main only once #373 merges, so this lands after it.
ac-pplx
force-pushed
the
ac/substrate-kind-sql-cluster-controller
branch
from
October 14, 2026 17:36
14e5b7b to
697926c
Compare
… SQL Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. The cluster controller scoped and gated on that value with raw string literals in 29 production SQL statements and one Rust comparison. Those statements now bind the value as a parameter and the comparison matches the parsed variant, so each statement stays syntactically whole and names the value through one Rust type. The shared `routable_execution_or_non_k8s` fragment, which the transient sweeper embeds, still writes `managed_by <> 'k8s'` and is unchanged here. common-core: - `SubstrateKind`, an alias of `NodeManagedBy`, names the substrate value. `NodeManagedBy` keeps its name. Its existing sqlx `Encode` writes the same text the literals spelled, so `managed_by = $n` with `.bind(SubstrateKind::Kubernetes)` matches the same rows. cluster controller: - Each `managed_by = 'k8s'` (or `'space'` in tombstone retention) becomes `managed_by = $n` bound to `SubstrateKind::Kubernetes` (or `::Space`), numbered after the statement's existing parameters. The started- and failed-restorable repair `row_state_guards` fragments take the placeholder as an argument, since their scan and apply number parameters differently. - The transient sweeper's failed-active-execution cleanup scoped by `s.managed_by = $own AND s.managed_by = 'k8s'`. It now returns early in Rust when the running substrate is not Kubernetes and binds a single predicate. Results are the same; the only observable difference is that a Direct-substrate controller no longer issues those three queries, which matched nothing there. Its list and rotate statements move to `const`s beside the existing repair `const`, so tests can run them. - `list_terminating_nodes` moves its statement to `terminating_nodes_query()`, beside `draining_nodes_query()`, for the same reason. - The orphaned-restore-pod classifier parses `managed_by` per row and matches `SubstrateKind::Kubernetes`. An unparseable value is treated as not Kubernetes, as before, and one bad row still cannot fail the batch. - Test callers of shared query builders bind the extra parameter. Other test-module SQL literals are unchanged. Tests: one Postgres-backed test per converted statement group shows a 'k8s' row included and an otherwise-identical 'space' row excluded, or for tombstone retention the reverse. Dropping any one converted predicate fails a test, except the terminating-node tombstone's snapshot SELECT: it only returns early ahead of the DELETE, which repeats the same guard, so dropping it alone changes no result. Two tests change a terminating node's `managed_by` under a row lock between that SELECT and the DELETE, so the DELETE re-checks the committed row: a node that became Direct is kept with no tombstone, and one still on Kubernetes is deleted. docs/rust/space-conventions.md and the space-rust-development skill say new code writes whole statements and binds substrate values; the `row_state_guards` helpers that take a placeholder are existing sites this change kept, not a pattern. A batch that must survive one bad row reads `managed_by` as text and parses per row. Test fixtures may write literal rows. Remaining raw-literal sites, including that fragment, are debt, not precedent. The agent plugin version is bumped to 0.2.32. No schema, wire format, metric or configuration changes. Statement text changes, so the `/*module_path:line*/` locators and the sqlx statement-cache keys and DBM query signatures of converted statements change too.
ac-pplx
force-pushed
the
ac/substrate-kind-sql-cluster-controller
branch
from
October 14, 2026 17:42
697926c to
bbe7ae1
Compare
Contributor
Author
|
Thanks. Reworked to bind the value, in bbe7ae1:
|
zibi-pplx
approved these changes
Oct 14, 2026
ac-pplx
added a commit
that referenced
this pull request
Oct 14, 2026
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and a module doc on direct.rs, so agents changing the substrate, or shared code that branches on it, know what Direct is for and where its contract ends. The AGENTS.md section holds the rules: - when the api selects Direct (no `kubernetes` feature, or SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure is an error, not a fallback; - implement new lifecycle operations for Kubernetes; Direct returns Unsupported unless the operation is one of the calls examples/first_sandbox.py makes (create, command session, list/inspect, delete); - make new SubstrateAdapter operations required methods, so the compiler demands each substrate's implementation and the forwarding arm in `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that shared code calls on every path keeps a default, and it still needs its arm; - where shared code needs a static fact about the substrate, add a SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read it through capabilities() instead of matching the substrate; where the substrate must act, add a required method (#368); - in new SQL, scope to a substrate's rows by binding the value (`substrate.managed_by()` for the running substrate's rows, or a fixed `SubstrateKind::Kubernetes`/`::Space` for one substrate's rows) instead of a 'k8s'/'space' literal; existing literals are debt (#373); - keep writes to the Direct-only tables in direct.rs and the Direct node registry; - don't cite a cluster-controller reconciler as the retry path for a failed Direct execution retire or repair, and don't change a known gap as a side effect of other work; - existing substrate matches, Direct-table joins in shared code and Unsupported trait defaults are debt, not precedent; - which test fixtures to use: Direct only for tests under `direct/` or substrate-independent behavior, `kubernetes::tests` helpers for shared tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both compile only with `--features kubernetes`. The direct.rs module doc records what Direct does today, so it is reviewed with the code: how it reaches workers, which lifecycle operations it implements, which it rejects as Unsupported (node-pool placement, forks, cluster pool configs), its no-op trait methods, its Direct-only admin routes, how admin drain/cordon actions fail on a Direct node, and which cluster-controller loops are Kubernetes-only: the drain, orphan, failed-row repair and node-sync loops never run for Direct, while the public-operation dispatcher runs for both substrates and retries transient operation failures. Nothing retries a failed Direct retire. It lists two gaps that succeed silently: one-shot create does not forward requested vCPU or memory (disk size is forwarded), and retire_execution does not enqueue the stopped-disk backup the trait documents. Correct the comment in Direct retire_execution that named the failed-cleanup pass as its retry path: that pass selects only 'k8s' rows, so a failed Direct retire is not retried. Correct the `resolves_cold_create_shape` field doc, which said that without the capability the requested shape reaches the node unchanged: Direct's one-shot create cannot carry vCPU or memory. Make the rules reachable from the code they govern: add the `controllers/substrate/CLAUDE.md` symlink, widen the pointer in crates/libs/api/AGENTS.md, and point the cluster-controller and internal-api AGENTS.md files at the section, with the cluster controller's binding rule for `managed_by` scoping stated there, including `tombstone_retention`'s fixed scope to Direct tombstones. The Direct-table rule names archcheck's `pattern_ratchet.direct_tables` rule (#358) as its enforcement, advisory today. The SQL-scoping rule also covers returning early in Rust when a fixed-scope statement may run under another substrate (#373), and names the shared `routable_execution_or_non_k8s` fragment as remaining debt. The Direct retire wording now says the sweeper's failed-cleanup retry does nothing unless the running substrate is Kubernetes, matching the code after #373. Documentation and comments only; no code, schema, wire or configuration change.
This was referenced Oct 14, 2026
ac-pplx
added a commit
that referenced
this pull request
Oct 16, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons or match arms on "k8s"/"space". It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches exactly the four converted lines; on this commit it matches nothing, so it has no baseline table and every behaviour is required. - Tests: Postgres-backed twins for each converted statement. A new Kubernetes builder-lookup test, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; Rust parses it and matches the variant (or calls the `SubstrateKind` method that names the question) and never compares the label string; and the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus three statements that landed on main since: internal-api's publisher-node lookup and Direct's create-target claim and release. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - internal-api: `publisher_node_for_identity` binds `n.managed_by = $3` (`Kubernetes`). - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons or match arms on "k8s"/"space". It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches exactly the seven converted lines; on this commit it matches nothing, so it has no baseline table and every behaviour is required. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, a relabelled-builder case in the tenant publication table, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; Rust parses it and matches the variant (or calls the `SubstrateKind` method that names the question) and never compares the label string; and the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons or match arms on "k8s"/"space". It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches exactly the six converted lines; on this commit it matches nothing, so it has no baseline table and every behaviour is required. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; Rust parses it and matches the variant (or calls the `SubstrateKind` method that names the question) and never compares the label string; and the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons or match arms on "k8s"/"space". It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches exactly the six converted lines; on this commit it matches nothing, so it has no baseline table and every behaviour is required. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; Rust parses it and matches the variant (or calls the `SubstrateKind` method that names the question) and never compares the label string; and the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons or match arms on "k8s"/"space". It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches exactly the six converted lines; on this commit it matches nothing, so it has no baseline table and every behaviour is required. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; Rust parses it and matches the variant (or calls the `SubstrateKind` method that names the question) and never compares the label string; and the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the common literal forms in production code: SQL comparisons of `managed_by` with 'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY, same-line CASE ... WHEN; any case, alias, prefix, cast or wrapping call), and Rust comparisons, label-string matches and match arms on "k8s"/"space" beside a substrate name. Its comment lists what it misses and its false positives. It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches the seven lines this commit converts; on this commit it matches nothing, so it has no baseline table. no_crossing, listed_no_growth and stale_entries are required; no_new_entries and no_raised_caps are advisory, so a deliberate exception is a reviewed baseline entry. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; in Rust a decision about a row's substrate calls a `SubstrateKind` method named for the question, a fact about the running substrate reads `capabilities()`, and code never compares the label string; and the archcheck rule catches the common literal forms. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the common literal forms in production code: SQL comparisons of `managed_by` with 'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY, same-line CASE ... WHEN; any case, alias, prefix, quoting, cast or wrapping call), and Rust comparisons with the label (raw strings included), a `NODE_MANAGED_BY_*` const or a variant's `.as_str()`/`.to_string()`, matches on the label string, and match arms on "k8s"/"space" beside a substrate name. Its comment lists what it misses and its false positives. It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches the seven lines this commit converts; on this commit it matches nothing, so it has no baseline table. no_crossing, listed_no_growth and stale_entries are required; no_new_entries and no_raised_caps are advisory, so a deliberate exception is a reviewed baseline entry. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; in Rust a decision about a row's substrate calls a `SubstrateKind` method named for the question, a fact about the running substrate reads `capabilities()`, and code never compares the label string; and the archcheck rule catches the common literal forms. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the common literal forms in production code: SQL comparisons of `managed_by` with 'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY, same-line CASE ... WHEN; any case, alias, prefix, quoting, cast or wrapping call), and Rust comparisons with the label (raw strings included), a `NODE_MANAGED_BY_*` const or a variant's `.as_str()`/`.to_string()`, matches on the label string, and match arms on "k8s"/"space" beside a substrate name. Its comment lists what it misses and its false positives. It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches the seven lines this commit converts; on this commit it matches nothing, so it has no baseline table. no_crossing, listed_no_growth and stale_entries are required; no_new_entries and no_raised_caps are advisory, so a deliberate exception is a reviewed baseline entry. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; in Rust a decision about a row's substrate calls a `SubstrateKind` method named for the question, a fact about the running substrate reads `capabilities()`, and code never compares the label string; and the archcheck rule catches the common literal forms. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
meefs
pushed a commit
to meefs/SPACE
that referenced
this pull request
Oct 18, 2026
…erplexityai#611) Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the substrate that owns a row, 'k8s' or 'space'. After perplexityai#373, perplexityai#383 and perplexityai#506 the remaining production literals were the shared routability fragment, the Kubernetes adapter's builder lookup and the Direct route-identity match, plus Direct's create-target claim and release, which landed on main since. They now bind `SubstrateKind` or match its parsed variant, and a new archcheck instance keeps new literals out of production code. No schema, wire format, metric or configuration changes, and no behavior change. - common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)` writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers after its own parameters: the transient sweeper's aged-transient list and restoring converge (`$5`), internal-api's restored-state convergence (`$4`) and the api's wake-completion CAS (`$2`). - api Kubernetes adapter: `builder_client_for_node` binds `n.managed_by = $2`. - api Direct: `claim_create_target` and `release_create_target` bind `managed_by = $3` (`Space`). - api Direct: the route identity's `substrate_kind` text is parsed as `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly 'space' and 'k8s' as before and rejects any other text with the same error. - ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the common literal forms in production code: SQL comparisons of `managed_by` with 'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY, same-line CASE ... WHEN; any case, alias, prefix, quoting, cast or wrapping call), and Rust comparisons with the label (raw strings included), a `NODE_MANAGED_BY_*` const or a variant's `.as_str()`/`.to_string()`, matches on the label string, and match arms on "k8s"/"space" beside a substrate name. Its comment lists what it misses and its false positives. It allows test-support modules and fixtures that archcheck's filename-based test rule counts as production. On the parent commit it matches the seven lines this commit converts; on this commit it matches nothing, so it has no baseline table. no_crossing, listed_no_growth and stale_entries are required; no_new_entries and no_raised_caps are advisory, so a deliberate exception is a reviewed baseline entry. - Tests: Postgres-backed twins for each converted statement. New Kubernetes builder-lookup and Direct create-target tests, a unit table for the route-identity labels, and an otherwise identical 'space' row added to the sweeper's converge, internal-api's restored convergence and the wake-completion redrive tests. Test callers of the sweeper's shared builders bind the extra parameter. - Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the space-rust-development skill say every crate's production SQL binds the value; in Rust a decision about a row's substrate calls a `SubstrateKind` method named for the question, a fact about the running substrate reads `capabilities()`, and code never compares the label string; and the archcheck rule catches the common literal forms. Plugin version 0.2.33 -> 0.2.34. Converted statements' text changes, so their sqlx statement-cache keys and query signatures change; statements below an added `.bind` move to new `sql!` locators.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What merging does
Control-plane Postgres
sandboxes.managed_byandnodes.managed_byhold the substrate that owns a row,'k8s'or'space'. The cluster controller scoped and gated on that value with raw string literals in 29 production SQL statements and one Rust comparison. After this PR, those statements (crates/libs/cluster/controller/server/src/) bind the value as a parameter, and the comparison matches the parsed variant. Each statement stays syntactically whole, and the value is named through one Rust type. One literal remains in the controller's statements: the transient sweeper'slist_aged_transient_queryandconverge_restoring_queryembed the sharedroutable_execution_or_non_k8sfragment (crates/libs/common/svc/src/sql/sandbox_routability.rs), which writes{alias}.managed_by <> 'k8s'. Internal-apipublished_state.rsand apiwake_completion.rsshare it. This PR leaves it unchanged, and the docs list it as debt. This is phase 2, PR 1 of containing the Direct/Kubernetes substrate split.There are no schema, wire format, metric or configuration changes. One behavior difference: a cluster controller running on the Direct substrate no longer issues the transient sweeper's three failed-active-execution cleanup queries, which matched nothing there (details below).
This revision follows the review: the substrate literal macros, their compile-time assertion and the
sql!(concat!(…))arm are gone, andsql!is as onmain.pplx-space-common-corepub type SubstrateKind = NodeManagedBy;next toNodeManagedBy(types/node.rs).NodeManagedBykeeps its name; new and converted code uses the alias.NodeManagedByalready implements sqlxType/Encode/Decodeas the column's text, so.bind(SubstrateKind::Kubernetes)sends exactlyk8s.Cluster controller
managed_by = 'k8s'becomesmanaged_by = $nwith.bind(SubstrateKind::Kubernetes). Tombstone retention's two'space'sites bindSubstrateKind::Space. Each new placeholder is numbered after the statement's existing ones.row_state_guardsinstarted_sandbox_repair/store.rsandfailed_restorable_repair/store.rsis shared by a scan and an apply that number their parameters differently. It now takes the managed-by placeholder as an argument:row_state_guards("$5", "$1")for the started scan,("$4", "$2")for the started apply,("$4", "$1")for the failed scan, and("$3", "$2")for the failed apply.list_failed_active_executions,note_failed_cleanup_attempt,repair_failed_active_execution). These scoped bys.managed_by = $own AND s.managed_by = 'k8s'. A newfailed_active_execution_scope(managed_by)returnsSome(SubstrateKind::Kubernetes)only on the Kubernetes substrate, and each method returns early (Vec::new(),(),false) otherwise. The statements keep a single bounds.managed_by = $n.LIST_FAILED_ACTIVE_EXECUTIONS_SQL/NOTE_FAILED_CLEANUP_ATTEMPT_SQLconsts beside the existingREPAIR_FAILED_ACTIVE_EXECUTION_SQL, so tests run the production text.node_drain:list_terminating_nodes's statement moves toterminating_nodes_query(), besidedraining_nodes_query(), for the same reason.SubstrateSinkneeds a kube client, and the test drives the free function.orphaned_restore_pod_gc::store::row_lifecyclechanges frommanaged_by != "k8s"to!matches!(managed_by.parse::<SubstrateKind>(), Ok(SubstrateKind::Kubernetes)). It still parses per row, so one bad row can't fail the batch. An unparseable value is treated as not Kubernetes (RowLifecycle::Unrecognized, skipped), exactly as the string comparison treated it.drained_node_terminator/store/{inflight_upload_terminate_gate,resident_census_gate}_tests.rs(delete_drained_node_if_orphan_query,mark_unrecoverable_sandboxes_lost_query,delete_refusal_diagnostics_query) andnode_drain/store/gentle_escalation_tests.rs(draining_nodes_query).transient_lifecycle_sweeper/tests.rsneeded no extra bind: with the early return,REPAIR_FAILED_ACTIVE_EXECUTION_SQLkeeps its two parameters. Other test-module SQL literals are unchanged.Docs agents read
docs/rust/space-conventions.md(new "Substrate values are bound parameters" subsection undersql!) and thespace-rust-developmentskill: new code writes whole statements with bound substrate values, asspace-data-store-developmentrequires. The tworow_state_guardshelpers that take the placeholder as an argument are existing fragment-based sites this conversion kept, not a pattern. Amanaged_bycolumn is read back asSubstrateKind. Where one bad row must not fail the batch, it is read as text and parsed per row, asorphaned_restore_pod_gcdoes, because a typed decode fails the wholefetch_allon an unrecognized value. Test fixtures may write literal rows. Remaining raw-literal sites are debt: theroutable_execution_or_non_k8sfragment above, plus about 40 production lines in 17 files, of whichcrates/libs/api/src/controllers/substrate/direct.rshas 12.plugin.yamland bothplugin.jsonfiles (.agents/plugins/AGENTS.md).claude plugin validatepasses; its one warning, a missingauthor, is also there onmain.Review items → changes
SubstrateKind. The literal macros,constassertion,sql!(concat!)arm and its doc/test are removed, sosql!is as onmain.SubstrateKindaliasrow_state_guardsis spliced into a scan and an apply with different numbering$ownand then add= 'k8s'.bindgentle_escalation_tests.rsand bothdrained_node_terminator/store/*_gate_tests.rs.transient_lifecycle_sweeper/tests.rsdidn't need one (see above).orphaned_restore_pod_gc/store.rs:23is still a string comparison'space'row excludedEXPLAIN (GENERIC_PLAN)forlist_candidates_afterandevaluate_gone_nodeRound 1 review → changes
node_drain/store/tombstone_tests.rschange the node'smanaged_byunder a row lock between the SELECT and the DELETE (see #15). Dropping the DELETE's predicate alone now failsdelete_skips_node_that_turned_direct_after_snapshot. No production code changed. The SELECT's predicate alone has no observable result (see Evidence); the existing test's comment says so.space-conventions.mdsays new code writes whole statements with bound values; therow_state_guardsplaceholder argument is only for the existing fragment-based sites this conversion kept.routable_execution_or_non_k8sas remaining debt, shared with internal-apipublished_state.rsand apiwake_completion.rs. The fragment is unchanged.SET managed_by = 'space'; typed decode fails a batchorphaned_restore_pod_gcdoes.ORDER BY id DESC LIMIT 1drained_node_terminator/store.rs(three) andorphaned_restore_pod_gc/store.rs(one).Evidence
A Postgres-backed test per converted statement
Each test seeds a row the statement would otherwise act on and shows the
'k8s'row included and an otherwise-identical'space'row excluded (for tombstone retention, the reverse). Most use the module's existing "eligible" fixture and change onlymanaged_by. All run on the shared nextest Postgres withSPACE_CI_REQUIRE_TEST_DEPS=1.Mutation check. For each statement, I replaced the converted predicate with
(managed_by = $n OR TRUE)(forrow_state_guards, the call site's placeholder becames.managed_by AND $n::text IS NOT NULL), ran the whole crate's suite, and reverted. "Before" is the crate's suite before this revision's new tests; "after" is this head. Every mutation fails at least one test, with one exception: #14, the snapshot SELECT indelete_terminating_node_and_record_tombstone. It repeats the DELETE's guard (#15). For a Direct row it returnsfalsebefore the DELETE runs, and without it the DELETE matches no row and returns the samefalsewith nothing written. The guard decides only whether the DELETE statement is issued, so no result can show it alone. #15 is now tested on its own, and dropping #14 and #15 together fails both tombstone substrate tests.crates/libs/cluster/controller/server/src/)drained_node_terminator/store.rsdelete_drained_node_if_orphan_querydrained_node_terminator::store::substrate_scope_tests::list_and_reap_skip_direct_nodesdelete_refusal_diagnostics_query(phase_drifted)phase_drifted)mark_unrecoverable_sandboxes_lost_querynodeEXISTS…::mark_lost_update_skips_direct_nodeslist_drained_nodes…::list_and_reap_skip_direct_nodesSELECT 1 FROM nodes … FOR SHAREre-check…::mark_lost_recheck_refuses_direct_nodes_before_claiming. A held row lock on the candidate shows the refusal happens before the claim; on'k8s'the give-up blocks on that lock.failed_restorable_repair/store.rsscan (row_state_guards)failed_restorable_repair::tests::f7_direct_rows_are_neither_listed_nor_repairedrow_state_guards)started_sandbox_repair/store.rsscan (row_state_guards)started_sandbox_repair::tests::postgres_backed::r13_direct_rows_are_not_bounded_listed_or_repairedrow_state_guards)current_max_sandbox_idnode_drain/store.rsdraining_nodes_querynode_drain::store::gentle_escalation_tests::draining_listing_skips_direct_nodesescalate_stale_gentle_drains…::gentle_escalation_tests::skips_rows_every_guard_excludesterminating_nodes_query(list_terminating_nodes)node_drain::store::tombstone_tests::terminating_direct_node_is_neither_listed_nor_deleteddelete_terminating_node_and_record_tombstonesnapshot SELECTdelete_terminating_node_and_record_tombstoneDELETE…::tombstone_tests::delete_skips_node_that_turned_direct_after_snapshot: a concurrent writer holds the row lock with an uncommittedmanaged_by = 'space', so the snapshot SELECT passes on the committed'k8s'row and the DELETE parks on the lock, then re-checks the committed'space'row and skips it. Its twindelete_removes_node_still_kubernetes_after_racing_writewrites'k8s', and the node is deleted and tombstoned. Both wait untilpg_blocking_pidsshows a backend blocked by the writer'spg_backend_pid(), polling every 25 ms. That wait and the join of the delete task are each bounded at 5 s, with a clear failure message.orphaned_nodes/store.rscandidate scanorphaned_nodes::tests::direct_nodes_are_neither_candidates_nor_deleteddelete_node_if_orphanorphaned_sandboxes.rscurrent_max_sandbox_idorphaned_sandboxes::tests::direct_rows_do_not_bound_the_sweep_or_become_candidateslist_candidates_afterorphaned_sandboxes/quarantine_recovery/store.rscandidatesorphaned_sandboxes::quarantine_recovery::tests::direct_rows_are_not_recovery_candidatesreconcile_queue_gc/store.rslist_gone_candidatesreconcile_queue_gc::tests::postgres::direct_nodes_are_neither_listed_nor_prunedevaluate_gone_nodesuspended_execution_reaper/store.rslist_candidatessuspended_execution_reaper::tests::postgres_backed::r11_direct_rows_do_not_bound_the_sweep_or_get_listedcurrent_max_sandbox_idtombstone_retention.rsdirect-node tombstone delete (SubstrateKind::Space)tombstone_retention::tests::store_prunes_and_counts_only_direct_node_tombstonesSubstrateKind::Space)transient_lifecycle_sweeper/store.rsREPAIR_FAILED_ACTIVE_EXECUTION_SQLtransient_lifecycle_sweeper::tests::postgres_backed::r21_failed_cleanup_statements_skip_direct_rows. The existing R16's direct row was excluded by placement, not by substrate.LIST_FAILED_ACTIVE_EXECUTIONS_SQLNOTE_FAILED_CLEANUP_ATTEMPT_SQLfailed_active_execution_scopeearly return (made to accept every substrate)…::r22_failed_cleanup_runs_only_on_the_kubernetes_substrate(unit)orphaned_restore_pod_gc/store.rsrow_lifecycle(if false && …)orphaned_restore_pod_gc::tests::postgres_backed::classify_sandboxes_reads_lifecycle_and_generationOn
main's tests, 2 of the 31 single mutations failed a test. At the previous head (14e5b7b), 29 of 31 did. Now 30 of 31 do, and the remaining one (#14) fails together with #15.Generic plans:
list_candidates_afterandevaluate_gone_nodeI seeded a scratch database migrated from the test template on the repository's test image (PostgreSQL 16.15) with synthetic, deliberately skewed data. There were 400 nodes (10%
'space', a quarter notready, somedraining), 200,000 sandboxes (5%'space', mixed lifecycles), an execution per'k8s'sandbox, and 2,000 stale queue refs, followed byANALYZE. Then I ranEXPLAIN (GENERIC_PLAN)on each statement's text frommain(literal) and from this head (bound).list_candidates_after. The plan shape is identical, and the only differences are the filter text and the row estimates. With the literal, Postgres usesmanaged_by's real frequency (95%'k8s'). With the bound parameter, the generic plan uses 1/ndistinct (50%), so estimated rows roughly halve, as the review predicted. The same index scan onsandboxes_pkey, merge join and anti join are chosen either way.evaluate_gone_node. The plans are identical, costs and row estimates included. The only line that differs is thecandidate_nodefilter,(managed_by = 'k8s'::text)vs(managed_by = $7), under the sameIndex Scan using nodes_pkey on nodes n/Index Cond: (id = $1). A primary-key lookup estimates one row either way.Not covered: the three
current_max_sandbox_idstatements (ORDER BY id DESC LIMIT 1). A generic plan sizes the bound value at 1/ndistinct. If the bound value were a minority of rows, the generic plan may prefer walkingsandboxes_pkeybackwards until a match, where the literal's real, low frequency would have avoided that walk. That matters only if a Kubernetes controller's database were mostly'space'sandboxes.There are no partial indexes on
managed_by(idx_nodes_managed_by,idx_nodes_role (managed_by, role)andidx_sandboxes_managed_byare full), so a generic plan has no index it could lose.Characterization tests pass unchanged
The 21 lifecycle-writer characterization tests from #323 pass in each run below. Their files, including
lifecycle_writer_test_support.rs, are unchanged.Decisions made unattended
SubstrateKind::Kubernetesat every scoped site, notsubstrate.managed_by()managed_by()now bind theSome(Kubernetes)thatfailed_active_execution_scopereturns.SubstrateAdapter::managed_by()returns&'static strand lives in the api crate, outside this PR's scope. Parsing it toSubstrateKindonce at the gate keeps the variant match in one place.terminating_nodes_query(), two sweeperconsts)draining_nodes_query,REPAIR_FAILED_ACTIVE_EXECUTION_SQL). The SQL text is unchanged apart from the bind.managed_by. The test-matrix doc entries are updated where a module keeps one.'k8s'/'space'like the rest of their harness.Locator note
The statement text changes (a placeholder replaces the literal), and some statements move lines, so the converted statements'
/*module_path:line*/locators, sqlx statement-cache keys and DBM query signatures change. Module paths don't change.Validation
Base
02e0656, head = this PR's single commit (bbe7ae1). It still merges cleanly onto currentorigin/main, so it was not rebased.cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features --locked -- -D warningsspace-public-reviewon the commit and this textcargo nextest run --locked -p pplx-space-common-core -p pplx-space-cluster-controller-server --retries 0,SPACE_CI_REQUIRE_TEST_DEPS=1, three runs697926c: 1378/1378 passed, 0 skipped, in each of runs 1, 2 and 3 (includes the 21 characterization tests). Atbbe7ae1, which changes only the two racing tests: the cluster-controller package once, 977/977 passed, 0 skipped; the two racing tests ten times in a row, 2/2 each time;cargo fmt --all -- --checkandcargo clippy -p pplx-space-cluster-controller-server --all-targets --all-features -- -D warningspass.cargo test --locked --doc -p pplx-space-common-core14e5b7b; common-core is unchanged sincescripts/check-rust-error-policy.sh;scripts/source_guards/*.py(at14e5b7b)thin_service_mains.py, which flags onlycrates/apps/*/main.rsfiles this PR doesn't touch: PRE-EXISTINGscripts/check_no_inline_rust_test_modules.py(at14e5b7b)crates/libs/observability/src/{otlp_endpoint,selection}.rs, untouched here: PRE-EXISTINGcargo arch(at14e5b7b; no production line counts change here)main. Growth:node_drain/store.rsgoes from 1897 to 1903 code lines andorphaned_sandboxes.rsfrom 1587 to 1590. Both are advisorylisted_no_growthcaps (1893, 1581) that the files already exceeded onmain.697926cfor #14 alone (977/977 pass), #15 alone (failsdelete_skips_node_that_turned_direct_after_snapshot), and both (also failsterminating_direct_node_is_neither_listed_nor_deleted). #15 alone rerun atbbe7ae1: fails the same test, 976/977.697926corbbe7ae1: this round changes docs, comments and one test module. The rows below are from14e5b7b.space-localcluster was created by this run's bring-up. Its services logversion: 14e5b7b419da(this head). The deployed/space/bin/space-cluster-controllermatches the artifact image's binary by sha256 (3a5672dd…), before and after the suites. That binary contains the boundWHERE s.managed_by = $9predicate and noAND s.managed_by = 'k8s'sweeper text.smoke-kind-e2e.sh(start)smoke-kind-typed.sh all(unsharded, CI's env: admin token file and cold-receipts dir)smoke-kind-leader-election-brownout.sh(SPACE_SMOKE_LEADER_ELECTION=1)smoke-kind-e2e.sh(end)Typed-suite non-passes, each rerun once:
local.telemetry_reaches_local_backends: ENVIRONMENTAL, reproduced on rerun. Tempo and Prometheus checks passed. Loki held zero lines because its WAL was throttling writes (disk usage exceeded threshold … usage_percent=90–91%) and rejected pushes. The Docker host device was at 92% during the suite. This PR touches no telemetry.local.rate_limiters_enforce_and_fail_open: FLAKY under the same disk pressure. The first run's preflight create timed out after 90 s without a response. On the rerun every assertion passed (cluster admission, per-client create and connect concurrency, fail-open); only its cleanup timed out killing the probe sandboxes, so the verdict wasINFRA_ERROR. This PR doesn't touch the API's limiters. I didn't compare againstmainbecause both failures have a cause that doesn't depend on the code, and the host had no disk left for a second stack.🤖 Generated with Claude Code