Skip to content

cluster-controller: bind the substrate value instead of writing it in SQL - #373

Merged
ac-pplx merged 1 commit into
mainfrom
ac/substrate-kind-sql-cluster-controller
Oct 14, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/substrate-kind-sql-cluster-controller

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What merging does

Control-plane Postgres sandboxes.managed_by and nodes.managed_by hold the substrate that owns a row, 'k8s' or 'space'. The cluster controller scoped and gated on that value with raw string literals in 29 production SQL statements and one Rust comparison. After this PR, those statements (crates/libs/cluster/controller/server/src/) bind the value as a parameter, and the comparison matches the parsed variant. Each statement stays syntactically whole, and the value is named through one Rust type. One literal remains in the controller's statements: the transient sweeper's list_aged_transient_query and converge_restoring_query embed the shared routable_execution_or_non_k8s fragment (crates/libs/common/svc/src/sql/sandbox_routability.rs), which writes {alias}.managed_by <> 'k8s'. Internal-api published_state.rs and api wake_completion.rs share it. This PR leaves it unchanged, and the docs list it as debt. This is phase 2, PR 1 of containing the Direct/Kubernetes substrate split.

There are no schema, wire format, metric or configuration changes. One behavior difference: a cluster controller running on the Direct substrate no longer issues the transient sweeper's three failed-active-execution cleanup queries, which matched nothing there (details below).

This revision follows the review: the substrate literal macros, their compile-time assertion and the sql!(concat!(…)) arm are gone, and sql! is as on main.

pplx-space-common-core

  • pub type SubstrateKind = NodeManagedBy; next to NodeManagedBy (types/node.rs). NodeManagedBy keeps its name; new and converted code uses the alias. NodeManagedBy already implements sqlx Type/Encode/Decode as the column's text, so .bind(SubstrateKind::Kubernetes) sends exactly k8s.

Cluster controller

  • managed_by = 'k8s' becomes managed_by = $n with .bind(SubstrateKind::Kubernetes). Tombstone retention's two 'space' sites bind SubstrateKind::Space. Each new placeholder is numbered after the statement's existing ones.
  • row_state_guards in started_sandbox_repair/store.rs and failed_restorable_repair/store.rs is shared by a scan and an apply that number their parameters differently. It now takes the managed-by placeholder as an argument: row_state_guards("$5", "$1") for the started scan, ("$4", "$2") for the started apply, ("$4", "$1") for the failed scan, and ("$3", "$2") for the failed apply.
  • Transient sweeper, failed-active-execution cleanup (list_failed_active_executions, note_failed_cleanup_attempt, repair_failed_active_execution). These scoped by s.managed_by = $own AND s.managed_by = 'k8s'. A new failed_active_execution_scope(managed_by) returns Some(SubstrateKind::Kubernetes) only on the Kubernetes substrate, and each method returns early (Vec::new(), (), false) otherwise. The statements keep a single bound s.managed_by = $n.
    • On Kubernetes the rows are the same.
    • On Direct the old statements matched nothing, so the results are the same. The only observable difference is one fewer query per call on Direct.
    • An unparseable substrate label is treated as not Kubernetes.
    • The list and rotate statements move to LIST_FAILED_ACTIVE_EXECUTIONS_SQL / NOTE_FAILED_CLEANUP_ATTEMPT_SQL consts beside the existing REPAIR_FAILED_ACTIVE_EXECUTION_SQL, so tests run the production text.
  • node_drain: list_terminating_nodes's statement moves to terminating_nodes_query(), beside draining_nodes_query(), for the same reason. SubstrateSink needs a kube client, and the test drives the free function.
  • orphaned_restore_pod_gc::store::row_lifecycle changes from managed_by != "k8s" to !matches!(managed_by.parse::<SubstrateKind>(), Ok(SubstrateKind::Kubernetes)). It still parses per row, so one bad row can't fail the batch. An unparseable value is treated as not Kubernetes (RowLifecycle::Unrecognized, skipped), exactly as the string comparison treated it.
  • Test callers of shared query builders bind the extra parameter. That's fixture plumbing, not an assertion change: drained_node_terminator/store/{inflight_upload_terminate_gate,resident_census_gate}_tests.rs (delete_drained_node_if_orphan_query, mark_unrecoverable_sandboxes_lost_query, delete_refusal_diagnostics_query) and node_drain/store/gentle_escalation_tests.rs (draining_nodes_query). transient_lifecycle_sweeper/tests.rs needed no extra bind: with the early return, REPAIR_FAILED_ACTIVE_EXECUTION_SQL keeps its two parameters. Other test-module SQL literals are unchanged.

Docs agents read

  • docs/rust/space-conventions.md (new "Substrate values are bound parameters" subsection under sql!) and the space-rust-development skill: new code writes whole statements with bound substrate values, as space-data-store-development requires. The two row_state_guards helpers that take the placeholder as an argument are existing fragment-based sites this conversion kept, not a pattern. A managed_by column is read back as SubstrateKind. Where one bad row must not fail the batch, it is read as text and parsed per row, as orphaned_restore_pod_gc does, because a typed decode fails the whole fetch_all on an unrecognized value. Test fixtures may write literal rows. Remaining raw-literal sites are debt: the routable_execution_or_non_k8s fragment above, plus about 40 production lines in 17 files, of which crates/libs/api/src/controllers/substrate/direct.rs has 12.
  • The agent plugin version goes from 0.2.31 to 0.2.32 in plugin.yaml and both plugin.json files (.agents/plugins/AGENTS.md). claude plugin validate passes; its one warning, a missing author, is also there on main.

Review items → changes

Review item Change
Bind the substrate value instead of splicing it into SQL text 29 statements bind SubstrateKind. The literal macros, const assertion, sql!(concat!) arm and its doc/test are removed, so sql! is as on main.
Keep the SubstrateKind alias Kept. Its doc now shows the bind.
Keep the docs/skill update, reworded for binding; keep the plugin bump Reworded (see above); 0.2.32 kept.
row_state_guards is spliced into a scan and an apply with different numbering The managed-by placeholder is a second argument.
Sweeper statements bind $own and then add = 'k8s' Return early unless Kubernetes; single bound predicate. Same results; one fewer query on Direct.
Test callers of shared query builders need the extra .bind Added in gentle_escalation_tests.rs and both drained_node_terminator/store/*_gate_tests.rs. transient_lifecycle_sweeper/tests.rs didn't need one (see above).
orphaned_restore_pod_gc/store.rs:23 is still a string comparison Parses each row and matches the variant. One bad row still can't fail the batch.
Word the convention as new code, existing sites as debt Done, with the rough count.
Postgres-backed test per converted statement showing a 'space' row excluded See Evidence. Every statement has one, and a mutation check shows each one fails when its predicate is dropped.
EXPLAIN (GENERIC_PLAN) for list_candidates_after and evaluate_gone_node See Evidence. Plan shapes are identical.

Round 1 review → changes

Finding Change
MINOR 1: tombstone test returns at the snapshot SELECT, so neither substrate predicate is tested alone (#14/#15) Two new tests in node_drain/store/tombstone_tests.rs change the node's managed_by under a row lock between the SELECT and the DELETE (see #15). Dropping the DELETE's predicate alone now fails delete_skips_node_that_turned_direct_after_snapshot. No production code changed. The SELECT's predicate alone has no observable result (see Evidence); the existing test's comment says so.
MINOR 2: docs endorse building a fragment around a placeholder space-conventions.md says new code writes whole statements with bound values; the row_state_guards placeholder argument is only for the existing fragment-based sites this conversion kept.
MINOR 3: "the cluster controller's production code follows this" overstates The doc, the skill bullet, the commit message and this body now name routable_execution_or_non_k8s as remaining debt, shared with internal-api published_state.rs and api wake_completion.rs. The fragment is unchanged.
MINOR 4: new tests write SET managed_by = 'space'; typed decode fails a batch One sentence exempts test fixtures, and the four tests are unchanged. Added: where one bad row must not fail the batch, read the column as text and parse per row, as orphaned_restore_pod_gc does.
NIT 5: EXPLAIN evidence leaves out ORDER BY id DESC LIMIT 1 Added a "Not covered" note under the generic plans.
NIT 6: four doc-comment lines over 100 columns Rewrapped in drained_node_terminator/store.rs (three) and orphaned_restore_pod_gc/store.rs (one).

Evidence

A Postgres-backed test per converted statement

Each test seeds a row the statement would otherwise act on and shows the 'k8s' row included and an otherwise-identical 'space' row excluded (for tombstone retention, the reverse). Most use the module's existing "eligible" fixture and change only managed_by. All run on the shared nextest Postgres with SPACE_CI_REQUIRE_TEST_DEPS=1.

Mutation check. For each statement, I replaced the converted predicate with (managed_by = $n OR TRUE) (for row_state_guards, the call site's placeholder became s.managed_by AND $n::text IS NOT NULL), ran the whole crate's suite, and reverted. "Before" is the crate's suite before this revision's new tests; "after" is this head. Every mutation fails at least one test, with one exception: #14, the snapshot SELECT in delete_terminating_node_and_record_tombstone. It repeats the DELETE's guard (#15). For a Direct row it returns false before the DELETE runs, and without it the DELETE matches no row and returns the same false with nothing written. The guard decides only whether the DELETE statement is issued, so no result can show it alone. #15 is now tested on its own, and dropping #14 and #15 together fails both tombstone substrate tests.

# Statement (crates/libs/cluster/controller/server/src/) Test (new unless noted) Predicate dropped: before → after
1 drained_node_terminator/store.rs delete_drained_node_if_orphan_query drained_node_terminator::store::substrate_scope_tests::list_and_reap_skip_direct_nodes passed → fails
2 … delete_refusal_diagnostics_query (phase_drifted) same (a direct row's refusal reads phase_drifted) passed → fails
3 … mark_unrecoverable_sandboxes_lost_query node EXISTS …::mark_lost_update_skips_direct_nodes passed → fails
4 … list_drained_nodes …::list_and_reap_skip_direct_nodes passed → fails
5 … mark-lost SELECT 1 FROM nodes … FOR SHARE re-check …::mark_lost_recheck_refuses_direct_nodes_before_claiming. A held row lock on the candidate shows the refusal happens before the claim; on 'k8s' the give-up blocks on that lock. passed → fails
6 failed_restorable_repair/store.rs scan (row_state_guards) failed_restorable_repair::tests::f7_direct_rows_are_neither_listed_nor_repaired passed → fails
7 … apply (row_state_guards) same passed → fails
8 started_sandbox_repair/store.rs scan (row_state_guards) started_sandbox_repair::tests::postgres_backed::r13_direct_rows_are_not_bounded_listed_or_repaired passed → fails
9 … apply (row_state_guards) same passed → fails
10 … current_max_sandbox_id same passed → fails
11 node_drain/store.rs draining_nodes_query node_drain::store::gentle_escalation_tests::draining_listing_skips_direct_nodes passed → fails
12 … escalate_stale_gentle_drains existing …::gentle_escalation_tests::skips_rows_every_guard_excludes fails → fails
13 … terminating_nodes_query (list_terminating_nodes) node_drain::store::tombstone_tests::terminating_direct_node_is_neither_listed_nor_deleted passed → fails
14 … delete_terminating_node_and_record_tombstone snapshot SELECT same alone: passed → passed (no observable effect alone, see above)
15 … delete_terminating_node_and_record_tombstone DELETE …::tombstone_tests::delete_skips_node_that_turned_direct_after_snapshot: a concurrent writer holds the row lock with an uncommitted managed_by = 'space', so the snapshot SELECT passes on the committed 'k8s' row and the DELETE parks on the lock, then re-checks the committed 'space' row and skips it. Its twin delete_removes_node_still_kubernetes_after_racing_write writes 'k8s', and the node is deleted and tombstoned. Both wait until pg_blocking_pids shows a backend blocked by the writer's pg_backend_pid(), polling every 25 ms. That wait and the join of the delete task are each bounded at 5 s, with a clear failure message. alone: passed → fails; #14 + #15 together: fails
16 orphaned_nodes/store.rs candidate scan orphaned_nodes::tests::direct_nodes_are_neither_candidates_nor_deleted passed → fails
17 … delete_node_if_orphan same passed → fails
18 orphaned_sandboxes.rs current_max_sandbox_id orphaned_sandboxes::tests::direct_rows_do_not_bound_the_sweep_or_become_candidates passed → fails
19 … list_candidates_after same passed → fails
20 orphaned_sandboxes/quarantine_recovery/store.rs candidates orphaned_sandboxes::quarantine_recovery::tests::direct_rows_are_not_recovery_candidates passed → fails
21 reconcile_queue_gc/store.rs list_gone_candidates reconcile_queue_gc::tests::postgres::direct_nodes_are_neither_listed_nor_pruned passed → fails
22 … evaluate_gone_node same passed → fails
23 suspended_execution_reaper/store.rs list_candidates suspended_execution_reaper::tests::postgres_backed::r11_direct_rows_do_not_bound_the_sweep_or_get_listed passed → fails
24 … current_max_sandbox_id same passed → fails
25 tombstone_retention.rs direct-node tombstone delete (SubstrateKind::Space) tombstone_retention::tests::store_prunes_and_counts_only_direct_node_tombstones passed → fails
26 … direct-node tombstone count (SubstrateKind::Space) same passed → fails
27 transient_lifecycle_sweeper/store.rs REPAIR_FAILED_ACTIVE_EXECUTION_SQL transient_lifecycle_sweeper::tests::postgres_backed::r21_failed_cleanup_statements_skip_direct_rows. The existing R16's direct row was excluded by placement, not by substrate. passed → fails
28 … LIST_FAILED_ACTIVE_EXECUTIONS_SQL same passed → fails
29 … NOTE_FAILED_CLEANUP_ATTEMPT_SQL same passed → fails
— … failed_active_execution_scope early return (made to accept every substrate) …::r22_failed_cleanup_runs_only_on_the_kubernetes_substrate (unit) passed → fails
— orphaned_restore_pod_gc/store.rs row_lifecycle (if false && …) existing orphaned_restore_pod_gc::tests::postgres_backed::classify_sandboxes_reads_lifecycle_and_generation fails → fails

On main's tests, 2 of the 31 single mutations failed a test. At the previous head (14e5b7b), 29 of 31 did. Now 30 of 31 do, and the remaining one (#14) fails together with #15.

Generic plans: list_candidates_after and evaluate_gone_node

I seeded a scratch database migrated from the test template on the repository's test image (PostgreSQL 16.15) with synthetic, deliberately skewed data. There were 400 nodes (10% 'space', a quarter not ready, some draining), 200,000 sandboxes (5% 'space', mixed lifecycles), an execution per 'k8s' sandbox, and 2,000 stale queue refs, followed by ANALYZE. Then I ran EXPLAIN (GENERIC_PLAN) on each statement's text from main (literal) and from this head (bound).

list_candidates_after. The plan shape is identical, and the only differences are the filter text and the row estimates. With the literal, Postgres uses managed_by's real frequency (95% 'k8s'). With the bound parameter, the generic plan uses 1/ndistinct (50%), so estimated rows roughly halve, as the review predicted. The same index scan on sandboxes_pkey, merge join and anti join are chosen either way.

-- before (literal)                                   -- after (bound)
Limit  (cost=0.84..8432.73 rows=412)                  Limit  (cost=0.84..6773.06 rows=217)
  ->  Nested Loop Anti Join (rows=4120)                 ->  Nested Loop Anti Join (rows=2172)
        ->  Merge Left Join (rows=4120)                       ->  Merge Left Join (rows=2172)
              Merge Cond: (s.id = e.sandbox_id)                     Merge Cond: (s.id = e.sandbox_id)
              ->  Index Scan using sandboxes_pkey on sandboxes s    ->  Index Scan using sandboxes_pkey on sandboxes s
                    (cost=0.42..31329.72 rows=4557)                       (cost=0.42..31329.72 rows=2402)
                    Index Cond: (id <= $3)                                Index Cond: (id <= $3)
                    Filter: (… AND (managed_by = 'k8s'::text) …)          Filter: (… AND (managed_by = $9) …)
              ->  Index Scan using idx_k8s_executions_active_sandbox  ->  (same)
              SubPlan 3 (same)                                      SubPlan 3 (same)
        ->  Seq Scan on sandbox_observed_states sos           ->  (same)

evaluate_gone_node. The plans are identical, costs and row estimates included. The only line that differs is the candidate_node filter, (managed_by = 'k8s'::text) vs (managed_by = $7), under the same Index Scan using nodes_pkey on nodes n / Index Cond: (id = $1). A primary-key lookup estimates one row either way.

Not covered: the three current_max_sandbox_id statements (ORDER BY id DESC LIMIT 1). A generic plan sizes the bound value at 1/ndistinct. If the bound value were a minority of rows, the generic plan may prefer walking sandboxes_pkey backwards until a match, where the literal's real, low frequency would have avoided that walk. That matters only if a Kubernetes controller's database were mostly 'space' sandboxes.

There are no partial indexes on managed_by (idx_nodes_managed_by, idx_nodes_role (managed_by, role) and idx_sandboxes_managed_by are full), so a generic plan has no index it could lose.

Characterization tests pass unchanged

The 21 lifecycle-writer characterization tests from #323 pass in each run below. Their files, including lifecycle_writer_test_support.rs, are unchanged.

Decisions made unattended

Decision Rationale
Bind SubstrateKind::Kubernetes at every scoped site, not substrate.managed_by() Each converted statement scoped to Kubernetes regardless of the running substrate, so the bound value is the constant. The sweeper sites that already bound managed_by() now bind the Some(Kubernetes) that failed_active_execution_scope returns.
Gate the sweeper cleanup with a free function over the label SubstrateAdapter::managed_by() returns &'static str and lives in the api crate, outside this PR's scope. Parsing it to SubstrateKind once at the gate keeps the variant match in one place.
Two small extractions for testability (terminating_nodes_query(), two sweeper consts) The sinks that own those statements need a kube client or a non-Direct substrate, and this file already uses that pattern (draining_nodes_query, REPAIR_FAILED_ACTIVE_EXECUTION_SQL). The SQL text is unchanged apart from the bind.
#14/#15 left as two predicates; #15 isolated with a concurrent writer, not by extracting the DELETE The snapshot SELECT and the guarded DELETE repeat one guard inside one transaction, and that defense in depth predates this PR. A concurrent writer between the two statements tests the DELETE through the production function without moving its text. The SELECT guard alone only decides whether the DELETE is issued, and the outcome is the same either way. Pinning that would mean asserting that no DELETE runs, for example by holding a conflicting table lock and timing out, which tests the implementation rather than a result, so I did not add it.
New tests next to each module's existing Postgres tests Each test reuses the module's eligible-row fixture, so the only difference between included and excluded rows is managed_by. The test-matrix doc entries are updated where a module keeps one.
Test-module SQL literals otherwise unconverted As before. Fixtures pass raw 'k8s'/'space' like the rest of their harness.

Locator note

The statement text changes (a placeholder replaces the literal), and some statements move lines, so the converted statements' /*module_path:line*/ locators, sqlx statement-cache keys and DBM query signatures change. Module paths don't change.

Validation

Base 02e0656, head = this PR's single commit (bbe7ae1). It still merges cleanly onto current origin/main, so it was not rebased.

Check Result
cargo fmt --all -- --check pass
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings pass
pre-commit (Gitleaks) at commit pass
space-public-review on the commit and this text no findings in the reviewed scope
cargo nextest run --locked -p pplx-space-common-core -p pplx-space-cluster-controller-server --retries 0, SPACE_CI_REQUIRE_TEST_DEPS=1, three runs at 697926c: 1378/1378 passed, 0 skipped, in each of runs 1, 2 and 3 (includes the 21 characterization tests). At bbe7ae1, which changes only the two racing tests: the cluster-controller package once, 977/977 passed, 0 skipped; the two racing tests ten times in a row, 2/2 each time; cargo fmt --all -- --check and cargo clippy -p pplx-space-cluster-controller-server --all-targets --all-features -- -D warnings pass.
cargo test --locked --doc -p pplx-space-common-core 383 passed at 14e5b7b; common-core is unchanged since
scripts/check-rust-error-policy.sh; scripts/source_guards/*.py (at 14e5b7b) pass, except thin_service_mains.py, which flags only crates/apps/*/main.rs files this PR doesn't touch: PRE-EXISTING
scripts/check_no_inline_rust_test_modules.py (at 14e5b7b) flags only crates/libs/observability/src/{otlp_endpoint,selection}.rs, untouched here: PRE-EXISTING
cargo arch (at 14e5b7b; no production line counts change here) exit 0. 59 advisory and 0 required findings, the same set as on main. Growth: node_drain/store.rs goes from 1897 to 1903 code lines and orphaned_sandboxes.rs from 1587 to 1590. Both are advisory listed_no_growth caps (1893, 1581) that the files already exceeded on main.
Mutation check (see Evidence) 30 of 31 single mutations fail a test; #14 fails together with #15. Rerun at 697926c for #14 alone (977/977 pass), #15 alone (fails delete_skips_node_that_turned_direct_after_snapshot), and both (also fails terminating_direct_node_is_neither_listed_nor_deleted). #15 alone rerun at bbe7ae1: fails the same test, 976/977.
Kind Not rerun at 697926c or bbe7ae1: this round changes docs, comments and one test module. The rows below are from 14e5b7b.
Kind: fresh cluster, build provenance A fresh space-local cluster was created by this run's bring-up. Its services log version: 14e5b7b419da (this head). The deployed /space/bin/space-cluster-controller matches the artifact image's binary by sha256 (3a5672dd…), before and after the suites. That binary contains the bound WHERE s.managed_by = $9 predicate and no AND s.managed_by = 'k8s' sweeper text.
smoke-kind-e2e.sh (start) pass (7/7)
smoke-kind-typed.sh all (unsharded, CI's env: admin token file and cold-receipts dir) ECW 76/78, native 23/23, none quarantined. The two non-passes are environmental; details below.
smoke-kind-leader-election-brownout.sh (SPACE_SMOKE_LEADER_ELECTION=1) pass: all three scenarios, including the apiserver outage (the holder resigned 44 s in, TTL 60 s)
smoke-kind-e2e.sh (end) pass (7/7)

Typed-suite non-passes, each rerun once:

  • local.telemetry_reaches_local_backends: ENVIRONMENTAL, reproduced on rerun. Tempo and Prometheus checks passed. Loki held zero lines because its WAL was throttling writes (disk usage exceeded threshold … usage_percent=90–91%) and rejected pushes. The Docker host device was at 92% during the suite. This PR touches no telemetry.
  • local.rate_limiters_enforce_and_fail_open: FLAKY under the same disk pressure. The first run's preflight create timed out after 90 s without a response. On the rerun every assertion passed (cluster admission, per-client create and connect concurrency, fail-open); only its cleanup timed out killing the probe sandboxes, so the verdict was INFRA_ERROR. This PR doesn't touch the API's limiters. I didn't compare against main because both failures have a cause that doesn't depend on the code, and the host had no disk left for a second stack.

🤖 Generated with Claude Code

@zibi-pplx zibi-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for a design reason, not a correctness problem. As written, the conversion is careful and complete:

  • all 27 statements and the Rust comparison are converted;
  • nothing is left in the crate's production code apart from the namespace defaults;
  • the byte-identical check is convincing;
  • the branch merges cleanly with main.

Request: bind the substrate value instead of splicing it into the SQL text

  • NodeManagedBy/SubstrateKind already implements sqlx Type/Encode/Decode. So managed_by = $n with .bind(SubstrateKind::Kubernetes) gives a single source of truth without the two macros, the const assertion, or the new sql!(concat!) arm.
  • It also follows space-data-store-development (SKILL.md:35-42): "Keep every SQL statement syntactically whole … values bound as parameters".
  • The concat! split cuts statements mid-predicate, so as written the Rust conventions doc and the store skill disagree.

What to keep:

  • the SubstrateKind alias: it names the bound value, and #368's console-label follow-up wants it;
  • the docs and skill update, reworded for binding;
  • the plugin version bump.

Specifics for the conversion

  • row_state_guards in started_sandbox_repair/store.rs:43 and failed_restorable_repair/store.rs:45 is spliced into a scan and an apply that number their parameters differently ("$1" and "$2"). It needs a second placeholder argument.
  • The sweeper statements already scope by s.managed_by = $own and then add = 'k8s' (transient_lifecycle_sweeper/store.rs:87-96, :1162-1172, :1218). Binding a second value there reads oddly. Returning early in Rust when the substrate isn't Kubernetes, then using a single bound predicate, gives the same result and skips a query that matches nothing on Direct.
  • Test callers. Some test callers of shared query builders will need the extra .bind, in node_drain/store/gentle_escalation_tests.rs, the two drained_node_terminator/store/*_gate_tests.rs files, and transient_lifecycle_sweeper/tests.rs.
  • orphaned_restore_pod_gc/store.rs:23 is still a string comparison. Please parse each row and match the variant, !matches!(managed_by.parse::<SubstrateKind>(), Ok(SubstrateKind::Kubernetes)). That keeps the "one bad row can't fail the batch" property this module already has for lifecycle.
  • Convention wording. Phrase it as applying to new code, with existing sites as debt. A few dozen production lines outside the cluster controller still compare managed_by with a raw 'k8s'/'space'; for example, api/.../substrate/direct.rs has about a dozen.

Evidence to replace the byte-identical proof

  • For each converted statement, a Postgres-backed test, or a pointer to an existing one, showing that a 'space' row is excluded.
  • EXPLAIN (GENERIC_PLAN) for list_candidates_after (orphaned_sandboxes.rs) and evaluate_gone_node (reconcile_queue_gc/store.rs).
    • Postgres may switch a prepared statement to a generic plan, which estimates managed_by = $n at 1/ndistinct instead of the real frequency.
    • There are no partial indexes on managed_by, so I don't expect a change, but it's worth confirming for these busy scans.

ac-pplx added a commit that referenced this pull request Oct 14, 2026
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and
a module doc on direct.rs, so agents changing the substrate, or shared
code that branches on it, know what Direct is for and where its contract
ends.

The AGENTS.md section holds the rules:

- when the api selects Direct (no `kubernetes` feature, or
  SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure
  is an error, not a fallback;
- implement new lifecycle operations for Kubernetes; Direct returns
  Unsupported unless the operation is part of the create, command,
  inspect and delete flow in docs/first-sandbox.md;
- make new SubstrateAdapter operations required methods, so the compiler
  demands each substrate's implementation and the forwarding arm in
  `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that
  shared code calls on every path keeps a default, and it still needs
  its arm;
- where shared code must differ, add a SubstrateCapabilities field, set
  it in DIRECT and KUBERNETES, and read it through capabilities() instead
  of matching the substrate (#368);
- scope queries to a substrate's own rows by binding the value
  (`substrate.managed_by()`, or `SubstrateKind::Kubernetes` in a
  Kubernetes-only loop) rather than a 'k8s' literal (#373);
- keep writes to the Direct-only tables in direct.rs and the Direct node
  registry;
- don't cite a reconciler as the retry path for a failed Direct
  operation, and don't change a known gap as a side effect of other work;
- which test fixtures to use: Direct only for tests under `direct/` or
  substrate-independent behavior, `kubernetes::tests` helpers for shared
  tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both
  compile only with `--features kubernetes`.

The direct.rs module doc records what Direct does today, so it is
reviewed with the code: which lifecycle operations it implements, which
it rejects as Unsupported (node-pool placement, forks, cluster pool
configs), its no-op trait methods, its Direct-only admin routes, how
admin drain/cordon actions fail on a Direct node, that the cluster
controller starts its drain, orphan, failed-row repair and node-sync
loops only for Kubernetes so nothing retries a failed Direct retire, and
two gaps that succeed silently: one-shot creates drop custom vcpu/memory,
and retire_execution does not enqueue the stopped-disk backup the trait
documents.

Correct the comment in Direct retire_execution that named the
failed-cleanup pass as its retry path: that pass selects only 'k8s'
rows, so a failed Direct retire is not retried.

Make the rules reachable from the code they govern: add the
`controllers/substrate/CLAUDE.md` symlink, widen the pointer in
crates/libs/api/AGENTS.md, and point the cluster-controller and
internal-api AGENTS.md files at the section, with the cluster
controller's binding rule for `managed_by` scoping stated there.

Documentation and comments only; no code, schema, wire or configuration
change.
@ac-pplx
ac-pplx force-pushed the ac/substrate-kind-sql-cluster-controller branch from 9d0cdc2 to 14e5b7b Compare October 14, 2026 15:54
@ac-pplx ac-pplx changed the title cluster-controller: name the substrate value once in Rust and SQL cluster-controller: bind the substrate value instead of writing it in SQL Oct 14, 2026
ac-pplx added a commit that referenced this pull request Oct 14, 2026
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and
a module doc on direct.rs, so agents changing the substrate, or shared
code that branches on it, know what Direct is for and where its contract
ends.

The AGENTS.md section holds the rules:

- when the api selects Direct (no `kubernetes` feature, or
  SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure
  is an error, not a fallback;
- implement new lifecycle operations for Kubernetes; Direct returns
  Unsupported unless the operation is one of the calls
  examples/first_sandbox.py makes (create, command session, list/inspect,
  delete);
- make new SubstrateAdapter operations required methods, so the compiler
  demands each substrate's implementation and the forwarding arm in
  `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that
  shared code calls on every path keeps a default, and it still needs
  its arm;
- where shared code needs a static fact about the substrate, add a
  SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read
  it through capabilities() instead of matching the substrate; where the
  substrate must act, add a required method (#368);
- in new SQL, scope to a substrate's rows by binding the value
  (`substrate.managed_by()` for the running substrate's rows, or a fixed
  `SubstrateKind::Kubernetes`/`::Space` for one substrate's rows) instead
  of a 'k8s'/'space' literal; existing literals are debt (#373);
- keep writes to the Direct-only tables in direct.rs and the Direct node
  registry;
- don't cite a cluster-controller reconciler as the retry path for a
  failed Direct execution retire or repair, and don't change a known gap
  as a side effect of other work;
- existing substrate matches, Direct-table joins in shared code and
  Unsupported trait defaults are debt, not precedent;
- which test fixtures to use: Direct only for tests under `direct/` or
  substrate-independent behavior, `kubernetes::tests` helpers for shared
  tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both
  compile only with `--features kubernetes`.

The direct.rs module doc records what Direct does today, so it is
reviewed with the code: how it reaches workers, which lifecycle
operations it implements, which it rejects as Unsupported (node-pool
placement, forks, cluster pool configs), its no-op trait methods, its
Direct-only admin routes, how admin drain/cordon actions fail on a
Direct node, and which cluster-controller loops are Kubernetes-only: the
drain, orphan, failed-row repair and node-sync loops never run for
Direct, while the public-operation dispatcher runs for both substrates
and retries transient operation failures. Nothing retries a failed
Direct retire. It lists two gaps that succeed silently: one-shot create
does not forward requested vCPU or memory (disk size is forwarded), and
retire_execution does not enqueue the stopped-disk backup the trait
documents.

Correct the comment in Direct retire_execution that named the
failed-cleanup pass as its retry path: that pass selects only 'k8s'
rows, so a failed Direct retire is not retried. Correct the
`resolves_cold_create_shape` field doc, which said that without the
capability the requested shape reaches the node unchanged: Direct's
one-shot create cannot carry vCPU or memory.

Make the rules reachable from the code they govern: add the
`controllers/substrate/CLAUDE.md` symlink, widen the pointer in
crates/libs/api/AGENTS.md, and point the cluster-controller and
internal-api AGENTS.md files at the section, with the cluster
controller's binding rule for `managed_by` scoping stated there,
including `tombstone_retention`'s fixed scope to Direct tombstones.

Documentation and comments only; no code, schema, wire or configuration
change. Names from #373 (`SubstrateKind`) exist on main only once #373
merges, so this lands after it.
@ac-pplx
ac-pplx force-pushed the ac/substrate-kind-sql-cluster-controller branch from 14e5b7b to 697926c Compare October 14, 2026 17:36
… SQL

Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold
the substrate that owns a row, 'k8s' or 'space'. The cluster controller
scoped and gated on that value with raw string literals in 29 production
SQL statements and one Rust comparison. Those statements now bind the
value as a parameter and the comparison matches the parsed variant, so each
statement stays syntactically whole and names the value through one Rust
type. The shared `routable_execution_or_non_k8s` fragment, which the
transient sweeper embeds, still writes `managed_by <> 'k8s'` and is
unchanged here.

common-core:
- `SubstrateKind`, an alias of `NodeManagedBy`, names the substrate value.
  `NodeManagedBy` keeps its name. Its existing sqlx `Encode` writes the
  same text the literals spelled, so `managed_by = $n` with
  `.bind(SubstrateKind::Kubernetes)` matches the same rows.

cluster controller:
- Each `managed_by = 'k8s'` (or `'space'` in tombstone retention) becomes
  `managed_by = $n` bound to `SubstrateKind::Kubernetes` (or `::Space`),
  numbered after the statement's existing parameters. The started- and
  failed-restorable repair `row_state_guards` fragments take the
  placeholder as an argument, since their scan and apply number
  parameters differently.
- The transient sweeper's failed-active-execution cleanup scoped by
  `s.managed_by = $own AND s.managed_by = 'k8s'`. It now returns early in
  Rust when the running substrate is not Kubernetes and binds a single
  predicate. Results are the same; the only observable difference is that
  a Direct-substrate controller no longer issues those three queries,
  which matched nothing there. Its list and rotate statements move to
  `const`s beside the existing repair `const`, so tests can run them.
- `list_terminating_nodes` moves its statement to
  `terminating_nodes_query()`, beside `draining_nodes_query()`, for the
  same reason.
- The orphaned-restore-pod classifier parses `managed_by` per row and
  matches `SubstrateKind::Kubernetes`. An unparseable value is treated as
  not Kubernetes, as before, and one bad row still cannot fail the batch.
- Test callers of shared query builders bind the extra parameter. Other
  test-module SQL literals are unchanged.

Tests: one Postgres-backed test per converted statement group shows a
'k8s' row included and an otherwise-identical 'space' row excluded, or for
tombstone retention the reverse. Dropping any one converted predicate
fails a test, except the terminating-node tombstone's snapshot SELECT: it
only returns early ahead of the DELETE, which repeats the same guard, so
dropping it alone changes no result. Two tests change a terminating node's
`managed_by` under a row lock between that SELECT and the DELETE, so the
DELETE re-checks the committed row: a node that became Direct is kept with
no tombstone, and one still on Kubernetes is deleted.

docs/rust/space-conventions.md and the space-rust-development skill say
new code writes whole statements and binds substrate values; the
`row_state_guards` helpers that take a placeholder are existing sites this
change kept, not a pattern. A batch that must survive one bad row reads
`managed_by` as text and parses per row. Test fixtures may write literal
rows. Remaining raw-literal sites, including that fragment, are debt, not
precedent. The agent plugin version is bumped to 0.2.32.

No schema, wire format, metric or configuration changes. Statement text
changes, so the `/*module_path:line*/` locators and the sqlx
statement-cache keys and DBM query signatures of converted statements
change too.
@ac-pplx
ac-pplx force-pushed the ac/substrate-kind-sql-cluster-controller branch from 697926c to bbe7ae1 Compare October 14, 2026 17:42
@ac-pplx

ac-pplx commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Reworked to bind the value, in bbe7ae1:

  • Binding. The cluster controller's 29 statements now use managed_by = $n with .bind(SubstrateKind::Kubernetes) (tombstone retention binds ::Space). The two macros, the const assertion and the sql!(concat!) arm are gone; sql! is as on main. SubstrateKind, the docs/skill update (reworded for binding) and the plugin bump are kept.
  • Your specifics. row_state_guards takes its placeholder as an argument for the scan and the apply. The sweeper returns early unless the substrate is Kubernetes and then uses one bound predicate (one fewer query per call on Direct; same results). The test callers of the shared builders bind the extra value. orphaned_restore_pod_gc parses each row and matches the variant.
  • Evidence. For each converted statement there is a Postgres-backed test, or a pointer to one, with a 'k8s' row included and an otherwise identical 'space' row excluded; making each predicate OR TRUE fails a test for 30 of 31. The tombstone DELETE's re-check is covered by a racing-writer test. EXPLAIN (GENERIC_PLAN) for list_candidates_after and evaluate_gone_node shows the same plan shapes; the PR body notes the case it doesn't cover (a minority bound value under ORDER BY id DESC LIMIT 1).
  • Convention wording. Bind substrate values in new code with whole statements; existing literals are debt, including the shared routable_execution_or_non_k8s fragment, which internal-api, trigger-queue, store: bind the substrate value instead of writing it in SQL #383 is taking up; test fixtures may write literal rows.

@ac-pplx
ac-pplx requested a review from zibi-pplx October 14, 2026 18:08
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit fc47b31 Oct 14, 2026
20 checks passed
@ac-pplx
ac-pplx deleted the ac/substrate-kind-sql-cluster-controller branch October 14, 2026 19:13
ac-pplx added a commit that referenced this pull request Oct 14, 2026
Add a "Direct Substrate Contract" section to the substrate AGENTS.md and
a module doc on direct.rs, so agents changing the substrate, or shared
code that branches on it, know what Direct is for and where its contract
ends.

The AGENTS.md section holds the rules:

- when the api selects Direct (no `kubernetes` feature, or
  SPACE_K8S_CONTROL_PLANE not truthy) and that a Kubernetes init failure
  is an error, not a fallback;
- implement new lifecycle operations for Kubernetes; Direct returns
  Unsupported unless the operation is one of the calls
  examples/first_sandbox.py makes (create, command session, list/inspect,
  delete);
- make new SubstrateAdapter operations required methods, so the compiler
  demands each substrate's implementation and the forwarding arm in
  `impl SubstrateAdapter for Substrate`; only a "nothing to do" hook that
  shared code calls on every path keeps a default, and it still needs
  its arm;
- where shared code needs a static fact about the substrate, add a
  SubstrateCapabilities field, set it in DIRECT and KUBERNETES, and read
  it through capabilities() instead of matching the substrate; where the
  substrate must act, add a required method (#368);
- in new SQL, scope to a substrate's rows by binding the value
  (`substrate.managed_by()` for the running substrate's rows, or a fixed
  `SubstrateKind::Kubernetes`/`::Space` for one substrate's rows) instead
  of a 'k8s'/'space' literal; existing literals are debt (#373);
- keep writes to the Direct-only tables in direct.rs and the Direct node
  registry;
- don't cite a cluster-controller reconciler as the retry path for a
  failed Direct execution retire or repair, and don't change a known gap
  as a side effect of other work;
- existing substrate matches, Direct-table joins in shared code and
  Unsupported trait defaults are debt, not precedent;
- which test fixtures to use: Direct only for tests under `direct/` or
  substrate-independent behavior, `kubernetes::tests` helpers for shared
  tests, `scripted_test_doubles.rs` inside `kubernetes/`, and that both
  compile only with `--features kubernetes`.

The direct.rs module doc records what Direct does today, so it is
reviewed with the code: how it reaches workers, which lifecycle
operations it implements, which it rejects as Unsupported (node-pool
placement, forks, cluster pool configs), its no-op trait methods, its
Direct-only admin routes, how admin drain/cordon actions fail on a
Direct node, and which cluster-controller loops are Kubernetes-only: the
drain, orphan, failed-row repair and node-sync loops never run for
Direct, while the public-operation dispatcher runs for both substrates
and retries transient operation failures. Nothing retries a failed
Direct retire. It lists two gaps that succeed silently: one-shot create
does not forward requested vCPU or memory (disk size is forwarded), and
retire_execution does not enqueue the stopped-disk backup the trait
documents.

Correct the comment in Direct retire_execution that named the
failed-cleanup pass as its retry path: that pass selects only 'k8s'
rows, so a failed Direct retire is not retried. Correct the
`resolves_cold_create_shape` field doc, which said that without the
capability the requested shape reaches the node unchanged: Direct's
one-shot create cannot carry vCPU or memory.

Make the rules reachable from the code they govern: add the
`controllers/substrate/CLAUDE.md` symlink, widen the pointer in
crates/libs/api/AGENTS.md, and point the cluster-controller and
internal-api AGENTS.md files at the section, with the cluster
controller's binding rule for `managed_by` scoping stated there,
including `tombstone_retention`'s fixed scope to Direct tombstones.

The Direct-table rule names archcheck's `pattern_ratchet.direct_tables`
rule (#358) as its enforcement, advisory today. The SQL-scoping rule also
covers returning early in Rust when a fixed-scope statement may run under
another substrate (#373), and names the shared
`routable_execution_or_non_k8s` fragment as remaining debt. The Direct
retire wording now says the sweeper's failed-cleanup retry does nothing
unless the running substrate is Kubernetes, matching the code after #373.

Documentation and comments only; no code, schema, wire or configuration
change.
ac-pplx added a commit that referenced this pull request Oct 16, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL
  comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=,
  IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons
  or match arms on "k8s"/"space". It allows test-support modules and
  fixtures that archcheck's filename-based test rule counts as production.
  On the parent commit it matches exactly the four converted lines; on this
  commit it matches nothing, so it has no baseline table and every behaviour
  is required.
- Tests: Postgres-backed twins for each converted statement. A new
  Kubernetes builder-lookup test, a unit table for the route-identity
  labels, and an otherwise identical 'space' row added to the sweeper's
  converge, internal-api's restored convergence and the wake-completion
  redrive tests. Test callers of the sweeper's shared builders bind the
  extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; Rust parses it and matches the variant (or calls the `SubstrateKind`
  method that names the question) and never compares the label string; and
  the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
three statements that landed on main since: internal-api's publisher-node
lookup and Direct's create-target claim and release. They now bind
`SubstrateKind` or match its parsed variant, and a new archcheck instance
keeps new literals out of production code. No schema, wire format, metric or
configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- internal-api: `publisher_node_for_identity` binds `n.managed_by = $3`
  (`Kubernetes`).
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL
  comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=,
  IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons
  or match arms on "k8s"/"space". It allows test-support modules and
  fixtures that archcheck's filename-based test rule counts as production.
  On the parent commit it matches exactly the seven converted lines; on
  this commit it matches nothing, so it has no baseline table and every
  behaviour is required.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, a relabelled-builder case in the tenant
  publication table, and an otherwise identical 'space' row added to the
  sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared builders bind the
  extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; Rust parses it and matches the variant (or calls the `SubstrateKind`
  method that names the question) and never compares the label string; and
  the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL
  comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=,
  IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons
  or match arms on "k8s"/"space". It allows test-support modules and
  fixtures that archcheck's filename-based test rule counts as production.
  On the parent commit it matches exactly the six converted lines; on
  this commit it matches nothing, so it has no baseline table and every
  behaviour is required.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; Rust parses it and matches the variant (or calls the `SubstrateKind`
  method that names the question) and never compares the label string; and
  the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL
  comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=,
  IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons
  or match arms on "k8s"/"space". It allows test-support modules and
  fixtures that archcheck's filename-based test rule counts as production.
  On the parent commit it matches exactly the six converted lines; on
  this commit it matches nothing, so it has no baseline table and every
  behaviour is required.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; Rust parses it and matches the variant (or calls the `SubstrateKind`
  method that names the question) and never compares the label string; and
  the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` matches SQL
  comparisons of `managed_by` with a 'k8s'/'space' literal (=, <>, !=,
  IS [NOT] DISTINCT FROM, [NOT] IN; any case and alias) and Rust comparisons
  or match arms on "k8s"/"space". It allows test-support modules and
  fixtures that archcheck's filename-based test rule counts as production.
  On the parent commit it matches exactly the six converted lines; on
  this commit it matches nothing, so it has no baseline table and every
  behaviour is required.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; Rust parses it and matches the variant (or calls the `SubstrateKind`
  method that names the question) and never compares the label string; and
  the archcheck rule enforces it. Plugin version 0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the
  common literal forms in production code: SQL comparisons of `managed_by`
  with 'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY,
  same-line CASE ... WHEN; any case, alias, prefix, cast or wrapping call),
  and Rust comparisons, label-string matches and match arms on "k8s"/"space"
  beside a substrate name. Its comment lists what it misses and its false
  positives. It allows test-support modules and fixtures that archcheck's
  filename-based test rule counts as production. On the parent commit it
  matches the seven lines this commit converts; on this commit it matches
  nothing, so it has no baseline table. no_crossing, listed_no_growth and
  stale_entries are required; no_new_entries and no_raised_caps are
  advisory, so a deliberate exception is a reviewed baseline entry.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; in Rust a decision about a row's substrate calls a `SubstrateKind`
  method named for the question, a fact about the running substrate reads
  `capabilities()`, and code never compares the label string; and the
  archcheck rule catches the common literal forms. Plugin version
  0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the
  common literal forms in production code: SQL comparisons of `managed_by`
  with 'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY,
  same-line CASE ... WHEN; any case, alias, prefix, quoting, cast or wrapping
  call), and Rust comparisons with the label (raw strings included), a
  `NODE_MANAGED_BY_*` const or a variant's `.as_str()`/`.to_string()`,
  matches on the label string, and match arms on "k8s"/"space" beside a
  substrate name. Its comment lists what it misses and its false positives. It allows test-support modules and fixtures that archcheck's
  filename-based test rule counts as production. On the parent commit it
  matches the seven lines this commit converts; on this commit it matches
  nothing, so it has no baseline table. no_crossing, listed_no_growth and
  stale_entries are required; no_new_entries and no_raised_caps are
  advisory, so a deliberate exception is a reviewed baseline entry.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; in Rust a decision about a row's substrate calls a `SubstrateKind`
  method named for the question, a fact about the running substrate reads
  `capabilities()`, and code never compares the label string; and the
  archcheck rule catches the common literal forms. Plugin version
  0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
ac-pplx added a commit that referenced this pull request Oct 17, 2026
Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the common
  literal forms in production code: SQL comparisons of `managed_by` with
  'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY,
  same-line CASE ... WHEN; any case, alias, prefix, quoting, cast or wrapping
  call), and Rust comparisons with the label (raw strings included), a
  `NODE_MANAGED_BY_*` const or a variant's `.as_str()`/`.to_string()`,
  matches on the label string, and match arms on "k8s"/"space" beside a
  substrate name. Its comment lists what it misses and its false positives.
  It allows test-support modules and fixtures that archcheck's filename-based
  test rule counts as production. On the parent commit it matches the seven
  lines this commit converts; on this commit it matches nothing, so it has no
  baseline table. no_crossing, listed_no_growth and stale_entries are
  required; no_new_entries and no_raised_caps are advisory, so a deliberate
  exception is a reviewed baseline entry.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; in Rust a decision about a row's substrate calls a `SubstrateKind`
  method named for the question, a fact about the running substrate reads
  `capabilities()`, and code never compares the label string; and the
  archcheck rule catches the common literal forms. Plugin version
  0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
meefs pushed a commit to meefs/SPACE that referenced this pull request Oct 18, 2026
…erplexityai#611)

Control-plane Postgres `sandboxes.managed_by` and `nodes.managed_by` hold the
substrate that owns a row, 'k8s' or 'space'. After perplexityai#373, perplexityai#383 and perplexityai#506 the
remaining production literals were the shared routability fragment, the
Kubernetes adapter's builder lookup and the Direct route-identity match, plus
Direct's create-target claim and release, which landed on main since. They
now bind `SubstrateKind` or match its parsed variant, and a new archcheck
instance keeps new literals out of production code. No schema, wire format,
metric or configuration changes, and no behavior change.

- common-svc: `routable_execution_or_non_k8s(alias, kubernetes_placeholder)`
  writes `{alias}.managed_by <> {placeholder}` instead of `<> 'k8s'`. Each
  caller binds `SubstrateKind::Kubernetes` at the placeholder it numbers
  after its own parameters: the transient sweeper's aged-transient list and
  restoring converge (`$5`), internal-api's restored-state convergence (`$4`)
  and the api's wake-completion CAS (`$2`).
- api Kubernetes adapter: `builder_client_for_node` binds
  `n.managed_by = $2`.
- api Direct: `claim_create_target` and `release_create_target` bind
  `managed_by = $3` (`Space`).
- api Direct: the route identity's `substrate_kind` text is parsed as
  `SubstrateKind` in `route_identity_substrate_kind`, which maps exactly
  'space' and 'k8s' as before and rejects any other text with the same
  error.
- ARCHITECTURE.toml: `pattern_ratchet.substrate_literals` catches the common
  literal forms in production code: SQL comparisons of `managed_by` with
  'k8s'/'space' (=, <>, !=, IS [NOT] DISTINCT FROM, [NOT] IN, = ANY,
  same-line CASE ... WHEN; any case, alias, prefix, quoting, cast or wrapping
  call), and Rust comparisons with the label (raw strings included), a
  `NODE_MANAGED_BY_*` const or a variant's `.as_str()`/`.to_string()`,
  matches on the label string, and match arms on "k8s"/"space" beside a
  substrate name. Its comment lists what it misses and its false positives.
  It allows test-support modules and fixtures that archcheck's filename-based
  test rule counts as production. On the parent commit it matches the seven
  lines this commit converts; on this commit it matches nothing, so it has no
  baseline table. no_crossing, listed_no_growth and stale_entries are
  required; no_new_entries and no_raised_caps are advisory, so a deliberate
  exception is a reviewed baseline entry.
- Tests: Postgres-backed twins for each converted statement. New
  Kubernetes builder-lookup and Direct create-target tests, a unit table for
  the route-identity labels, and an otherwise identical 'space' row added to
  the sweeper's converge, internal-api's restored convergence and the
  wake-completion redrive tests. Test callers of the sweeper's shared
  builders bind the extra parameter.
- Docs: `docs/rust/space-conventions.md`, the substrate AGENTS.md and the
  space-rust-development skill say every crate's production SQL binds the
  value; in Rust a decision about a row's substrate calls a `SubstrateKind`
  method named for the question, a fact about the running substrate reads
  `capabilities()`, and code never compares the label string; and the
  archcheck rule catches the common literal forms. Plugin version
  0.2.33 -> 0.2.34.

Converted statements' text changes, so their sqlx statement-cache keys and
query signatures change; statements below an added `.bind` move to new
`sql!` locators.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants