[ac-pplx]

Mechanical move, no functional change. The sandboxes row writes listed below move into pplx-space-sandbox-store with their SQL byte-for-byte; callers keep their locks, transactions and error handling. Everything else is doc comments. Part of the wave-2 stack (8 of 10, on #1333); it lands with the stack.

Summary

Three cluster-controller loops still wrote the sandboxes row with inline SQL: the orphaned-sandbox reconciler, tombstone retention and the transient-lifecycle sweeper. Their last five statements move into pplx-space-sandbox-store, one function per transition, each running on the caller's transaction or executor. Each doc names the fact it writes; all five are inference.

• orphan (new): recovery to suspended and quarantine. Each is one statement whose CTEs also retire the active execution and reset open port exposures. The classify-guard fragments move too and are public, because the reconciler's plan read (SELECT … FOR UPDATE) uses the same guards. The store takes an OrphanGuard with today's primitive binds; the reconciler converts its ClassifyGuard at the call site. Note: the CTEs re-check the plan read's guard and eligibility. The plan read locks the sandbox row (and, for a pod guard, the execution) but not the snapshot rows the eligibility reads. No test reaches these re-checks apart from the plan read.
• tombstone (new): the prune of rows deleted past retention. It is not delete_with_dependents, so the caller still deletes the claimed warm-pool slots.
• sweeper (extended): the per-lifecycle fail CAS and the failed-active-execution repair. The CAS moves with FailGuard, is_activation_owned and the shared evacuation_guards, which the candidate select still uses. The controller re-exports FailGuard and is_activation_owned.

SQL text is byte-identical and each statement is built as before. Locking reads, transactions, the quarantine inserts and the slot delete stay at the call sites.

Moved (old → new; store crate pplx_space_sandbox_store):

• orphaned_sandboxes::{recover_orphan_to_suspended, quarantine_orphan, classify_guard_sql, classify_guard_from_sql, classify_guard_lock_sql, fetch_guarded_resolution} → orphan:: (same names); OrphanResolutionRow → orphan::OrphanResolution
• the DELETE FROM sandboxes in TombstoneRetentionStore::prune_tombstones → tombstone::prune_deleted_sandboxes
• transient_lifecycle_sweeper::store::{fail_candidate_query, evacuation_guards, REPAIR_FAILED_ACTIVE_EXECUTION_SQL} → sweeper:: (same names), bound by the new sweeper::fail_aged_candidate and sweeper::repair_failed_active_execution
• transient_lifecycle_sweeper::{FailGuard, is_activation_owned} → sweeper::

sql! locators change: the moved statements now report pplx_space_sandbox_store::{orphan,tombstone,sweeper}, visible in pg_stat_activity and query monitoring.

Baseline: the sandbox_writes entries for the three controller files are removed; the file_size cap for orphaned_sandboxes.rs drops from 1581 to 1418.

Validation

On this head:

• cargo nextest run -p pplx-space-sandbox-store -p pplx-space-cluster-controller-server --no-fail-fast (Postgres setup binary built first): 1211 passed, 0 skipped.
• cargo clippy -p pplx-space-sandbox-store -p pplx-space-cluster-controller-server --all-targets -- -D warnings, RUSTDOCFLAGS="-D warnings" cargo doc --no-deps (same packages), cargo fmt --check: clean.
• cargo arch --base-baseline … --base-architecture … --renames …: 0 required findings, none from these files.
• Test files change only use paths: tests.rs 6 +-, tests/activation_owner.rs 3 +-, tests/crashed_create.rs 5 +-.
• Mutations (not committed), each caught by a named test:
  • prune also takes started rows → tombstone_retention::tests::store_prunes_only_expired_deleted_sandboxes_of_every_stored_lifecycle
  • fail CAS without the pending-receipt guard → crashed_create::postgres_backed::only_a_create_no_receipt_can_finish_is_failed_alive
  • repair without placement = 'on_node' → postgres_backed::r16_failed_active_repair_cas_requires_k8s_failed_active_execution
  • recover keeps target_node_id → orphaned_sandboxes::lifecycle_writer_tests::classification_moves_only_the_lifecycles_its_guard_owns
  • Forgotten guard without state = 'forgotten' → …::classification_acts_only_on_the_execution_its_guard_names
• Kind: fresh cluster (up.sh, 12m08s), then scripts/tests/smoke-kind-typed.sh all (52m33s). These ran on a tree that differs from this head only in doc comments.
  • ECW: 85 passed, 0 failed, 3 inconclusive. Native: 35 passed.
  • ecw.cold_create_environment and ecw.warm_pool_environment rerun with SPACE_SMOKE_KIND_COLD_RECEIPTS: both passed.
  • Not run (need otel telemetry; bench ran datadog): local.restarted_btrfs_send_is_rejected_and_retried (inconclusive); skipped local.btrfs_send_restore_pulls_parent_template, local.leader_election_brownout_keeps_one_leader, local.telemetry_reaches_local_backends.

Severity and risk

Severity: does not apply (refactor). Risk: a bind mistake could change a guarded write; the SQL is byte-identical, and the tests and mutations above exercise each statement. Roll back by reverting; no schema or data change.

Rollout

Same statements, so old and new pods can run together; only the sql! locators differ.

Open PRs that will need to rebase

• [space] fence every placement writer against the provider-terminate path #559: classify_guard_sql → orphan::classify_guard_sql; fail_candidate_query → sweeper::fail_candidate_query. Its new activation_receipt_pending helper, which fail_candidate_query and its new abandoned-create claim both call, has to move into the store too, because the store cannot call the controller. Its new UPDATE sandboxes statements in orphaned_sandboxes.rs no longer have a ratchet entry, so they belong in orphan.
• [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826: REPAIR_FAILED_ACTIVE_EXECUTION_SQL → sweeper::REPAIR_FAILED_ACTIVE_EXECUTION_SQL.
• api: close six admin API gaps the admin console works around #1303, [api] trigger a stopped sandbox's runtime upgrade from a test hook, and smoke it on Kind #1322, [cluster-controller] record pod-create intent so the pod-less drain give-up can release a stalled recovery #1323: same files, but nothing they touch moved.
• Store lib.rs: api: close six admin API gaps the admin console works around #1303, Add idempotent ECW network policy replacement and live status #1297, Add trusted lifecycle registration and advisory candidate inventory #579. architecture-baseline.toml: [cluster-controller] record pod-create intent so the pod-less drain give-up can release a stalled recovery #1323, api: page and filter the admin sandbox list, count spaced versions, and share one sandbox ID parser #1321, Add idempotent ECW network policy replacement and live status #1297, archcheck: delete file_size and the checks the layers rule covers #1222, archcheck: give every database table one declared owning store #1164, archcheck: keep the domain layer free of I/O and its models off protocols #1162, archcheck: keep database and object-storage access in the storage layer #1161, archcheck: add a layers rule for crate placement and dependency direction #1152, api: refuse a cold create with a 503 when every eligible node's network slots are held #1061, api: expose process recovery sessions #976, spaced: gate routed process recovery on VM activation #969, spaced: advertise process output gap recovery #889, Enforce template lookup scope after compatible rollout #806, [space] fence every placement writer against the provider-terminate path #559.
• PRs that move other sandboxes writes into the store (sandbox-store: move the node drain's sandbox row writes into the store #1328, api: move the create path's sandboxes writes into sandbox-store #1329, cluster-controller: gauge sandboxes wedged in a transient lifecycle #1332, sandbox-store: move the publish path's stopped, activity, absent-delete and rollback writes #1333) each add a store module line or edit their own baseline entries; those conflicts are textual.
• architecture-baseline.toml also conflicts textually with [cluster-controller] give up on pod-less drain residents only when the node vouches #705 (merged), which added a sandbox_writes entry for node_drain/store/pod_less_giveup.rs; a rebase keeps that entry.

AI assistance: Claude Code (an AI agent) wrote the code changes, ran the validation and drafted this description.

After the wave-2 restack

This PR is one layer of a 10-PR stack, restacked onto main c5fdf0c7 (which includes #826) and then rebased mechanically onto 42a99002, which changed only baseline context lines, in order #1329 → #1336 → #1338 → #1339 → #1342 → #1347 → #1333 → #1335 → #1328 → #1332. The validation earlier in this description ran at pre-restack heads; this layer's head is 8892c3ae.

• [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826: Main's [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 made REPAIR_FAILED_ACTIVE_EXECUTION_SQL also drop a disk artifact pin. The moved sweeper::REPAIR_FAILED_ACTIVE_EXECUTION_SQL is byte-identical to main's, doc paragraph included. For every file [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 and this stack share, each SQL line [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 added was checked to be present verbatim at the top head, in place or in the store module it moved to.
• At this head: cargo fmt --all -- --check, cargo check --workspace --all-targets --all-features and cargo arch (0 required findings, with main's layers and table-owner rules), and the ratchet and caps this layer changes are recounted exactly.
• At the top of the stack (e90aa284 before that rebase; nextest passed 4370 tests again at 77fbae1e): cargo clippy --workspace --locked --all-targets --all-features -- -D warnings is clean, and SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --all-features passed 4355 tests across the store, api, internal-api server, cluster-controller server, metrics, db-test-fixtures, common-core, common-svc and portability crates, [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's suites included.
• Kind at the top of the stack, fresh clusters with the otel telemetry mode and CI's cold-receipt directory: scripts/tests/smoke-kind-typed.sh all gave ECW 91/91 and native 42/42, including [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's native.forced_drain_re_places_stopped_sandbox and its re-drain and restarted variants. The Direct substrate lane's all gave ECW 51/51 and native 27/27. No flakes and no reruns.
• Equivalence: apart from the [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 port named above, the PR's own +/- lines and its SQL string literals match its reviewed head (range-diff plus a literal multiset compare).

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.