[ac-pplx]

Mechanical move, no functional change. The sandboxes row writes listed below move into pplx-space-sandbox-store with their SQL byte-for-byte; callers keep their locks, transactions and error handling. Everything else is doc comments. Part of the wave-2 stack (9 of 10, on #1335); it lands with the stack.

Summary

The node drain, its pod-less give-up and the drained-node terminator wrote sandboxes with inline SQL. 16 statements move into a new pplx-space-sandbox-store module, drain, and the terminator's mark-lost moves into the existing lost module. When it flips rows, the terminator's transaction then releases their placement through lost::release_lost_placement; lost::mark_started_lost is the same give-up for started rows, which lost_started_sandboxes runs in its own transaction. Each is one function per transition, on the caller's executor or transaction, and its doc names the fact it writes.

Moved (node_drain/store.rs → sandbox_store::drain, same name unless shown):

• Operation steps: claim_evacuation_suspend, revert_evacuation_suspend_claim, claim_stopped_evacuation (with StoppedEvacuationClaim); the suspend commit in evacuate_sandbox → commit_evacuation_suspend; the finalize in restore_evacuated_and_finalize → finalize_evacuation_restore; schedule_restore_retry_query → schedule_restore_retry.
• Observed projection: record_stopped_evacuation_artifact.
• Inferences: redrive_stranded_suspending_query, mark_sandbox_lost_query, mark_sandbox_lost_on_gone_node_query, adopt_remote_snapshot_on_gone_node_query, mark_sandbox_failed_on_node_query, adopt_remote_snapshot_for_drain_query, abandon_stale_resident_sandboxes_query, fail_resident_sandboxes_on_gone_node_query (each builder keeps its name and gains a binding function).
• node_drain/store/pod_less_giveup.rs::revert_to_suspended → drain::revert_pod_less_resident_to_suspended (inference, &mut Transaction).
• drained_node_terminator/store.rs::mark_unrecoverable_sandboxes_lost_query → lost::mark_unrecoverable_sandboxes_lost_query + lost::mark_unrecoverable_sandboxes_lost (inference, &mut Transaction).
• gone_node_identity_guards is public (the snapshots give-up uses it).

Unchanged:

• SQL text byte-for-byte (all 17 compared against main); constructions kept.
• Same binds: PHASE_DRAINING → NodeDrainPhase::Draining (same text). The three lifecycle sets are passed as wire strings; two are the drain's, the resident set is pplx-space-common-svc's.
• The standalone FOR UPDATE/FOR SHARE reads, candidate lists, node lock, lock timeout, transactions, and the pod-less precheck and its other writes stay with the callers. The in-statement FOR UPDATE SKIP LOCKED of the failed-on-node flip, the stale-resident abandon and the gone-node resident failure moved with their statements. No new dependency edges.

Thin guards, documented, unchanged: the suspend commit's match is unchecked; the claim revert re-checks neither node nor marker; the finalize guards only on restoring; the pod-less revert has no lifecycle/node guard and relies on the precheck under the caller's locks.

sql! locators change. They now name pplx_space_sandbox_store::drain, and pplx_space_sandbox_store::lost for the terminator's statement, in pg_stat_activity and query monitoring.

Docs that named the old locations now name the store functions. sandbox_writes entries removed: node_drain/store.rs 15, pod_less_giveup.rs 3, drained_node_terminator/store.rs 1. node_drain/store.rs's file_size cap (1893) stays.

Validation

• cargo nextest run -p pplx-space-sandbox-store -p pplx-space-cluster-controller-server -p pplx-space-db-test-fixtures --all-features --no-fail-fast (Postgres running): 1257 passed, 0 skipped.
• Mutations (not committed): dropping AND NOT {durable} from lost::mark_unrecoverable_sandboxes_lost_query fails mark_unrecoverable_sandboxes_lost_guards and mark_lost_yields_to_concurrent_snapshot_finalize; OR TRUE on the claim's drain_mode guard fails evacuation_claim_refuses_a_node_that_turned_gentle; AND FALSE on the pod-less revert fails 7 pod_less_giveup_tests.
• cargo clippy --all-targets --all-features -- -D warnings (store, controller server, portability, api), cargo fmt --check, RUSTDOCFLAGS="-D warnings" cargo doc --no-deps (store, controller server): clean.
• cargo arch against main: 0 required findings (110 advisory, as on main).
• Test files: only use lines (3 files, +9/−2).
• Kind, fresh cluster from this commit (up.sh 6m59s, every image at this commit except the Kind-only STS bridge helper): local.forced_drain_evacuates_sandbox passed in 34s, local.gentle_drain_leaves_sandbox_resident in 55s. Postgres statement logging and the controller's logs and sandbox events show that the forced drain changed rows through the evacuation suspend claim, the suspend commit and the restore finalize. The stranded-suspend redrive and the stale-resident abandon ran but matched no row. Stopped evacuation is off on Kind. The give-ups, the retry backoff, the pod-less revert and the terminator's mark-lost matched no row there, so only the Postgres tests cover them. For schedule_restore_retry and the two remote-snapshot adoptions, the Postgres tests bind the builders themselves, so their production binds were checked against main by reading. The full typed suite was not rerun this round.

Severity and risk

Severity: does not apply (refactor). Risk: a moved statement binding differently; SQL and binds checked one by one. Rollback: revert.

Rollout

Old and new controllers issue identical statements and binds; only the locator differs.

Open PRs that will need to rebase

• [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826: claim_stopped_evacuation (new $6, format!, docs) and record_stopped_evacuation_artifact (docs, call site): node_drain/store.rs → sandbox/store/src/drain.rs.
• api: refuse a cold create with a 503 when every eligible node's network slots are held #1061, [space] fence every placement writer against the provider-terminate path #559: edit these files but none of the moved statements. No open PR touches lost.rs.

AI assistance: Claude Code wrote the code, ran the validation and drafted this description.

After the wave-2 restack

This PR is one layer of a 10-PR stack, restacked onto main c5fdf0c7 (which includes #826) and then rebased mechanically onto 42a99002, which changed only baseline context lines, in order #1329 → #1336 → #1338 → #1339 → #1342 → #1347 → #1333 → #1335 → #1328 → #1332. The validation earlier in this description ran at pre-restack heads; this layer's head is 8932644f.

• [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826: Main's [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 changed claim_stopped_evacuation: a new reused_durable_artifact parameter, the $6 predicate built from the portability crate's durable-backup fragment, and two doc rewrites. All of it is carried into drain::claim_stopped_evacuation, byte-identical to main's statement and construction, and the call site passes the new argument as on main. For every file [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 and this stack share, each SQL line [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 added was checked to be present verbatim at the top head, in place or in the store module it moved to.
• At this head: cargo fmt --all -- --check, cargo check --workspace --all-targets --all-features and cargo arch (0 required findings, with main's layers and table-owner rules), and the ratchet and caps this layer changes are recounted exactly.
• At the top of the stack (e90aa284 before that rebase; nextest passed 4370 tests again at 77fbae1e): cargo clippy --workspace --locked --all-targets --all-features -- -D warnings is clean, and SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --all-features passed 4355 tests across the store, api, internal-api server, cluster-controller server, metrics, db-test-fixtures, common-core, common-svc and portability crates, [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's suites included.
• Kind at the top of the stack, fresh clusters with the otel telemetry mode and CI's cold-receipt directory: scripts/tests/smoke-kind-typed.sh all gave ECW 91/91 and native 42/42, including [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's native.forced_drain_re_places_stopped_sandbox and its re-drain and restarted variants. The Direct substrate lane's all gave ECW 51/51 and native 27/27. No flakes and no reruns.
• Equivalence: apart from the [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 port named above, the PR's own +/- lines and its SQL string literals match its reviewed head (range-diff plus a literal multiset compare).

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.