Skip to content

sandbox-store: move the restore-lease and pod-tracking sandbox row writes into the store - #1339

Merged
ac-pplx merged 1 commit into
ac/lifecycle-move-suspendfrom
ac/lifecycle-move-restore-lease
Nov 3, 2026
Merged

ac-pplx merged 1 commit into
ac/lifecycle-move-suspendfrom
ac/lifecycle-move-restore-lease

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Mechanical move, no functional change. The sandboxes row writes listed below move into pplx-space-sandbox-store with their SQL byte-for-byte; callers keep their locks, transactions and error handling. Everything else is doc comments, plus one test's pg_stat_activity match pattern updated to the moved statement's new module path. Part of the wave-2 stack (4 of 10, on #1338); it lands with the stack.

Problem
The Kubernetes substrate's store wrote the control-plane sandboxes row with inline SQL in kubernetes/store/restore_lease.rs (8 statements) and kubernetes/store/pod_tracking.rs (4 statements and an artifact-marker renderer call). Every write of the row belongs in pplx-space-sandbox-store.

Change
All 13 ratchet matches move into two new store modules, one function per transition: 12 statements plus the clear_current_local_artifact_marker_assignments renderer call. The SQL text is byte-for-byte the same, and each statement is built the way it was, including its portability and routability fragments. Binds keep their order and Postgres types. The claims keep their unused parameters (the evacuation claim's $4, the user claim's $5), so each statement is prepared with the same parameter list.

The claims, the completion and the release take impl PgExecutor and still run on the pool. The reverts and the pod-tracking writes take &mut Transaction because they rely on the caller's row lock. These stay in the Kubernetes store: the revert's lock and evidence read, the ownership snapshot, the transactions and the execution writes. The K8sStore methods now call the store.

Each function's doc names the fact it writes, its guards, and what its caller must have done first:

  • Operation step: both restore claims, the markerless lease claim, the completion, the release, the three reverts and both unplaces.
  • Inference: the three recoveries.
  • The reverts and unplaces have no lifecycle predicate of their own; they rely on the caller's lock.

Moved (api/…/kubernetes/store/ → pplx_space_sandbox_store::):

  • restore_lease.rs::begin_restore_lifecycle_for (user arm) → restore_lease::claim_user_restore
  • restore_lease.rs::begin_restore_lifecycle_for (evacuation arm) → restore_lease::claim_evacuation_restore
  • ::claim_markerless_restoring_restore_lease → restore_lease::claim_markerless_restoring_lease
  • ::complete_owned_user_restore → restore_lease::complete_owned_user_restore
  • ::release_user_restore_lease → restore_lease::release_user_restore_lease
  • ::revert_restore_claim_to_suspended, 3 arms → restore_lease::revert_restore_claim_{retiring_placement,clearing_placement_clock,lifecycle}
  • RestoreClaimRow → restore_lease::RestoreClaimRow
  • pod_tracking.rs::clear_pod_tracking_in, 2 statements → pod_tracking::unplace_after_pod_teardown{,_preserving_placement_clock}
  • ::recover_markerless_restoring_after_execution_cleanup_in → pod_tracking::recover_markerless_restoring_after_execution_cleanup
  • ::recover_started_with_current_suspend_snapshot_in (with clear_current_local_artifact_marker_assignments) → pod_tracking::recover_started_after_{execution_retired,failed_recovery}

sql! locators: the moved statements' locators now name pplx_space_sandbox_store::restore_lease and ::pod_tracking. Queries in pg_stat_activity and query monitoring that filter on the old kubernetes::store module path stop matching them.

Test change: in concurrent_wakes_for_one_sandbox_hold_one_claim_connection, one pg_stat_activity pattern changes from %kubernetes::store% to %pplx_space_sandbox_store::restore_lease:%, because that test finds the claim by its sql! locator. No other test or in-repo monitor keys on a moved statement's locator.

Ratchet: in sandbox_writes, both entries are removed (pod_tracking.rs 5, restore_lease.rs 8). No file_size entries were affected.

Validation (base 37cdc80d, head 064ffba1)

  • cargo nextest run -p pplx-space-sandbox-store -p pplx-space-api -p pplx-space-cluster-controller-server --all-features: 3360 passed, 1 skipped (an #[ignore] benchmark). Postgres setup ran, with no dependency skips.
  • cargo clippy on the same three packages with --all-targets --all-features -- -D warnings: clean.
  • cargo arch --base-baseline … --base-architecture …: exit 0, 0 required findings. The 111 advisory findings are pre-existing, and none are in touched files.
  • Test files: git diff 37cdc80d --stat -- '*test*' shows only kubernetes/tests.rs | 2 +- (the pattern above).
  • The claim-herd test against the moved code, before its pattern edit: fails with Elapsed(()) (no contender found). After the edit: passes.
  • Mutation checks, all reverted afterwards:
    • Dropping AND evacuation_restore_target IS NULL from the user claim's restoring arm fails k8s_store_postgres_behaviors (begin_restore_lifecycle_preserves_retry_and_marker_claim_rules).
    • Dropping AND s.lifecycle = 'restoring' from the markerless recovery fails cleanup_recoveries_revert_only_their_stored_lifecycles.
    • Dropping AND restore_lease_token = $2 from the owned completion fails k8s_store_postgres_behaviors (owned_restore_completion_tests.rs).
  • Kind, fresh cluster: infra/space/kind/scripts/up.sh took 4:36. The space-api and space-cluster-controller images carry revision 064ffba14f0f.
  • SPACE_SMOKE_KIND_COLD_RECEIPTS=<tmpdir> scripts/tests/smoke-kind-typed.sh all, wall time 51:46:
    • ECW: 87 passed, 0 failed, 1 inconclusive.
    • Native: 35/35 passed.
    • The inconclusive one is local.restarted_btrfs_send_is_rejected_and_retried: it needs the otel telemetry mode, and the bench runs datadog.

Risk and rollback: the statements and their order inside each transaction are unchanged. Roll back by reverting the commit.

Mixed-version: old and new pods send identical SQL; only the locator comments differ.

Open PRs that will need to rebase:

AI assistance: an AI agent (Claude Code) wrote this change and its validation.

After the wave-2 restack

This PR is one layer of a 10-PR stack, restacked onto main c5fdf0c7 (which includes #826) and then rebased mechanically onto 42a99002, which changed only baseline context lines, in order #1329 → #1336 → #1338 → #1339 → #1342 → #1347 → #1333 → #1335 → #1328 → #1332. The validation earlier in this description ran at pre-restack heads; this layer's head is b70c03b5.

@github-actions github-actions Bot added the triage:pending Waiting for a maintainer to triage label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

@ac-pplx
ac-pplx force-pushed the ac/lifecycle-move-restore-lease branch 2 times, most recently from 064ffba to bee8332 Compare November 2, 2026 23:25
@ac-pplx
ac-pplx changed the base branch from main to ac/lifecycle-move-suspend November 3, 2026 03:50
@ac-pplx
ac-pplx added this pull request to stack #1382 November 3, 2026 03:50
@ac-pplx
ac-pplx marked this pull request as ready for review November 3, 2026 03:51
@ac-pplx
ac-pplx marked this pull request as draft November 3, 2026 03:52
@ac-pplx
ac-pplx force-pushed the ac/lifecycle-move-restore-lease branch from bee8332 to 5a41347 Compare November 3, 2026 06:00
@ac-pplx
ac-pplx marked this pull request as ready for review November 3, 2026 06:29
…ites into the store

The Kubernetes substrate's store wrote the control-plane `sandboxes` row
from inline SQL in `restore_lease.rs` and `pod_tracking.rs`. All twelve
statements move into two new `pplx-space-sandbox-store` modules, one
function per transition:

- `restore_lease`: the user and evacuation restore claims, the markerless
  `restoring` lease claim, the owned `restoring -> started` completion,
  the lease release, and the three reverts of an abandoned claim.
- `pod_tracking`: both forms of the unplace after pod teardown (placement
  clock bumped or preserved), the markerless `restoring` recovery, and the
  `started` recovery onto a current suspend snapshot (execution-retired
  and failed-recovery proofs, with its artifact-marker SET fragment).

Statement text is unchanged byte for byte, and each keeps its
construction, including the portability and routability fragments it
interpolates. Binds keep their order and Postgres types; the claims keep
their unused parameters, so each statement is prepared with the same
parameter list as before. The claims, the completion and the release take
any Postgres executor and still run on the pool; the reverts and the
pod-tracking writes take the caller's transaction, since they rely on the
row lock the caller takes. The revert's lock and evidence read, the
ownership snapshot, the transactions and the execution writes stay in the
Kubernetes store, whose methods now delegate. Each function's doc names
the fact it writes, the guard it relies on, and what its caller must have
done first.

The moved statements' `sql!` locators now name
`pplx_space_sandbox_store::restore_lease` and
`pplx_space_sandbox_store::pod_tracking`. The one test that finds a
statement by its locator, the claim-herd test that counts lock-blocked
user restore claims in `pg_stat_activity`, now matches the new module
path. The `sandbox_writes` ratchet drops both entries (`pod_tracking.rs`
5, `restore_lease.rs` 8).
@ac-pplx
ac-pplx force-pushed the ac/lifecycle-move-restore-lease branch from 5a41347 to b70c03b Compare November 3, 2026 21:23
@ac-pplx
ac-pplx added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 50bb9d5 Nov 3, 2026
27 checks passed
@ac-pplx
ac-pplx deleted the ac/lifecycle-move-restore-lease branch November 3, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage:pending Waiting for a maintainer to triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants