Repository navigation
sandbox-store: move the restore-lease and pod-tracking sandbox row writes into the store - #1339
Merged
ac-pplx merged 1 commit intoNov 3, 2026
Conversation
|
Thanks for opening this pull request! A SPACE maintainer will take a look. CONTRIBUTING.md explains how we review outside contributions. |
ac-pplx
force-pushed
the
ac/lifecycle-move-restore-lease
branch
2 times, most recently
from
November 2, 2026 23:25
064ffba to
bee8332
Compare
This was referenced Nov 3, 2026
Merged
Merged
ac-pplx
marked this pull request as ready for review
November 3, 2026 03:51
ac-pplx
marked this pull request as draft
November 3, 2026 03:52
ac-pplx
force-pushed
the
ac/lifecycle-move-restore-lease
branch
from
November 3, 2026 06:00
bee8332 to
5a41347
Compare
ac-pplx
marked this pull request as ready for review
November 3, 2026 06:29
kumarpallav-pplx
requested review from
kumarpallav-pplx and
nikhilbirmiwal-pplx
November 3, 2026 06:30
nikhilbirmiwal-pplx
approved these changes
Nov 3, 2026
…ites into the store The Kubernetes substrate's store wrote the control-plane `sandboxes` row from inline SQL in `restore_lease.rs` and `pod_tracking.rs`. All twelve statements move into two new `pplx-space-sandbox-store` modules, one function per transition: - `restore_lease`: the user and evacuation restore claims, the markerless `restoring` lease claim, the owned `restoring -> started` completion, the lease release, and the three reverts of an abandoned claim. - `pod_tracking`: both forms of the unplace after pod teardown (placement clock bumped or preserved), the markerless `restoring` recovery, and the `started` recovery onto a current suspend snapshot (execution-retired and failed-recovery proofs, with its artifact-marker SET fragment). Statement text is unchanged byte for byte, and each keeps its construction, including the portability and routability fragments it interpolates. Binds keep their order and Postgres types; the claims keep their unused parameters, so each statement is prepared with the same parameter list as before. The claims, the completion and the release take any Postgres executor and still run on the pool; the reverts and the pod-tracking writes take the caller's transaction, since they rely on the row lock the caller takes. The revert's lock and evidence read, the ownership snapshot, the transactions and the execution writes stay in the Kubernetes store, whose methods now delegate. Each function's doc names the fact it writes, the guard it relies on, and what its caller must have done first. The moved statements' `sql!` locators now name `pplx_space_sandbox_store::restore_lease` and `pplx_space_sandbox_store::pod_tracking`. The one test that finds a statement by its locator, the claim-herd test that counts lock-blocked user restore claims in `pg_stat_activity`, now matches the new module path. The `sandbox_writes` ratchet drops both entries (`pod_tracking.rs` 5, `restore_lease.rs` 8).
ac-pplx
force-pushed
the
ac/lifecycle-move-restore-lease
branch
from
November 3, 2026 21:23
5a41347 to
b70c03b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Kubernetes substrate's store wrote the control-plane
sandboxesrow with inline SQL inkubernetes/store/restore_lease.rs(8 statements) andkubernetes/store/pod_tracking.rs(4 statements and an artifact-marker renderer call). Every write of the row belongs inpplx-space-sandbox-store.Change
All 13 ratchet matches move into two new store modules, one function per transition: 12 statements plus the
clear_current_local_artifact_marker_assignmentsrenderer call. The SQL text is byte-for-byte the same, and each statement is built the way it was, including its portability and routability fragments. Binds keep their order and Postgres types. The claims keep their unused parameters (the evacuation claim's$4, the user claim's$5), so each statement is prepared with the same parameter list.The claims, the completion and the release take
impl PgExecutorand still run on the pool. The reverts and the pod-tracking writes take&mut Transactionbecause they rely on the caller's row lock. These stay in the Kubernetes store: the revert's lock and evidence read, the ownership snapshot, the transactions and the execution writes. TheK8sStoremethods now call the store.Each function's doc names the fact it writes, its guards, and what its caller must have done first:
Moved (
api/…/kubernetes/store/→pplx_space_sandbox_store::):restore_lease.rs::begin_restore_lifecycle_for(user arm) →restore_lease::claim_user_restorerestore_lease.rs::begin_restore_lifecycle_for(evacuation arm) →restore_lease::claim_evacuation_restore::claim_markerless_restoring_restore_lease→restore_lease::claim_markerless_restoring_lease::complete_owned_user_restore→restore_lease::complete_owned_user_restore::release_user_restore_lease→restore_lease::release_user_restore_lease::revert_restore_claim_to_suspended, 3 arms →restore_lease::revert_restore_claim_{retiring_placement,clearing_placement_clock,lifecycle}RestoreClaimRow→restore_lease::RestoreClaimRowpod_tracking.rs::clear_pod_tracking_in, 2 statements →pod_tracking::unplace_after_pod_teardown{,_preserving_placement_clock}::recover_markerless_restoring_after_execution_cleanup_in→pod_tracking::recover_markerless_restoring_after_execution_cleanup::recover_started_with_current_suspend_snapshot_in(withclear_current_local_artifact_marker_assignments) →pod_tracking::recover_started_after_{execution_retired,failed_recovery}sql!locators: the moved statements' locators now namepplx_space_sandbox_store::restore_leaseand::pod_tracking. Queries inpg_stat_activityand query monitoring that filter on the oldkubernetes::storemodule path stop matching them.Test change: in
concurrent_wakes_for_one_sandbox_hold_one_claim_connection, onepg_stat_activitypattern changes from%kubernetes::store%to%pplx_space_sandbox_store::restore_lease:%, because that test finds the claim by itssql!locator. No other test or in-repo monitor keys on a moved statement's locator.Ratchet: in
sandbox_writes, both entries are removed (pod_tracking.rs5,restore_lease.rs8). Nofile_sizeentries were affected.Validation (base
37cdc80d, head064ffba1)cargo nextest run -p pplx-space-sandbox-store -p pplx-space-api -p pplx-space-cluster-controller-server --all-features: 3360 passed, 1 skipped (an#[ignore]benchmark). Postgres setup ran, with no dependency skips.cargo clippyon the same three packages with--all-targets --all-features -- -D warnings: clean.cargo arch --base-baseline … --base-architecture …: exit 0, 0 required findings. The 111 advisory findings are pre-existing, and none are in touched files.git diff 37cdc80d --stat -- '*test*'shows onlykubernetes/tests.rs | 2 +-(the pattern above).Elapsed(())(no contender found). After the edit: passes.AND evacuation_restore_target IS NULLfrom the user claim'srestoringarm failsk8s_store_postgres_behaviors(begin_restore_lifecycle_preserves_retry_and_marker_claim_rules).AND s.lifecycle = 'restoring'from the markerless recovery failscleanup_recoveries_revert_only_their_stored_lifecycles.AND restore_lease_token = $2from the owned completion failsk8s_store_postgres_behaviors(owned_restore_completion_tests.rs).infra/space/kind/scripts/up.shtook 4:36. Thespace-apiandspace-cluster-controllerimages carry revision064ffba14f0f.SPACE_SMOKE_KIND_COLD_RECEIPTS=<tmpdir> scripts/tests/smoke-kind-typed.sh all, wall time 51:46:local.restarted_btrfs_send_is_rejected_and_retried: it needs the otel telemetry mode, and the bench runs datadog.Risk and rollback: the statements and their order inside each transaction are unchanged. Roll back by reverting the commit.
Mixed-version: old and new pods send identical SQL; only the locator comments differ.
Open PRs that will need to rebase:
UPDATE sandboxesand aDELETE FROM sandboxestokubernetes/store/pod_tracking.rsand raises that file's entry to 7. With the entry gone, archcheck'sno_crossingrule flags them (notlisted_no_growth), so they belong inpplx_space_sandbox_store::pod_tracking.pplx_space_sandbox_portabilityimport block inkubernetes/store.rs. Once both land,clear_current_local_artifact_marker_assignmentshas no user in the api crate, so whichever lands second must drop it from that import, or clippy-D warningsfails.kubernetes/store/pod_tracking.rs([api] record the activating client from Direct's start, wake and restore receipts #1175 is stacked on [api] record the activating client and label sandbox pods with it #1068); none touches a moved function. sandbox-store: move the wake path's writes of the sandboxes row into a wake module #1336 rewrites the doc comment on thecomplete_owned_user_restorewrapper inkubernetes/store/restore_lease.rs; its hunk doesn't overlap this PR's.sandbox/store/src/lib.rsnear the module list.AI assistance: an AI agent (Claude Code) wrote this change and its validation.
After the wave-2 restack
This PR is one layer of a 10-PR stack, restacked onto main
c5fdf0c7(which includes #826) and then rebased mechanically onto42a99002, which changed only baseline context lines, in order #1329 → #1336 → #1338 → #1339 → #1342 → #1347 → #1333 → #1335 → #1328 → #1332. The validation earlier in this description ran at pre-restack heads; this layer's head isb70c03b5.git range-diffshows=). For every file [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 and this stack share, each SQL line [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 added was checked to be present verbatim at the top head, in place or in the store module it moved to.cargo fmt --all -- --check,cargo check --workspace --all-targets --all-featuresandcargo arch(0 required findings, with main's layers and table-owner rules), and the ratchet and caps this layer changes are recounted exactly.e90aa284before that rebase; nextest passed 4370 tests again at77fbae1e):cargo clippy --workspace --locked --all-targets --all-features -- -D warningsis clean, andSPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --all-featurespassed 4355 tests across the store, api, internal-api server, cluster-controller server, metrics, db-test-fixtures, common-core, common-svc and portability crates, [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's suites included.scripts/tests/smoke-kind-typed.sh allgave ECW 91/91 and native 42/42, including [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826'snative.forced_drain_re_places_stopped_sandboxand its re-drain and restarted variants. The Direct substrate lane'sallgave ECW 51/51 and native 27/27. No flakes and no reruns.+/-lines and its SQL string literals match its reviewed head (range-diff plus a literal multiset compare).