Repository navigation
cluster-controller: characterize the lifecycle writers (tests only) - #323
Conversation
92d7d97 to
6550b0a
Compare
Pin, against the real Postgres fixture, which stored values of Postgres `sandboxes.lifecycle` the transient-lifecycle sweeper, node drain and the orphaned-sandbox reconciler move, what they move them to, and the columns that move with them, so a later change to lifecycle ownership has an executable spec. No production code changes; the only edits to production files are the `#[cfg(test)] mod` lines that wire the new modules and their shared fixtures (`lifecycle_writer_test_support`). Each writer gets one sweep over every `SandboxLifecycle` variant plus the stored non-variants `deleted` and `cleanup_required`, with the inputs production can pass. Refusals no existing test pins sit beside an accepting control; refusals an existing query-level test already pins are not repeated. - Transient-lifecycle sweeper, through `SubstrateSweepSink` over the direct substrate: the aged fail CAS and its undo, the restoring converge, revert-to-suspended and its undo, the unrestorable-restoring fail (with its node-column refusals), and the command-ready-pending reap (with its marker guard), its undo (with its write token) and heal. - Node drain, through `NodeDrainSink` on `SubstrateSink` with its in-process suspend: the evacuation claim and its residency and drain-phase guards, the revert, node drain's own commit after the Kubernetes substrate's persisted suspend handoff, the stranded-suspend redrive, the cohort restore's deferrals beside the restore-pending listing, mark-lost, and the gone-node give-up. - Orphaned-sandbox reconciler, through `classify_orphan`, for each classification guard, including the forgotten guard's latest-generation fence, the pod guard's pod-name fence and the young-create window. - Row-lock interleavings: the sweeper's fail CAS and the evacuation claim, each on a one-connection pool whose backend is polled through `pg_blocking_pids` under a timeout, decide on the row a concurrent writer commits and claim when it rolls back. `evacuation_reverts_claim_when_suspend_fails_before_commit` inserted its node without a forced drain, so the claim was refused before the suspend and the revert never ran. Its node is now forced draining, and the test asserts the suspend ran once. Every guard was mutation-checked against the whole package: dropping the predicate or widening the from-set failed the pinned test, splitting a CAS into an unlocked guard plus an update by id or adding SKIP LOCKED failed only the interleaving test, and each mutation was reverted.
6550b0a to
28f6226
Compare
Review — 4 findingsMedium
Low
Verified against |
the-gigi-pplx
left a comment
the-gigi-pplx
left a comment
There was a problem hiding this comment.
Approved. 4 non-blocking findings in the review comment above.
Adds characterization tests for the cluster-controller's writers of the Postgres control-plane
sandboxes.lifecyclecolumn and the columns that move with it. The tests pin, for each writer below, which stored lifecycles it moves, what it moves them to and what else changes, so a later change to lifecycle ownership has an executable spec. Nothing here touches the node-local sbx-mgr SQLitesandboxestable.Tests only. No production code changes. The only edits to production files are four
#[cfg(test)] modlines:lib.rs: the shared fixture modulelifecycle_writer_test_support;transient_lifecycle_sweeper/mod.rs,node_drain/store.rsandorphaned_sandboxes.rs: onelifecycle_writer_testsmodule each.One existing test is corrected (see below).
What the tests cover
Each characterized writer gets one sweep over all 16
SandboxLifecyclevariants plus the stored non-variantsdeletedandcleanup_required, with the inputs production can pass. Refusals not already pinned elsewhere sit beside an accepting control. Refusals already pinned by an existing test are not repeated; the inventory names that test, and the mutation table shows it failing. Assertions read the persisted Postgres rows, not SQL text, logs, metrics or the error taxonomy.SubstrateSweepSinkover the direct substrate (managed_by = 'space'):startedrow), its undo (including a later write onfailed) and heal.NodeDrainSinkonSubstrateSinkwith its existing in-process suspend (SubstrateSink::with_suspend_execution):started/paused, with residency and drain-phase refusals;SubstrateActionSink::classify_orphanfor eachClassifyGuard:suspended;sandbox_quarantine_staterow and theentered_quarantinehistory row;deletingrow;pg_blocking_pidsfor that backend, and bounds the wait and the join with a timeout.SELECTplus anUPDATEby id (I1, I3), or taking its lock withSKIP LOCKED(I2, I4), fails only the interleaving test. The package's other 954 tests pass.Shared fixtures (
stored_lifecycles, the sandbox, node, execution and snapshot inserts, the one-backend pool and the bounded lock wait) live inlifecycle_writer_test_support.Changes to existing tests
node_drain::store::tests::evacuation_reverts_claim_when_suspend_fails_before_commitinserted its node with no drain phase or mode. The claim requires a forced drain, so it was refused, and the assertions held without the revert running: with the revert disabled (R1) the test still passed. Its node is nowdraining/forced, and it asserts that the suspend ran once, so a refused claim fails it and R1 fails it.Inventory: cluster-controller writers of Postgres
sandboxes.lifecycleAll writes go to the Postgres control-plane store. The
BEFORE UPDATEtriggers on Postgressandboxesfire on every one:sandboxes_lifecycle_changed_atstampslifecycle_changed_at.sandboxes_clear_evacuation_restore_targetclears the marker on a move outsidesuspending/suspended/restoring, and resets the restore backoff whenever the marker is NULL.sandboxes_clear_restore_leaseclears the restore lease andpreserve_local_artifact_on_restore_failureoutsiderestoring.sandboxes_clear_command_ready_pendingclears that flag on a move away fromstarted.No trigger enqueues reconcile work or sends
NOTIFY; the crate onlyLISTENs, onspace_evacuation_restore_ready. "New" means this PR's tests; "existing" names the test that already pins a guard, and the mutation table below shows it failing.transient_lifecycle_sweeper/store.rsfail_candidate_query→SweepSink::fail_aged_candidatecreating, starting, pausing, resuming, suspending, restoring, stopping, checkpointing)failedmanaged_by= substrate; age; marker NULL unlesssuspending; a markedsuspendingrow is spared while recoverable (a published local artifact withactive_node_idNULL, or a current-cyclecreating/readysnapshot);restoringneeds the lease quiet for a full age windowpreserve_local_artifact_on_restore_failure; returns a write tokenr1, substrater4, markedrestoringr6, current-cycle snapshotr7, leaser13/pu9undo_fail_aged_candidate_query→undo_fail_aged_candidatefailedmanaged_byan_age_out_is_not_undone_where_the_undo_would_be_lossy)converge_restoring_query→converge_restoringrestoringk8s),managed_by; the CAS carries no evacuation exemption (the candidate select applies it)preserve_local_artifact_on_restore_failurer10, routabilityr18revert_to_suspended_query→revert_to_suspendedrestoringsuspendedmanaged_bypreserve_local_artifact_on_restore_failureclearedr11/r15, leaser13undo_revert_to_suspended_query→undo_revert_to_suspendedsuspendedrestoring)managed_bypreserve_local_artifact_on_restore_failure, not the leasethe_revert_undo_only_matches_its_own_writefail_unrestorable_restoring_query→fail_unrestorable_restoringrestoringfailedactive_node_idandtarget_node_idNULL; no active execution; no execution with a node; no retry-source snapshot; no in-flight upload;placement_changed_ataged;managed_by; deliberately no lease guardpu1–pu9REAP_COMMAND_READY_PENDING_SQL→reap_command_ready_pendingstartedwithcommand_ready_pendingfailedmanaged_bycommand_ready_pendingundo_reap_command_ready_pending_query→undo_reap_command_ready_pendingfailedstartedmanaged_bycommand_ready_pendingbackHEAL_COMMAND_READY_PENDING_SQL→heal_command_ready_pendingmanaged_by, flag setcommand_ready_pendingREPAIR_FAILED_ACTIVE_EXECUTION_SQL→repair_failed_active_executionfailedstartedmanaged_by = 'k8s',placement = 'on_node', active executionr16(query level; it never matches through the direct substrate)node_drain/store.rsclaim_evacuation_suspend→NodeDrainSink::evacuate_sandboxstarted/paused, orsuspendingalready marked for that lifecyclesuspending, marker = listed lifecycleactive_node_id= node; nodedrain_phase = 'draining'anddrain_mode = 'forced'evacuation_claim_refuses_a_node_that_turned_gentlerevert_evacuation_suspend_claim(suspend failed before the node commit)suspendingsuspended; no node or marker guardevacuation_reverts_claim_when_suspend_fails_before_commit(fixed here)finish_suspended_executionin the api crate (the Kubernetes substrate's suspend)suspending/suspendedwith the published artifactsuspendedplacement = 'unplaced',active_node_idNULL,command_ready_pendingcleared, artifact marker written, execution retiredsuspended, open port exposures reset;target_node_idkeptevacuate_sandboxsuspendingsuspendedevacuation_after_the_suspend_handoff_leaves_the_handed_off_rowredrive_stranded_suspending_query→redrive_stranded_suspending_evacuationssuspendingsuspendedactive_node_idNULL; a local artifact, or for a row without one a current-cycle ready snapshotredrive_legacy_rows_require_ready_snapshot,redrive_skips_when_only_prior_cycle_ready_snapshot,redrive_waits_for_current_upload_over_stale_readyrestore_evacuated_and_finalize→restore_evacuated_sandboxrestoring(afterrestore_evacuated_executionmovessuspended→restoringin the api crate)started/paused,placement = 'on_node', marker NULLenqueue_sandbox_reconcile(Postgresnode_reconcile_queue, sandbox desired state)schedule_restore_retry_query→schedule_restore_retryrestore_attempt_count,next_restore_attempt_atdurable_upload_gate_testsmark_sandbox_lost_query→mark_sandbox_lostsuspendedfailedtarget_node_id= node; no copy durable for source release; no in-flight upload of the current suspendmark_sandbox_lost_waits_for_inflight_uploadmark_sandbox_lost_on_gone_node_query,mark_sandbox_failed_on_node_query,abandon_stale_resident_sandboxes_query, the two remote-snapshot adoptionssuspended;started/paused; the non-evacuable resident setfailed(adoptions: no lifecycle change)durable_upload_gate_testsfail_resident_sandboxes_on_gone_node_query→fail_resident_sandboxes_on_gone_nodecreating, starting, started, pausing, paused, resuming, stopping, suspending, restoring, checkpointingfailedactive_node_id= node; markedrestoringspared; markedsuspendingspared while recoverable; node draining with the observedk8s_node_uidgone_node_bulk_fail_flips_dead_residents_onlyorphaned_sandboxes.rsrecover_orphan_to_suspended→classify_orphanfailed, quarantined, cleanup_required, deleted, plus each guard's set. Pod guard: the active execution with the observed pod name and UID, and eitherstarted, paused, resuming, pausing, stoppingwithe.state NOT IN ('creating','recovering')or an execution updated 300 s ago, orcreating, startingwith an execution updated 300 s ago. Forgotten guard:creating, starting, started, paused, resuming, pausing, stoppingwhose latest execution isforgotten.suspendedcreatingsnapshotplacement = 'unplaced', both node columns NULL, active execution retiredsuspended. Not asserted here:placement_changed_atand marker clearing, port-exposure reset, reconcile-ref deletioncreating_snapshot_defers_dispositionquarantine_orphan→classify_orphanquarantinedquarantined, Postgressandbox_quarantine_state.entered_from_lifecycleand oneentered_quarantinehistory row. Not asserted here: port-exposure reset, reconcile-ref deletionfinalize_sandbox_db_delete_tx→classify_orphancreatingwithout a current snapshot; legacydeletingorphaned_sandboxes/quarantine_recovery/store.rsrecoverquarantined(publication-failed quarantine)suspendedquarantine_recovery/tests.rsNot characterized here, with their existing per-module Postgres tests:
started_sandbox_repair(started→suspended);failed_restorable_repair(failed→suspended);drained_node_terminator(suspended→failed);suspended_execution_reaper(placement only);tombstone_retention(deletesdeletedrows);The Direct substrate does not reach node drain: the reconciler builds
SubstrateSinkonly from aKubernetesSubstrate, and its node scans aremanaged_by = 'k8s'. Direct's ownsuspend_executionwrites no Postgressandboxes.lifecycle: it records the artifact marker and clears the direct route identity (clear_direct_route_identity).Inputs not pinned (defensive, unreachable)
evacuate_sandboxandrestore_evacuated_sandboxwith a non-evacuable listed lifecycle or target: the only callers filter onEVACUABLE_LIFECYCLES.suspendingorrestoringage-out: the reconciler never requests it.restoring: the reconciler only revertsrestoringcandidates.Tests → guard → mutation
Each mutation was applied to production code, the whole cluster-controller package was run against Postgres (955 tests), and the file was restored. The harness ran one mutation at a time with its own backup file. The table lists every test that failed, so a guard pinned by an existing test shows that test.
How to read it:
Run predicate by predicate, the fail CAS's interleaving test fails only for the lease predicate (S6). A committed lifecycle move is refused both by
lifecycle = $2and by the age window that move resets, so neither alone breaks it. What only the interleaving tests catch is the recheck itself: I1–I4 each fail exactly one test, the matching interleaving test, while the other 954 pass.S12 survives by design. It drops the revert's
lifecycle = 'restoring'hard pin, which only matters for a listed lifecycle other thanrestoring, and no caller passes one (see the defensive inputs above).transient_lifecycle_sweeper/store.rs: fail CAS: lifecycle = $2 widened to every sweepable lifecyclefail_cas_claims_only_the_lifecycle_it_was_listed_undertransient_lifecycle_sweeper/store.rs: fail CAS: managed_by predicate droppedr4_candidate_select_scoped_to_substrate_managed_bytransient_lifecycle_sweeper/store.rs: fail CAS: marked-restoring exemption droppedr6_evacuation_exemption_spares_restoring_but_not_suspendingtransient_lifecycle_sweeper/store.rs: fail CAS: age window zeroedr13_revert_and_fail_cas_respect_restore_lease,r1_candidate_select_and_fail_cas_leave_drifted_rows_alone,r20_stranded_stopping_row_without_a_node_is_selected_fenced_and_failedtransient_lifecycle_sweeper/store.rs: fail CAS: marked suspending with recoverable state no longer sparedfail_cas_clears_the_moved_columns_and_spares_a_published_evacuation,r7_suspending_candidates_guard_inflight_evacuation_uploadtransient_lifecycle_sweeper/store.rs: fail CAS: restore-lease quiet window droppedfail_cas_waiting_on_a_row_lock_decides_on_the_committed_row,pu9_aged_fail_cas_defers_on_lease_where_unrestorable_cas_claims,r13_revert_and_fail_cas_respect_restore_leasetransient_lifecycle_sweeper/store.rs: fail undo: from-set widenedfail_undo_reverts_only_its_own_failed_writetransient_lifecycle_sweeper/store.rs: fail undo: write-token predicate droppedfail_undo_reverts_only_its_own_failed_writetransient_lifecycle_sweeper/store.rs: converge CAS: from-set widenedconverge_cas_moves_only_restoring_rows,r10_converge_cas_respects_restore_leasetransient_lifecycle_sweeper/store.rs: converge CAS: restore-lease guard droppedr10_converge_cas_respects_restore_leasetransient_lifecycle_sweeper/store.rs: converge CAS: age predicate droppedr10_converge_cas_respects_restore_leasetransient_lifecycle_sweeper/store.rs: revert CAS: lifecycle = 'restoring' pin droppedtransient_lifecycle_sweeper/store.rs: revert CAS: snapshot predicate droppedr11_revert_cas_requires_ready_snapshot_with_artifact_identity,r15_revert_cas_refuses_late_branch_restore_artifacttransient_lifecycle_sweeper/store.rs: revert undo: from-set widenedrevert_undo_reverts_only_its_own_suspended_writetransient_lifecycle_sweeper/store.rs: unrestorable fail CAS: from-set widenedunrestorable_fail_cas_moves_only_unplaced_restoring_rows_of_its_substratetransient_lifecycle_sweeper/store.rs: unrestorable fail CAS: managed_by droppedunrestorable_fail_cas_moves_only_unplaced_restoring_rows_of_its_substratetransient_lifecycle_sweeper/store.rs: reap CAS: from-set widenedreap_moves_only_marked_started_rows_and_heal_never_moves_a_lifecycletransient_lifecycle_sweeper/store.rs: heal: also writes lifecyclereap_moves_only_marked_started_rows_and_heal_never_moves_a_lifecycletransient_lifecycle_sweeper/store.rs: heal: managed_by droppedreap_moves_only_marked_started_rows_and_heal_never_moves_a_lifecycletransient_lifecycle_sweeper/store.rs: reap undo: from-set widenedreap_undo_reverts_only_its_own_failed_writetransient_lifecycle_sweeper/store.rs: revert CAS: age window zeroedrevert_cas_moves_only_restoring_rowstransient_lifecycle_sweeper/store.rs: revert CAS: managed_by droppedrevert_cas_moves_only_restoring_rowstransient_lifecycle_sweeper/store.rs: converge CAS: managed_by droppedconverge_cas_moves_only_restoring_rowsnode_drain/store.rs: mark lost: target node droppedmark_lost_fails_only_suspended_rows_without_a_durable_copynode_drain/store.rs: mark lost: durable-copy guard droppedmark_lost_fails_only_suspended_rows_without_a_durable_copynode_drain/store.rs: gone-node give-up: recoverable marked suspend no longer sparedgone_node_bulk_fail_flips_dead_residents_onlynode_drain/store.rs: gone-node give-up: active_node_id residency droppedgone_node_give_up_fails_only_resident_rows_of_the_nodesandbox-portability lib.rs: recoverable-suspend predicate: published-artifact branch dropped (sandbox-portability)abandon_sweep_spares_redrivable_marked_suspend,fail_cas_clears_the_moved_columns_and_spares_a_published_evacuationnode_drain/store.rs: claim: lifecycle = $2 widened to both evacuable lifecyclesevacuation_claims_only_the_listed_lifecycle_or_its_own_marked_suspendnode_drain/store.rs: claim: resume branch no longer requires its own markerevacuation_claims_only_the_listed_lifecycle_or_its_own_marked_suspendnode_drain/store.rs: claim: forced drain mode droppedevacuation_claim_refuses_a_node_that_turned_gentlenode_drain/store.rs: claim: active_node_id residency droppedevacuation_claim_requires_residency_on_a_draining_node,evacuation_claim_waiting_on_a_row_lock_decides_on_the_committed_rownode_drain/store.rs: claim: draining phase droppedevacuation_claim_requires_residency_on_a_draining_nodenode_drain/store.rs: revert: from-set widenedevacuation_revert_moves_only_a_claim_still_suspending_and_uncommittednode_drain/store.rs: revert: committed-execution guard droppedevacuation_revert_moves_only_a_claim_still_suspending_and_uncommitted,evacuation_redrives_pod_retirement_after_suspend_commitnode_drain/store.rs: commit: lifecycle = 'suspending' droppedevacuation_after_the_suspend_handoff_leaves_the_handed_off_rownode_drain/store.rs: redrive: from-set widenedredrive_legacy_rows_require_ready_snapshot,redrive_moves_only_marked_suspending_rowsnode_drain/store.rs: redrive: marker droppedredrive_legacy_rows_require_ready_snapshotnode_drain/store.rs: redrive: active_node_id IS NULL droppedredrive_legacy_rows_require_ready_snapshotnode_drain/store.rs: redrive: legacy ready-snapshot gate droppedredrive_legacy_rows_require_ready_snapshot,redrive_skips_when_only_prior_cycle_ready_snapshot,redrive_waits_for_current_upload_over_stale_readynode_drain/store.rs: restore: eligibility marker check droppedcohort_restore_defers_rows_the_restore_sweep_would_not_listnode_drain/store.rs: mark lost: from-set droppedmark_lost_fails_only_suspended_rows_without_a_durable_copynode_drain/store.rs: mark lost: in-flight upload guard droppedgive_up_path_gated_on_state_and_freshness,mark_sandbox_lost_waits_for_inflight_uploadnode_drain/store.rs: gone-node give-up: from-set widenedgone_node_bulk_fail_flips_dead_residents_only,gone_node_give_up_fails_only_resident_rows_of_the_nodenode_drain/store.rs: gone-node give-up: node uid identity droppedgone_node_bulk_fail_flips_dead_residents_onlynode_drain/store.rs: gone-node give-up: marked-restoring exemption droppedgone_node_bulk_fail_flips_dead_residents_onlytransient_lifecycle_sweeper/store.rs: reap CAS: command_ready_pending guard droppedreap_moves_only_marked_started_rows_and_heal_never_moves_a_lifecycletransient_lifecycle_sweeper/store.rs: unrestorable fail CAS: active_node_id IS NULL droppedunrestorable_fail_cas_moves_only_unplaced_restoring_rows_of_its_substratetransient_lifecycle_sweeper/store.rs: unrestorable fail CAS: target_node_id IS NULL droppedunrestorable_fail_cas_moves_only_unplaced_restoring_rows_of_its_substratetransient_lifecycle_sweeper/store.rs: reap undo: write-token predicate droppedreap_undo_reverts_only_its_own_failed_writeorphaned_sandboxes.rs: forgotten guard: latest-generation predicate droppedclassification_acts_only_on_the_execution_its_guard_namesorphaned_sandboxes.rs: forgotten guard: latest execution need not be forgottenclassification_acts_only_on_the_execution_its_guard_namesorphaned_sandboxes.rs: pod guard: active execution pod-name fence droppedclassification_acts_only_on_the_execution_its_guard_namesnode_drain/store.rs: restore-pending scan: suspended rows dropped (the accepting control)redrive_legacy_rows_require_ready_snapshot,restore_pending_gates_on_next_restore_attempt_at,restore_pending_ignores_stale_windows_when_backoff_disabled,restore_pending_orders_attempt_tiers_then_recency,restore_pending_selects_only_evacuation_marked_ready_sandboxes,restore_pending_waits_for_current_upload_over_stale_snapshot,cohort_restore_defers_rows_the_restore_sweep_would_not_listorphaned_sandboxes.rs: pod guard: live-pod lifecycle set widenedclassification_moves_only_the_lifecycles_its_guard_ownsorphaned_sandboxes.rs: forgotten guard: lifecycle set widenedclassification_moves_only_the_lifecycles_its_guard_ownsorphaned_sandboxes.rs: pod guard: pending-create age droppedclassification_acts_only_on_the_execution_its_guard_namesorphaned_sandboxes.rs: plan: in-flight snapshot deferral droppedcreating_snapshot_defers_dispositionorphaned_sandboxes.rs: plan: abandoned-create delete droppedclassification_moves_only_the_lifecycles_its_guard_owns,abandoned_initial_create_is_finalized_instead_of_quarantined,lifecycle_change_during_cleanup_prevents_database_delete,unconfirmed_cleanup_keeps_abandoned_create_for_retryorphaned_sandboxes.rs: legacy-deleting guard widenedclassification_moves_only_the_lifecycles_its_guard_ownstransient_lifecycle_sweeper/store.rs: fail CAS split into an unlocked guard SELECT plus an UPDATE by idfail_cas_waiting_on_a_row_lock_decides_on_the_committed_rowtransient_lifecycle_sweeper/store.rs: fail CAS takes its row lock with SKIP LOCKEDfail_cas_waiting_on_a_row_lock_decides_on_the_committed_rownode_drain/store.rs: evacuation claim split into an unlocked guard SELECT plus an UPDATE by idevacuation_claim_waiting_on_a_row_lock_decides_on_the_committed_rownode_drain/store.rs: evacuation claim takes its row lock with SKIP LOCKEDevacuation_claim_waiting_on_a_row_lock_decides_on_the_committed_rownode_drain/store.rs: the claim revert disabled (WHERE FALSE)evacuation_claim_requires_residency_on_a_draining_node,evacuation_claim_waiting_on_a_row_lock_decides_on_the_committed_row,evacuation_claims_only_the_listed_lifecycle_or_its_own_marked_suspend,evacuation_revert_moves_only_a_claim_still_suspending_and_uncommitted,evacuation_reverts_claim_when_suspend_fails_before_commitFindings
Behavior that looks wrong, or is redundant, is listed here and not pinned.
suspend_executionfinalizes throughfinish_suspended_execution→finalize_suspended_sandbox_in. That already persists Postgressandboxes.lifecycle = 'suspended', clearsactive_node_id, unplaces the row, writes the artifact marker and retires the execution. Soevacuate_sandbox's ownUPDATE sandboxes SET lifecycle = 'suspended' WHERE id = $1 AND lifecycle = 'suspending'matches no row on a normal drain.sandbox_k8s_executionsand placement "but NOTsandboxes.lifecycle".rows_affectedcan't be checked: doing so would fail every normal drain. The update is a simplification candidate.evacuate_sandboxemitsSuspendCompletedand enqueues the artifact backup whatever the row holds after the handoff.suspended, unplaced row between the handoff and the backup enqueue leaves a success event and a backup request for a row that is no longersuspended. The reachable one is the API's delete claim todeleting. The backup is wasted work for a sandbox being deleted, and the event stream shows a completed suspend after the delete began.abandon_stale_resident_sandboxescannot land in that window, because its lifecycle set excludessuspended. If it fails the row while the suspend is still in flight, the handoff's own lifecycle check refuses, the suspend errors, and evacuation reports the failure.active_node_idis NULL.evacuation_after_the_suspend_handoff_leaves_the_handed_off_rowpins the persisted rows, including that a write landing after the handoff stands.Observations, pinned or noted as today's contract rather than defects:
restoringrows.Not characterized here
These need a
KubernetesSubstrate, and this crate has no injectable seam for it. Their SQL keeps its existing query-level tests.mark_sandbox_failed_on_nodeandabandon_stale_resident_sandboxes, which call the Kubernetes execution delete as soon as they claim;repair_failed_active_execution.Host safety
The tests use the Postgres fixture only.
nodesrow, so the post-suspend backup enqueue resolves no endpoint. That means no node-api connection, no Kubernetes call and no token-file read.Overlap with open pull requests
node_drain/store.rs; this PR adds three lines beside the test-module declarations at the end of that file.lib.rs; this PR adds two lines in the module list.Validation
ci/scripts/run-rust-checks.sh fmt(env contract, error policy,cargo fmt --check)cargo clippy --locked --workspace --all-targets --all-features -- -D warningscargo nextest run --locked -p pplx-space-cluster-controller-server --retries 0,SPACE_CI_REQUIRE_TEST_DEPS=1, three consecutive runsscripts/source_guards/{env_test_hook_boundary,node_selector_extras_smoke,observability_vendor_boundary}.pyscripts/check_no_inline_rust_test_modules.py,scripts/source_guards/thin_service_mains.pyobservability/src/{otlp_endpoint,selection}.rsandapps/vol/mgr/svc/src/main.rs, which this PR does not touch#[cfg(test)] modlinesAI-assisted: written with Claude Code; every result above was produced by the commands listed.
🤖 Generated with Claude Code