[ac-pplx]

Adds characterization tests for node publication's writes to the Postgres control-plane sandboxes.lifecycle column and what moves with it. Node publication is the bridge between the two stores: sbx-mgr publishes its node-local SQLite state, and internal-api moves Postgres lifecycle from those reports. Through publish_sandbox_states against the real Postgres fixture, the tests pin which stored lifecycles each projection moves, what it moves them to, and what else changes:

• placement;
• active_node_id and target_node_id;
• the sandbox_k8s_executions row;
• the Postgres sandbox_quarantine_state and sandbox_quarantine_history rows;
• open port exposures.

Every publication in these tests, including the extended convergence sweep, runs in one transaction, as publish_node_state runs it, and rows are read after the commit. The tests stand in for the node by building the PublishedSandboxState it would send; the sbx-mgr SQLite store is not touched.

Tests only. No production code changes. The new module is wired by one mod line in the test file published_state_tests.rs, and the existing convergence sweep in that file is extended.

What the tests cover

Each projection gets:

• one sweep over all 16 SandboxLifecycle variants plus the stored non-variants deleted and cleanup_required;
• for W3, W4, W5 (the suspended report), W7, W8 and W14, a report from a second ready node that the row is not active on, which must leave the row unchanged. W9's node refusal is pinned by the existing node_inventory_only_recovers_sbx_manager_pause_failed_quarantines (tests/published_state.rs), and W2's by publishing_started_sandbox_from_non_active_node_does_not_converge. W6's node guard has no refusing test, here or elsewhere;
• the refusals specific to it.

Assertions read the persisted Postgres rows, not SQL text, logs or metrics.

• node_publication_converges_only_from_its_source_lifecycles (existing) now sweeps k8s rows with a routable execution as well as space rows, for the restoring → started and → stopped convergences.
• failed_report_moves_every_live_k8s_lifecycle_to_failed
• quarantined_report_moves_every_unquarantined_lifecycle_and_records_it pins deleting → quarantined, which the projection documents as intended. cleanup_required is not swept, since no Postgres writer stores it.
• activation_completion_receipt_projects_its_target_outside_the_suspend_window: receipts for another attempt, another generation or another node release nothing and move nothing.
• activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle:
  • target_node_id is seeded NULL so its backfill is observable;
  • one open and one closed port exposure show that only the open one is reset;
  • a generation-8 receipt and another node's receipt are refused.
• suspended_report_projects_only_out_of_running_and_restoring_lifecycles: target_node_id is seeded NULL; space rows carry no execution.
• ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles drives project_ready_suspend_snapshot in a transaction, as finalize_artifact_backup does:
  • a suspending row is projected only when its execution is suspended and its marker names the backed-up artifact;
  • each of those alone is refused;
  • a backup whose source node is not the row's active node is refused.
• recovery_receipt_suspends_only_a_still_quarantined_row
• healthy_report_recovers_only_a_pause_failed_quarantine
• publication_waiting_on_a_row_lock_decides_on_the_committed_row:
  • the cases: a stopped report and an activation-completion receipt behind a node-drain evacuation claim; a quarantined report behind a committed stopped; and a quarantined report behind a committed move of active_node_id to another node with the lifecycle unchanged, as persist_pod_node_placement makes;
  • setup: the publication runs on a connection acquired first, so its backend pid is known; the test polls writer = ANY(pg_blocking_pids(publication)) and bounds both the wait and the join with a timeout;
  • what is pinned: each publication decides on the row the writer commits, and moves the row as usual on rollback.

Inventory: internal-api writers of Postgres sandboxes.lifecycle

publish_node_state → publish_sandbox_states runs W1–W9 in this order inside one transaction, then W12 and W13. Each statement sees the earlier ones' writes; W4 and W5 null active_node_id, so later node-guarded projections miss those rows.

The BEFORE UPDATE triggers on Postgres sandboxes fire on every write:

• they stamp lifecycle_changed_at;
• they clear evacuation_restore_target outside suspending/suspended/restoring;
• they clear the restore lease and preserve_local_artifact_on_restore_failure outside restoring;
• they clear command_ready_pending on a move away from started.

#	Writer (file → entry point)	From-set	Target	Other guards	Moves with it / side effects	Characterized by
W1	published_state.rs absent-row delete	stopped	row deleted	active_node_id = node, managed_by = 'space', published present = false	snapshot storage detached; warm-pool slots, executions and quarantine state removed (cascade); history kept	existing publishing_absent_stopped_space_sandbox_deletes_central_row
W2	… restored convergence	restoring	started	node; marker NULL; no activation-completed receipt; routable execution (k8s)	RestoreCompleted event	extended node_publication_converges_only_from_its_source_lifecycles; existing routability and marker tests
W3	sandbox_quarantine_store.rs consume_activation_completion_receipts	anything but suspending, suspended, quarantined, deleting, deleted, cleanup_required; a created target only on created	receipt target started/paused (placement = 'on_node') or created	node; active execution with the receipt's generation and attempt, a route, the same target, and for started/paused running with a pod IP; no marker guard	the attempt is released even when the lifecycle write is refused	activation_completion_receipt_projects_its_target_outside_the_suspend_window (with attempt, generation and node refusals), interleaving test (see finding 3)
W4	select_activation_rollback_candidates + apply_activation_rollback_candidates	anything but quarantined, deleted, deleting, cleanup_required	receipt target created/stopped/suspended	node; the execution's exact generation and attempt (FOR UPDATE); published status = target	placement = 'unplaced', active_node_id NULL, target_node_id backfilled, command_ready_pending cleared; execution retired suspended or replaced; open port exposures reset; a lost row after selection aborts the publication	activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle (with port exposures, a NULL target_node_id, generation and node refusals)
W5	select_suspended_projection_candidates + apply_suspended_k8s_finalization	started, paused, restoring with a current durable suspend snapshot; suspending with committed or exact node evidence; quarantined through a recovery receipt	suspended	node (FOR UPDATE); no managed_by filter; a restoring row with an unconsumed recovery receipt waits for it	unplaced, active_node_id NULL, an empty target_node_id backfilled from it, artifact marker set or cleared; execution retired suspended; open port exposures reset; warm-pool slots retired; recovery receipts write a recovered_to_suspended history row and delete the quarantine state	suspended_report_projects_only_out_of_running_and_restoring_lifecycles (with node refusal), recovery_receipt_suspends_only_a_still_quarantined_row; existing evidence tests
W6	published_state.rs stopped promotion	started, paused, suspended, stopping, restoring	stopped	node	placement, nodes and execution untouched	extended sweep; interleaving test
W7	… failed projection	anything but failed, quarantined, deleting, deleted, cleanup_required	failed	node, managed_by = 'k8s'; the from-set is read by an unlocked CTE	placement, nodes and execution untouched; failure-episode facet	failed_report_moves_every_live_k8s_lifecycle_to_failed (see finding 1)
W8	… quarantined projection	anything but quarantined, deleted	quarantined	node; the CAS re-checks the claimed lifecycle	Postgres quarantine state upsert and entered_quarantine history	quarantined_report_moves_every_unquarantined_lifecycle_and_records_it (including deleting, which the projection documents as intended), interleaving test
W9	recover_pause_failed_from_inventory	quarantined with an sbx-mgr pause_failed quarantine	reported started/paused	node; exact quarantine id, source and reason (FOR UPDATE)	history recovered_to_*; quarantine state deleted; observed failure reason cleared; a lost CAS aborts the publication	healthy_report_recovers_only_a_pause_failed_quarantine; existing source/reason/node refusals
W10	… activity clocks	— (no lifecycle write)	—	node, one-minute throttle	fires the triggers	existing
W12	status.rs prune_absent_node_sandboxes	anything but deleted, deleting	deleted tombstone	node, managed_by = 'space', absent from a complete census	unplaced, both nodes NULL, lease cleared	existing complete_inventory_marks_missing_space_sandbox_deleted
W13	auto_lifecycle_delete_completions.rs	non-transient live lifecycles, under the receipt's fences	row deleted	execution generation and route, node name, policy fingerprint, quarantine ownership, no creating snapshot	finalize_sandbox_db_delete_tx	existing auto_lifecycle_delete_completions/tests.rs
W14	project_ready_suspend_snapshot (via finalize_artifact_backup)	started, paused, restoring; suspending whose own suspend committed	suspended	the snapshot's source node is the row's active node; ready, durable, suspend-origin	W5's finalization, retiring any active execution	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles (each commit fact alone refused; another source node refused)

Publication also enqueues node reconcile triggers (enqueue_triggers_conn) for new inventory. Those writes go to the Postgres trigger queue and do not touch sandboxes.lifecycle.

Tests → guard → mutation

Each mutation was applied to production code, the whole internal-api package was run against Postgres (316 tests), and the file was restored. The harness ran one mutation at a time with its own backup file. The table lists every test that failed.

Four mutations survive by construction, because production repeats the guard in a statement that runs first or re-checks the row:

• Q4 drops quarantined from the rollback apply's exclusion. The rollback select already excludes it, so the apply never sees such a row. Q5, the select-side mutation, is killed.
• P9 drops the quarantined projection's node guard from its claim, a plain SELECT without a row lock. The CAS repeats the guard on the row version it writes. P10, which drops the CAS's copy, is killed by the interleaving case in which active_node_id moves while the report waits. P13 drops both copies and is also killed.
• Q12 and Q13 each drop the rollback's generation fence from one of its two statements (the select and the retire). Q16 drops it from both and is killed.

#	Mutation (reverted after the run)	Tests that failed (whole package run)	Result
P1	published_state.rs: failed: managed_by = 'k8s' dropped	failed_report_moves_every_live_k8s_lifecycle_to_failed, publishing_direct_failed_sandbox_does_not_mark_central_lifecycle_failed	killed
P2	published_state.rs: failed: quarantined no longer excluded	failed_report_moves_every_live_k8s_lifecycle_to_failed	killed
P3	published_state.rs: failed: reporting-node guard dropped	failed_report_moves_every_live_k8s_lifecycle_to_failed, publishing_failed_sandbox_from_non_active_node_does_not_mark_central_lifecycle_failed	killed
P4	published_state.rs: quarantined: claim no longer excludes quarantined	quarantined_report_moves_every_unquarantined_lifecycle_and_records_it	killed
P5	published_state.rs: quarantined: CAS no longer rechecks the claimed lifecycle	publication_waiting_on_a_row_lock_decides_on_the_committed_row	killed
P6	published_state.rs: restoring -> started: k8s rows excluded (only the k8s extension of the sweep sees it)	healthy_report_recovers_only_a_pause_failed_quarantine, node_publication_converges_only_from_its_source_lifecycles, publishing_owned_started_sandbox_converges_restoring_lifecycle	killed
Q1	sandbox_quarantine_store.rs: completion: suspending no longer excluded	activation_completion_receipt_cannot_overwrite_a_newer_suspend_claim, activation_completion_receipt_projects_its_target_outside_the_suspend_window, publication_waiting_on_a_row_lock_decides_on_the_committed_row	killed
Q2	sandbox_quarantine_store.rs: completion: placement no longer set on_node	activation_completion_receipt_projects_its_target_outside_the_suspend_window	killed
Q3	sandbox_quarantine_store.rs: completion: execution generation fence dropped	activation_completion_receipt_projects_its_target_outside_the_suspend_window	killed
Q5	sandbox_quarantine_store.rs: rollback select: quarantined no longer excluded	activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle	killed
Q6	sandbox_quarantine_store.rs: rollback: target_node_id backfill dropped	activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle	killed
Q7	sandbox_quarantine_store.rs: rollback: execution always retired as replaced	activation_rollback_receipts_converge_stable_targets_with_attempt_fence, activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle	killed
Q8	sandbox_quarantine_store.rs: pause-failed recovery select: lifecycle guard dropped	healthy_report_recovers_only_a_pause_failed_quarantine	killed
X1	sandbox-portability lib.rs: suspended projection: from-set widened (sandbox-portability, shared by the publication and ready-backup projections)	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles, suspended_report_projects_only_out_of_running_and_restoring_lifecycles	killed
P7	published_state.rs: ready-backup projection: committed-suspend evidence never matches	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles, finalize_suspend_artifact_projects_committed_suspend	killed
Q9	sandbox_quarantine_store.rs: recovery receipt: lifecycle = 'quarantined' dropped	recovery_receipt_suspends_only_a_still_quarantined_row	killed
Q4	sandbox_quarantine_store.rs: rollback apply: quarantined no longer excluded (shadowed by the select, which excludes it first)	—	survived (expected: shadowed, see above)
P8	published_state.rs: quarantined: deleting excluded	quarantined_report_moves_every_unquarantined_lifecycle_and_records_it	killed
P9	published_state.rs: quarantined: claim node guard dropped (the CAS re-checks it)	—	survived (expected: shadowed, see above)
P10	published_state.rs: quarantined: CAS node guard dropped	publication_waiting_on_a_row_lock_decides_on_the_committed_row	killed
P11	published_state.rs: ready-backup projection: source-node = active-node guard dropped	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles	killed
P12	published_state.rs: ready-backup projection: committed-artifact evidence always matches	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles, finalize_suspend_artifact_without_committed_marker_preserves_suspend	killed
Q10	sandbox_quarantine_store.rs: completion: reporting-node guard dropped	activation_completion_receipt_projects_its_target_outside_the_suspend_window	killed
Q11	sandbox_quarantine_store.rs: rollback select: reporting-node guard dropped	activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle	killed
Q12	sandbox_quarantine_store.rs: rollback select: generation fence dropped (the retire repeats it)	—	survived (expected: shadowed, see above)
Q13	sandbox_quarantine_store.rs: rollback retire: generation fence dropped (the select repeats it)	—	survived (expected: shadowed, see above)
Q14	sandbox_quarantine_store.rs: rollback: port-exposure reset no longer limited to open exposures	activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle	killed
Q15	sandbox_quarantine_store.rs: suspended projection: reporting-node guard dropped	suspended_report_projects_only_out_of_running_and_restoring_lifecycles	killed
X2	sandbox-portability lib.rs: suspended finalization: target_node_id backfill dropped (sandbox-portability)	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles, suspended_report_projects_only_out_of_running_and_restoring_lifecycles	killed
X3	sandbox-portability lib.rs: suspended projection: suspending row no longer needs its execution suspended (sandbox-portability)	ready_backup_projects_suspended_only_out_of_running_and_restoring_lifecycles	killed
P13	published_state.rs: quarantined: node guard dropped from both the claim and the CAS	quarantined_report_moves_every_unquarantined_lifecycle_and_records_it	killed
Q16	sandbox_quarantine_store.rs: rollback: generation fence dropped from both the select and the retire	activation_rollback_receipt_converges_its_target_out_of_every_live_lifecycle	killed

Findings

Reproduced against the Postgres fixture with throwaway tests and not pinned. Neither is fixed here.

1. A failed report can overwrite a lifecycle committed while it waits for the row lock.
   • Cause: W7 reads its from-set, its node guard and managed_by = 'k8s' in a CTE without a row lock, then updates WHERE s.id = claimed.id. After a lock wait, Postgres re-checks only the UPDATE target's own conditions, so none of the CTE's filters is re-checked.
   • What it overwrites: a concurrently committed deleting, a concurrently committed quarantined, and a row a concurrent suspend finalization left suspended and unplaced (active_node_id NULL). All become failed.
   • Contrast: the quarantined CAS re-checks its claimed lifecycle and holds in the same race, which the interleaving test pins.
   • Window: narrow. Publication runs at READ COMMITTED, and the API's delete claim holds its row lock until it commits.
   • Damage: usually transient, because the delete finalize ignores the lifecycle. But if teardown fails, the delete's compensating WHERE lifecycle = 'deleting' misses, the API returns 500, and the row stays failed.
   • Reproduction: hold UPDATE sandboxes SET lifecycle = 'deleting' (or 'quarantined', or 'suspended' with active_node_id = NULL, placement = 'unplaced') uncommitted on a started k8s row. Publish a failed report from its node. Wait until pg_blocking_pids shows the publication blocked, then commit. The row ends failed.
2. An activation-completion receipt clears the evacuation marker of a node-drain cohort restore.
   • Cause: W2 deliberately spares an evacuation-marked restoring row, because node drain owns it. W3 has no marker guard: the drain's restore activation completes with a receipt whose target equals the drain's target, so W3 moves the row to that target and the trigger clears evacuation_restore_target. The final lifecycle is right.
   • Harm: the drain's finalize (WHERE lifecycle = 'restoring') then misses. It emits LifecycleTransitionCasLost and reports a restore failure, and it skips the post-restore port-exposure reconcile. The row also drops out of the restore-pending scan.
   • Reproduction: set evacuation_restore_target = 'paused' on a restoring row with a matching activation attempt targeting paused, and publish the completion receipt. The row becomes paused with no marker. The same row with a plain started report stays restoring.

Not a defect: a quarantined report moves a deleting row to quarantined. The projection documents it ("Quarantine is a stronger hold than cleanup/delete intent … Deleted still wins"), and a later delete re-claims deleting from any lifecycle, so the delete intent is not lost. It is pinned. cleanup_required is unreachable in Postgres (only sbx-mgr's SQLite stores it), so the quarantined sweep leaves it out.

Pre-existing: cargo nextest run -p pplx-space-internal-api-server does not compile on its own. data_plane_sessions/tests.rs uses tokio::runtime::Builder::start_paused, whose test-util feature the crate's dev-dependencies do not enable; it compiles in workspace runs through feature unification. Package runs here therefore also select pplx-space-cluster-controller-server for the feature and filter to package(pplx-space-internal-api-server).

Not characterized here

These keep their existing tests and are not swept by lifecycle here:

• W1 absent-row delete;
• W12 census prune;
• W13 auto-lifecycle delete completions;
• the W5 node-evidence suspending branch;
• W10 activity clocks.

Host safety

The tests use the Postgres fixture only. Publication is called in process with built reports. No sockets, node-api clients, host paths, token files or processes are involved, on the normal or the mutation runs.

Overlap with open pull requests

• Of node-mgr: validate LoadSnapshot requests before staging host state #269, sbx-mgr: keep quarantined rows through inventory and bound recovery that can never succeed #270, sbx-mgr: characterize stop, delete and create core RPC paths (tests only) #271, api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273, docs: localized AGENTS.md for sbx-mgr, node-mgr and the api crate #275, api: release a Kubernetes create's claimed execution on every early exit #277 and tests: keep three tests off host state they don't own #278, only sbx-mgr: keep quarantined rows through inventory and bound recovery that can never succeed #270 is open, and it touches sbx-mgr only.
• Prepare tenant publication grants for rolling upgrades #165 also edits published_state_tests.rs, in a one-line hunk near the top. This PR edits the convergence sweep in the middle of the file and appends the module line at its end.
• [sbx-mgr] record the capture host CPU on snapshots and publish it as a node label #203 and [api] restore full snapshots only onto CPU-compatible hosts, behind a default-off gate #176 edit published_state.rs, and fix(sandbox): wait before retrying failed activations #3 edits sandbox_quarantine_store.rs and tests/published_state.rs. This PR touches none of those files.

Validation

Check	Result
ci/scripts/run-rust-checks.sh fmt (env contract, error policy, cargo fmt --check)	pass
cargo clippy --locked --workspace --all-targets --all-features -- -D warnings	pass
cargo nextest run --locked -p pplx-space-internal-api-server -p pplx-space-cluster-controller-server --retries 0 -E 'package(pplx-space-internal-api-server)', SPACE_CI_REQUIRE_TEST_DEPS=1, three consecutive runs	316 passed, 0 skipped, each run
pre-commit (gitleaks) over the branch	pass
scripts/source_guards/{env_test_hook_boundary,node_selector_extras_smoke,observability_vendor_boundary}.py	pass
scripts/check_no_inline_rust_test_modules.py, scripts/source_guards/thin_service_mains.py	fail, PRE-EXISTING: they flag observability/src/{otlp_endpoint,selection}.rs and apps/vol/mgr/svc/src/main.rs, which this PR does not touch
Files vs the merge base	only published_state_tests.rs and the new published_state_tests/lifecycle_writers.rs
Mutation table above	32 mutations: 28 killed, 4 expected survivors (shadowed guards, each covered as described above)
Kind	not run: tests-only change

AI-assisted: written with Claude Code; every result above was produced by the commands listed.

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 4 findings

Low

1. crates/libs/internal/api/server/src/published_state_tests/lifecycle_writers.rs:996 — the (Report::Quarantined, true) arm expects stopped for Concurrent::EvacuationClaim too, and that's the wrong oracle.
   No case reaches it today, so it's dead code. I added (Report::Quarantined, "started", Concurrent::EvacuationClaim) locally and the row ended suspending with the marker, because W8's CAS refuses the claimed started. I'd add that case with the right expectation, since it pins W8 against an evacuation claim too, or make the arm unreachable!().
2. crates/libs/internal/api/server/src/published_state_tests/lifecycle_writers.rs:803 — the name and docstring of healthy_report_recovers_only_a_pause_failed_quarantine say "only" sbx-mgr's pause_failed, but every cell seeds a sbx_manager/pause_failed quarantine.
   So the source/reason refusal can't fail here. As the PR body says, it lives in node_inventory_only_recovers_sbx_manager_pause_failed_quarantines. Either narrow the docstring to the lifecycle sweep, or seed one non-pause_failed quarantine on a quarantined row.
3. crates/libs/internal/api/server/src/published_state_tests/lifecycle_writers.rs:863 — if the spawned publication fails before it blocks, wait_until_blocked times out after 10s with "publication not blocked", and the publication's own error is lost.
   The ? at line 975 returns before the JoinHandle is awaited. A tokio::select! over the poll and the handle would report the real cause.
4. crates/libs/internal/api/server/src/published_state_tests.rs:578 — W6 still has no foreign-node refusal, as the body notes.
   The helper is already here, so it's one call: assert_other_node_report_is_inert(pool, id, published_sandbox_state(id, "stopped")) on a started row.

Verified against 6e526a34d5a43b6dc988208f0edbfd61867b91c0. Reproduced locally: the eight new tests plus the extended convergence sweep, against the Postgres fixture (all pass). Then three mutations: dropping the recovery receipt's lifecycle = 'quarantined' guard (started/paused cells then move to suspended, so the guard is what refuses them), dropping the suspended projection's node guard (killed by the foreign-node check), and the extra interleaving case above.

[the-gigi-pplx]

Approved. 4 non-blocking findings in the review comment above.