[ac-pplx]

Node publication (PublishNodeState) projects each node's report onto the control-plane Postgres public.sandboxes row. Other writers in api, internal-api and cluster-controller share that row. This PR makes the failed projection decide on the row it actually writes. sbx-mgr's node-local SQLite sandboxes table (state) is not involved. There is no schema change and no new metric.

The failed projection overwrote a lifecycle committed while it waited

Mechanism. The projection read its candidates in an unlocked CTE (claimed: the reporting node's k8s rows outside failed, quarantined, deleted, cleanup_required, deleting). It then ran UPDATE sandboxes s SET lifecycle = 'failed' FROM claimed WHERE s.id = claimed.id. If another transaction held the row, for example a user DELETE that takes FOR UPDATE and commits deleting, the UPDATE waited for it. Under READ COMMITTED, Postgres then re-checks only the UPDATE's own WHERE, here s.id = claimed.id, so it wrote failed over the committed row. The same race overwrote a committed quarantined, and a row that had meanwhile been unplaced as suspended (active_node_id NULL). The SandboxFailed event also reported a stale from-state.

Impact.

• A delete usually survived, because finalize ignores lifecycle. If teardown then failed, the DELETE's compensating update (WHERE lifecycle = 'deleting', crates/libs/api/src/controllers/sandbox/lifecycle.rs) matched nothing. The API returned 500 and the row stayed failed.
• A quarantine hold became failed.
• A suspended, unplaced row became failed.

Fix. The claim now locks before it checks, as the suspended projection does: FOR NO KEY UPDATE OF s in the CTE. A row another writer holds is re-read at its committed version once the lock is granted. The node, managed_by and lifecycle guards are evaluated against that version, and claimed.lifecycle is the lifecycle the UPDATE replaces.

Why this rather than repeating the guards in the UPDATE's WHERE:

• Accurate from-state. With repeated guards, RETURNING claimed.lifecycle would still be the pre-wait value whenever the row changed but still qualified (for example started → paused). Postgres before 18 cannot return OLD.
• Same lock mode, same rows. FOR NO KEY UPDATE is the mode the UPDATE already takes on these rows (lifecycle is in no unique index). The statement locks the same rows in the same mode as before; the lock is only taken earlier, during the claim's scan. Unlike FOR UPDATE, it adds no conflict with the FOR KEY SHARE locks that foreign-key checks take.
• Deadlock review. No new deadlock cycle was found:
  • The other multi-row lockers in the same publish transaction select rows with other published statuses, so none of them holds a failed-status row.
  • The concurrent single-row claims this can wait on lock one row.
  • Node drain's multi-row give-ups use FOR UPDATE SKIP LOCKED.
  • Lock order within the statement is plan-dependent, as it was before. The input is id-sorted.
• Behavior otherwise identical. The projection moved into project_node_published_failed_sandboxes so its result can be tested. The caller emits the same SandboxFailed events and the same warning; only the from-state is now accurate.

crates/libs/internal/api/server/AGENTS.md no longer lists the failed projection as an exception to its recheck-under-lock rule. It now names FOR NO KEY UPDATE as an accepted lock, lists the activation-completion receipt as the existing exception (it rechecks only the lifecycle and is fenced by the execution attempt it releases: generation, attempt, active), and calls the convergence "restoring-to-target".

Test

published_state_lifecycle_race_tests::failed_projection_decides_on_the_row_committed_while_it_waits is a new module, declared by one #[cfg(test)] mod line in published_state.rs, and runs against the real Postgres fixture.

• Setup: a started k8s row and a failed report.
• Interleaving: a conflicting writer holds the row in its own transaction. The test captures the projection's pg_backend_pid() and polls $holder = ANY(pg_blocking_pids($waiter)) until the projection is blocked; both that wait and the join are bounded by tokio::time::timeout. Then the writer commits.
• Assertions: the persisted lifecycle and active_node_id, plus the transitions the projection returns.

Concurrent commit	Expected	Main's SQL	This PR
DELETE: FOR UPDATE, then deleting	stays deleting, nothing reported	became failed, reported from started	pass
quarantined	stays quarantined	became failed	pass
suspend that unplaced the row	stays suspended, active_node_id NULL	became failed	pass
paused (control: still qualifies)	becomes failed, reported as leaving paused	reported from started	pass

Mutation. M1 drops FOR NO KEY UPDATE OF s, which leaves exactly main's statement. The test fails in all four cases.

The test's row and interleaving helpers duplicate helpers in #324's lifecycle_writers.rs. Once both land, these cases fold into #324's interleaving table as a failed report.

Not fixed here: node drain's evacuation marker

Design input, not part of this change:

• The problem. Three writers can move a restoring row that still carries evacuation_restore_target to started/paused before node drain's finalize runs: the activation-completion receipt, complete_wake_lifecycle, and fenced_completion_cas.
• The consequences. The sandboxes_clear_evacuation_restore_target trigger then clears the marker. Drain's finalize (WHERE lifecycle = 'restoring') misses, reports a restore failure and skips re-pointing port exposures. The row leaves the restore-pending scan, so nothing retries.
• Why a receipt-only patch is not enough. It leaves joiners of drain's activation attempt failing the fence with Unavailable, and it leaves the two wake writers.
• Conclusion. Completing an evacuation restore has no single owner. That should be resolved in the lifecycle redesign rather than by exemptions in each writer.

Interaction with #324

#324 (ac/char-tests-internal-api-lifecycle-writers, checked at 6e526a3) is tests only, and nothing in it breaks with this change. Its revised test files were overlaid on this head in a scratch worktree, without touching this branch. All 26 published_state_tests, including its 9 lifecycle_writers tests, pass.

• failed_report_moves_every_live_k8s_lifecycle_to_failed publishes without a concurrent writer, so locking before checking changes no outcome. The from-set is unchanged.
• Its interleaving table (publication_waiting_on_a_row_lock_decides_on_the_committed_row) has no failed report, so it does not cover this race; the cases above are meant to join it.
• This PR no longer changes the activation-completion receipt, so internal-api: characterize node publication's lifecycle writers (tests only) #324's receipt tests see main's behavior.

This PR does not touch #324's files.

Overlaps

• internal-api: characterize node publication's lifecycle writers (tests only) #324 edits published_state_tests.rs and published_state_tests/lifecycle_writers.rs; this PR edits neither.
• cluster-controller: characterize the lifecycle writers (tests only) #323 (cluster-controller): this PR has no cluster-controller changes (git diff origin/main -- crates/libs/cluster is empty).
• docs: localized AGENTS.md for spaced, net-gateway, cluster-controller, internal-api and vol-mgr #316 (merged) added the internal-api AGENTS.md that this PR amends.
• sbx-mgr: keep quarantined rows through inventory and bound recovery that can never succeed #270 (sbx-mgr) and api: let tests inject the Kubernetes readiness and suspend budgets #330 (api Kubernetes adapter) touch none of these files.

Validation

Head 7dbc89e, on current main.

Check	Result
ci/scripts/run-rust-checks.sh fmt	pass
cargo clippy --locked --workspace --all-targets --all-features -- -D warnings	pass
pre-commit (gitleaks)	pass
cargo nextest run --locked -p pplx-space-internal-api-server -p pplx-space-cluster-controller-server --retries 0, SPACE_CI_REQUIRE_TEST_DEPS=1, three consecutive runs	1245 passed, 0 failed, 0 skipped each. The known pre-existing flake tests::concurrent_begin_artifact_backup_is_idempotent did not fire.
Red/green, M1	red on main's statement in all four cases, green with the fix; M1 killed
#324 at 6e526a3, overlaid	26/26 pass
Kind	not rerun. The failed-projection code and its SQL are byte-identical to bd25eca, whose fresh-cluster run covered it: smoke-kind-e2e.sh passed; smoke-kind-typed.sh all passed ECW 77/77 and native 23/23 with cold receipts set; drain and delete scenarios passed individually; and the deployed internal-api binary was shown to carry this statement.
CI	see the checks on this PR

🤖 Generated with Claude Code