[ac-pplx]

Stacked on #418 (on #438, on main). Base: ac/sbxmgr-sw-pr4-create-warm-start @ 6a1c00f. Review #438 and #418 first; this PR is one commit on top of #418. Stack, bottom to top: #438 → #418 → #413 → #416 → #421 → #417 → #414. #372 and #388 have merged to main.

Every state named here is sbx-manager's node-local SQLite sandboxes.state, not the control-plane Postgres public.sandboxes.lifecycle.

This is plan PR 3 (reports/state-write-helper-plan.md §5): it fixes §3 finding 2.

What merging does

Call sites converted

Call site	Before	After	Edge
resume_after_wake_lock, the branch without an activation policy (service/auth_lookup.rs, previously ~812)	require_state_write(update_state(&db, id, Resuming), "begin resident resume")	require_state_write_from(update_state_from(&db, id, &edges::RESUME_BEGIN), "begin resident resume"), returned directly, in the new begin_resident_resume (service/auth_lookup/resume_begin.rs), using #438's shared helper	RESUME_BEGIN: Request, {paused} → resuming (existing, unchanged)

resume_after_wake_lock validates paused with transition(Resuming) before the gate, so paused is exactly what the write legitimately expects.

Other lines named in the task, traced:

• ~639 stable_policy_update_state and ~987 set_auto_lifecycle_policy_exact do not write sandboxes.state. One is a predicate; the other gates an auto_lifecycle_* column update (update_auto_lifecycle_with_state) behind that predicate under the wake lock. Nothing to convert.
• The policy branch, begin_resident_resume_with_policy (db/state_transitions.rs), moves into begin_resident_resume unchanged. It already re-reads paused and the execution generation inside its own IMMEDIATE transaction before its internal update_state(Resuming), so it cannot race. It is left as is: converting that internal call is behavior-preserving, but would grow state_transitions.rs past the cap sbx-mgr: add the persisted state-transition table and update_state_from (no callers) #388 just lowered. It remains for PR 6, which retires update_state.
• No other update_state/update_state_best_effort caller lives in auth_lookup.rs.

Locks, transactions and side columns. The callers' mutex and lifecycle-lock scopes (wake and artifact locks, held by the Resume RPC and the Wake paths) are unchanged. update_state(Resuming) ran as autocommit statements. In autocommit, update_state_from opens its own IMMEDIATE transaction around its guard reads and the UPDATE, which is harmless on the single connection. The policy branch keeps its own IMMEDIATE transaction. Side columns are update_state's: update_state_from clears the same failure reason, restore columns and non-matching receipts.

Race fixed: plan finding 2

The race. A Resume without activation_auto_lifecycle, or a Wake of a paused sandbox without an activation policy, works like this:

1. It reads paused.
2. It runs the pre-wake network policy gate.
3. It writes resuming with the generic writer.

Stop takes neither the wake nor the artifact lock. Its runtime gate admits a sandbox whose fc.pid resolves, which a paused VM's does, and it can settle stopped between steps 1 and 3. The resume then wrote resuming over stopped and asked node-manager to thaw the VM. With the VM gone and unobservable, finish_resident_resume_after_rpc_error quarantined the sandbox as resident_resume_failed.

Now the write refuses any row that is not paused. The resume returns Aborted before any node-manager call, with every row unchanged. The Stop's stopped row stands.

Red/green test. service::auth_lookup::resume_begin::tests::policy_free_resume_writes_resuming_only_over_paused:

• Setup. It drives the real Resume RPC (wake and artifact locks included) on a paused warm-pool member whose runtime columns are populated: the test process's pid, route generation 3, execution generation 7. Nothing on the resume path probes them before the write. A pool member has no network identity, so the gate needs no net-gateway. A test-only task_local barrier (precedent: cleanup_executor.admission_barrier) holds the resume between the gate and the write. Node-manager is a FakeNodeManager that answers only ResumeSandbox, with the error a VM that is gone gets.
• The sweep. While the barrier holds, the row is moved to each SandboxState, or deleted with db::delete_sandbox. The expected outcome comes from a wildcard-free table.
  • stopped is reached by Stop's own writers, sbx-mgr: characterize the state writers before the state-write helper (tests only) #372's documented sequence: update_state_superseding_activation_in_transaction(Stopping) plus set_cleanup_intent(mark_stopped), then complete_row_intent_cleanup_for_job(MarkStopped).
  • Every other state is written directly. The deleted row is the Missing arm.
• What it asserts.
  • paused (the accepting control) resumes: the write applies and node-manager receives ResumeSandbox.
  • Every other case answers Aborted with its exact message, leaves the whole database unchanged (sbx-mgr: characterize the state writers before the state-write helper (tests only) #372's snapshot), and makes no node-manager call.
  • Mismatches are collected across all 19 cases and asserted once, so a failure lists every wrong case.
• Red on sbx-mgr: add the persisted state-transition table and update_state_from (no callers) #388's code (the generic update_state call restored): the 12 states the generic guards admit all fail. A disposable probe restricted to stopped showed the exact finding: the row ends quarantined with resident_resume_failed, node-manager received PrepareSandboxArtifactDir, ResumeSandbox and GetFirecrackerVmState, and the client got Internal "resume failed: …".
• Green with this change.

Behavior changes

What the write finds	Before	After
paused	writes resuming, thaws	unchanged
stopped (the race), and creating, created, starting, started, pausing, resuming, checkpointing, suspending, suspended, restoring, failed, when no cleanup episode owns the row	wrote resuming and tried to thaw. Depending on node-manager's answer the row ended started, rolled back to paused, or was quarantined resident_resume_failed (Internal/Unavailable)	Aborted, begin resident resume: sandbox state write was superseded: sandbox is <state>; no row change and no node-manager call
stopping, deleting, cleanup_required, cleanup_failed, and any row a cleanup episode owns (in production a starting or restoring row whose activation episode keeps the row)	Aborted, begin resident resume: sandbox state write was not applied: superseded_by_cleanup_episode	unchanged (same code and message)
quarantined	Aborted, …: blocked_by_quarantine	unchanged
row deleted	Aborted, …: missing	unchanged

Reachability. Under the callers' wake and artifact locks, the reachable entries in the second row are stopped (this race) and, in principle, rows a Stop and a later cold Start moved on: started, or a starting row, which answers through the superseded_by_cleanup_episode guard path while its activation episode owns it. The other entries are listed because the from-set now refuses them. The test forces those states without episodes, so it sees the from-set answer.

Client-visible changes:

• New sbx-mgr message, for a row outside the from-set: Aborted, begin resident resume: sandbox state write was superseded: sandbox is <state> (sbx-mgr: one refusal-to-status mapping for update_state_from #438's canonical text). It replaces a thaw attempt that ended in Internal, Unavailable or success, depending on node-manager. Every other resume message is unchanged. The policy branch still answers FailedPrecondition "resident execution changed before resume".
• Native API: a native Resume that reaches this branch and loses the race now reports 409 Conflict (Aborted) where it reported 500 (Internal) after quarantining the sandbox. The mapping is http_status_for_grpc_code in crates/libs/api/src/controllers/mod.rs. ECW resume maps to SPACE restore and does not reach this path.
• Wake and implicit activation: the same policy-free branch is reached by Wake and by implicit command/access activation (activation.rs). There, a resume that loses to Stop now returns Aborted instead of quarantining and returning Internal. The data plane treats Aborted as retryable and cold-starts on retry.

Edges, allowlist and #438's helper

• Deleted: RESUME_BEGIN_AFTER_STOP (KnownRace, {stopped} → resuming).
• Added: none. RESUME_BEGIN was reused, and its doc comment now names begin_resident_resume's update_state_from.
• Allowlist: removed (Stopped, Resuming, WriteForbidden, "known race: policy-free resume after Stop"). No other edge writes stopped → resuming, and can_transition_to forbids it, so the pair no longer disagrees. No mixed reason names this race; the remaining race reasons name pause, process-snapshot rollback and template snapshot (sbx-mgr: create completion and warm start write only over the state they expect #418, below this PR, removed the warm-start ones). 79 entries remain.
• Four KnownRace edges remain, so the kind and its any KnownRace assertion stay; sbx-mgr: pause writes only over the state it expects #417, above this PR, retires them.
• sbx-mgr: one refusal-to-status mapping for update_state_from #438's dead_code expectations: sbx-mgr: create completion and warm start write only over the state they expect #418, below this PR, already removed them from the five items this PR reaches through require_state_write_from: the function itself, StateWriteRejected, StateWriteGuard, state_write_result and impl StateWriteRejected (into_status). update_state_from_best_effort keeps its expectation, because this PR does not call it. This PR does not touch cleanup_support.rs.

Other changes

• service/auth_lookup.rs drops from 1242 to 1221 code lines on main's base. It was already over its 1237 archcheck cap; the begin step moved to resume_begin.rs. The cap is lowered to 1221.
• The barrier (BEGIN_BARRIER) is #[cfg(test)] only. Production code waits on nothing new.

Overlaps with the other PRs in the stack

• sbx-mgr: pause writes only over the state it expects #417 (PR 2, pause), above this PR: its deletions of PAUSE_ROLLBACK_AFTER_STOP/PAUSE_COMPLETE_AFTER_STOP sit directly above this PR's deleted edge in db/state_write.rs, and its allowlist deletions ((Stopped, Started), (Stopped, Paused)) sit directly above the removed (Stopped, Resuming) entry. Its rebase onto this PR kept both deletions.
• expect(dead_code) on sbx-mgr: one refusal-to-status mapping for update_state_from #438's items: sbx-mgr: create completion and warm start write only over the state they expect #418 (below) and sbx-mgr: snapshot completion and rollback write only over the state they expect #421 (above) remove them; this PR touches none.
• PR 6 (retire update_state): must still convert begin_resident_resume_with_policy's internal update_state(Resuming) (see above). It also has the other internal db/ callers.
• architecture-baseline.toml: only the auth_lookup.rs line changes.
• No other PR in the stack has call sites in auth_lookup.rs or resume_begin.rs, as far as the plan's split shows.

Mutation proof

M1, M2, M9 and M10 (the lines this round's rebase touched) were re-run on the final head; M3–M8 ran on 5b14c80 (the round-1 head before the final #438 rebase), whose mutated lines the rebase did not change. Each mutation was applied alone from a backup, and every SQL ?N stayed bound. The new test and the state_write_table suite were run, then the file was restored and touched before the next. "Cases" lists the cases the test reported.

#	Mutation	Failing tests (cases)
M1	Restore #388's call: require_state_write(update_state(&db, id, Resuming), …)	policy_free_resume_… (the 12 states the guards admit)
M2	Pass StateSet::ALL (update_state_from_states(ALL, Resuming)) instead of RESUME_BEGIN	policy_free_resume_… (same 12)
M3	Report FromState with the guard's superseded_by_cleanup_episode text	policy_free_resume_… (same 12)
M4	Answer FromState with FailedPrecondition instead of Aborted	policy_free_resume_… (same 12)
M5	Report a quarantine refusal as superseded_by_cleanup_episode	policy_free_resume_… (quarantined)
M6	Widen RESUME_BEGIN to {paused, stopped}	policy_free_resume_… (stopped), persisted_edges_disagree_with_the_admission_machine_only_as_allowlisted
M7	Restore the RESUME_BEGIN_AFTER_STOP edge	persisted_edges_disagree_…
M8	Restore the (Stopped, Resuming) allowlist entry	persisted_edges_disagree_… (stale entry)
M9	Map Missing to Ok in state_write_result	policy_free_resume_… (Missing)
M10	Answer Missing with NotFound (same text) in into_status	policy_free_resume_… (Missing)

All 10 were caught, and all were restored. M3–M5, M9 and M10 mutate #438's shared mapping and show that this PR's test pins the call site's statuses.

#372 / #388 / #438 tests

No #372 matrix, accepted-shape or race expectation changes, no #388 table, equivalence or binding test changes apart from the deleted allowlist entry above, and no #438 test changes. Pins green: paused_suspend_serializes_resume_behind_wake_and_artifact_locks, the auth_lookup tests (service::auth_lookup::tests, auth_lookup::wait_ready_policy_tests, service::auth_lookup_tests), resident_resume_refusal_status_for_every_state, resume_fails_closed_when_network_policy_cannot_confirm and service::cleanup_support::tests.

Decisions made without an operator

• Reused RESUME_BEGIN, whose from-set is exactly {paused}, rather than adding an edge.
• sbx-mgr: one refusal-to-status mapping for update_state_from #438's require_state_write_from as is. Every refusal is Aborted, which is what the plan's row asks for a lost race, so no from.contains(found) branch and no typed StateWriteRejected match is needed.
• A new module instead of growing auth_lookup.rs, which was already over its cap. The begin step, both branches, moved whole.
• task_local barrier instead of a service field. It keeps SandboxManagerService, its constructor and service/mod.rs (also over its cap) untouched.
• Stop at the DB level. The Stop RPC's watchdog removal stats the fixed sidecar pid path (see cleanup/stop_admission_tests.rs), which the host-safety rules forbid in tests, so the test runs the writers the Stop RPC runs, in sbx-mgr: characterize the state writers before the state-write helper (tests only) #372's documented order.
• Not converting begin_resident_resume_with_policy's internal call (see above).

Validation of the restacked stack

The stack was restacked linearly on main (b18051f) after #372 and #388 merged: #438 → #418 → #413 → #416 → #421 → #417 → #414. The per-PR validation further down ran on this PR's pre-restack head.

This PR's head (38e424c): pull CI is green. fmt, workspace clippy --all-targets --all-features -D warnings and cargo arch (0 required findings) all pass. SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0 passed 1639/1639, with 0 dependency skips. Tested at 5cefe54; the final head differs from it only in architecture-baseline.toml, which only cargo arch reads, and cargo arch was rerun on the final head. Kind Pull CI ran on #417, which contains this PR: all 4 Firecracker smoke shards passed.

The top of the stack (#414 @ 9a0a251, which contains this PR):

Check	Result
sbx-mgr nextest ×3	1650/1650 each, 0 dependency skips
Whole-workspace nextest	8985/9021 passed, and every API crate passed. The 36 failures are all outside sbx-mgr. 34 fail identically on main: spaced-guest and spaced-host-cli process tests (the test box can't switch identity) and one net-gateway test. The other 2 are common-grpc retry-deadline tests that pass on rerun on both heads.
The stack's 30 new or changed tests, ×30	900/900
cargo test -p pplx-space-sbx-mgr (shared process)	1650/1650
Mutation spot-check on the top head	#417's quarantine-miss mutation and #418's warm-start admission mutation are both caught
Kind, fresh cluster; the deployed sbx-mgr binary is from this head	smoke-kind-e2e.sh passed. smoke-kind-typed.sh all: native 23/23; ECW 76 passed, 2 infra errors. local.ecw_template_build_from_oci_boots passed, and so did the leader-election brownout.
The 2 ECW infra errors	node_group_boundary_schedules and node_selector_extras_render_and_enforce. Both scenarios restart the API Deployment, and the same failure family appears on main in an A/B on the same cluster. In the failed runs the kill never reached sbx-mgr.
Lifecycle race soak on Kind	11 concurrent operation pairs × 40 iterations, on the stack and on main on the same fresh cluster, checking that sbx-mgr's SQLite sandboxes.state, control-plane Postgres lifecycle and the VM agree. No change-induced violation: across 880 sandboxes, no quarantine, panic, restart or stuck state. Every violation reproduces on main with the same signature: an ECW pause that loses to Stop answers 503 instead of 409; after Suspend vs Delete, the node row can outlive the control-plane row until the next eviction pass (pass timing, not the build); and a refused quarantine write after Pause vs Delete is still logged. The soak's 0–200 ms API-level offsets did not reproduce the interleavings this stack closes, so the unit-level race tests remain the direct evidence for those.

Validation on the pre-restack head

Check	Result
cargo fmt --all -- --check, scripts/check-rust-error-policy.sh	pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings	pass (0 warnings; the five removed dead_code expectations and update_state_from_best_effort's remaining one are all consistent)
cargo arch	pass, 0 required findings. auth_lookup.rs at 1221 (cap lowered 1237 → 1221); resume_begin.rs and cleanup_support.rs under the 1000-line limit; no listed file grew
Gitleaks (pre-commit)	pass on f1d1c9b..HEAD
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0, SPACE_TEST_DEP_SKIP_LOG set	on 5b14c80: 1631/1631 × 3; on this head after the final #438 rebase: 1631/1631 × 1. 0 dependency skips every run; 1 skipped = the #[ignore] privileged transfer_directory::tests::distinct_service_uids_can_exchange_downloads, unrelated
Red/green	red with #388's update_state restored (M1: 12 cases), green with this change
Mutation proof	M1–M10: 10/10 caught, all restored
Public-safety review of the diff and commit text	no findings in the reviewed scope
Kind (round 1, on 5b14c80; the final rebase changed only #438's return type and this call site's ?; Ok(()), so per the owner Kind was not rerun)	fresh cluster (none on the box beforehand), brought up by smoke-kind-e2e.sh on the current harness
Kind: binary provenance	Tilt ran from this worktree; synced source matches; space-sbxmgr-svc in space-rust-build-kind:local and /proc/<pid>/exe in both sandbox-manager pods share sha256 4af260dd…5afc25, and the binary contains sandbox state write was superseded: sandbox is, which production reaches only through this call site. Same hash, 0 restarts after the typed run
Kind: Tilt quiet	every resource ok / not_applicable before the smokes
smoke-kind-e2e.sh	pass
smoke-kind-typed.sh all (CI env incl. SPACE_SMOKE_KIND_COLD_RECEIPTS, unsharded)	ECW 78 passed, native 23 passed; 0 failed, 0 inconclusive, 0 quarantined; 1 IGNORED by design (native_contract.command_replay_credentials); ecw.cold_create_environment passed. No rerun needed
Kind teardown	infra/space/kind/scripts/teardown.sh: Tilt stopped, cluster deleted
CI	see the pull request checks

🤖 Generated with Claude Code

[nikhilgupta-pplx]

Approve. {paused} is exactly what transition(Resuming) admits, the refusal fires before any node-manager call or watchdog edge, and side columns match update_state. Removing RESUME_BEGIN_AFTER_STOP and its allowlist pair is consistent.

Non-blocking (description only): the client-visible section covers the native API 500→409, but the policy-free branch is also reached by Wake and implicit command/access activation (activation.rs). There a resume losing to Stop now returns Aborted instead of quarantining + Internal; the data plane treats Aborted as retryable and cold-starts on retry, so it is an improvement. Worth one line in the body.

[the-gigi-pplx]

Review — 5 findings

The fix does what it says: RESUME_BEGIN is exactly {paused}, Stop's RPC (service/grpc_core.rs:571) takes neither the wake nor the artifact lock, and the policy-free write now refuses a row a concurrent Stop settled instead of thawing a VM that's gone. Nothing below blocks.

Low

1. crates/libs/sbx/mgr/server/src/service/auth_lookup/resume_begin.rs:50 — the same lost-to-Stop race now answers 412 or 409 depending on timing and on whether a policy was sent.
   If Stop lands before resume_after_wake_lock's pre-gate check, transition(Resuming) fails and From<StateTransitionError> (state.rs:431) answers FailedPrecondition (412 on the native API). If it lands during the gate, the policy-free branch now answers Aborted (409) and the policy branch answers FailedPrecondition "resident execution changed before resume" (412). The end state is stopped in all three. A client that branches on 409 vs 412 has to treat both as "the sandbox moved". That's worth either a sentence in the PR's client-visible section or picking one code for the post-gate refusal in both branches.

2. crates/libs/sbx/mgr/server/src/service/auth_lookup/resume_begin/tests.rs:205 — the title covers wake, but the test only drives the Resume RPC.
   Both Wake callers (service/activation.rs:678 blocking and :823 non-blocking) bind the activation operation before resume_after_wake_lock and propagate its error through finish_resident_wake_after_resume. A Wake that loses to Stop now leaves with Aborted, whereas the same Wake arriving a moment later would take the loop's Created | Stopped arm and start the sandbox. Under the repo's wake vocabulary (accept any valid start, resume or restore path), re-reading the row and continuing the loop on this refusal may be the better answer. Either way, one case driving the Wake path through the barrier would pin whichever behavior you pick.

3. crates/libs/sbx/mgr/server/src/service/auth_lookup/resume_begin/tests.rs:258 — the accepting control doesn't check that resuming was written.
   Expected::Resumes matches Err(Code::Internal, _) plus a ResumeSandbox call. The fake answers NotFound whether or not the row moved to resuming first, so a change that skips the write on the paused path and still calls node-manager satisfies it. I ran that mutation and the test passes. Asserting the row's state at the point node-manager is called (the fake's handler can read it), or the post-failure row, would make the control load-bearing.

4. crates/libs/sbx/mgr/server/src/service/auth_lookup/resume_begin.rs:28 — begin_resident_resume is async only for the #[cfg(test)] barrier.
   In production its body has no await point. That's harmless today, but the signature suggests an async dependency the write doesn't have. Hooking the barrier at the call site in resume_after_wake_lock would let the function stay synchronous.

5. PR description — it cites reports/state-write-helper-plan.md (§3 finding 2, §5, "PR 6"), which isn't in the repository, so a reader can't follow the plan references. Inlining the one-line finding it fixes would make the description self-contained.

What I checked and found sound: the pre-gate transition(Resuming) admits only paused, so RESUME_BEGIN is exactly what the write expects. Both Wake paths and the Resume RPC hold the artifact lock across the gate, and Pause takes it (grpc_lifecycle/pause_resume.rs:163), so a Stop → Start → Pause back to paused with a new execution can't land in the window, and the policy-free branch doesn't need a generation fence. The node-api simulator answers a non-paused resume with FailedPrecondition, which matches real sbx-mgr when the row is already non-paused at the pre-gate read. Removing RESUME_BEGIN_AFTER_STOP and its allowlist entry is consistent: nothing else writes stopped → resuming.

Verified against 38e424cfe69c. Reproduced locally: cargo clippy -p pplx-space-sbx-mgr --all-targets --all-features -D warnings (clean) and cargo nextest run -p pplx-space-sbx-mgr -p pplx-space-db-test-fixtures --all-features (1650 passed, 0 failed) at this head; for finding 3, a mutation that returns Ok(()) without writing when the row is paused leaves policy_free_resume_writes_resuming_only_over_paused passing.

[the-gigi-pplx]

Approved. 5 non-blocking findings in the review comment above.