[ac-pplx]

What merging this does

sbx-mgr's fatal Kill (SIGKILL/SIGTERM) is admitted from exactly the node-local SQLite sandboxes.state values that SandboxState::can_transition_to(Stopping) (state.rs) accepts. That rule is widened from started/paused to every state whose VM is live or launching, mid-operation included. Kill still refuses deleting and quarantined.

Stop is unchanged in every one of the 18 states, live and not live, blocking and non-blocking. Stop does not consult the widened rule. Making it do so would change Stop's answer for 7 states (shown below), and the owner decision says Stop must not change. The rule's doc comment explains why Stop's runtime-gated admission is separate.

Everything here is sbx-mgr's node-local SQLite sandboxes.state. The control-plane Postgres public.sandboxes.lifecycle and its API gating are not touched.

Analysis

Target resolution and gates

	Stop (service/grpc_core.rs stop)	Kill (service/grpc_lifecycle/kill.rs)
Target lookup	resolve_lifecycle_target(.., running_only = true) (cleanup_targets.rs): the row for one exact UUID, or a rowless cache directory. Honors the template filter; space-tmplmgr may only target template-owned rows	lookup_sandbox: the row must exist (NotFound otherwise). No template filter, no cache-directory fallback
Live-runtime gate	Yes. running_only selects nothing unless fc.pid resolves. Returns Ok with no stopped_ids and writes nothing	None. Kill reads only the row; a dead or missing runtime does not change its answer
State gate before admission	None	Before: matches!(Started | Paused). The transition(Stopping) call after it had the same set, so it never refused. After: fatal signals use can_transition_to(Stopping); non-fatal signals keep Started | Paused
Cleanup admission	admit_selected_cleanup (unfenced). The supersede writer refuses deleting/quarantined with Unavailable	admit_lifecycle_cleanup_for_observed_execution. Same supersede writer, plus an execution-generation check: Aborted if the generation changed since lookup
Signal	None; cleanup terminates the VM	StopProcess(signal) under the admission fence, then the same MarkStopped cleanup
Intent persisted	Non-blocking only	Non-blocking only

Sweep of all 18 states (pinned by the new tests)

Stop. Every state, live and not live, blocking and not, behaves the same before and after this change:

• Admitted, blocking: stopped_ids = [id]; the row ends stopped with no intent or job left.
• Admitted, non-blocking: stopped_ids = [id]; the row is stopping with the MarkStopped/keep-filesystem intent and an active cleanup job.
• Not live: Ok with no ids, and the whole database is unchanged.
• Refused: Unavailable, and the whole database is unchanged.

Kill. Same answer live or not live, and for both signals 9 and 15:

• Admitted, blocking: StopProcess(signal) once; the row ends stopped.
• Admitted, non-blocking: StopProcess(signal) once; the row is stopping with intent and job.
• Refused: FailedPrecondition, no signal sent, and the whole database is unchanged.

state	Stop, live runtime	Stop, no live runtime	fatal Kill before	fatal Kill after	non-fatal Kill (unchanged)
creating	admitted	nothing selected	refused	refused	refused
created	admitted	nothing selected	refused	refused	refused
starting	admitted	nothing selected	refused	admitted	refused
started	admitted	nothing selected	admitted	admitted	delivered
stopping	admitted	nothing selected	refused	admitted	refused
stopped	admitted	nothing selected	refused	refused	refused
deleting	refused (Unavailable)	nothing selected	refused	refused	refused
pausing	admitted	nothing selected	refused	admitted	refused
paused	admitted	nothing selected	admitted	admitted	delivered
resuming	admitted	nothing selected	refused	admitted	refused
checkpointing	admitted	nothing selected	refused	admitted	refused
suspending	admitted	nothing selected	refused	admitted	refused
suspended	admitted	nothing selected	refused	refused	refused
restoring	admitted	nothing selected	refused	admitted	refused
cleanup_required	admitted	nothing selected	refused	refused	refused
cleanup_failed	admitted	nothing selected	refused	refused	refused
quarantined	refused (Unavailable)	nothing selected	refused	refused	refused
failed	admitted	nothing selected	refused	refused	refused

Stop admits a live runtime under creating/created/stopped/suspended only when a runtime has outlived its row's lifecycle, such as a leaked VM. It is a repair path, and it is kept.

Evidence for the "before" column: the new Kill sweep was run against the parent commit's kill.rs and state.rs. With the after-expectations it fails, first at starting. With the expectation set to started/paused it passes. On the parent commit, the Stop sweep and the non-fatal sweep pass unchanged.

The widened set

can_transition_to(Stopping) was {started, paused}. It is now these 9 states:

{starting, started, stopping, pausing, paused, resuming, checkpointing, suspending, restoring}

That adds 7 edges into stopping, from starting, stopping (a self-edge), pausing, resuming, checkpointing, suspending and restoring. The rule is a private, wildcard-free predicate (can_stop), so a new state does not compile until it is placed.

These states stay excluded even though Stop admits them with a live runtime. Kill has no runtime gate, so admitting them would act on rows whose VM is not known to be live:

• failed: mutation M2 shows Kill would admit it and end it stopped. A failed row that carries the retained-filesystem-only-copy marker would become erase-safe, and a later Delete could erase the only copy.
• cleanup_required / cleanup_failed: superseding drops the cleanup episode and the MarkSuspended intent that keeps SuspendPublicationPending.
• suspended: Kill would discard the suspended lifecycle.
• creating / created / stopped: there is no VM to signal. M5 shows Kill would convert creating to stopped without the gate.

Other callers of transition(Stopping) / can_transition_to(Stopping)

caller	effect of widening
service/grpc_lifecycle/kill.rs	The intended change (fatal path only)
db.rs generic_writer_transition_blocked	Consults transition(target) only when the previous state is quarantined. quarantined → stopping stays false, so it is unchanged
db/tests.rs every_allowed_state_transition_tracks_the_change_time	Iterates can_transition_to; now also drives the 6 new non-self edges through update_state. Passes
state.rs required_states()	The Stopping text names the new sources. It is only used in transition()'s Invalid error, which no production caller of Stopping reaches now
Every other transition(..) / can_transition_to(..)	Fixed targets other than Stopping (Deleting, Starting, Checkpointing, Quarantined, Pausing, Resuming); unaffected. SandboxState is not used outside the sbx-mgr crate

No other behavior changes. The refusal code for every still-refused state stays FailedPrecondition; the message for a refused fatal kill now reads cannot kill: sandbox is <state>, which cannot stop. Stop is not a caller.

Why Stop does not consult the rule

Mutation M8b temporarily made Stop refuse any state the rule rejects, with expectations set to the 9-state rule; the Stop sweep then passes. So consulting the rule would turn Stop on a live runtime from admitted to refused in exactly 7 states: creating, created, stopped, suspended, cleanup_required, cleanup_failed and failed. Widening the rule to Stop's 16 states instead would make Kill act on those 7 states with no runtime gate (above). Per the owner decision, Stop keeps its runtime-gated admission.

Kill's behavior change and its consequences

Kill is newly admitted from starting, stopping, pausing, resuming, checkpointing, suspending and restoring. In each case Kill:

1. claims stopping through the same supersede writer Stop uses, which drops any activation or cleanup episode;
2. persists a MarkStopped cleanup job;
3. sends StopProcess(signal) under the admission fence;
4. removes watchdog and snapshot scheduling;
5. runs cleanup to stopped, blocking or non-blocking.

The in-flight operation's own completion write then races Kill's cleanup. Stop already has every one of these races for a live runtime; this change gives Kill the same exposure. Making the in-flight writer lose cleanly is the job of state-write PRs 2–5.

in-flight state	owner / fencing	its completion writer	outcome when Kill wins admission
starting	Start holds the cleanup fence through launch	mark_start_succeeded: compare-and-set on starting plus the activation episode	Kill waits during launch. In readiness, the supersede deletes the episode, the CAS fails, and start returns Aborted. Clean
restoring	Same fencing as start	mark_restore_succeeded: CAS on restoring plus the episode	Clean, as for starting
resuming	Wake/artifact locks, no cleanup fence	restore_resident_resume_state: CAS on resuming	Clean (Aborted)
checkpointing (scheduled / CreateSnapshot)	Publication locks	restore_checkpoint_state: CAS on checkpointing	Clean; an artifact that was already captured is still published
stopping	Stop/Kill cleanup	MarkStopped completion	Kill waits for a synchronous cleanup's fence, then re-claims stopping and stops again. Redundant and harmless. A StopProcess against a VM that is already gone can return an error after cleanup is admitted
pausing (Pause RPC)	artifact_lock only, no cleanup fence	Generic update_state(Paused), or Started on failure	Finding. Generic update_state refuses started/paused over stopping, but not over stopped. A pause that finishes after Kill's cleanup reached stopped rewrites the row to paused/started with the VM dead
checkpointing (CreateTemplateBaseSnapshot)	wake_lock only	Generic update_state(Paused), success or failure	Finding. The same overwrite of stopped with paused; the long snapshot window makes it realistic
suspending (node-local)	The cleanup fence is held through capture and finalize	finalize_suspended_runtime (re-checks ownership)	Kill waits for the fence. Finding: Kill admission checks only the execution generation, and suspend keeps it, so a Kill that looked up suspending then converts the freshly suspended row to stopped. Kill already has this check-then-act gap for a started/paused lookup that a suspend overtakes
suspending (remote-before-teardown)	The fence is dropped once the artifact is staged; backup may run for a long time	finalize_suspended_runtime returns Aborted once Kill owns the row	Finding. The row ends stopped, which sandbox_erase_safety treats as safe. The staged artifact and teardown job remain, and the permanently-failed marker can no longer be stamped (it requires suspending). A later Delete could erase the filesystem before the remote backup lands. Stop already has the same exposure for a live runtime

Also: Kill on starting/restoring can return Aborted ("cleanup execution changed") when launch attaches a new execution generation between Kill's lookup and its admission. A caller may retry.

The four Finding rows come from reading the code; no test reproduces them.

Possible follow-ups, not in this PR:

• make the Pause and template-base-snapshot completion writers compare-and-set against pausing/checkpointing (state-write PRs 2–5);
• have Kill's admission re-check the rule against the state it supersedes under the fence;
• keep erase-safety evidence for a row that left suspending because a stop superseded it.

Coordination

State-write PR 1 (ac/sbxmgr-state-write-pr1-helper, stacked on #372; not yet pushed when this was opened) adds a parity test between a persisted-edge table and can_transition_to. This PR adds 7 (X, Stopping) edges, listed above, so whichever lands second updates that test's allowlist.

#372 is not merged. This PR ports only what it needs from #372's service/cleanup/stop_admission_tests.rs: the expected-outcome table, the restoring seed and a whole-database unchanged check. It does not touch #372's files.

Tests

• state/tests.rs stopping_is_reachable_exactly_from_live_states: a wildcard-free sweep of all 18 states through can_transition_to(Stopping) and transition(Stopping). It replaces the single started_to_stopping case.
• service/tests/stop_kill_admission_tests.rs (new). Expected outcomes are written as wildcard-free matches, independent of the code under test:
  • fatal_kill_admits_every_state_a_stop_may_supersede: 18 states × signals 9 and 15 × live and not live × blocking and not. Admitted cases check the response, the delivered signal, and the row (stopped, or stopping with intent and job). Refused cases check FailedPrecondition, no signal, and an unchanged whole database.
  • non_fatal_kill_signals_only_a_started_or_paused_sandbox: 18 states × live and not live; the signal is forwarded only to started/paused, and nothing is ever written.
  • stop_admits_a_live_runtime_from_every_state_but_deleting_or_quarantined: 18 states × live and not live × blocking and not. It covers the Stop RPC end to end, including the runtime gate.
• grpc_lifecycle/backup_policy_tests.rs kill_refuses_signals_it_cannot_deliver_before_cleanup_admission now takes its fatal-signal expectation from the same oracle, fatal_kill_admits.
• Host safety:
  • real in-memory SQLite (db::test_conn);
  • services built from for_test_in(tempdir);
  • node cleanup only through the executor's test dispatch;
  • node-manager is the in-process FakeNodeManager on a socket in its own temporary directory;
  • liveness from a tempdir fc.pid;
  • each test runs in an isolated child process.

Mutation proofs

Each mutation was applied alone from a single backup, run against the targeted tests, then restored and verified by hash, strictly in sequence.

#	mutation	result
M1	can_stop: suspending → false	FAIL: the state sweep (suspending), the Kill sweep, and the backup-policy Kill test
M2	can_stop: failed → true	FAIL: the state sweep; the Kill sweep, where Kill admits failed; the backup-policy test
M3	can_stop: deleting → true	FAIL: the state sweep; the Kill sweep, where deleting gets Unavailable from the supersede writer instead of FailedPrecondition; the backup-policy test
M4	Kill's non-fatal gate uses the stop rule	FAIL: the non-fatal sweep (starting signalled)
M5	Kill's fatal gate removed	FAIL: the Kill sweep (creating admitted) and the backup-policy test
M6	Stop running_only = false	FAIL: the Stop sweep (creating, not live, admitted)
M7	Generic writer no longer blocks a quarantined source	FAIL: the Stop sweep (quarantined, live, not refused)
M8	Stop refuses what the rule refuses	FAIL: the Stop sweep (creating, live, refused)
M8b	M8, with Stop's expectations set to the 9-state rule	PASS, so the Stop delta is exactly the 7 states above
red-first	Parent commit's kill.rs/state.rs	Kill and state sweeps FAIL at starting; Stop and non-fatal sweeps PASS. With started/paused expectations, every sweep PASSes

Validation

check	result
cargo fmt --all	clean
cargo clippy --locked --workspace --all-targets -- -D warnings	clean
pre-commit (hook installed; pre-commit run --from-ref 64afd0b --to-ref HEAD)	Passed (gitleaks)
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0, run 1	1589 passed, 0 failed, 1 skipped (#[ignore] root-only distinct_service_uids_can_exchange_downloads, not a dependency skip)
same, run 2	1589 passed, 0 failed, same 1 ignored
same, run 3	1589 passed, 0 failed, same 1 ignored
Mutation proofs M1–M8, M8b, red-first	all as in the table above
Kind: fresh cluster	none existed beforehand; smoke-kind-e2e.sh bootstrapped space-local from this checkout
Kind: deployed sbx-mgr is this build	all 4 running space-sbxmgr-svc processes (2 sandbox nodes, 2 builders) have /proc/<pid>/exe sha256 cd6c8771…e785a, equal to the artifact from this checkout; that binary contains the new Kill refusal text and the new required_states text
Kind: Tilt quiet	50 UI resources, none pending or building; every workload rolled out
smoke-kind-e2e.sh	PASS (round trip ok)
smoke-kind-typed.sh all, ECW (SPACE_SMOKE_KIND_COLD_RECEIPTS set)	77 passed, 1 failed: local.telemetry_reaches_local_backends (classified ENVIRONMENTAL, below)
smoke-kind-typed.sh all, native	23 passed, 0 failed
Rerun of local.telemetry_reaches_local_backends	INCONCLUSIVE: the smoke's Loki port-forward closed after Loki was restarted
Kind teardown (infra/space/kind/scripts/teardown.sh)	done; cluster deleted, Tilt stopped

ENVIRONMENTAL: local.telemetry_reaches_local_backends.

• What failed: only the Loki phase. Sandbox create, the command, the Tempo trace (which includes space-sandbox-manager) and the Prometheus phases passed.
• Cause: the Docker data disk backing the Kind nodes reached 94%. Loki's WAL crossed its 90% threshold, and from then on the ingester answered every push with 500 Ingester is shutting down, so no pod log lines reached Loki.
• Recovery: the two authorized commands ran: docker builder prune -af (reported 21.88GB; 17,583,058,944 bytes freed on the filesystem) and docker image prune -f (0B). The ingester did not recover until Loki was restarted in this disposable cluster. After that, Alloy push errors stopped.
• Rerun: the single rerun was inconclusive because the smoke's port-forward still targeted the replaced Loki pod. No further rerun was made.
• Why it is not this change: nothing in the change touches log shipping.

🤖 Generated with Claude Code

[ac-pplx]

Closing for now. Review showed Stop and Kill can't share one admission rule as scoped: they answer different questions.

• Stop checks for a live runtime and admits 16 states.
• Kill checks the state rule.
• Neither checks whether the in-flight operation yields.

Widening Kill alone admits restoring rows with no VM (a remote-restore pull). It also exposes the pause and template-snapshot writers, which can rewrite a stopped row. And a Kill that waits behind a blocking Stop re-claims the stopped row and returns Internal.

We'll revisit after the state-write conversions make in-flight writers compare-and-set, as part of the lifecycle redesign. The branch ac/sbxmgr-stop-kill-shared-admission is kept for reference.