Skip to content

sbx-mgr: create completion and warm start write only over the state they expect - #418

Merged
ac-pplx merged 1 commit into
ac/sbxmgr-sw-pr1b-refusal-statusfrom
ac/sbxmgr-sw-pr4-create-warm-start
Oct 14, 2026
Merged

ac-pplx merged 1 commit into
ac/sbxmgr-sw-pr1b-refusal-statusfrom
ac/sbxmgr-sw-pr4-create-warm-start

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #438 (on main). Base: ac/sbxmgr-sw-pr1b-refusal-status @ bf43fd5. This PR's diff is one commit on top of it. Stack, bottom to top: #438 → #418 → #413 → #416 → #421 → #417 → #414. #372 and #388 have merged to main.

Everything here is sbx-manager's node-local SQLite sandboxes.state, not the control-plane Postgres public.sandboxes.lifecycle.

This is plan §5 PR 4. It converts the create-completion writes and the warm-snapshot Start write from update_state to update_state_from, which closes plan findings 3 and 4. Outcomes are mapped with #438's require_state_write_from and state_write_result.

What merging does

Call sites converted

Call site Before After Edge
service/create.rs create_with_id, create completion require_state_write(update_state(&db, &id, Created), "complete sandbox create") write_create_completed(&db, &id) CREATE_COMPLETE: {creating} → created (Completion)
service/create.rs create_with_id, base-snapshot completion require_state_write(update_state(&db, &id, Suspended), "complete restored sandbox create") write_base_snapshot_create_completed(&db, &id) CREATE_BASE_SNAPSHOT_COMPLETE: {created} → suspended (Completion)
service/start.rs try_start_from_warm_snapshot, restage write require_state_write(update_state(&db, &sandbox.id, Suspended), "prepare recovered sandbox restore") write_warm_start_restaged(&db, &sandbox.id) WARM_START_RESTAGE: {created, stopped} → suspended (Request)
service/start.rs try_start_from_warm_snapshot, entry no state check (Start refuses only failed earlier) admit_warm_start(sandbox)? before anything is staged — (read-state check)

All four functions are in service/create_support.rs, next to stage_base_snapshot_runtime_metadata, which both paths already share. Each write_* function is the call site's whole state write, so a test can drive exactly what the service runs. Past the durable insert, Create and a warm Start create directories under the fixed host sandbox cache, so no host-safe service test reaches these writes.

Create completions. Both return require_state_write_from(..) (Result<(), Status>) directly. Every refusal is Aborted:

  • A row outside the from-set: "<context>: sandbox state write was superseded: sandbox is <found>".
  • A generic-guard refusal or a missing row: update_state's message.

Warm start: two checks.

  • Before staging. admit_warm_start refuses a row read outside {created, stopped, suspended} with FailedPrecondition. This mirrors start_from_checkpoint's own state check. Before, such a row first had the base snapshot staged into its canonical suspend snapshot directory: runtime.json, plus reflinked vmstate and memory. Restore treats that directory as local suspend evidence.
  • At the restage write. write_warm_start_restaged checks again, because a row can move while the snapshot is staged. It uses the typed state_write_result:
    • FromState { found: suspended } is success with no write.
    • Any other FromState is the same FailedPrecondition.
    • Guard refusals and Missing go through into_status, which keeps update_state's Aborted. A suspended, starting or restoring row owned by cleanup therefore gets Aborted "superseded_by_cleanup_episode", not a self-contradictory FailedPrecondition.

Verifying the warm-start from-set. {created, stopped} holds:

  • Every production from_warm Start targets a suspended row: the API's direct substrate and the template pool manager after a base-snapshot Create, the pool manager after a donor clone (complete_pool_clone leaves suspended), and Create's own base-snapshot autostart. The existing sandbox.state != Suspended check skips the write for those rows.
  • Only an operator CLI --from-warm Start of a created or stopped row reaches the write.

Locks and side columns. Caller mutex and lifecycle-lock scopes are unchanged, and so are each writer's side columns. In autocommit, update_state_from adds its own IMMEDIATE transaction around its reads and the UPDATE, which update_state did not; that is harmless on the single connection.

Logging. The three converted sites no longer log update_state's "ignored generic sandbox state update because …" warning on a guard refusal. Every one of these writes is required, so the reason reaches the client status instead. No logging was added.

Races and unvalidated writes fixed

Plan finding 4: create completion over Stop's stopping (defense in depth).

  • The race. Create holds no lock between its insert and its completion writes. A Stop admitted in between leaves the row stopping, with the create episode superseded and a mark_stopped intent.
  • Before. update_state(Created) or update_state(Suspended) overwrote stopping. The row then survived the failed create as created or suspended, with a stale mark_stopped intent that refuses every activation (sbx-mgr: characterize the state writers before the state-write helper (tests only) #372 Findings 1).
  • Now. The write is refused. Create returns Aborted and fails through its cleanup guard, whose episode request is Superseded. Stop completion settles the row stopped.
  • Reachability. As sbx-mgr: add the persisted state-transition table and update_state_from (no callers) #388 shows, the Stop RPC cannot produce this, because a row during create has no runtime. The conversion is kept as defense in depth.
  • Red/green test. create_completion_leaves_a_row_a_stop_admitted_to_the_stop runs sbx-mgr: characterize the state writers before the state-write helper (tests only) #372's DB-level sequence (create insert with an armed delete_row episode, then Stop's admission transaction) for both completion writes. It asserts:
    • the Aborted status and its exact message;
    • every table unchanged;
    • the cleanup guard's request_cleanup_episode returns Superseded, with every table still unchanged;
    • Stop completion leaves stopped with no intent.

Plan finding 3: a warm-snapshot Start writes suspended from an unvalidated state. This needs no concurrency. Before, start_sandbox_with_authority rejected only failed ahead of the warm path. For example, Start { from_warm: true } on a started row staged the base snapshot into the live sandbox's suspend directory, rewrote the row to suspended, and went on to a checkpoint restore.

Red/green tests:

  • start_refusals_before_activation_leave_the_row_state_unchanged (start/tests.rs) gains started and paused cases. These are service-level and host-safe: the refusal comes before staging, and a regression fails closed at the test service's unreachable template-manager. Restoring sbx-mgr: add the persisted state-transition table and update_state_from (no callers) #388's start.rs, or dropping the pre-check, fails them.
  • completion_and_warm_start_writes_answer_every_source_state sweeps all three writers over a plain row in every SandboxState.
    • Each row carries a failure reason, a receipt for another target and backdated timestamps, so a write that should not happen shows in the row. In particular, an already-suspended warm-start row must stay byte-identical, and putting back the old same-state write fails.
    • The expected outcomes, with exact messages, come from wildcard-free matches written independently of the edge table.
    • The accepting controls are: creating, then created, for the two completions; created and stopped for the warm start.
    • Every refusal leaves every table unchanged.
    • The warm-start failed row is the writer's contract only, because Start refuses failed earlier.
  • warm_start_admits_only_its_sources_and_suspended_rows runs admit_warm_start on a row in every SandboxState. A wildcard-free expected match accepts exactly created, stopped and suspended, and every other state must be refused with the exact FailedPrecondition text. This pins the accepting set, including suspended, which every production warm Start targets.
  • warm_start_write_keeps_a_guard_refusal_aborted covers rows a concurrent cold Start or restore claimed with an activation episode, plus stopped and suspended rows owned by an episode. All four keep the guard's Aborted. The two hand-rewound rows are the writer's contract only.

Behavior changes

Compared with #388's update_state call sites:

Path Row Before After
Create completion found outside creating and not guard-owned (e.g. stopping after a Stop) wrote created; Create then returned Aborted "sandbox create cleanup episode was no longer current" Aborted "complete sandbox create: sandbox state write was superseded: sandbox is <found>"; the row stays with its owner
Base-snapshot completion found outside created and not guard-owned (e.g. stopping) wrote suspended; then the same Aborted Aborted "complete restored sandbox create: sandbox state write was superseded: sandbox is <found>"
Base-snapshot completion found suspended because an operator CLI warm Start restaged the created row inside Create's sub-second window between its two writes same-state write applied; complete_create succeeded Aborted (message above, suspended). Create's cleanup_after_failure then requests the still-armed delete_row create episode, which deletes the row
Warm Start read as creating, starting, started, stopping, pausing, paused, resuming, checkpointing, suspending, restoring, cleanup_required, cleanup_failed with no owning episode or delete_row intent staged the base snapshot, wrote suspended, then a checkpoint restore FailedPrecondition "cannot start sandbox <id> from a warm snapshot while local state is <found>; expected suspended, created or stopped", before anything is staged
Warm Start read as a guard-owned row: deleting, cleanup_required/cleanup_failed with a delete_row intent, any row with an owning activation episode (e.g. starting/restoring), or quarantined staged the base snapshot, then Aborted "…superseded_by_cleanup_episode" or "…blocked_by_quarantine" FailedPrecondition (same new message), before anything is staged
Warm Start read as created/stopped, moved during staging to a non-guard-owned state outside {created, stopped, suspended} (e.g. stopping) wrote suspended FailedPrecondition (same message)
Warm Start read as created/stopped, moved to suspended during staging same-state write: cleared failure_reason and non-matching receipts, stamped updated_at, then the restore no write, then the restore
all three missing row; guard refusal at the write Aborted unchanged: Aborted, same "sandbox state write was not applied: missing / blocked_by_quarantine / superseded_by_cleanup_episode" messages

New client-visible messages:

  1. Aborted "complete sandbox create: sandbox state write was superseded: sandbox is <found>"
  2. Aborted "complete restored sandbox create: sandbox state write was superseded: sandbox is <found>"
  3. FailedPrecondition "cannot start sandbox <id> from a warm snapshot while local state is <found>; expected suspended, created or stopped"

Messages 1 and 2 are #438's canonical FromState text.

Other consequences:

  • A warm Start retried after its first attempt succeeded finds started and now gets FailedPrecondition. Before, it restaged that row for a second restore.
  • Recorded residual. The concurrent case still stages without a lock. If a row moves after the pre-check, the base snapshot is staged into its suspend snapshot directory before the restage write refuses, and that refused staging is left in place. A failure after staging already left it there before this PR.

Edges, allowlist and fixtures

  • Deleted. WARM_START_UNVALIDATED (KnownRace). No edges added.
  • Doc comments. CREATE_COMPLETE, CREATE_BASE_SNAPSHOT_COMPLETE and WARM_START_RESTAGE now name their new writer functions. The table is not reordered or reformatted.
  • Finding 4. No KnownRace edge to delete: sbx-mgr: add the persisted state-transition table and update_state_from (no callers) #388 already dropped finding 4's edges as unreachable.
  • ALLOWLIST removals. The five pairs only the deleted edge wrote: started, pausing, paused, resuming and checkpointing → suspended.
  • ALLOWLIST reason refreshes. The class is unchanged; the warm-start race mention is dropped from:
    • creating → suspended
    • starting → suspended
    • restoring → suspended
    • cleanup_required → suspended
    • cleanup_failed → suspended
    • stopping → suspended, which now reads "known race: process-snapshot rollback in the gate-to-fence window"
  • Other mixed reasons, re-read. created → suspended and stopped → suspended name the warm start as a legitimate Request and stay.
  • KnownRace kind. Five KnownRace edges remain on this branch, so the kind and its any KnownRace assertion stay. sbx-mgr: pause writes only over the state it expects #417, which deletes the last two, retires them.
  • Fixtures. db/tests/state_write_fixtures.rs's insert_row, force_state, insert_creating, set_failure_reason, seed_receipt and backdate become pub(crate) (test-only module), so the service tests reuse them.
  • sbx-mgr: one refusal-to-status mapping for update_state_from #438 dead-code expectations. This PR removes the cfg_attr(not(test), expect(dead_code)) from the items it calls in production: require_state_write_from, StateWriteRejected (and its into_status impl), StateWriteGuard and state_write_result. update_state_from_best_effort keeps its expectation; this PR has no best-effort write.
  • sbx-mgr: characterize the state writers before the state-write helper (tests only) #372 tests. None of sbx-mgr: characterize the state writers before the state-write helper (tests only) #372's matrix or accepted-shape tests changed; they are green.

Overlaps with the other conversion PRs in the stack (2, 3, 5)

Mutation proof

Each mutation was applied alone from a clean backup of the final source. The new and changed tests were run against it (-E 'test(/state_write_tests|start_refusals_before_activation/)'), and the source was restored before the next. "ALL" widens the edge's from-set in the table to StateSet::ALL.

# Mutation Failing tests
C1 Create completion: restore require_state_write(update_state(Created)) source-state sweep; Stop race
C2 CREATE_COMPLETE from-set → ALL source-state sweep; Stop race
C3 Base-snapshot completion: restore require_state_write(update_state(Suspended)) source-state sweep; Stop race
C4 CREATE_BASE_SNAPSHOT_COMPLETE from-set → ALL source-state sweep; Stop race
W1 Restage write: restore require_state_write(update_state(Suspended)) source-state sweep
W2 WARM_START_RESTAGE from-set → ALL source-state sweep; start_refusals_before_activation…
W3 Restage write: put back the same-state write over an already-suspended row source-state sweep (side columns changed)
W4 Restage write: accept any rejection that found suspended (drop the FromState predicate) guard-refusal test (suspended case)
W5 Restage write: answer guard refusals outside the from-set with FailedPrecondition source-state sweep; guard-refusal test
W6 start.rs: drop the admit_warm_start pre-staging check start_refusals_before_activation…
W7 start.rs: restore #388's file (generic update_state, no pre-check) start_refusals_before_activation…
A1 admit_warm_start: drop suspended admission sweep (only this test)
A2 admit_warm_start: drop created admission sweep
A3 admit_warm_start: drop stopped admission sweep; start_refusals_before_activation…

All 14 were caught and restored. The guard-refusal test fails only under W4 and W5: the generic guards answer before the from-set, so restoring update_state or widening the from-set does not change its outcome.

Decisions made without an operator

  • Already-suspended warm start. A warm Start that read created/stopped but finds suspended at the write treats it as done, which is the existing pre-check's skip. Edges cannot list their target as a source, so it does not write. In that interleaving only, the old same-state write's side effects (failure_reason, receipts, updated_at) no longer happen. That is the one deviation from D4, and it is confined to a no-op transition.
  • Create keeps Aborted. Both create refusals stay Aborted, matching complete_create's supersession answer.
  • Pre-check scope. quarantined and every guard-owned state fall outside {created, stopped, suspended}, so the pre-check refuses them with FailedPrecondition before staging, as the review asked. Nothing is staged into a quarantined sandbox's directory.
  • Test layer. The DB-level tests drive the call-site functions, because the service paths need the fixed host cache. The service-level pre-check case runs where it is host-safe.

Self-assessed severity

Medium. A warm Start of a running or paused sandbox could stage a snapshot over its suspend directory and restage it for a second restore with the VM still live. Its callers are the operator CLI and retries. The create race is unreachable through the Stop RPC.

The change and this description were drafted with an AI coding agent (Claude Code); a human owner reviews before merge.

Validation of the restacked stack

The stack was restacked linearly on main (b18051f) after #372 and #388 merged: #438 → #418 → #413 → #416 → #421 → #417 → #414. The per-PR validation further down ran on this PR's pre-restack head.

This PR's head (6a1c00f): pull CI is green. fmt, workspace clippy --all-targets --all-features -D warnings and cargo arch (0 required findings) all pass. SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0 passed 1638/1638, with 0 dependency skips. Kind Pull CI ran on #417, which contains this PR: all 4 Firecracker smoke shards passed.

The top of the stack (#414 @ 9a0a251, which contains this PR):

Check Result
sbx-mgr nextest ×3 1650/1650 each, 0 dependency skips
Whole-workspace nextest 8985/9021 passed, and every API crate passed. The 36 failures are all outside sbx-mgr. 34 fail identically on main: spaced-guest and spaced-host-cli process tests (the test box can't switch identity) and one net-gateway test. The other 2 are common-grpc retry-deadline tests that pass on rerun on both heads.
The stack's 30 new or changed tests, ×30 900/900
cargo test -p pplx-space-sbx-mgr (shared process) 1650/1650
Mutation spot-check on the top head #417's quarantine-miss mutation and #418's warm-start admission mutation are both caught
Kind, fresh cluster; the deployed sbx-mgr binary is from this head smoke-kind-e2e.sh passed. smoke-kind-typed.sh all: native 23/23; ECW 76 passed, 2 infra errors. local.ecw_template_build_from_oci_boots passed, and so did the leader-election brownout.
The 2 ECW infra errors node_group_boundary_schedules and node_selector_extras_render_and_enforce. Both scenarios restart the API Deployment, and the same failure family appears on main in an A/B on the same cluster. In the failed runs the kill never reached sbx-mgr.
Lifecycle race soak on Kind 11 concurrent operation pairs × 40 iterations, on the stack and on main on the same fresh cluster, checking that sbx-mgr's SQLite sandboxes.state, control-plane Postgres lifecycle and the VM agree. No change-induced violation: across 880 sandboxes, no quarantine, panic, restart or stuck state. Every violation reproduces on main with the same signature: an ECW pause that loses to Stop answers 503 instead of 409; after Suspend vs Delete, the node row can outlive the control-plane row until the next eviction pass (pass timing, not the build); and a refused quarantine write after Pause vs Delete is still logged. The soak's 0–200 ms API-level offsets did not reproduce the interleavings this stack closes, so the unit-level race tests remain the direct evidence for those.

Validation on the pre-restack head

Check Result
cargo fmt --all -- --check; scripts/check-rust-error-policy.sh pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings pass, 0 warnings
cargo arch pass: 0 required findings, advisory only. Against #388's head, create.rs drops 1203 → 1199 code lines and start.rs 2320 → 2316; both were already over their caps
pre-commit (Gitleaks) on the final commit pass
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0 × 3 on 5f09798 (on #438 @ 7e2e41c), then once after rebasing onto #438's final f1d1c9b and once on 69423eb (1633/1633 each), and once on the final head 9d8e974 (1634/1634, with the admission sweep). Every run passed; 0 dependency skips (skip log never written); 1 skipped: the #[ignore] privileged transfer_directory::tests::distinct_service_uids_can_exchange_downloads, unrelated
Mutation proof (C1–C4, W1–W7, A1–A3) C1–C4 and W1–W7 caught on 5f09798; C1 and C3 rerun and caught on 69423eb; A1–A3 caught on the final head; all restored
Public-safety review of the commit no findings in the reviewed scope
Kind: fresh cluster none existed (kind get clusters empty); bootstrapped from this checkout on the current harness (smoke-kind-e2e.sh, which runs bootstrap_kind_stack)
Kind: deployed sbx-mgr is this build (5f09798) services report version 5f097988d319. Both worker space-sandbox-manager pods' running space-sbxmgr-svc (/proc/<pid>/exe) hash 6e991ba3…dec, equal to artifacts/bin/space-sbxmgr-svc in the space-rust-build-kind:local image Tilt built. The synced build source matches HEAD for every changed file, and the binary contains both new message texts
Kind: Tilt quiet bring-up returned only after Tilt had nothing pending. Before the typed smoke, every UI resource was ok with no build in progress, and every pod was Running or Completed
smoke-kind-e2e.sh (baseline, DATA_PLANE_URL set as CI does) PASS
smoke-kind-typed.sh all (SKIP_BOOTSTRAP=1, admin token file, SPACE_SMOKE_KIND_COLD_RECEIPTS as CI sets them; unsharded on one stack) ECW 78 passed, native 23 passed; 0 warned, 0 failed, 0 inconclusive; 0 quarantined withheld. No reruns needed
Kind vs final head Kind ran on 5f09798. The final head 9d8e974 differs only by the admission sweep test and by #438's final API (require_state_write_from returns Result<(), Status>, plus #438's own test and doc changes) and by dropping two now-redundant .map(drop) calls. Per the owner, Kind was not rerun for that rebase; nextest was
Kind teardown infra/space/kind/scripts/teardown.sh: Tilt stopped, cluster deleted
Disk ran docker builder prune -af once (authorized), between scenarios with Tilt idle, when the Docker data disk reached 77%

🤖 Generated with Claude Code

@ac-pplx
ac-pplx force-pushed the ac/sbxmgr-sw-pr4-create-warm-start branch from 0e0901d to 5f09798 Compare October 14, 2026 00:49
@ac-pplx
ac-pplx changed the base branch from ac/sbxmgr-state-write-pr1-helper to ac/sbxmgr-sw-pr1b-refusal-status October 14, 2026 00:50
@ac-pplx
ac-pplx force-pushed the ac/sbxmgr-sw-pr4-create-warm-start branch 2 times, most recently from 69423eb to 9d8e974 Compare October 14, 2026 02:16
…hey expect

Convert three generic update_state callers of the node-local SQLite
sandboxes.state column to update_state_from with an expected from-set,
and map the outcomes with require_state_write_from and
state_write_result. This does not touch the control-plane Postgres
public.sandboxes.lifecycle.

- Create completion (create_with_id): creating -> created through
  CREATE_COMPLETE, and, in base-snapshot mode, created -> suspended
  through CREATE_BASE_SNAPSHOT_COMPLETE (write_create_completed,
  write_base_snapshot_create_completed). Create holds no lock between its
  insert and these writes. A row another transition moved, such as a
  Stop's stopping, is now refused with Aborted "<context>: sandbox state
  write was superseded: sandbox is <found>", and Create fails through its
  cleanup guard while the Stop keeps the row and settles it stopped.
  Before, the write overwrote stopping and left a created or suspended
  row with a stale mark_stopped intent that refuses every activation. The
  Stop RPC cannot reach a row during create (no runtime), so this is
  defense in depth.
- Warm-snapshot Start (try_start_from_warm_snapshot):
  - admit_warm_start refuses a row read outside created, stopped and
    suspended with FailedPrecondition before the base snapshot is staged
    into the sandbox's suspend snapshot directory. Before, such a row was
    staged and rewritten to suspended for a checkpoint restore, and
    started and paused need no concurrency to reach that.
  - The restage write, created or stopped -> suspended through
    WARM_START_RESTAGE (write_warm_start_restaged), checks again. A row
    that moved to suspended needs no write; a row moved anywhere else is
    the same FailedPrecondition. Generic-guard refusals and a missing row
    keep require_state_write_from's Aborted.

Guard refusals and missing rows keep update_state's Aborted messages.
Caller mutex and lifecycle-lock scopes and each writer's side columns
are unchanged; in autocommit update_state_from adds its own IMMEDIATE
transaction around its reads and the UPDATE, which update_state did not
(harmless on the single connection). The three converted sites no longer
log update_state's "ignored generic sandbox state update" warning; the
refusal reaches the client status instead.

The WARM_START_UNVALIDATED known-race edge is deleted. The parity
allowlist drops the five pairs only it wrote (started, pausing, paused,
resuming, checkpointing -> suspended) and the warm-start mention from six
mixed reasons. The edge doc comments name the new writer functions. The
state_write_fixtures helpers the tests reuse become crate-visible, and
require_state_write_from, StateWriteRejected, StateWriteGuard and
state_write_result lose their dead-code expectation.

Tests:
- service/create_support/state_write_tests.rs (DB level, because both
  paths create directories under the fixed host cache before these
  writes):
  - a Stop admitted before either create completion write: the write is
    refused with Aborted and changes nothing, Create's cleanup request is
    superseded, and Stop completion settles stopped;
  - each of the three writers over a plain row in every SandboxState,
    carrying a failure reason, a receipt and backdated timestamps, with
    wildcard-free expected outcomes and accepting controls; an
    already-suspended warm-start row stays byte-identical;
  - warm-start rows a concurrent activation or cleanup episode owns keep
    the guard's Aborted, including a suspended one;
  - admit_warm_start over a row in every SandboxState accepts exactly
    created, stopped and suspended and refuses the rest with the exact
    FailedPrecondition.
- start/tests.rs: a warm Start of a started or paused row is refused
  before staging and leaves the row unchanged.
The Stop-race test and the source-state sweep fail with a writer's generic
update_state call restored or its edge's from-set widened to every state.
The guard-refusal test fails only when the warm start mishandles a guard
refusal: the guards answer before the from-set. The start/tests.rs cases
fail without the pre-staging check, and the admission sweep fails when
any of created, stopped or suspended is dropped from it.

@nikhilgupta-pplx nikhilgupta-pplx left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. {creating}/{created} completions and WARM_START_RESTAGE {created, stopped} + the suspended no-write cover every production warm Start (API base-snapshot, pool manager create/clone, Create autostart); only CLI --from-warm reaches the write. admit_warm_start runs before ensure_snapshot_restore_supported and any staging, which closes finding 3.

Non-blocking:

  • A warm Start on a transient row (restoring, stopping, …) now gets FailedPrecondition, while the cold path answers the same row Unavailable via transition(Starting). Current callers treat every error alike, so no impact today; consider Unavailable when found.is_transient(), or note the choice as deliberate.
  • One doc line on why Guard { found: Suspended } stays Aborted while FromState { Suspended } is success would help the next reader.

@ac-pplx
ac-pplx added this pull request to the merge queue Oct 14, 2026
@the-gigi-pplx

Copy link
Copy Markdown
Collaborator

Review — 2 findings

Low

  1. crates/libs/sbx/mgr/server/src/service/create_support.rs:290 — the write_create_completed doc comment describes a race that the edge table rules out.
    It says "Create holds no lock between its insert and this write, so a row a Stop moved to stopping refuses it". But STOP_ADMISSION (db/state_write.rs:293-296) is StateSet::ALL.without(Creating)…, and its own doc says "a creating row has none". So no Stop writer can produce stopping from the row this write expects. The PR body and the test docstring both say the Stop RPC can't reach a row during create, and that this is defense in depth. The comment on the production function is the only place that reads as if the race is live. Saying it's defense in depth against a writer that skips the edge, or naming the writers that actually can race here (Delete admission, which the generic guard answers), would keep the next reader from going looking for a Stop path.
  2. crates/libs/sbx/mgr/server/src/service/create_support.rs:316 — the refusal text hardcodes "expected suspended, created or stopped", while admit_warm_start (line 325) derives the accepted set from WARM_START_RESTAGE.from() plus suspended.
    If the edge's from-set changes, admission follows it and the message doesn't. The tests also hardcode the string (and warm_start_admitted lists the set by hand, on purpose), so nothing fails when the two drift apart. Formatting the message from the same set would leave only the independent test oracle to update.

What I checked and found sound:

  • The from-sets: CREATE_COMPLETE {creating} and CREATE_BASE_SNAPSHOT_COMPLETE {created} match the order of create_with_id's two writes (create.rs:414-425, create.rs:526-538), both before complete_create, so the create episode is still armed when a refusal goes through cleanup_after_failure.
  • The production warm-Start callers all reach a suspended row synchronously: the API direct substrate after a base-snapshot Create (substrate/direct.rs:3605), the template pool manager after Create and after clone_from_donor, whose complete_pool_clone writes suspended before the RPC returns (grpc_pool_wait_clone.rs:860-873, db/state_transitions.rs:800), and Create's own autostart. None of them reaches the new admission refusal. Both API and pool callers delete the sandbox on any Start error, so the Aborted → FailedPrecondition change for guard-owned rows doesn't change a retry decision there.
  • write_warm_start_restaged: the FromState { found: Suspended } arm is the only one that skips the write. Guard refusals and Missing go through into_status and keep update_state's Aborted text.
  • The expect(dead_code) removals match exactly what this PR starts calling. update_state_from_best_effort keeps its expectation.
  • The ALLOWLIST removals are exactly the five → suspended pairs only WARM_START_UNVALIDATED wrote.

I deliberately didn't raise the staged files left in place when the restage write refuses after staging. The PR body records it as a residual, and before this PR the same interleaving wrote suspended over the moved row, which is worse.

Verified against 6a1c00f38587. Reproduced locally: cargo clippy -p pplx-space-sbx-mgr --all-targets --all-features -D warnings (clean) and cargo nextest run -p pplx-space-sbx-mgr -p pplx-space-db-test-fixtures --all-features (1649 passed, 0 failed) at this head.

@the-gigi-pplx the-gigi-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. 2 non-blocking findings in the review comment above.

Merged via the queue into main with commit 7c6489d Oct 14, 2026
20 checks passed
@ac-pplx
ac-pplx deleted the ac/sbxmgr-sw-pr4-create-warm-start branch October 14, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants