Repository navigation
sbx-mgr: create completion and warm start write only over the state they expect - #418
Conversation
0e0901d to
5f09798
Compare
69423eb to
9d8e974
Compare
…hey expect
Convert three generic update_state callers of the node-local SQLite
sandboxes.state column to update_state_from with an expected from-set,
and map the outcomes with require_state_write_from and
state_write_result. This does not touch the control-plane Postgres
public.sandboxes.lifecycle.
- Create completion (create_with_id): creating -> created through
CREATE_COMPLETE, and, in base-snapshot mode, created -> suspended
through CREATE_BASE_SNAPSHOT_COMPLETE (write_create_completed,
write_base_snapshot_create_completed). Create holds no lock between its
insert and these writes. A row another transition moved, such as a
Stop's stopping, is now refused with Aborted "<context>: sandbox state
write was superseded: sandbox is <found>", and Create fails through its
cleanup guard while the Stop keeps the row and settles it stopped.
Before, the write overwrote stopping and left a created or suspended
row with a stale mark_stopped intent that refuses every activation. The
Stop RPC cannot reach a row during create (no runtime), so this is
defense in depth.
- Warm-snapshot Start (try_start_from_warm_snapshot):
- admit_warm_start refuses a row read outside created, stopped and
suspended with FailedPrecondition before the base snapshot is staged
into the sandbox's suspend snapshot directory. Before, such a row was
staged and rewritten to suspended for a checkpoint restore, and
started and paused need no concurrency to reach that.
- The restage write, created or stopped -> suspended through
WARM_START_RESTAGE (write_warm_start_restaged), checks again. A row
that moved to suspended needs no write; a row moved anywhere else is
the same FailedPrecondition. Generic-guard refusals and a missing row
keep require_state_write_from's Aborted.
Guard refusals and missing rows keep update_state's Aborted messages.
Caller mutex and lifecycle-lock scopes and each writer's side columns
are unchanged; in autocommit update_state_from adds its own IMMEDIATE
transaction around its reads and the UPDATE, which update_state did not
(harmless on the single connection). The three converted sites no longer
log update_state's "ignored generic sandbox state update" warning; the
refusal reaches the client status instead.
The WARM_START_UNVALIDATED known-race edge is deleted. The parity
allowlist drops the five pairs only it wrote (started, pausing, paused,
resuming, checkpointing -> suspended) and the warm-start mention from six
mixed reasons. The edge doc comments name the new writer functions. The
state_write_fixtures helpers the tests reuse become crate-visible, and
require_state_write_from, StateWriteRejected, StateWriteGuard and
state_write_result lose their dead-code expectation.
Tests:
- service/create_support/state_write_tests.rs (DB level, because both
paths create directories under the fixed host cache before these
writes):
- a Stop admitted before either create completion write: the write is
refused with Aborted and changes nothing, Create's cleanup request is
superseded, and Stop completion settles stopped;
- each of the three writers over a plain row in every SandboxState,
carrying a failure reason, a receipt and backdated timestamps, with
wildcard-free expected outcomes and accepting controls; an
already-suspended warm-start row stays byte-identical;
- warm-start rows a concurrent activation or cleanup episode owns keep
the guard's Aborted, including a suspended one;
- admit_warm_start over a row in every SandboxState accepts exactly
created, stopped and suspended and refuses the rest with the exact
FailedPrecondition.
- start/tests.rs: a warm Start of a started or paused row is refused
before staging and leaves the row unchanged.
The Stop-race test and the source-state sweep fail with a writer's generic
update_state call restored or its edge's from-set widened to every state.
The guard-refusal test fails only when the warm start mishandles a guard
refusal: the guards answer before the from-set. The start/tests.rs cases
fail without the pre-staging check, and the admission sweep fails when
any of created, stopped or suspended is dropped from it.
9d8e974 to
6a1c00f
Compare
nikhilgupta-pplx
left a comment
There was a problem hiding this comment.
Approve. {creating}/{created} completions and WARM_START_RESTAGE {created, stopped} + the suspended no-write cover every production warm Start (API base-snapshot, pool manager create/clone, Create autostart); only CLI --from-warm reaches the write. admit_warm_start runs before ensure_snapshot_restore_supported and any staging, which closes finding 3.
Non-blocking:
- A warm Start on a transient row (
restoring,stopping, …) now getsFailedPrecondition, while the cold path answers the same rowUnavailableviatransition(Starting). Current callers treat every error alike, so no impact today; considerUnavailablewhenfound.is_transient(), or note the choice as deliberate. - One doc line on why
Guard { found: Suspended }staysAbortedwhileFromState { Suspended }is success would help the next reader.
Review — 2 findingsLow
What I checked and found sound:
I deliberately didn't raise the staged files left in place when the restage write refuses after staging. The PR body records it as a residual, and before this PR the same interleaving wrote Verified against |
the-gigi-pplx
left a comment
There was a problem hiding this comment.
Approved. 2 non-blocking findings in the review comment above.
Stacked on #438 (on main). Base:
ac/sbxmgr-sw-pr1b-refusal-status@bf43fd5. This PR's diff is one commit on top of it. Stack, bottom to top: #438 → #418 → #413 → #416 → #421 → #417 → #414. #372 and #388 have merged to main.Everything here is sbx-manager's node-local SQLite
sandboxes.state, not the control-plane Postgrespublic.sandboxes.lifecycle.This is plan §5 PR 4. It converts the create-completion writes and the warm-snapshot Start write from
update_statetoupdate_state_from, which closes plan findings 3 and 4. Outcomes are mapped with #438'srequire_state_write_fromandstate_write_result.What merging does
Call sites converted
service/create.rscreate_with_id, create completionrequire_state_write(update_state(&db, &id, Created), "complete sandbox create")write_create_completed(&db, &id)CREATE_COMPLETE:{creating} → created(Completion)service/create.rscreate_with_id, base-snapshot completionrequire_state_write(update_state(&db, &id, Suspended), "complete restored sandbox create")write_base_snapshot_create_completed(&db, &id)CREATE_BASE_SNAPSHOT_COMPLETE:{created} → suspended(Completion)service/start.rstry_start_from_warm_snapshot, restage writerequire_state_write(update_state(&db, &sandbox.id, Suspended), "prepare recovered sandbox restore")write_warm_start_restaged(&db, &sandbox.id)WARM_START_RESTAGE:{created, stopped} → suspended(Request)service/start.rstry_start_from_warm_snapshot, entryfailedearlier)admit_warm_start(sandbox)?before anything is stagedAll four functions are in
service/create_support.rs, next tostage_base_snapshot_runtime_metadata, which both paths already share. Eachwrite_*function is the call site's whole state write, so a test can drive exactly what the service runs. Past the durable insert, Create and a warm Start create directories under the fixed host sandbox cache, so no host-safe service test reaches these writes.Create completions. Both return
require_state_write_from(..)(Result<(), Status>) directly. Every refusal isAborted:<context>: sandbox state write was superseded: sandbox is<found>".update_state's message.Warm start: two checks.
admit_warm_startrefuses a row read outside{created, stopped, suspended}withFailedPrecondition. This mirrorsstart_from_checkpoint's own state check. Before, such a row first had the base snapshot staged into its canonical suspend snapshot directory:runtime.json, plus reflinkedvmstateandmemory. Restore treats that directory as local suspend evidence.write_warm_start_restagedchecks again, because a row can move while the snapshot is staged. It uses the typedstate_write_result:FromState { found: suspended }is success with no write.FromStateis the sameFailedPrecondition.Missinggo throughinto_status, which keepsupdate_state'sAborted. Asuspended,startingorrestoringrow owned by cleanup therefore getsAborted"superseded_by_cleanup_episode", not a self-contradictoryFailedPrecondition.Verifying the warm-start from-set.
{created, stopped}holds:from_warmStart targets asuspendedrow: the API's direct substrate and the template pool manager after a base-snapshot Create, the pool manager after a donor clone (complete_pool_cloneleavessuspended), and Create's own base-snapshot autostart. The existingsandbox.state != Suspendedcheck skips the write for those rows.--from-warmStart of acreatedorstoppedrow reaches the write.Locks and side columns. Caller mutex and lifecycle-lock scopes are unchanged, and so are each writer's side columns. In autocommit,
update_state_fromadds its ownIMMEDIATEtransaction around its reads and theUPDATE, whichupdate_statedid not; that is harmless on the single connection.Logging. The three converted sites no longer log
update_state's "ignored generic sandbox state update because …" warning on a guard refusal. Every one of these writes is required, so the reason reaches the client status instead. No logging was added.Races and unvalidated writes fixed
Plan finding 4: create completion over Stop's
stopping(defense in depth).stopping, with the create episode superseded and amark_stoppedintent.update_state(Created)orupdate_state(Suspended)overwrotestopping. The row then survived the failed create ascreatedorsuspended, with a stalemark_stoppedintent that refuses every activation (sbx-mgr: characterize the state writers before the state-write helper (tests only) #372 Findings 1).Abortedand fails through its cleanup guard, whose episode request isSuperseded. Stop completion settles the rowstopped.create_completion_leaves_a_row_a_stop_admitted_to_the_stopruns sbx-mgr: characterize the state writers before the state-write helper (tests only) #372's DB-level sequence (create insert with an armeddelete_rowepisode, then Stop's admission transaction) for both completion writes. It asserts:Abortedstatus and its exact message;request_cleanup_episodereturnsSuperseded, with every table still unchanged;stoppedwith no intent.Plan finding 3: a warm-snapshot Start writes
suspendedfrom an unvalidated state. This needs no concurrency. Before,start_sandbox_with_authorityrejected onlyfailedahead of the warm path. For example,Start { from_warm: true }on astartedrow staged the base snapshot into the live sandbox's suspend directory, rewrote the row tosuspended, and went on to a checkpoint restore.Red/green tests:
start_refusals_before_activation_leave_the_row_state_unchanged(start/tests.rs) gainsstartedandpausedcases. These are service-level and host-safe: the refusal comes before staging, and a regression fails closed at the test service's unreachable template-manager. Restoring sbx-mgr: add the persisted state-transition table and update_state_from (no callers) #388'sstart.rs, or dropping the pre-check, fails them.completion_and_warm_start_writes_answer_every_source_statesweeps all three writers over a plain row in everySandboxState.suspendedwarm-start row must stay byte-identical, and putting back the old same-state write fails.creating, thencreated, for the two completions;createdandstoppedfor the warm start.failedrow is the writer's contract only, because Start refusesfailedearlier.warm_start_admits_only_its_sources_and_suspended_rowsrunsadmit_warm_starton a row in everySandboxState. A wildcard-free expected match accepts exactlycreated,stoppedandsuspended, and every other state must be refused with the exactFailedPreconditiontext. This pins the accepting set, includingsuspended, which every production warm Start targets.warm_start_write_keeps_a_guard_refusal_abortedcovers rows a concurrent cold Start or restore claimed with an activation episode, plusstoppedandsuspendedrows owned by an episode. All four keep the guard'sAborted. The two hand-rewound rows are the writer's contract only.Behavior changes
Compared with #388's
update_statecall sites:creatingand not guard-owned (e.g.stoppingafter a Stop)created; Create then returnedAborted"sandbox create cleanup episode was no longer current"Aborted"complete sandbox create: sandbox state write was superseded: sandbox is <found>"; the row stays with its ownercreatedand not guard-owned (e.g.stopping)suspended; then the sameAbortedAborted"complete restored sandbox create: sandbox state write was superseded: sandbox is <found>"suspendedbecause an operator CLI warm Start restaged thecreatedrow inside Create's sub-second window between its two writescomplete_createsucceededAborted(message above,suspended). Create'scleanup_after_failurethen requests the still-armeddelete_rowcreate episode, which deletes the rowcreating,starting,started,stopping,pausing,paused,resuming,checkpointing,suspending,restoring,cleanup_required,cleanup_failedwith no owning episode ordelete_rowintentsuspended, then a checkpoint restoreFailedPrecondition"cannot start sandbox <id> from a warm snapshot while local state is <found>; expected suspended, created or stopped", before anything is stageddeleting,cleanup_required/cleanup_failedwith adelete_rowintent, any row with an owning activation episode (e.g.starting/restoring), orquarantinedAborted"…superseded_by_cleanup_episode" or "…blocked_by_quarantine"FailedPrecondition(same new message), before anything is stagedcreated/stopped, moved during staging to a non-guard-owned state outside{created, stopped, suspended}(e.g.stopping)suspendedFailedPrecondition(same message)created/stopped, moved tosuspendedduring stagingfailure_reasonand non-matching receipts, stampedupdated_at, then the restoreAbortedAborted, same "sandbox state write was not applied: missing / blocked_by_quarantine / superseded_by_cleanup_episode" messagesNew client-visible messages:
Aborted"complete sandbox create: sandbox state write was superseded: sandbox is <found>"Aborted"complete restored sandbox create: sandbox state write was superseded: sandbox is <found>"FailedPrecondition"cannot start sandbox <id> from a warm snapshot while local state is <found>; expected suspended, created or stopped"Messages 1 and 2 are #438's canonical
FromStatetext.Other consequences:
startedand now getsFailedPrecondition. Before, it restaged that row for a second restore.Edges, allowlist and fixtures
WARM_START_UNVALIDATED(KnownRace). No edges added.CREATE_COMPLETE,CREATE_BASE_SNAPSHOT_COMPLETEandWARM_START_RESTAGEnow name their new writer functions. The table is not reordered or reformatted.ALLOWLISTremovals. The five pairs only the deleted edge wrote:started,pausing,paused,resumingandcheckpointing→suspended.ALLOWLISTreason refreshes. The class is unchanged; the warm-start race mention is dropped from:creating → suspendedstarting → suspendedrestoring → suspendedcleanup_required → suspendedcleanup_failed → suspendedstopping → suspended, which now reads "known race: process-snapshot rollback in the gate-to-fence window"created → suspendedandstopped → suspendedname the warm start as a legitimate Request and stay.any KnownRaceassertion stay. sbx-mgr: pause writes only over the state it expects #417, which deletes the last two, retires them.db/tests/state_write_fixtures.rs'sinsert_row,force_state,insert_creating,set_failure_reason,seed_receiptandbackdatebecomepub(crate)(test-only module), so the service tests reuse them.cfg_attr(not(test), expect(dead_code))from the items it calls in production:require_state_write_from,StateWriteRejected(and itsinto_statusimpl),StateWriteGuardandstate_write_result.update_state_from_best_effortkeeps its expectation; this PR has no best-effort write.Overlaps with the other conversion PRs in the stack (2, 3, 5)
db/state_write.rs. The deletedWARM_START_UNVALIDATEDsat betweenRESUME_BEGIN_AFTER_STOP(sbx-mgr: wake and resident resume write only over the state they expect #413, PR 3, deletes it) andPROCESS_SNAPSHOT_ROLLBACK_AFTER_STOP(sbx-mgr: snapshot completion and rollback write only over the state they expect #421, PR 5a, deletes it). Both PRs sit above this one; their adjacent-hunk conflicts were resolved by keeping every deletion.ALLOWLIST. Thestopping → suspendedreason now names only the process-snapshot rollback race. When sbx-mgr: snapshot completion and rollback write only over the state they expect #421 (PR 5a), above this PR, deletes that edge, no writer is left for the pair and sbx-mgr: snapshot completion and rollback write only over the state they expect #421 deletes the entry.service/cleanup_support.rs. This PR is the first in the stack to call these items, so it removes theirexpect(dead_code)attributes; the PRs above it no longer touch those lines.db/tests/state_write_fixtures.rs. The PRs above this one that widen the same fixtures build on itspub(crate)lines.service/start.rsandservice/create.rs. No other PR in the stack converts call sites in these files. The plan lists open PRs Add an opt-in Firecracker balloon device per template #212 and Preserve authenticated scope in cached template lookups #217 as touchingstart.rs.Mutation proof
Each mutation was applied alone from a clean backup of the final source. The new and changed tests were run against it (
-E 'test(/state_write_tests|start_refusals_before_activation/)'), and the source was restored before the next. "ALL" widens the edge's from-set in the table toStateSet::ALL.require_state_write(update_state(Created))CREATE_COMPLETEfrom-set → ALLrequire_state_write(update_state(Suspended))CREATE_BASE_SNAPSHOT_COMPLETEfrom-set → ALLrequire_state_write(update_state(Suspended))WARM_START_RESTAGEfrom-set → ALLstart_refusals_before_activation…suspendedrowsuspended(drop theFromStatepredicate)suspendedcase)FailedPreconditionstart.rs: drop theadmit_warm_startpre-staging checkstart_refusals_before_activation…start.rs: restore #388's file (genericupdate_state, no pre-check)start_refusals_before_activation…admit_warm_start: dropsuspendedadmit_warm_start: dropcreatedadmit_warm_start: dropstoppedstart_refusals_before_activation…All 14 were caught and restored. The guard-refusal test fails only under W4 and W5: the generic guards answer before the from-set, so restoring
update_stateor widening the from-set does not change its outcome.Decisions made without an operator
suspendedwarm start. A warm Start that readcreated/stoppedbut findssuspendedat the write treats it as done, which is the existing pre-check's skip. Edges cannot list their target as a source, so it does not write. In that interleaving only, the old same-state write's side effects (failure_reason, receipts,updated_at) no longer happen. That is the one deviation from D4, and it is confined to a no-op transition.Aborted. Both create refusals stayAborted, matchingcomplete_create's supersession answer.quarantinedand every guard-owned state fall outside{created, stopped, suspended}, so the pre-check refuses them withFailedPreconditionbefore staging, as the review asked. Nothing is staged into a quarantined sandbox's directory.Self-assessed severity
Medium. A warm Start of a running or paused sandbox could stage a snapshot over its suspend directory and restage it for a second restore with the VM still live. Its callers are the operator CLI and retries. The create race is unreachable through the Stop RPC.
The change and this description were drafted with an AI coding agent (Claude Code); a human owner reviews before merge.
Validation of the restacked stack
The stack was restacked linearly on main (
b18051f) after #372 and #388 merged: #438 → #418 → #413 → #416 → #421 → #417 → #414. The per-PR validation further down ran on this PR's pre-restack head.This PR's head (
6a1c00f): pull CI is green. fmt, workspace clippy--all-targets --all-features -D warningsandcargo arch(0 required findings) all pass.SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0passed 1638/1638, with 0 dependency skips. Kind Pull CI ran on #417, which contains this PR: all 4 Firecracker smoke shards passed.The top of the stack (#414 @
9a0a251, which contains this PR):cargo test -p pplx-space-sbx-mgr(shared process)smoke-kind-e2e.shpassed.smoke-kind-typed.sh all: native 23/23; ECW 76 passed, 2 infra errors.local.ecw_template_build_from_oci_bootspassed, and so did the leader-election brownout.node_group_boundary_schedulesandnode_selector_extras_render_and_enforce. Both scenarios restart the API Deployment, and the same failure family appears on main in an A/B on the same cluster. In the failed runs the kill never reached sbx-mgr.sandboxes.state, control-plane Postgreslifecycleand the VM agree. No change-induced violation: across 880 sandboxes, no quarantine, panic, restart or stuck state. Every violation reproduces on main with the same signature: an ECW pause that loses to Stop answers 503 instead of 409; after Suspend vs Delete, the node row can outlive the control-plane row until the next eviction pass (pass timing, not the build); and a refused quarantine write after Pause vs Delete is still logged. The soak's 0–200 ms API-level offsets did not reproduce the interleavings this stack closes, so the unit-level race tests remain the direct evidence for those.Validation on the pre-restack head
cargo fmt --all -- --check;scripts/check-rust-error-policy.shcargo clippy --workspace --locked --all-targets --all-features -- -D warningscargo archcreate.rsdrops 1203 → 1199 code lines andstart.rs2320 → 2316; both were already over their capsSPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0#[ignore]privilegedtransfer_directory::tests::distinct_service_uids_can_exchange_downloads, unrelatedkind get clustersempty); bootstrapped from this checkout on the current harness (smoke-kind-e2e.sh, which runsbootstrap_kind_stack)5f097988d319. Both workerspace-sandbox-managerpods' runningspace-sbxmgr-svc(/proc/<pid>/exe) hash6e991ba3…dec, equal toartifacts/bin/space-sbxmgr-svcin thespace-rust-build-kind:localimage Tilt built. The synced build source matches HEAD for every changed file, and the binary contains both new message textsokwith no build in progress, and every pod was Running or Completedsmoke-kind-e2e.sh(baseline,DATA_PLANE_URLset as CI does)smoke-kind-typed.sh all(SKIP_BOOTSTRAP=1, admin token file,SPACE_SMOKE_KIND_COLD_RECEIPTSas CI sets them; unsharded on one stack)require_state_write_fromreturnsResult<(), Status>, plus #438's own test and doc changes) and by dropping two now-redundant.map(drop)calls. Per the owner, Kind was not rerun for that rebase; nextest wasinfra/space/kind/scripts/teardown.sh: Tilt stopped, cluster deleteddocker builder prune -afonce (authorized), between scenarios with Tilt idle, when the Docker data disk reached 77%🤖 Generated with Claude Code