Skip to content

sbx-mgr: retire the generic update_state - #498

Merged
ac-pplx merged 1 commit into
mainfrom
ac/sbxmgr-sw-pr6-retire-update-state
Oct 15, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/sbxmgr-sw-pr6-retire-update-state

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Follows the merged sbx-mgr state-write stack (#438 → #418 → #413 → #416 → #421 → #417 → #414). This is PR 6 of the state-write plan (reports/state-write-helper-plan.md on ac/state-write-helper-plan, §5 row 6).

All state here is sbx-mgr's node-local SQLite sandboxes.state. Nothing touches the control-plane Postgres public.sandboxes.lifecycle.

What changes

Every persisted write of the node-local SQLite sandboxes.state now goes through update_state_from with an explicit edge, or through a named specialized writer. The generic update_state is deleted. So are SandboxStateWriteOutcome, the legacy require_state_write, and CleanupCompletionError::StateWriteNotApplied, which nothing constructed. Net production change: 9 files, +203 / −273.

Stop and Delete admission is not on an edge. Owner rule: if (c) is not behavior-preserving with any exact edge, leave it on a private, clearly named non-generic helper and report why.

  • Stop and Delete admission (update_state_superseding_activation_in_transaction) legitimately re-admits a row already in its target:
  • The edge table forbids an edge from listing its own target (no_edge_lists_its_own_target_as_a_source). It also says a same-state write "changes no state and has no edge".
  • So no exact edge preserves behavior. The write moves to write_cleanup_admission_state, a private writer for stopping and deleting only. It writes over every state the generic guards admit, as admission always has. It uses update_state_from's statement, guards and IMMEDIATE handling. The guard warnings are update_state's, moved into this writer: update_state_from emits none.
  • STOP_ADMISSION and DELETE_ADMISSION keep listing the transitions it makes.

Scope

Item What I did
1a — deleting branch inside update_state No production caller: admission always writes inside its own transaction. Deleted with update_state. update_state_from already opens IMMEDIATE in autocommit for every target, so the deletion and its teardown_jobs::fail_active still commit together (pinned by a_standalone_deletion_rolls_back_when_its_job_failure_does). Test callers that wrote deleting now use edges::DELETE_ADMISSION.
1b — begin_resident_resume_with_policy update_state_from(&tx, id, &edges::RESUME_BEGIN) (Request, {paused} → resuming) inside the same IMMEDIATE policy transaction. The edge refuses a non-paused row, so the duplicate state != Paused pre-check is deleted. The generation fence, the transaction and the Ok(false) on any refusal are unchanged. Edge doc updated. Log-only change: update_state warned when a guard refused a paused row, and update_state_from is silent. That case is likely unreachable here.
1c — Stop/Delete admission writer Moved to write_cleanup_admission_state, as described above. Lock, transaction, client messages and side columns are unchanged. The writer also returns InvalidQuery for a target other than stopping or deleting. That duplicates the debug_assert still in update_state_superseding_activation_in_transaction, only matters in release builds, and no production caller passes another target.
2 — retire update_state Deleted. rg -P '(?<![\w])update_state\(' finds nothing in any file. Of the #372 matrix, only the update_state writer arm and its two tests are deleted. The same grids, row diffs and shapes already run through update_state_from with every state expected, standalone and in a transaction (#388), so every guard cell stays pinned. In db/tests.rs, the guard tests now target production edges. The failed-target assertion is dropped because the matrix's X column pins it. Seed-only test callers use an edge where production has one, force_state where only the state matters, and an explicit-source update_state_from_states for the two started → restoring seeds.
3 — legacy outcome path SandboxStateWriteNotApplied is now Guard(StateWriteGuard) | Missing. Its Display holds the three client-visible literals, byte for byte. into_status, checkpoint admission and cleanup admission render through it. In the #438 parity tests, the legacy comparison and update_state_outcomes_keep_their_statuses are dropped; the literal-text assertions stay.
4 — collapse the guard enums db::StateWriteGuard { Quarantined, CleanupOwned }. generic_state_write_refusal returns Option<StateWriteGuard>. StateWriteRefusal::{FromState, Guard(StateWriteGuard)}. Every match is wildcard-free, and the "guards never answer FromState" arm is gone.
5 — Stop fixture admit_stop, complete_stop and stop in db/tests/state_write_fixtures.rs replace the copies in pause_resume/supersession_tests.rs, auth_lookup/resume_begin/tests.rs, create_support/state_write_tests.rs, grpc_state_snapshot/tests.rs and grpc_spaced_process_snapshots/tests.rs. reconcile/tests.rs's blocking Stop (no intent) is a different fixture, so it stays.
6 — sbx-mgr AGENTS.md Every persisted write goes through update_state_from with an explicit edge, or through a named specialized writer. A new edge needs a kind and a doc comment naming its writer. Map outcomes with require_state_write_from / state_write_result.
New coverage delete_admission_answers_every_source_state. Delete re-admission of a deleting row was unpinned: narrowing the Delete from-set by deleting failed no test.
Review comment from the stack approval — 1 update_state_from_best_effort logs Missing with its own message ("found no sandbox row").
Review comment from the stack approval — 2 require_state_write_from takes context: &str, as into_status does.
Review comment from the stack approval — 3 The pause comment now says mark_quarantined_best_effort also answers false on a poisoned connection or a DB error. No behavior change.
Review comment from the stack approval — 4 A refused startup recovery write is StateWriteRefused { edge, found, refusal }, rendered explicitly. A missing row is StateWriteMissing { edge }. No {outcome:?}.
Review comment from the stack approval — 5 delete_admitted_during_auto_lifecycle_stop_keeps_its_row: a deleting row with a current delete_row intent, which the CleanupOwned guard refuses, is skipped and kept.
Review comment from the stack approval — 6 One-line comments on suspend's guest-prep rollback refusal and auto-pause refusal: both are unreachable under suspend's fence and locks, so they answer Internal. No behavior change.
Review comment from the stack approval — 7 The write_warm_start_restaged doc says why Guard { found: Suspended } stays Aborted while FromState { Suspended } succeeds. It also says FailedPrecondition for a transient row, versus cold Start's Unavailable, is deliberate (plan §5).
Round-1 review: admission's same-state write was unpinned activation_supersession_retires_the_episode_before_the_generic_write now re-admits a decorated stopping and a decorated deleting row: failure reason, a started receipt, backdated timestamps, and an active teardown job for deleting. It asserts the whole-database diff.
Round-1 review: the cleanup-owned refusal text was unpinned The Stop sweep asserts the full Unavailable message for deleting (superseded_by_cleanup_episode) and quarantined (blocked_by_quarantine).
Round-1 review: the matrix adapter's catch-all update_state_matrix's writer adapter matches every outcome explicitly and asserts the seeded state separately.
Round-1 review: stale doc references Dropped the plan parenthetical in create_support.rs. The three test docs that named update_state now say "the generic guards' message" or "a same-state generic write".
Review comment from the stack approval — 8 Checkpoint admission no longer rebuilds the legacy outcome. The text is byte-identical: the existing guard literals are pinned, and checkpoint_admission_of_a_missing_row_reports_it pins missing.
cargo arch db/state_transitions.rs cap lowered 1140 → 1018. No cap raised. My change shrinks service/cleanup.rs by 2 lines; its advisory growth over the cap is already on main.

Changes beyond the refactor

  • Behavior change, accepted: a row whose state label this binary cannot parse now fails Stop admission with FromSqlConversionFailure, where it used to be overwritten with stopping. Delete is unchanged: it already failed earlier, in get_sandbox. This is reachable only after a downgrade or with a corrupt row. No tolerance was added.
  • Log-target rename: admission's two guard warnings ("ignored generic sandbox state update because …") moved with the code. Their text and fields are unchanged, but their tracing target is now pplx_space_sbx_mgr::db::state_write instead of …::db::state_transitions. The old target is not pinned.

Out of scope, as instructed: finding 8's dead and test-only writers (including mark_stopping_superseding_activation, mark_deleting_superseding_activation and update_state_superseding_activation), the unreachable cleanup-admission branches, and D2/D4. Side columns are untouched.

Mutation proof

Each mutant was applied alone, run against the full sbx-mgr suite with --retries 0, then reverted.

Mutant Failing tests
(b) widen RESUME_BEGIN to all states resident_policy_resume_is_generation_fenced_and_arms_only_after_thaw, policy_free_resume_writes_resuming_only_over_paused, and two edge-table tests
(b) narrow RESUME_BEGIN by paused (empty) the same two behavioral tests, plus cleanup_intent_cannot_be_replaced_by_suspend_or_resident_completion, start_arms_access_idle_only_after_guest_readiness and an edge-table test
(c) widen Identity: the writer already writes over every state the guards admit.
(c) narrow the Stop from-set by stopping stop_admission_answers_every_source_state
(c) narrow the Delete from-set by deleting delete_admission_answers_every_source_state (new). Before that test existed, nothing failed.
(a) drop the autocommit IMMEDIATE wrapper a_standalone_deletion_rolls_back_when_its_job_failure_does
Review 5: auto-lifecycle Stop skips only FromState delete_admitted_during_auto_lifecycle_stop_keeps_its_row, auto_lifecycle_stop_writes_only_over_a_still_suspended_row
Round 1: admission skips a same-state write (if found == target { return Ok(Ok(())) }) activation_supersession_retires_the_episode_before_the_generic_write
Round 1: admission maps both guards to Quarantined stop_admission_answers_every_source_state

Validation

Check Result
cargo fmt --all --check pass
cargo clippy --workspace --all-targets --all-features -- -D warnings pass, before and after the rebase
pre-commit (gitleaks) pass
cargo arch 0 required findings; the only stale entry was lowered
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-sbx-mgr --retries 0 ×3 (pre-rebase head) 1650/1650 each run, 0 dependency skips. The 1 skip is the #[ignore]d root-only distinct_service_uids_can_exchange_downloads.
The same, once after the rebase onto main 1652/1652, same single #[ignore] skip
The same, once after the round-1 fixes 1652/1652, same single #[ignore] skip
Kind: fresh cluster, pre-rebase head 099710b All 4 sbx-mgr pods log git_sha=099710b2e779, and the deployed space-sbxmgr-svc contains strings only this change adds. Tilt was quiet (46 ok, 4 not applicable, none pending). smoke-kind-e2e.sh: PASS. smoke-kind-typed.sh all with SPACE_SMOKE_KIND_ADMIN_TOKEN_FILE and SPACE_SMOKE_KIND_COLD_RECEIPTS set as CI does: ECW 78/78 and native 23/23, 0 failed, 0 inconclusive, 0 quarantined. Torn down with infra/space/kind/scripts/teardown.sh.

Kind ran on the pre-rebase commit. The rebase onto main applied without conflicts, and this change's diff is the same on both bases. The round-1 fixes touch only tests, comments and this text, so Kind was not rerun.

🤖 Generated with Claude Code

@ac-pplx
ac-pplx force-pushed the ac/sbxmgr-sw-pr6-retire-update-state branch from ab56b47 to 3434db0 Compare October 14, 2026 22:19
Every write of sbx-mgr's node-local SQLite `sandboxes.state` now goes
through `update_state_from` with an explicit edge, or through a named
specialized writer. Nothing here touches the control-plane Postgres
`public.sandboxes.lifecycle`.

The three remaining db-internal `update_state` callers:

- The `deleting` re-entry inside `update_state` had no production caller:
  Stop and Delete admission always write inside their own transaction.
  `update_state_from` already opens an `IMMEDIATE` transaction in
  autocommit for every target, so the deletion and the teardown jobs it
  fails still commit together.
- `begin_resident_resume_with_policy` writes `edges::RESUME_BEGIN`
  (`paused` -> `resuming`) inside its policy transaction. The edge now
  refuses a non-paused row, so the separate state pre-check is gone; the
  generation fence, the lock and the transaction are unchanged.
- Stop and Delete admission
  (`update_state_superseding_activation_in_transaction`) cannot use an
  exact edge without changing behavior. Admission re-admits a row that is
  already in its target: a `stopping` row for a repeated Stop of a live
  runtime, and a `deleting` row for a repeated Delete. No edge lists its
  own target. The write therefore moves to `write_cleanup_admission_state`,
  a private writer for only those two targets. It writes over every state
  the generic guards admit, as admission always has, using
  `update_state_from`'s statement and guards. `update_state`'s guard
  warnings move with it; their text and fields are unchanged, but the log
  target is now `db::state_write`. `STOP_ADMISSION` and `DELETE_ADMISSION`
  keep listing the transitions it makes. One behavior change: a row whose
  state label this binary cannot parse, reachable only after a downgrade or
  with a corrupt row, now fails Stop admission instead of being overwritten
  with `stopping`. Delete already failed earlier, in `get_sandbox`.

`update_state`, `SandboxStateWriteOutcome`, the legacy
`require_state_write` and `CleanupCompletionError::StateWriteNotApplied`
are deleted. The guard types collapse: `db::StateWriteGuard { Quarantined,
CleanupOwned }`, `generic_state_write_refusal` returns
`Option<StateWriteGuard>`, and `StateWriteRefusal` is
`FromState | Guard(StateWriteGuard)`. That removes the unreachable
"guards never answer FromState" arm. `SandboxStateWriteNotApplied` becomes
`Guard(StateWriteGuard) | Missing` and renders the same client-visible text
byte for byte. `into_status`, checkpoint admission and cleanup admission
all render through it.

Tests:

- The `update_state` outcome matrix already ran through `update_state_from`
  with every state expected, standalone and in a transaction (#388). Only
  its `update_state` arm is dropped, so every guard cell stays pinned. The
  guard tests in `db/tests.rs` now target production edges, and the #438
  status tests keep their literal texts without the legacy comparison.
- New: admission's same-state write. Re-admitting a `stopping` or
  `deleting` row clears the failure reason and another target's receipt and
  advances `updated_at`. For `deleting`, it also fails the active teardown
  jobs. The Stop sweep now pins the full client text for `deleting` and
  `quarantined` rows.
- New: Delete admission from every source state. It admits every plain row,
  a `deleting` row included, except a `suspending` row whose suspend has
  not reached the cluster. Nothing pinned Delete re-admission before.
- Test-only callers use an edge where production has one, `force_state`
  where only the state matters, and an explicit-source
  `update_state_from_states` for the two `started` -> `restoring` seeds.
- The five copies of the Stop fixture (superseding write, `mark_stopped`
  intent, row-intent completion) are now `admit_stop`, `complete_stop` and
  `stop` in `db/tests/state_write_fixtures.rs`.

Review comments from the stack's approval, folded in:

- `update_state_from_best_effort` logs a missing row with its own message,
  instead of "superseded".
- `require_state_write_from` takes `context: &str`, as `into_status` does.
- Pause's comment on `mark_quarantined_best_effort` now says it also
  answers false on a poisoned connection or a database error.
- A refused startup recovery write names the edge, the found state and the
  refusal, instead of printing the outcome's `Debug` form. A missing row
  has its own error.
- Auto-lifecycle Stop gains a case for a Delete admitted mid-write: a
  `deleting` row with a current `delete_row` intent, which the
  cleanup-ownership guard refuses and the Stop skips.
- Suspend's guest-prep rollback refusal and auto-pause refusal still answer
  `Internal`. Both are unreachable under suspend's fence and locks, and a
  comment now says so.
- Warm start's doc says why a guarded `suspended` row stays `Aborted`, and
  that answering a transient row with `FailedPrecondition` is deliberate.
- Checkpoint admission renders through `SandboxStateWriteNotApplied`, not
  the legacy outcome. A missing-row case pins its text alongside the
  existing guard literals.

The sbx-mgr AGENTS.md writer rule is updated, and the archcheck cap for
`db/state_transitions.rs` is lowered to its new size.
@ac-pplx
ac-pplx force-pushed the ac/sbxmgr-sw-pr6-retire-update-state branch from 3434db0 to cd14056 Compare October 14, 2026 22:30
@ac-pplx
ac-pplx marked this pull request as ready for review October 14, 2026 22:54
@the-gigi-pplx

Copy link
Copy Markdown
Collaborator

Review: 1 finding

Low

  1. crates/libs/sbx/mgr/server/src/db/startup_recovery.rs:384: refusal_text is a second hand-maintained StateWriteGuard → text map.
    guard_label in db/state_write.rs:456 already maps the same two variants, and the two disagree on wording (blocked_by_quarantine vs "blocked by quarantine", superseded_by_cleanup_episode vs "owned by cleanup"). write_cleanup_admission_state adds a third per-variant match for its two warnings. A new guard variant means touching all three, and nothing ties their wording together. One option: give StateWriteGuard an as_str() (the client-visible literal), and have refusal_text and the admission warning use it. Take it or leave it. The startup-recovery text isn't client-visible, so the divergence costs only grep-ability.

Checked and not reported

  • Stop admission over a deleting row that the eviction claim wrote (no cleanup_completion set). After admission deletes the episode, the delete_row intent clause still refuses it: CleanupCompletion::owns_intent (db.rs:202-203) treats DeleteRow as owned whenever state == deleting, so the doc on write_cleanup_admission_state saying the guards refuse a Stop of a deleting row holds.
  • write_state_from_in_transaction against the deleted update_state. It runs the same SET list, the same guards (now reading found instead of previous), and the same teardown_jobs::fail_active on deleting. The only differences are the unparseable-label error and the always-IMMEDIATE autocommit wrapper, both called out in the description. StateSet::ALL = 18 bits against an exhaustive bit() match, so a new state can't silently fall outside it.
  • begin_resident_resume_with_policy: dropping the state != Paused pre-check is covered by RESUME_BEGIN's from-set, which is {paused}, inside the same transaction.
  • update_state_from_states is #[cfg(test)], so the new AGENTS.md rule ("update_state_from with an edge, or a named specialized writer") holds for production code. No references to update_state(, SandboxStateWriteOutcome or require_state_write remain outside test docs.

Verified against cd14056ba04d18f4cfb4ed8998407c33af624241. I didn't run anything locally. Every point above comes from reading the source at that commit. I relied on CI's green run (Rust tests, clippy, format) for the suite and didn't re-run the mutation table.

@the-gigi-pplx the-gigi-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. 1 non-blocking finding in the review comment above.

@ac-pplx
ac-pplx added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit 6ef2d61 Oct 15, 2026
20 checks passed
@ac-pplx
ac-pplx deleted the ac/sbxmgr-sw-pr6-retire-update-state branch October 15, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants