[the-gigi-pplx]

Summary

Adds SubstrateAdapter::upgrade_stopped_runtime, which moves a stopped Kubernetes sandbox to the cluster's default runtime environment and leaves it stopped, on the node that holds its disk. An upgrade that doesn't finish leaves a disk backup owed, which every later activation of the sandbox pays before it takes the execution. Nothing calls the operation yet: the test-only trigger and a bench run come in PR #1322, at the top of the stack.

Terms

• Stay-stopped upgrade: upgrade_stopped_runtime, which boots a stopped sandbox's disk under the target environment and stops it again, so the row stays stopped throughout.
• Execution: the sandbox's active sandbox_k8s_executions row, which names its pod, its node and its generation.
• Claim: an activation attempt assigned to the execution (activation_attempt_id, with activation_attempt_target_state and an expiry). Its lease is ACTIVATION_ATTEMPT_LEASE_TTL_SECS, 300 s. A live claim makes every other activation refuse; an expired one can be taken over.
• Recorded upgrade: runtime_upgrade_operation_id on the execution, which the stay-stopped upgrade writes in its claim's transaction, before any pod exists, so a retry renders the same operation.
• Owed backup: the current-instance disk backup an unfinished stay-stopped upgrade may still owe. It's owed while the execution still records the upgrade and its claim has expired.
• Disk pin: the node the sandbox's placement says holds its local disk: active_node_id, with no other target_node_id chosen.
• Stopped evacuation marker: evacuation_restore_target = 'stopped', which a forced drain writes to claim a stopped sandbox for evacuation.

The problem

The only way the control plane moves a sandbox to a new runtime environment today is the decision that a start or a resume makes (PR #1708), and that always ends with the sandbox running. A campaign that upgrades every sandbox in a cluster would therefore have to start sandboxes their owners stopped.

Booting a stopped sandbox without starting it breaks assumptions the Kubernetes substrate relies on:

1. An execution exists while the row is stopped, which the claim, retirement and rollback paths don't expect. The ordinary retire clears the sandbox's placement, so a drain would no longer see that node holds the sandbox's only local disk.
2. The boot rewrites the disk, so the durable copy is stale until a backup of the upgraded disk is enqueued. An upgrade that's interrupted after the boot and before the backup leaves no trace of that debt. The next start would clear the record and boot over a disk with no matching backup.
3. kubelet keeps retrying the upgrade pod's create after the upgrade's own call gives up, and each retry is another boot. Anything that captures the disk has to make sure no boot can follow the capture.

How this fixes it

• Run the upgrade in place. kubernetes/adapter/stopped_runtime_upgrade.rs runs the prechecks first: the sandbox is stopped and unencrypted, isn't already on an allowed environment, has its disk on the disk pin's node, and that node can take work. It then claims the execution and records the upgrade in one transaction. It boots through the runtime-upgrade pod mode pinned to that node and reads the outcome as PR [api] boot a start's or resume's runtime upgrade in its pod and read its outcome from the node #1708's start and resume do. It then backs up the upgraded disk and retires the execution without touching the placement, so a drain still lists, waits for and evacuates the sandbox.
• Keep the claim honest. In kubernetes/store/activation_fence.rs, kubernetes/store/execution_rows.rs and kubernetes/store/runtime_upgrade.rs, the claim refuses when another operation's record is present, the disk pin moved, or a stopped evacuation is marked. A retry takes over the expired claim and keeps its record. Only the upgrade's own claim keeps the record; any other claim clears it, and only once its owed backup is enqueued.
• Pay the owed backup before any other activation. A stopped or restoring materialization first takes over the expired claim, then deletes the upgrade's pod and waits until it's gone. It then has the disk node abandon the operation (AbandonRuntimeUpgrade), enqueues the backup on that node, and retires the upgrade's execution. Only then does it claim. A disk node that's gone or holds no such sandbox answers 412, and so does one that refuses the abandon as invalid (it records the operation for another sandbox, or an id doesn't parse), since no retry changes that. Every other abandon failure stays retryable.
• Keep a recorded execution through pod cleanup. In kubernetes/pod_ops/ensure_pod.rs and kubernetes/pod_ops/started_recovery.rs, a start never reuses an unfinished upgrade's placeholder pod. A failed create or a stale-pod cleanup keeps an execution that records an upgrade, so its backup debt survives.
• Give the drain what it needs. kubernetes/stopped_evacuation.rs and kubernetes/store/stopped_evacuation.rs add the substrate's side of a drain's stopped evacuation: open it only when no upgrade holds a live claim, find the execution an upgrade left, remove its pod and abandon its operation. stopped_execution_retirement_refused (crates/l30_storage/sandbox/store/src/pod_tracking.rs) lets that evacuation retire an expired upgrade's execution. disk_pin_node_name (crates/l30_storage/sandbox/store/src/restore_source.rs) reads the disk pin with no fallback to execution history.
• Keep the placement on rollback. Internal-api's activation rollback of a stay-stopped upgrade retires only the execution and leaves the sandbox placed on its disk node (crates/l40_api/internal/api/server/src/sandbox_quarantine_store.rs).

Direct answers Unsupported, with a gap row in direct.rs. Starts, resumes and restores are unchanged for an execution that records no stay-stopped upgrade, which is every execution until something calls the operation. The cluster controller's drain doesn't call the new evacuation helpers yet; the next pull request in the stack, PR #1733, does.

Validation

At this head, stacked on PR #1708 at d99f199, on a Linux build box:

• cargo clippy --locked --all-targets --all-features -- -D warnings for pplx-space-sandbox-store, pplx-space-api, pplx-space-internal-api-server and pplx-space-cluster-controller-server: clean. The cluster controller compiles unchanged on top of this.
• cargo nextest run --all-features for the same four crates plus pplx-space-db-test-fixtures, with the api and sbx-mgr test builds at opt-level 0 as CI builds them, against Postgres: 4721 passed, 1 skipped.
• New tests, against Postgres, a scripted node-api and a scripted Kubernetes API:
  • kubernetes::tests::stopped_runtime_upgrade (129 tests), split into outcomes.rs, backup.rs, pods.rs, retire.rs, claims.rs and prechecks.rs, on the shared fixture in tests/runtime_upgrade_fixture.rs. These include every_upgrade_outcome_leaves_the_sandbox_placed_on_its_disk_node, an_owed_backup_captures_only_once_the_node_abandoned_the_upgrade, an_owed_backup_waits_until_the_upgrades_pod_is_gone, an_owed_backup_whose_operation_the_node_records_for_another_sandbox_is_refused_for_good, a_start_never_reuses_an_unfinished_upgrades_placeholder_pod and an_open_stopped_evacuation_refuses_the_upgrade.
  • kubernetes::tests::leftover_upgrade_pod: recorded UID versus carried operation, an already-deleting pod, a 409 or 404 delete, read and delete failures, and a gone pod going on to the node's abandon.
  • substrate::tests::a_not_yet_stopped_backup_refusal_is_the_conflict_it_was.
  • Internal-api: activation_completion_receipt_never_moves_a_stopped_runtime_upgrade and activation_rollback_of_a_stopped_runtime_upgrade_keeps_its_placement.

Not validated here: a Kind run. Nothing in this pull request calls the operation, so no Kind scenario reaches it. PR #1322, which adds the test-only trigger, carries the bench run.

Not covered

• The drain doesn't call the evacuation helpers here yet. open_stopped_evacuation, leftover_stopped_upgrade and settle_expired_stopped_upgrade have no production caller in this pull request, and the cluster controller's stopped evacuation still claims through claim_stopped_evacuation directly. The next pull request of the stack wires them in. Until then no execution records a stay-stopped upgrade, since nothing calls upgrade_stopped_runtime, so the loosened retirement check in stopped_execution_retirement_refused matches no row.

• The stay-stopped orchestration (decision, claim, takeover, backup ordering, retire and the owed backup) lives in the Kubernetes adapter, kubernetes/adapter/stopped_runtime_upgrade.rs. The substrate contract (crates/l40_api/api/src/controllers/substrate/AGENTS.md) puts new core logic above the adapter, so this is debt, not precedent. Moving it means a shared stay-stopped controller under controllers/sandbox/ and new required SubstrateAdapter methods for the Kubernetes effects, with Direct returning Unsupported under gap rows. It's left for before Direct decides runtime upgrades or the campaign that calls this lands.

• Nothing reclaims an abandoned stay-stopped upgrade's placeholder pod and execution that nobody retries. They stay until a user start, a drain or a delete; the campaign that calls the operation owns retrying them.

• The owed-backup check runs only before a stopped or restoring materialization claims the execution.

• A start can't proceed from the durable copy when the disk node is gone for good while a backup is owed. That needs the stop path to enqueue and confirm a stopped-disk backup first, and an ordinary Kubernetes stop enqueues none today (run_stop_or_kill in crates/l40_api/api/src/controllers/sandbox/lifecycle.rs). Until then the owed-backup gate answers 412 and the operator path is to delete the sandbox.

• The operation's timeout bounds the boot and its outcome wait only. The backup enqueue, pod delete and database writes that follow run under their own bounds, so the call can return later than its timeout by their duration.

• A user start of a stopped sandbox during an open stopped evacuation can take over an expired stay-stopped upgrade's claim for its owed backup (reserve_expired_stopped_upgrade in kubernetes/store/stopped_evacuation.rs doesn't read the evacuation marker). That's deliberate: the marker blocks only the upgrade's own claims and takeovers (stopped_evacuation_marked_in in kubernetes/store/activation_fence.rs), and a user start is a takeover of the evacuation itself, which clears the marker once the row leaves stopped (claim_stopped_evacuation in crates/l30_storage/sandbox/store/src/drain.rs). Each order of the two goes through. The owed backup deletes the same pod and abandons the same operation as the evacuation, both idempotent. Its enqueue doesn't move the node's disk pin, since a stopped sandbox's disk backup mints a pin only from a pinless row and otherwise reuses the existing one, so the evacuation's minted artifact stays pinned. Its live reservation defers the evacuation's teardown only until it retires the upgrade's execution.

• The success path enqueues the upgraded disk's backup before it deletes the upgrade's pod (retire_upgrade_boot in kubernetes/adapter/stopped_runtime_upgrade.rs), the reverse of the owed backup's order. That's sound here because it runs only after the node reported the upgrade reached its target and published the sandbox stopped under it (boot_for_upgrade). A kubelet retry of that pod's create can't write the disk afterwards. Sandbox-manager answers a retired operation as a replay that never touches the sandbox, and one whose sandbox is already on the target as already on target, without booting (upgrade in crates/l40_api/sbx/mgr/server/src/service/runtime_upgrade.rs). The owed backup deletes the pod first because its upgrade may not have reached that point.

Dependencies

• PR [api] boot a start's or resume's runtime upgrade in its pod and read its outcome from the node #1708 is this pull request's stacked base. It sits on PR [sbx-mgr] keep a pod's runtime upgrade finality when its sandbox row is deleted #1707, which sits on the chain PR [sbx-mgr] answer a backup of a sandbox not yet at rest as retryable #1750 → PR [sbx-mgr] hold suspends while a runtime upgrade that ends stopped runs #1751 → PR [sbx-mgr] publish the execution a rollback restores while a runtime upgrade fences a sandbox #1752 → PR [sbx-mgr] add the per-attempt runtime upgrade fields and AbandonRuntimeUpgrade to the wire #1753 → PR [sbx-mgr] record runtime upgrade outcomes per attempt id in the store #1754 → PR [sbx-mgr] make a runtime upgrade rollback or refusal final for its attempt #1755 → PR [sbx-mgr] add AbandonRuntimeUpgrade, so no late call for an abandoned operation fences or boots #1756 → PR [sbx-mgr] keep a pod's runtime upgrade finality when its sandbox row is deleted #1707. Merge them in that order, then PR [api] boot a start's or resume's runtime upgrade in its pod and read its outcome from the node #1708, then retarget this pull request to main.
• After this: PR cluster-controller: evacuate a stopped sandbox an unfinished stay-stopped runtime upgrade left, and test the sweepers against it #1733 (the cluster controller's drain and sweepers), then PR [api] trigger a stopped sandbox's runtime upgrade from a test hook, and smoke it on Kind #1322 (the test-only trigger, Kind settings, smoke scenario and docs).

Severity and risk

Severity: medium. Without this there's no way to upgrade a stopped sandbox's runtime without starting it.

Risk: low. Nothing calls upgrade_stopped_runtime, so nothing records a stay-stopped upgrade. The owed-backup check, the claim's gate and internal-api's rollback projection run on the start, restore and rollback paths, but they act only on an execution that records a stay-stopped upgrade. Roll back by reverting. No Postgres migration is involved.

Docs

None in this pull request. "Upgrading a stopped sandbox without starting it" in docs/execution-environments.md, the long form of all of this, lands with PR #1322.

AI assistance: Claude Code split this pull request out of PR #1322 and drafted this description.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

Failed checks: 2 inconclusive

Check name	Status	Explanation	Resolution
Title check		This assessment has not completed. The check will be retried on the next pre-merge check run.
Weakened Safeguards Are Justified		The check asks whether this diff removes or weakens a test, assertion, lint, CI check, or repository guard without a stated reason. I could not establish the diff. The local checkout is a single root …	1. Fetch the PR base d99f199 (or the stack base PR #1708 head) and run git diff d99f199e2 HEAD so the full patch can be read. 2. Check every added line for #[allow(, #[expect(, #[ignore, should_panic, pytest.mark.skip/xfail,…

Passed checks: 4 passed

Check name	Status	Explanation
Description Follows The Pr Template		The PR description contains all four required items with real content: 1. Summary: A ## Summary section states that the change adds SubstrateAdapter::upgrade_stopped_runtime. It moves a stoppe…
Bug Fixes Name A Regression Test		The check applies only when the PR fixes a defect in application behavior under crates/, shim/, or ui/. This PR does not fix a defect. Its title and summary describe a new capability: `SubstrateAdapte…
Agent Tooling Runs No New Commands		The check fails only when the PR diff adds or changes agent tooling that runs commands or starts servers. The repository checkout is a single squashed commit (c473b42, no parent), so the exact PR diff…
Public-Safe Content		Summary: no content in the title, description, or the PR's named source files matches the public-safe failure conditions. Scope and limits: - The checkout is an orphan commit (c473b42) with no base. `…

Full details: Description Follows The Pr Template

Explanation

The PR description contains all four required items with real content: 1. Summary: A ## Summary section states that the change adds SubstrateAdapter::upgrade_stopped_runtime. It moves a stopped Kubernetes sandbox to the default runtime environment and leaves it stopped. 2. Validation: A ## Validation section names the commands run (cargo clippy, cargo nextest run, and the new test suites) and their results (clippy clean; 4721 passed, 1 skipped). It also states what was not validated. 3. Severity and risk: A ## Severity and risk section gives "Severity: medium" and "Risk: low", with the reasoning for each. 4. AI assistance: A line begins with "AI assistance:" and names the tool and its part: "Claude Code split this pull request out of PR #1322 and drafted this description." No required section is missing or limited to template comments. Presence is the only criterion, so I did not assess quality. No repository tools were needed for this description-only check.

Full details: Bug Fixes Name A Regression Test

Explanation

The check applies only when the PR fixes a defect in application behavior under crates/, shim/, or ui/. This PR does not fix a defect. Its title and summary describe a new capability: SubstrateAdapter::upgrade_stopped_runtime. The "problem" section describes a missing capability, not a failing behavior. The PR says that no caller exists yet and that the operation has no production path. The hazards it lists, such as an execution that exists while the row is stopped or an owed backup after an interrupted boot, are design constraints for the new operation. They are not regressions in shipped behavior. The check is therefore inapplicable, and the PR passes. I did not gather further evidence because the PR metadata alone decides the check.

Full details: Agent Tooling Runs No New Commands

Explanation

The check fails only when the PR diff adds or changes agent tooling that runs commands or starts servers. The repository checkout is a single squashed commit (c473b42, no parent), so the exact PR diff could not be isolated. The result does not depend on that gap. I checked the post-change state of every agent-tooling file and found no failing content, so the PR cannot have introduced any: - Plugin manifests (.agents/plugins/pplx-space/.claude-plugin/plugin.json, .codex-plugin/plugin.json, plugin.yaml) contain only name, version, description, author, interface text, and the skills path. No hooks, MCP, LSP, or command entries. - .agents/plugins/marketplace.json contains only the plugin source and policy. - Every SKILL.md frontmatter (9 skills) has only name and description. No hooks or allowed-tools keys. - No !`` inline shell substitution appears in any .mdfile in the repository, including skills, references, assets, and theAGENTS.md/CLAUDE.mdguidelines. - Theagents/openai.yamlfiles contain only display and prompt text. - The onlyhook/mcp/lsphits in guideline files are the Cargotest-hooksfeature name and unrelated prose. The Rust-focused PR description does not mention agent files. Scripts underskills/*/scripts/` exist, but they are not manifest entries or substitutions. The check does not list them, and I did not find any agent-file reference that makes them run automatically. The PR's own description and Rust changes do not touch them.

Full details: Public-Safe Content

Explanation

Summary: no content in the title, description, or the PR's named source files matches the public-safe failure conditions. Scope and limits: - The checkout is an orphan commit (c473b42) with no base. main shares no history with it, so the exact PR patch could not be isolated. - I checked the title and the full description directly. - I scanned the files the description names, plus their test directories, for credentials, private keys, tokens, ARNs, 12-digit account IDs, internal hosts, private URLs, personal absolute paths, and cluster or environment names. Findings: - Title and description: no URLs, hostnames, credentials, account IDs, personal paths, production logs or measurements, or incident details. PR numbers and file paths are public repository references. The description describes known limitations and design behavior. It does not disclose an unfixed security vulnerability. - Named source and test files: no secret or private-value matches. The only hits were benign. space.pplx.net/node-api-port is the project's own annotation key. "staging" appears only in a generic doc comment. "kind-" and similar terms do not appear as environment names. - Test fixtures use placeholder values such as 00000000-… UUIDs and sha256:0707…. Residual gap: files in the PR that the description does not name were not individually scanned. The check does not require a missing-diff FAIL. Based on the evidence gathered, no failure condition is established.

Full details: Weakened Safeguards Are Justified

Explanation

The check asks whether this diff removes or weakens a test, assertion, lint, CI check, or repository guard without a stated reason. I could not establish the diff. The local checkout is a single root commit (c473b42) with no parent. The stated base, PR #1708 commit d99f199, is not present, and main is an unrelated older commit. So removed tests, assertions, and config lines cannot be compared against the base. Evidence checked on the current tree: - The new and touched PR modules have no #[ignore], #[should_panic], #[allow], or #[expect] markers. This covers stopped_runtime_upgrade.rs, stopped_evacuation.rs, the stopped_runtime_upgrade/ test split, and runtime_upgrade_fixture.rs. - No config or baseline file (ARCHITECTURE.toml, architecture-baseline.toml, deny.toml, clippy.toml, .gitleaks.toml, ci/kind-pull-paths.txt, ci/kind-direct-known-failures.txt, .config/nextest.toml) references any new PR path. SPACE_TEST_DEPS_OPTIONAL is set only in docs and scripts, not in CI or test config. - Non-test #[allow] at crates/l40_api/api/src/controllers/substrate/kubernetes/store/pod_tracking.rs:573 (#[allow(clippy::too_many_arguments)] on finish_execution_and_clear_pod_tracking_if_generation, lines 573-583). It has no reason = "..." and no adjacent comment. The file is touched by this PR (upgrade-record logic at lines 108-122, 338-371, 484-519). The function's existing arguments (activation_attempt_id, PodTrackingCleanupCause, PlacementClock) look older than this PR, but I could not confirm that. If the PR added this allow, it fails the check. The PR description does not mention it. - The #[allow] at pod_placement.rs:113 has a reason. The other too_many_arguments allows in the tree predate the check's scope or cannot be dated. - The CONTRIBUTING.md and AGENTS.md text about severity and approval exists on the current tree. I could not tell whether this PR changed it. The PR's own "Severity and risk" section sits in the description, not in a repository guidance file. - The description says stopped_execution_retirement_refused is loosened. That is a runtime business rule in pod_tracking.rs, not a repository guard, so I did not count it. Missing base means I cannot establish that nothing was removed or weakened. I cannot establish PASS.

Resolution

1. Fetch the PR base d99f199 (or the stack base PR #1708 head) and run git diff d99f199e2 HEAD so the full patch can be read. 2. Check every added line for #[allow(, #[expect(, #[ignore, should_panic, pytest.mark.skip/xfail, continue-on-error: true, and changes to .config/nextest.toml, ci/kind-pull-paths.txt, ci/kind-direct-known-failures.txt, deny.toml, clippy.toml, ARCHITECTURE.toml, architecture-baseline.toml, and [lints] tables. 3. Confirm whether crates/l40_api/api/src/controllers/substrate/kubernetes/store/pod_tracking.rs:573 (#[allow(clippy::too_many_arguments)]) is new in this PR. If it is new, either add reason = "..." with the cause, or remove the allow by reducing the argument count. Either fix clears the item. 4. Check removed or loosened assertions in any test file in the diff, and confirm each removal has a replacement in the same diff. 5. Check whether CONTRIBUTING.md, AGENTS.md, CLAUDE.md, .agents/, or docs/rust/ gained or changed text on reviews, merge-risk ratings, or approvals.