[the-gigi-pplx]

Summary

Adds the cluster controller's runtime_upgrade reconciler, which drives sandbox upgrade campaigns: it finds the sandboxes on a campaign's source environment that can move to its target, the cluster default, and hands them one at a time to an upgrade dispatcher under the campaign's per-node and per-cluster caps. It pauses a campaign that reaches its failure budget, re-drives items whose worker died, and completes the campaign once its sandboxes are drained. It drives sandboxes through a dispatcher trait and is tested against a fake. It ships disabled and isn't started yet: its production dispatcher, over the stay-stopped upgrade in #1322, and its spawn from run_reconcilers come once that PR has the node-side half it now needs (see Dependencies).

Terms

• Execution environment: a named runtime identity (runtime, version, kernel, snapshot state format) that a sandbox last ran under. The cluster policy (runtime_environment_policy, runtime_environment_states) names a default and gives each environment a state: allowed, deprecated (still executes, but should move) or blocked.
• Campaign: a row of runtime_upgrade_campaigns naming the environment to move off (from), the target (to), the kind (sandboxes or templates), its pacing (max_in_flight_per_node, max_in_flight_per_cluster), a failure_budget, include_running, and a phase: canary, rolling, paused, completed or aborted.
• Item: a row of runtime_upgrade_items, one sandbox (or template release) in a campaign, in phase pending, in_flight, done, failed or skipped.
• Lease: an in_flight item's lease_token and lease_expires_at. The schema requires both exactly while the item is in flight; an outcome is recorded only under the matching token.
• Stranded: a sandbox on the campaign's source with no certified upgrade edge to the target, as the start-time decision (decide in crates/l40_api/api/src/controllers/substrate/runtime_upgrade.rs) would find it.

The problem

PR #981 added the campaign and item tables, and PR #1015 decides an upgrade when a stopped sandbox starts or a suspended one resumes. Nothing moves a sandbox that isn't activated, so an environment an operator deprecates or blocks keeps its stopped sandboxes until each happens to start. Retiring an environment needs its population at zero, and nothing drives it there.

Nothing reads or writes the campaign tables yet, either. The schema leaves the store a set of obligations: write finished_at and lease_expires_at from the database clock, bound lease length (a live lease blocks the down migration), fence outcomes by lease token, keep subject_kind consistent with the campaign's kind, truncate last_error to 4096 characters, and lock campaigns before items, in the down migration's order.

How this fixes it

• A campaign store (crates/l30_storage/runtime/upgrade/campaign/store), the tables' first reader and writer. Executor-generic verbs with typed rows and closed vocabularies:
  • sandbox_candidates: one page, in id order, of the substrate's published sandboxes in the given lifecycles that the campaign has no item for, with the execution each last published. A record bound to another environment is left out in SQL; bound and unbound records that could be on the source are returned for attribution.
  • insert_pending_sandboxes: records items only into an unfinished sandbox campaign, which it locks for the insert. A skipped item found again is made pending while it has attempts left, keeping its count; items in other phases are left alone.
  • lease_sandbox_items: owns the pacing. Under the campaign row lock it leases pending items up to the cluster cap, counting items already in flight, and up to the per-node cap by the node that last published each sandbox. A canary leases one item at a time and nothing once one is done, so it waits for an operator's resume. Leases are clamped to an hour and expire on the database clock. It leases an item whose sandbox is published on a node only while that node's status is reachable and evidence of its environments; an item on any other node stays pending, which holds the campaign open. An item with no published node is leased, and the worker skips it, since it's on neither end of the campaign.
  • renew_lease and finish_item: act only under the item's current token, so a worker that lost its lease records nothing. A renewal also needs the lease unexpired on the database clock; an outcome still records under an expired lease nobody has re-driven, since nobody else holds the item.
  • redrive_expired: returns lapsed in-flight items to pending, or to failed once they've been leased the maximum number of times.
  • create_campaign: starts a campaign, refusing while an unfinished campaign of the same kind moves the same source, or a finished one (aborted, say) still has an upgrade in flight under a live lease, so two campaigns never act on one sandbox. Creators for one kind and source serialize on a transaction-scoped advisory lock and look for an unfinished campaign after taking it; no constraint or migration.
  • pause_if_over_budget and complete_if_drained: guarded phase changes that never overwrite a phase an operator wrote first. Completion locks the campaign before it reads the items, so it can't complete a campaign under items inserted meanwhile. It refuses while any item is pending or in flight, while an item with attempts left is skipped for a sandbox that's still a candidate (the substrate's, in a lifecycle the passes scan, published, and not bound elsewhere, as sandbox_candidates reads it), and while the campaign is over its failure budget, which leaves it to pause. The counts come from one snapshot after the lock, so an outcome committed late is either visible and judged by its sandbox's current state, or still in flight. A skipped sandbox that was deleted, started under a stopped-only campaign, or moved elsewhere doesn't hold the campaign open.
  • retire_skipped_sandboxes: uses up the attempts of skipped items whose unbound execution the catalog now attributes elsewhere, so they neither come back as candidates nor hold completion.
  • Leasing re-checks the failure budget under the campaign lock, so a failure committed after the tick's own check stops the next batch.
  • Every verb that opens a transaction commits or rolls it back before it returns, on errors too, so a pooled connection never keeps the campaign row locked until its next use.
• The reconciler (crates/l40_api/cluster/controller/server/src/runtime_upgrade/). Each tick, for each unfinished sandbox campaign, it re-drives lapsed leases and pauses a campaign at its failure budget. Then it examines one page of candidates, completes a drained campaign, and leases and dispatches items.
  • A campaign is decided as a start would be. PR [api] decide the runtime upgrade at resume and start, and stage pod-less sandboxes on the node #1015's decide, for a sandbox on the campaign's source, says what the campaign does. A campaign is held while its target isn't the cluster default (the node-side upgrade only boots under the default) or its source is allowed again. Its sandboxes are stranded when no certified edge reaches the target: they're counted, not recorded, nothing is leased, and the campaign stays open until an edge is certified or an operator aborts it. Each candidate's published execution is attributed with PR [api] decide the runtime upgrade at resume and start, and stage pod-less sandboxes on the node #1015's attribute, so a campaign moves exactly what a start would move, and, as at start, a publish that carried no execution is attributed nowhere.
  • Stopped by default. Running and paused sandboxes are candidates only when the campaign sets include_running and the move's authorization accepts memory loss, which only a block of the source gives. A deprecated environment still executes, so its residents are left alone.
  • A dispatcher trait. SandboxUpgradeDispatcher has two calls: upgrade a stopped sandbox to the campaign's target, leaving it stopped, and stop a started or paused one so it can take that path. The stop names the campaign's source, and its contract is a guarded transition: it declines unless the sandbox is still started or paused on that source, so a sandbox restarted elsewhere after the reconciler's read keeps running, and its item is skipped. The upgrade decides afresh where it acts, as a start would, and a move to anything but the campaign's target (the default changed since the lease) isn't recorded as the campaign's. Just before the upgrade, the worker re-checks where the sandbox is: one started and stopped again under another environment since it was found is skipped, while one on the target still goes to the upgrade, which finishes an earlier lease's upgrade that published the target but didn't finish its backup or retire. The reconciler re-reads each leased sandbox first: a deleted one is skipped, one in a transitional lifecycle is retried, and one that started since it was found is skipped unless the campaign sets include_running.
  • A stop is decided on what holds when it's made. Stopping a running sandbox loses its processes and memory, and nothing later gives them back. So just before the stop, the worker re-reads the policy, the catalog and the sandbox's published execution. It stops the sandbox only if it's still on the source and the policy still blocks the source with an edge to the target. Otherwise the item is skipped: a sandbox a start has since cold-booted onto the target is left running. A sandbox mid-transition is released back to pending without spending an attempt. Retryable dispatch errors go back to pending and fail the item after five leases; other errors fail it.
  • Each dispatch on its own task. Leased items are dispatched on tasks that outlive the tick, so a slow upgrade holds only its own slot; each tick leases against what's free, since the store counts items still in flight. Stopping the loop aborts them, and their leases lapse to be re-driven.
  • A deadline per dispatch. A dispatch that hasn't finished SPACE_RUNTIME_UPGRADE_ITEM_DEADLINE_SECS (30 minutes by default, from 15 minutes to a day) after its lease was requested is dropped, however many renewals it spans. The item fails, counting against the budget, and its lease is released, so a dispatch that never returns can't hold its slot. The deadline is per lease: a retried item gets a new one on its next lease, and its five attempts bound the total.
  • Lease renewal. While the dispatcher works, the task renews the lease every third of its five-minute length (the shared claim_renewal_interval). When the lease is lost, or no renewal succeeds by its end (a hung renewal included), the work is dropped and nothing is recorded; nothing it waits on, sleep or renewal, runs past the lease's end.
  • Completion. A rolling campaign that moves sandboxes completes on the tick a whole pass of candidates ends, with every page read under one plan and set of attribution inputs, once the store finds it drained: nothing pending or in flight, and no skipped item that's still a candidate with attempts left. It never completes partway through a pass, or on a tick whose read or write failed. A pass whose plan or attribution inputs change partway (an edge certified, the source newly blocked, a catalog change, the legacy unbound setting) starts over. A sandbox that stops after its part of the pass was scanned is left to the start-time upgrade.
  • Skipped sandboxes are found again, a bounded number of times. A sandbox skipped because it had started by the time it was leased stays a candidate. When a later pass finds it back on the source before the campaign completes, its item is pending again, keeping its attempt count, so a sandbox the node keeps declining is dispatched at most five times.
• A node status store (crates/l30_storage/node/status/store), pplx-space-node-status-store, which declares node_status_snapshots as its state. Its writers, the internal API's node status handler and the Direct substrate, stay where they are for now. It holds environment_evidence(alias), the one rule for when a node's published environments count as evidence, as a whole SQL predicate a caller builds into its statement once, as pplx-space-sandbox-portability does: the latest execution inventory is complete, the snapshot is fresh by stale_after, and the environment arrays were received within the node's freshness window (stale_after - observed_at). It's never NULL, and it leaves reachability to the caller. The campaign lease is its first caller; placement, which today checks its own freshness columns inline, can move onto it.
• The store skill (space-data-store-development) lists environment_evidence beside the portability and residency predicates as a fragment interpolated whole rather than re-spelled.
• Default-disabled. SPACE_RUNTIME_UPGRADE_RECONCILER_ENABLED (the shared flag vocabulary; unset or anything unrecognized is off) gates the loop. Off, it waits for shutdown and never touches the campaign tables.
• Telemetry (crates/l40_api/cluster/controller/server/src/metrics/runtime_upgrade.rs, beside the controller's other families): gauges for unfinished campaigns by kind and phase, each campaign's items by phase, and the stranded count of its last full pass (sandboxes on the source it found unrecorded, plus pending items), every combination emitted including zeros; counters for item outcomes (including lost leases), re-drives, policy holds, and auto-pauses. Their tag values are metric_vocabulary! enums.
• Shared reads instead of copies. The catalog read with edges moves into the catalog store as list_unretired_entries, and the parsing of a recorded execution into RecordedExecution::from_record. The start-time decision's store and the reconciler both use them.

Template campaigns are listed and reported but not driven: moving a template release needs the node-side template caches keyed by environment first. The start-time and resume-time upgrade is unchanged.

Validation

On a Linux x86_64 build host, at this head on 85b68f39d:

• run-rust-checks.sh clippy and isolated-crates passed, and so did cargo arch against main's baseline and architecture (no required findings; the advisory layers.policy_changed ones are the store grants under Docs).
• 113 of 113 targeted tests passed (both stores, the catalog store, and every runtime_upgrade and env_config test in the API and the cluster controller), and check-test-durations.py passed: no test of this change takes 2 s.
• run-rust-checks.sh tests-unprivileged: 12,538 of 12,538 nextest tests passed, plus doctests and feature isolation.
• run-rust-checks.sh tests-privileged, as root inside with-nocow-test-filesystem.sh: 70 of 76 passed. The six failures come from the build host: three node-runtime-core fs-verity tests (no fs-verity in the host kernel), two node-runtime-supervisor contained_launcher::linux tests (the checkout's ancestors aren't root-owned), and spaced-guest-worker embedding (it needs a private cgroup namespace). CI's privileged lane runs them.

New coverage, all against Postgres:

• Node status store (node/status/store/src/tests.rs): a_snapshot_is_evidence_only_while_every_condition_holds, across a complete, fresh snapshot (reachable or not), no inventory report, an incomplete one, a stale snapshot, one whose arrays came after its observation, no arrays, and arrays inside and past a window measured from observed_at; every case decodes into a plain bool, so the predicate is never NULL.
• Store (runtime/upgrade/campaign/store/src/tests.rs): only_items_on_reachable_nodes_with_environment_evidence_are_leased (an unreachable node, a stale one and one with no status get nothing, an unpublished sandbox is leased, and the scan limit counts only what may be leased); a_gated_node_is_leased_from_once_it_qualifies_again; a verb that fails after locking the campaign releases the lock on the connection it held; a skipped item with attempts left is a candidate again and is made pending with its count kept, while one out of attempts, and done, failed and pending items, are left alone; a release gives its lease's attempt back and a retry keeps it; an expired lease isn't renewed but still records its outcome; a_skipped_sandbox_that_is_still_a_candidate_holds_completion: a skipped candidate holds it (a non-string environment included), and one out of attempts, retired, bound elsewhere, deleted, started under a stopped-only scan, of another substrate, or unpublished doesn't, while a started one does under a scan that takes running sandboxes; retirement leaves other sandboxes, phases and campaigns alone; a campaign over its budget neither leases nor completes; one_unfinished_campaign_per_kind_and_source, a_finished_campaign_holds_its_source_while_an_upgrade_is_in_flight, and of_two_concurrent_creates_for_one_source_exactly_one_succeeds, which holds the first create's transaction open across the second; campaign validation, including a per-node cap equal to the cluster cap; candidates bound to another environment left out, and a non-string environment treated as unbound; the cluster cap counting items already in flight; the per-node cap, and an unpublished sandbox counting toward the cluster cap only; subject_sandbox reading a published, an unpublished and a missing sandbox; the canary; nothing leased from a paused or template campaign; leases on the database clock and clamped; renewals and outcomes only under the current token; lapsed leases re-driven until their attempts run out; the budget pause and that it leaves an operator's phase alone; completion only when rolling and drained, including with a connection holding the campaign across refused calls; candidate scoping, paging and exclusion; error truncation.
• Reconciler (runtime_upgrade/tests.rs, tests/leases.rs, tests/disabled.rs), against a fake dispatcher that records concurrency:
  • stopped sandboxes on the source upgraded with at most two in flight and one per node, while sandboxes on another environment, a running one, and another substrate's are left alone; the campaign completes;
  • running and paused sandboxes stopped first and then upgraded only under include_running on a blocked source;
  • the budget pause, with no further dispatch;
  • retryable errors retried until the attempts run out;
  • holds for a target that isn't the default and a source that's allowed again;
  • stranded sandboxes counted, not recorded, with the campaign left open, and an item recorded before the edge went away left pending, not leased, and counted stranded too;
  • a_plan_that_changes_mid_pass_restarts_the_pass: an edge certified mid-pass doesn't let the pass complete the campaign without the sandboxes it counted as stranded earlier;
  • a_sandbox_the_node_keeps_declining_is_dispatched_a_bounded_number_of_times;
  • a_sandbox_declined_once_on_a_multi_page_pass_is_moved_before_completion;
  • a_catalog_change_mid_pass_restarts_the_pass;
  • an unbound execution attributed through the catalog;
  • the canary waiting for resume;
  • deleted, restarted, settling and not-moved sandboxes;
  • template campaigns reported but not driven, with every gauge series emitted;
  • completion only after a whole multi-page pass, and a_campaign_completes_only_when_a_pass_ends_not_partway_through_the_next: a sandbox that stops after a pass ended, behind the next pass's first page, is still upgraded before the campaign completes;
  • a_sandbox_skipped_because_it_had_started_is_upgraded_once_it_stops;
  • a_failed_scan_after_a_pass_ended_doesnt_complete_the_campaign: an insert that fails on the tick after a pass ended leaves the campaign rolling;
  • finished dispatch tasks collected on the next tick;
  • a_declined_sandbox_is_found_again_and_a_deleted_one_doesnt_hold_completion: a declined sandbox seeded beside a deleted one is found again and moved, and the campaign then completes with the deleted one's item still skipped;
  • a_skipped_sandbox_the_catalog_attributes_elsewhere_is_retired_not_waited_on;
  • a_running_sandbox_is_stopped_only_while_it_and_the_policy_still_call_for_it: an include_running item whose sandbox a start has moved onto the target, and one whose source is only deprecated now, are skipped without a stop;
  • a_stop_the_substrate_declines_skips_the_item_without_an_upgrade;
  • every upgrade and every stop guarded on the campaign's source, and every upgrade asked for its target;
  • a_dispatch_that_never_returns_fails_at_the_deadline: the item fails with its lease released, and the failure pauses a campaign with a budget of one;
  • a dead worker's item re-driven;
  • a long upgrade renewing its lease once a renewal interval, not continuously;
  • a lost lease dropping the work;
  • failing renewals dropping it at the lease's end, not before;
  • a re-drive that uses up the budget pausing before anything more is leased;
  • a_publish_without_an_execution_is_attributed_nowhere: with the legacy setting off and a source on the initial Firecracker versions, an empty record is neither found nor upgraded;
  • an_unreachable_node_with_fresh_evidence_gets_no_dispatch: the item stays pending across ticks and the campaign stays rolling;
  • a_stopped_sandbox_goes_to_the_upgrade_only_on_the_campaigns_source_or_target: one on another environment is skipped, and one on the target is handed to the upgrade;
  • and a_disabled_reconciler_never_touches_the_campaign_tables: with both tables held ACCESS EXCLUSIVE by another transaction, the disabled loop leaves no lock waiter, while the enabled loop's first read waits on the lock.

Mutation check, on the store and reconciler code of this head (the store's and most of the reconciler's before the move to the layer directories, which didn't change their code; the worker's lease and dispatch code again after the lease tests' rescale): cargo-mutants over the campaign store (82 caught, 13 unviable) and over the reconciler module (93 caught, 32 unviable, counting the held-back dispatcher's mutants), over the worker after the rescale (20 caught, 8 unviable), the node evidence predicate's three conditions and its never-NULL default, the lease gate's reachable, evidence and unpublished-node clauses, the stranded count's pending items, the no-evidence attribution guard, and the draining refusal, each mutated by hand and each caught, and each of 51 SQL predicate guards mutated by hand. The survivors that later test additions cover (the subject read, the stranded campaign's lease guard, completion's string-environment check, retirement's sandbox list, the worker's source-or-target check, and a boot-timeout constant since held back) were rerun by hand and are caught. Every remaining survivor is equivalent:

• two conditions in the reconciler that only decide whether a log line is written, and the pass-state read that only feeds one;
• the phase predicates in renew_lease, finish_item and redrive_expired, which the schema's runtime_upgrade_items_lease_phase_check already implies through the token or expiry they also test;
• the lease UPDATE's phase = pending guard, since only the lease verb moves an item out of pending, under the campaign lock;
• completion's subject_kind = sandbox, since no verb writes a template item into a sandbox campaign.

Not validated: any run on a Kind or deployed cluster. The reconciler isn't started anywhere yet, and a campaign also needs certified edges in runtime_environments and an admin route to create it.

Dependencies

• [api] trigger a stopped sandbox's runtime upgrade from a test hook, and smoke it on Kind #1322, the stay-stopped upgrade operation the production dispatcher will call, and [db] record a stay-stopped runtime upgrade decision on its Kubernetes execution #1304, the migration it's stacked on. That operation's boot needs a node-side change first. Once it's rebased onto that change, this branch stacks on it, adds SandboxUpgradeDispatcher over it (an upgrade to the campaign's target is done, the operation's final refusals are skips, every other error is retried, every stop is declined) and spawns the loop from run_reconcilers for both substrates.
• PR [db] add runtime upgrade campaign and item tables #981 (campaign tables) and PR [api] decide the runtime upgrade at resume and start, and stage pod-less sandboxes on the node #1015 (the start-time decision) have merged.

Not covered

• Template campaigns, which need environment-keyed node template caches (template: partition template caches by execution environment #1075, a draft).
• Suspended sandboxes on a blocked environment that never resume.
• Creating, resuming, pausing and aborting campaigns: no admin route exists yet, so what :resume does with failed items already counted against the budget is that route's decision.
• Running and paused sandboxes under include_running. The production dispatcher declines every stop, so they're left running and their items skipped. A guarded stop, which stops a sandbox only while it's still on the campaign's source, is a follow-up change; once it exists they'll be stopped, upgraded and left stopped, not restarted.
• A sandbox stuck mid-transition keeps its campaign open: each lease releases it without spending an attempt, until the transient-lifecycle sweeper settles it. In canary, where one item is leased at a time in subject order, such a sandbox holds the canary on it too.
• A skipped item whose unbound sandbox the catalog attributes nowhere for one pass is retired for good, so a later catalog fix doesn't bring it back to the campaign; where the activation-time upgrade is on, a start still moves it.
• The per-campaign gauges keep their last value once a campaign finishes.
• Skips and retryable errors share an item's attempts: a sandbox skipped on earlier leases (it was running, say) has fewer retries left for a transient error, and on its last attempt one fails it against the budget. Separate counts need a column on items.
• Sandboxes whose node no longer publishes them (their sandbox_observed_states row is gone) aren't candidates; they're left to the start-time upgrade.
• The candidate query reads sandboxes and sandbox_observed_states from the campaign store rather than through their own stores, since it needs them joined with the campaign's items.
• Backoff between retries of a retryable dispatch error, such as an unavailable node. At the 30-second tick, five come within about two minutes, and the item then fails against the budget. A backoff needs a not-before column on items, which is a migration of its own.

Severity and risk

Severity: does not apply. This adds a capability that ships disabled.

Risk: low. Nothing starts the reconciler, and enabled it does nothing without a campaign, which nothing creates yet. The start-time decision's evidence reads now go through the shared catalog verb and parser, with its existing tests unchanged and passing. Roll back by reverting this commit.

Docs

docs/configuration.md gains SPACE_RUNTIME_UPGRADE_RECONCILER_ENABLED and SPACE_RUNTIME_UPGRADE_ITEM_DEADLINE_SECS. The first says that stopped sandboxes are upgraded and stay stopped, that running and paused ones are left running and skipped for now, and that the controller doesn't start the loop yet. The new store lives in crates/l30_storage/ and declares the campaign tables its state. architecture-baseline.toml drops them from its unowned tables, and node_status_snapshots too, which the new node status store now declares. ARCHITECTURE.toml adds the campaign store as the node status store's user, and the cluster controller as a user of the catalog, policy and campaign stores and the API as a user of the catalog store, which layers.policy_changed reports: both read the catalog and policy to decide upgrades, and the controller owns the campaign tables. The schema's comment on runtime_upgrade_campaigns.include_running still says the upgrade "restarts" them; changing a column comment is a migration, so it's corrected with the next migration that touches the table.

AI assistance: Claude Code drafted the code, the tests and this description, under the author's direction and review.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true