Repository navigation
[api] boot a start's or resume's runtime upgrade in its pod and read its outcome from the node - #1708
Conversation
|
Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
|
6c0968a to
762ec97
Compare
a1819ed to
8556b10
Compare
83f1ca8 to
8781a20
Compare
8556b10 to
77e1fcc
Compare
7d37cef to
8451efa
Compare
1c43156 to
5ad7ee2
Compare
8451efa to
6d843e7
Compare
f00949e to
825bb34
Compare
6d843e7 to
5e9cc27
Compare
825bb34 to
1410439
Compare
5e9cc27 to
e49aafc
Compare
1410439 to
de4fb94
Compare
e49aafc to
913b4c7
Compare
de4fb94 to
6697735
Compare
913b4c7 to
ccbd14c
Compare
6697735 to
2660b4f
Compare
ccbd14c to
1a3116a
Compare
2660b4f to
87822d1
Compare
1a3116a to
1fc4517
Compare
87822d1 to
7c91758
Compare
1fc4517 to
4a2b483
Compare
3bbfe7e to
66f2fe8
Compare
4a2b483 to
4fe67bc
Compare
6e11a12 to
9248cdf
Compare
4fe67bc to
0371077
Compare
9248cdf to
aef571a
Compare
5c6f945 to
7d398d3
Compare
aef571a to
9eedd3e
Compare
7d398d3 to
3c3b124
Compare
9eedd3e to
9cedcd5
Compare
3c3b124 to
2366fc0
Compare
9cedcd5 to
7a564dc
Compare
2366fc0 to
ddc8ba1
Compare
…outcome from the node
7a564dc to
d99f199
Compare
Summary
With
SPACE_RUNTIME_UPGRADE_AT_ACTIVATION_ENABLEDon, a Kubernetes start or resume that decides a runtime upgrade boots that upgrade in the sandbox's pod and reads its outcome from the node before the Wake, instead of failing. With the flag off, which is the default, start and resume behave as before.Api paths below are under
crates/l40_api/api/src/controllers/substrate/. "Booting an upgrade in its pod" and "Upgrading at resume and start" indocs/execution-environments.mdare the long form.Terms
activation_runtime_upgrade(kubernetes/runtime_upgrade.rs) and carried out by sandbox-manager'sUpgradeRuntime.runtime-upgradepod mode: a restore pod whosespace.pplx.net/checkpointisruntime-upgradeand whosespace.pplx.net/runtime-upgrade-*annotations carry the operation (PR [sbx-mgr] boot a Kubernetes runtime upgrade in its pod's shim through a runtime-upgrade checkpoint #1380). The pod's shim callsUpgradeRuntimefrom its container create.sandbox_k8s_executionsfor a sandbox. It records the pod name (which fixes the generation), the pod UID once a materialization persists it, and the activation attempt holding its claim.restoringsandbox keeps whatever a later resume decides, since its upgrade may have retired the checkpoint the resume read.FailedPrecondition(412).The problem
PR #1015 decides a runtime upgrade at start and resume and sends it on the activation's Wake as
WakeRequest.runtime_upgrade. On Kubernetes a sandbox boots in its pod's containerd shim at container create, never from the Wake, so an upgrade sent there can't take effect. PR #1380 made sandbox-manager refuse that field for a Kubernetes route identity, and added theruntime-upgradepod mode for the shim to boot it instead. Nothing renders that mode yet, so with the flag on, every Kubernetes start or resume that decides an upgrade fails.Moving the upgrade onto the pod raises questions the Wake path never had to answer:
suspendedcould interrupt a boot or lose a checkpoint the upgrade retired.How this fixes it
Render the mode for start and resume.
PodCreateContext::upgrading_runtime(kubernetes/pod_ops.rs,kubernetes/pod_ops/create_request.rs) attaches the decided upgrade, with target statestarted, to the user-restore context a start or resume creates. A cross-node re-drive of an unschedulable source-pinned pod carries the same upgrade (kubernetes/pod_ops/redrive.rs). Every other pod kind is built without one.EnvironmentMove::wake_upgradeis gone (runtime_upgrade.rs), so no Kubernetes Wake carriesruntime_upgrade.Read the outcome before the Wake.
await_pod_runtime_upgrade(kubernetes/pod_runtime_upgrade.rs) pollsGetRuntimeUpgradeon the pod's node by node name, throughnode_worker_client(kubernetes/pod_ops/worker_clients.rs, which teardown's retire-by-node route now shares), since a pod whose create failed has no address. Each read names the pod's attempt (GetRuntimeUpgradeRequest.attempt_id, which on Kubernetes is the pod UID the shim names its attempt by: the pod at the name or, once it's gone, the one the wait knew), so a node from this release answers that attempt's own outcome, and absent in place of another attempt's; an older node answers the operation's latest, and the table below still settles only on a report naming the pod. In the table, "names this pod" means the report'sattempt_idorrefused_attempt_idis the pod's UID. Each report is read with the pod at the execution's name:startedtargetrefused_attempt_idnames this pod: refused (retryable 503), and the pod is deleted. Otherwise it's reached once the pod's container runs, refused (503) once the container has ended or the pod is gone, and it waits until thenattempt_idnames this pod: settled whatever its containers show (a rollback 503, a refusal 412), and the pod is deleted under its UID. Pod gone: settled only whenattempt_idnames the UID the wait last saw, or the one the execution recorded. A report naming another pod or none waits, unless the node the pod is bound to (or, with the pod gone, the node a look saw it bound to) reports the operation abandoned since, which settles as not startedrefused_attempt_id): settled as that refusal (retryable 503), and the pod is deleted. Otherwise not started (503) once the pod's container has terminated, since the shim's call has returned by then. A failed pod phase, or a pod that's gone, proves nothing and waits. Absent and abandoned settles as not startedThe Wake then carries no upgrade and only acknowledges the route the pod's execution runs under.
Settle only on this pod's own outcome, and fail closed. On every look the wait reads the execution that owns the pod's name. A pod being deleted, or one that isn't the pod the execution recorded, settles nothing, and a settled rollback or refusal deletes its pod under its UID, settling only once that delete lands or fails. A joiner reads the pod from the API on every look and doesn't settle on a missing pod while the execution records none (
MissingPod::NotCreatedYet). A report naming no pod, from a sandbox-manager that predates the field, runs to the deadline and keeps the pod and its claim. Each observation runs under max(wait deadline, now + 1 s) (NODE_READ_FLOOR).Answer a settled outcome before any Wake.
settle_pod_runtime_upgrade_for_attemptgives up the activation's own attempt, so the retry gets a new pod. A rollback, or a pod the node refused after the target was reached, answers a retryable 503. A refusal the node recorded answers 412, as a stranded sandbox does: the node records only a refusal no retry changes, and a retry decides the same operation from the same inputs.Keep the restore when the upgrade may have retired its checkpoint. A resume whose upgrade reached the target before its pod failed (
PodUpgradeFailure::LeftOnTarget) staysrestoringon its node rather than reverting tosuspended. The node's runtime-exit cleanup stops the sandbox and keeps its disk, and its publication converges the row tostopped. A resume the node rolled back or refused reverts tosuspendedfirst and gives its attempt up only afterwards, since the revert retires the restore's execution only while it still carries the attempt (revert_restore_claim_to_suspended).Treat a pod that may run as node work (
kubernetes/pod_ops.rs,kubernetes/execution.rs,kubernetes/pod_ops/ensure_pod.rs). A materialization that fails after it created or kept a pod carrying its upgrade isn't a clean failure: a restore staysrestoringand keeps its pod, and a start keeps its attempt's claim until the lease ends. The same holds for a create whose answer was lost and whose pod a confirming read doesn't find yet. The attempt holding acreatingexecution's claim creates a missing upgrade pod at the claim's name (create_missing_upgrade_pod). A pod already there carrying the operation is hydrated from an uncached read, since an interrupted re-drive leaves a successor at the same name and a reflector can still hold the deleted predecessor.One stale-pod rule for upgrade pods (
runtime_upgrade_stale_reason,kubernetes/pod_ops/ensure_pod.rs). On every path that could adopt an existing pod:stopped-target placeholder is stale to anything that doesn't render its operation;stopped. Arestoringsandbox keeps it, and a resume that takes over such a restore settles on the upgrade the pod carries (carried_runtime_upgrade,kept_runtime_upgrade), even when its own decision now strands the sandbox. A kept pod the reflector doesn't hold as the execution records it is confirmed from the API, both when the takeover looks for it and before the materialization refuses to replace it, so a replica that hasn't seen the pod still settles it. That covers a pod whose activation created it and failed to persist it: the execution is stillcreatingwith no UID, and the pod at its name counts while its annotations name the execution's generation and it isn't being deleted. The node doesn't read the control plane's policy, so kubelet's retries of that pod can boot the upgrade whatever a later decision says. The takeover reads the kept pod's upgrade before it materializes anything, and settles it first unless its own decision renders that same operation, whether the decision refused, renders no upgrade or renders another. The kept upgrade only settles its pod (PodMaterializationOwnership::runtime_upgrade_kept): the pod is kept whatever its phase, failed or being deleted included, it isn't re-driven, and no new pod is created for it, since the node takes a new pod as a new attempt and a plain replacement could restore from a checkpoint the upgrade retired. A resume that finds the kept pod gone answers retryably and keeps the restorerestoring.Stable operation ids and runtime names (
runtime_upgrade.rs,kubernetes/runtime_upgrade.rs,kubernetes/store/runtime_upgrade.rs). Astartedtarget derives its operation id from its inputs, and every runtime name read into the decision's evidence passes throughcanonical_runtime, so the node's legacyfccolumns match the catalog'sfirecracker.A start still falls back to a plain start only when its decision changes, and when the upgrade didn't take effect the published environment stays the old one. The flag applies only to Kubernetes, so Direct activations are unchanged. The wait already reads a
stoppedtarget, which nothing here renders yet; that's the next pull request's.Validation
At this head, on a Linux build box:
cargo clippy --workspace --locked --all-targets --all-features -- -D warningsandcargo fmt --all --check: clean.ci/scripts/run-rust-checks.sh tests-unprivilegedand, as root underci/scripts/with-nocow-test-filesystem.sh,tests-privileged: 13027 unprivileged tests passed, plus doctests and the feature checks. The privileged lane had 6 failures, all from the build box rather than this change: three fs-verity tests (node-runtime-coreverityandartifact_store::execution_cache, whose kernel has no fs-verity), twonode-runtime-supervisorcontained_launcher::linuxtests (unsafe ancestors of the mirrored path), andspaced-guest-worker::embedding(needs a private cgroup namespace). CI's privileged lane covers them.kubernetes::pod_runtime_upgrade::tests: the outcome table above, row by row.kubernetes::tests::runtime_upgrade_dispatch: start and resume against a scripted node and Kubernetes API. The pod carries the decided upgrade and the Wake carries none; a rollback and a refused create answer a retryable 503, and a recorded refusal a 412, each deleting the pod; a retry gets a new pod; a joined claim doesn't settle on a missing pod, and a start's joiner settles once the owner's pod it saw is gone; a resume left on its target staysrestoring; a rolled-back resume gives its attempt up after the revert.kubernetes::tests::runtime_upgrade_dispatch::api_faults: a settled pod is never a later attempt's owner pod; a settled pod on a legacy route-less execution keeps it claimable; a settled pod settles only on its own delete; a pod read that never answers holds the wait at mostNODE_READ_FLOORpast its deadline; a lost upgrade create keeps its claim; a stale upgrade pod that can't be deleted keeps the claim; a kept restore pod settles on the upgrade it carries, even once its own decision strands the sandbox, including one whose execution records no UID, cached or not; a stranded resume never settles on a pod at the name its execution doesn't own (another generation, one being deleted, or arecoveringexecution's); a resume hydrates a re-drive's successor, not a cached predecessor carrying the same operation; a takeover never places a new pod for its kept upgrade, whether its decision strands the sandbox or, with the flag off, renders no upgrade, and whether the kept pod failed, is being deleted, is unschedulable, is gone from the API, or is missing from this replica's reflector, with a rollback or the upgrade still in progress; a joiner never takes a cached earlier pod for its owner's; the wait asks the node for the pod it reads; a refused create the node kept after the sandbox's row went settles a start or a resume as it would with the report standing. Each guard oncarried_runtime_upgrade's path for an execution with no recorded UID (thecreatingstate, the generation match, the deleting check, the read on a cache miss), the uncached hydrate read, the kept upgrade's keep-the-pod and no-create guards, settling the kept upgrade on a decision that succeeds, the API reads that confirm a kept pod the reflector misses, and the kept refusal an absent report carries, was mutated away in turn, and a test fails for each. The kept upgrade's no-re-drive guard isn't reached by the scripted fixture, so no test fails without it.kubernetes::store::k8s_store_postgres_tests::runtime_upgrade_evidence_tests: the legacyfcruntime spelling matches the catalog.Not validated: a Kind run. The end-to-end Kind scenario for the start arm (
space_smoke) is in a later pull request in the stack, with the trigger it needs.Not covered
None of these loses a sandbox's disk or checkpoint, and every path that reverts a restore to
suspendedstill needs the node's recorded outcome for that exact pod.restoring. A kept pod that's unschedulable, or being deleted with no outcome the node names for it, keeps every stranded takeover waiting out its readiness budget until the pod goes or an operator allows the environment.note_upgrade_podmarks a pod only when it carries the rendered operation. A start that finds an earlier start's pod carrying another operation, and can't delete it, gives up its attempt instead of keeping it until the lease ends.restoringuntil its lease expires.restoringrow settles on a kept pod's upgrade. A fresh resume, or a joiner, that finds one answers retryably, and the next takeover settles it. That includes a pod a settled rollback couldn't delete, which a revert tosuspendedleaves in place with abandoned-restore retirement off: a fresh resume that then renders no upgrade waits out its readiness budget and leases on it.settle_pod_runtime_upgrade_for_attempt, the restore path inexecution.rs), alongside the restore leasescontrollers/substrate/AGENTS.mdalready lists as adapter debt.Dependencies
This is stacked on PR #1707, its base branch, and merges after it. PR #1707 is the top of an eight-PR chain that merges in order: PR #1750 → PR #1751 → PR #1752 → PR #1753 → PR #1754 → PR #1755 → PR #1756 → PR #1707. Together they make sandbox-manager's report name the pod each outcome is for, make a rollback or refusal final for its pod, and add
AbandonRuntimeUpgrade, all of which the wait here relies on.Deploy that chain first. This pull request adds no migrations or stored state, so it rolls out with any api release after it. Turn the flag on only once every worker's shim and sandbox-manager know the
runtime-upgradecheckpoint (PR #1380) and run the chain above. An older shim fails the pod without booting anything. Against an older sandbox-manager, whose report names no pod, the wait never settles a rollback: it runs to the activation's deadline and keeps the pod and its claim. Set the flag the same on every api and cluster-controller replica, and give both node-api's policy-mutation trust, whichGetRuntimeUpgradeneeds (docs/configuration.md).PR #1714 is stacked on this one, and the rest of the stay-stopped upgrade builds on it.
Severity and risk
Severity: medium. With
SPACE_RUNTIME_UPGRADE_AT_ACTIVATION_ENABLEDon, every Kubernetes start or resume that decides an upgrade fails today.Risk: low with the flag off, which is the default.
activation_runtime_upgradethen decides nothing, no pod carries an upgrade, and start and resume run as before. Two changes apply regardless of the flag:node_worker_client, under the same connect budget;runtime-upgradeannotations, and no pod carries them unless the flag was on when it was created.Roll back by reverting this pull request: it adds no migrations or stored state.
Docs
docs/execution-environments.md: "Booting an upgrade in its pod" and "Upgrading at resume and start" now describe the pod dispatch for start and resume, the outcome read and how it settles, the kept-pod and stale-pod rules, and the restore that staysrestoringwhen its upgrade left it on the target.docs/configuration.md: the flag's entry now says a decided upgrade boots in the activation's pod, that the flag waits on every worker's shim and sandbox-manager knowing theruntime-upgradecheckpoint, that node-api's policy-mutation trust is forGetRuntimeUpgrade, and that with the flag off a resume taking over an unsettled restore still settles its kept pod's upgrade first.AI assistance: Claude Code split this pull request out of a larger change, wrote the code and tests with review, and drafted this description.