Skip to content

api: carry restore decisions in one RestoreIntent value - #1660

Merged
ac-pplx merged 1 commit into
mainfrom
ac/restore-intent
Nov 6, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/restore-intent

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

User restores and the node drain's evacuation restores share the Kubernetes restore executor, which told them apart by testing evacuation_placement.is_some(), or passing evacuation: bool, at each decision: which claim to take, whether to join an in-flight or a completed restore, the charge, the wake coalescer, the snapshot and the pod placement, the runtime upgrade, the started-recovery redrive, and who commits restoring -> started. One Option<Option<EvacuationPlacement>> meant both "this is the drain" and "which cohort", and another kind of restore would have had to add a condition to every one of those branches.

Merging this builds a RestoreIntent once: RestoreIntent::user on the user restore and wake paths, and RestoreIntent::drain inside restore_evacuated_execution, whose signature is unchanged. The executor, ensure-pod, the suspend restore plan and the restore-source selector read its named fields (target, claim, snapshot, pod_placement, allows_runtime_upgrade, charge, coalesces, owner_commits_started, redrives_started_recovery) instead. The charge moves from RestoreClaimHandling into the intent, and the wake drops it for a preclaimed caller, which paid for its claim itself, as preclaimed_with_cleanup did.

This changes no behaviour. Each constructor sets every field to what the old test gave its caller. Branches keyed on the target state or on the claim variant keep that key, and branches that never fire for the drain keep the drain's old value.

Validation

At this head:

  • Flipping each field in its constructor fails an existing executor test for most fields. the_drains_re_placement_of_a_backed_off_stopped_evacuation_still_places_a_pod now passes a rollout cohort and checks the restore pod's cohort toleration and required node affinity, which no test covered before; routing the cohort to the no-cohort arms in ensure_pod_exists fails it. The drain's coalesces, owner_commits_started and redrives_started_recovery can't fire today: the drain is always preclaimed, its claim has no lease token, and its row is never started.
  • Future sizes at opt-level 0, base to head: no future on the path is larger than before, except the detached restore drive, 16 bytes larger, which is now boxed.
  • Pull CI: Rust format, workspace clippy with the repository structure check, and the full Rust test inventory (both groups) pass.
  • Locally, pplx-space-api and pplx-space-cluster-controller-server tests at CI's test profile (opt-level 0), with Postgres and Redis running: 3802 of 3803 pass. routes::e2b::tests::detail_reports_authoritative_federated_ownership_not_metadata fails on a JSON snapshot's key order when only these packages are built; it passes in Pull CI's full run.
  • Kind, at this head: local.suspend_restore_cross_node_from_object_storage, local.concurrent_restore_converges_to_one_execution and native_contract.command_wakes_idle_suspended_sandbox (the user constructor: lease claim, join and coalescing, remote snapshot, the owner committing started), and local.forced_drain_evacuates_sandbox and native.forced_drain_re_places_stopped_sandbox (the drain constructor: the marker claim, the newest remote snapshot, no source pin, the stopped materialization). All pass.

Severity and risk

Severity: does not apply; a refactor with no behaviour change.

Risk: a field set to the wrong value in a constructor would change that decision for every restore of that kind. The existing executor suites catch a flip of most fields, and the drain test above checks that the cohort reaches the restore pod; no test catches a user restore placed as an evacuation (see Not covered). Roll back by reverting this commit.

Not covered

AI assistance: Claude Code made the change, ran the validation and drafted this description. A human reviews before merge.

@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Restore behavior is now represented by RestoreIntent, with separate policies for user restores and drain restores. Restore planning accepts explicit snapshot choices instead of an evacuation boolean. Pod creation derives evacuation tolerations and affinity from the intent’s placement. Callers and tests now pass the intent or snapshot choice explicitly.

Possibly related PRs

Merge Risk: Moderate · up to 25b41

This refactor changes how restore decisions are carried and is described as behavior-preserving. No concrete defect was found. The charge, redrive and source-placement paths have no scenario coverage, so check them before merging or accept that gap explicitly.

Security architecture risk: Low · up to 25b41

The inspected changes preserve the separation between user restores and privileged drain operations. No introduced security issue was identified. Residual risk is concentrated in preserving lifecycle and recovery behavior across this policy refactor.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The sensitive surface is shared sandbox lifecycle orchestration: claim ownership, snapshot-source selection and Kubernetes pod placement. The inspected operations remain keyed to a sandbox identity; no introduced path to select another sandbox's snapshot or acquire drain authority was identified.

Trust Boundaries and Controls

  • observed — User lifecycle calls reach fixed user-restore policy rather than accepting an intent from request input. The inspected production caller of restore_evacuated_execution is the node-drain controller, which derives placement and target from drain reconciliation. The executor separately selects the evacuation-marker claim and rejects a stopped target under user claim mode.

Resilience and Maintainability Implications

  • observed — Claim winners retain deferred charging and claim release on charge rejection. A preclaimed wake makes its cloned intent charge-exempt before detached execution, preserving the previous protection against charging the same activation again. Started-recovery supersession continues through the existing bounded redrive paths for user intent.
Pre-merge checks: Passed 6

Check name Status Explanation
Title check Passed The title meets the format requirements and describes the main restore intent refactor.
Description Follows The Pr Template Passed The description contains all four required items. It describes the change, reports validation results, states severity and risk, and identifies the AI tool and its contributions.
Bug Fixes Name A Regression Test Passed This PR refactors restore decisions without fixing a stated application defect. The check does not apply.
Weakened Safeguards Are Justified Passed The changed drain test uses a cohort instead of no cohort. The description explains that this adds coverage for cohort placement. The existing assertions remain, and the diff adds cohort assertions. N…
Agent Tooling Runs No New Commands Passed The diff changes only Kubernetes Rust source and tests. It does not change agent manifests, skills, guideline files, or agent command files.
Public-Safe Content Passed The diff, title, and description contain no identified restricted content. The cohort domain in the test uses a reserved example domain. The reported measurements are from validation, not production.
Full details: Weakened Safeguards Are Justified

Explanation

The changed drain test uses a cohort instead of no cohort. The description explains that this adds coverage for cohort placement. The existing assertions remain, and the diff adds cohort assertions. No other safeguard weakening was found.

Recent review info

Automatic approval policy

Commit: 25b4176ccc15.

Setting Effective value Source
Automatic approvals Enabled Global configuration
Merge risk Moderate Global configuration
Security risk Not set Global configuration

No overrides matched. The global limits apply.

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

  • Source mode: Auto
  • Repository: perplexityai/SPACE
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: f72ba6a
  • Layers: Repository YAML; Enforced overrides: Organization
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 53d0efba-169d-4690-8488-65ca74af20fe
Commits

Reviewing files that changed from the base of the PR and between f72ba6a and 25b4176.

Files selected for processing: 13

  • crates/l40_api/api/src/controllers/substrate/kubernetes/adapter/substrate_adapter.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/adapter_characterization_tests/create_and_teardown.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/execution.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/mod.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/pod_ops/ensure_pod.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/pod_ops/restore.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/pod_ops/tests.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/restore_intent.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/store.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/store/k8s_store_postgres_tests.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/store/k8s_store_postgres_tests/restore_revert_tests.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/store/restore_source.rs
  • crates/l40_api/api/src/controllers/substrate/kubernetes/tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 1 remain after this review.

@github-actions github-actions Bot added the triage:pending Waiting for a maintainer to triage label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

User restores and the node drain's evacuation restores share the
Kubernetes restore executor, which told them apart by testing
`evacuation_placement.is_some()`, or passing `evacuation: bool`, at each
decision: which claim to take, whether to join an in-flight or a
completed restore, the charge, the wake coalescer, the snapshot and the
pod placement, the runtime upgrade, the started-recovery redrive, and
who commits `restoring -> started`. One
`Option<Option<EvacuationPlacement>>` meant both "this is the drain" and
"which cohort", and another kind of restore would have had to add a
condition to every one of those branches.

Build a `RestoreIntent` once: `RestoreIntent::user` on the user restore
and wake paths, and `RestoreIntent::drain` inside
`restore_evacuated_execution`. The executor, ensure-pod, the suspend
restore plan and the restore-source selector read its named fields
(`target`, `claim`, `snapshot`, `pod_placement`,
`allows_runtime_upgrade`, `charge`, `coalesces`,
`owner_commits_started`, `redrives_started_recovery`) instead. The
charge moves from `RestoreClaimHandling` into the intent, and the wake
drops it for a preclaimed caller, which paid for its claim itself, as
`preclaimed_with_cleanup` did.

This changes no behaviour. Each constructor sets every field to what the
old test gave its caller. Branches keyed on the target state or on the
claim variant keep that key, and branches that never fire for the drain
keep the drain's old value. `restore_evacuated_execution` keeps its
signature. A drain restore test now checks that a rollout cohort reaches
the restore pod. At opt-level 0 no future on the path is larger than
before, except the detached restore drive, 16 bytes larger, which is now
boxed.
@ac-pplx
ac-pplx marked this pull request as ready for review November 6, 2026 19:58
@ac-pplx
ac-pplx added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit 6bfcdf8 Nov 6, 2026
51 checks passed
@ac-pplx
ac-pplx deleted the ac/restore-intent branch November 6, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage:pending Waiting for a maintainer to triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant