Skip to content

Bind backup RPCs to verified historical worker identity - #345

Closed
zibi-pplx wants to merge 1 commit into
pr319from
codex/verified-backup-caller
Closed

zibi-pplx wants to merge 1 commit into
pr319from
codex/verified-backup-caller

Conversation

@zibi-pplx

@zibi-pplx zibi-pplx commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Bind Begin/Touch/Finalize backup RPCs to TokenReview identity, a bounded live pod lookup, the registered worker incarnation, and the attempt's durable source. A same-name replacement cannot inherit an old upload. Exact verified retries retain their capture time across relocation and use a separate attempt-liveness timestamp.

The injected verifier replaces the process-global design and shares pure identity checks with credential-manager without changing credential-vending policy. Verification occurs before database locks. Preserve main's retired-but-unsuperseded admission, typed supersession verdict, and verdict-only rollback; legacy rows never acquire guessed historical authority. Direct workers retain explicit peer-based compatibility.

Start in shadow mode, then enable compatibility certification after all readers deploy, and strict mode only after legacy inventory and writer rollout are verified. Shadow preserves legacy progress during identity infrastructure failures, while already-verified attempts and retries remain enforced in every mode. Keep identity-aware readers on rollback once proof has been issued. Existing bounded upload retries and retention remain the owners of liveness.

Validation: the full internal API nextest inventory passed all 389 tests with zero skips; real RPC/TokenReview/live-pod/PostgreSQL tests cover rollout modes, retired capture, relocation, immutable freshness, verification outages, static fallback, and replacement refusal. Common authentication and credential-manager identity suites passed. Workspace all-target/all-feature Clippy and affected aarch64 Linux musl compilation/lint pass. Public diff review and Gitleaks passed.

Stack (land in order): #318 schema → #319 materialization ownership → #345 backup caller identity → #346 upload evidence → #283 retained-artifact recovery.

Rebuilt against d1abc66fe48bccaaa715e09593f875f7c6c4d11d. The schema is separate because current CI requires migration-only PRs. All existing PRs are reused.

Scope boundary: no file overlap with #828 (2a4988b6), #829 (1b696768), or #837 (58ceff04). Both compositions—#828+#829 and #837+#829—passed aarch64 Linux musl Clippy and all 98 selected snapshot/teardown tests each. These are component tests, not live Firecracker recovery evidence. Capture, encrypted sealing, teardown handoff, and source cleanup remain with those workstreams. Linux/KVM held-volume recovery, manager-restart/handoff failure, and live restore of the same artifact have not been validated here; issuance must remain disabled until that deployment evidence exists.

AI assistance: Codex contributed implementation, tests, and this description.

@zibi-pplx
zibi-pplx added this pull request to stack #347 October 13, 2026 01:20
@zibi-pplx
zibi-pplx removed this pull request from stack #347 October 13, 2026 06:10
@zibi-pplx
zibi-pplx added this pull request to stack #371 October 13, 2026 06:10
Base automatically changed from codex/encrypted-backup-validation to main October 13, 2026 17:13
@zibi-pplx
zibi-pplx removed this pull request from stack #371 October 13, 2026 17:22
@zibi-pplx zibi-pplx changed the title Preserve verified Kubernetes identity for backup callers Preserve reviewed Kubernetes caller identity for backup authorization Oct 13, 2026
@zibi-pplx
zibi-pplx force-pushed the codex/verified-backup-caller branch from 0ef5424 to ad4d067 Compare October 13, 2026 17:35
@zibi-pplx
zibi-pplx marked this pull request as ready for review October 13, 2026 17:35
@zibi-pplx
zibi-pplx added this pull request to stack #387 October 13, 2026 17:35

@shrikant-pplx shrikant-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed as part of stack #345 → #318 → #319 → #346, in two passes (correctness, then security). No P0/P1 findings in this PR. One P2 and two P3s are inline.

Ran locally: cargo test -p pplx-space-common-svc --features k8s --lib -- auth:: (105 passed, including both new tests).

  1. Fixes the stated problem and root cause? Partly. The TokenReview pod/node binding is retained on both the fresh and cached paths, but no RPC calls verify_node_bound_caller yet (inline P2).
  2. Covers every affected caller and path? Partly. Both TokenReview paths carry the binding. No sensitive RPC consumes it anywhere in the stack.
  3. Every change necessary, no scope creep? Partly. node_identity.rs has no caller in the stack, and it overlaps the credential manager's existing pod-identity checks.
  4. Do the tests reproduce the failure and prove the fix and edge cases? Partly. The retention, pod-comparison and budget tests are good. The entry point's error mapping and IPv4-mapped peers are untested.
  5. Regression? No. The new field defaults to None, static-token and cache behavior are unchanged, and the verifier is not reachable yet.

Checked and found no problem:

  • The cache key (token digest plus audience) covers the binding, because the binding comes from immutable token claims.
  • Missing, empty or multi-valued extras yield None.
  • deletionTimestamp is checked after the UID, node, service account, phase and IP checks, so a replaced pod that is terminating still gets PermissionDenied.
  • The timeout wraps permit acquisition, cancellation returns the permit, and advisory mode uses try_acquire.
  • Peer comparison normalizes IPv4-mapped addresses on both sides.

/// at most eight checks execute concurrently per process. Unavailable identity
/// infrastructure never grants authority.
#[cfg(feature = "k8s")]
pub async fn verify_node_bound_caller<T>(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [doc] Nothing calls the live verifier, in this PR or anywhere in the stack

At the top of the stack (#346), verify_node_bound_caller and NodeApiAuthenticatedCaller::kubernetes_pod() are referenced only by this module, the pub use in auth/mod.rs, and their own tests. No internal-api or node-api handler changes.

So two lines in the description don't describe anything an RPC does after this stack lands:

  • "Sensitive RPCs verify the live daemon's UID, node, service account, phase and peer IP"
  • "A matching terminating daemon returns retryable Unavailable"

The Required/Advisory semantics also can't be evaluated without a caller.

Fix (either):

  • Wire the verifier into the intended RPCs, with RPC-level tests.
  • Land node_identity.rs together with its first caller, and reword this PR as "retains the TokenReview pod/node binding; no RPC behavior changes".

}

#[cfg(feature = "k8s")]
fn verify_observed_pod(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [func] A second pod-identity verifier that disagrees with the credential manager's

crates/libs/credential/mgr/server/src/k8s_identity.rs already parses the same four authentication.kubernetes.io/* extras (single_extra_value). It also already checks pod UID, Running phase and peer IP (check_pod_uid, check_pod_running, check_peer_ip, normalize_ip).

The two verifiers differ:

Credential manager This verifier
Pod lookup reflector cache live pods.get on every call, through a process-global client
Node check node name and UID against registered nodes (check_node_registration) spec.node_name only
Pod with deletionTimestamp, still Running allowed Unavailable

That's one trust decision with two possible outcomes. Once this is wired in, it also adds an apiserver round trip to every sensitive RPC.

Fix: when the first caller lands, move the shared parsing and pod checks into common-svc, or reuse the credential manager's reflector approach.

.get(identity.pod_name().as_str())
.await
.map_err(|error| match error {
kube::Error::Api(response) if response.code == 404 => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [test] The entry point is untested, and hard to test

No test covers:

  • a missing caller extension
  • a missing remote_addr
  • the error mapping: 404 → PermissionDenied, any other error → Unavailable
  • a client initialization failure
  • an IPv4-mapped peer (::ffff:192.0.2.1)
  • a pod with several IPs

The static OnceCell<kube::Client> also means a fake apiserver can't be injected.

Fix: take the client as a parameter, as CredmgrTokenReviewer::new does. Then add tests for the error mapping and the mapped-peer case.

Backup retries must not inherit a replacement worker's authority or manufacture a fresh capture time. Retain TokenReview pod/node bindings, verify the live daemon before database locks, and enforce persisted source identity in Begin, Touch, and Finalize. Share credential-manager's pure identity checks without changing its policy. Exact verified retries survive relocation and preserve capture freshness while advancing the separate heartbeat clock. Preserve retired-unsuperseded admission, verdict-only rollback, and main's typed supersession response.

Consumes the separate source identity schema prerequisite. Initial SPACE_BACKUP_IDENTITY_MODE=shadow preserves legacy admission during live-verification outages and issues no new proof until all identity-aware readers deploy. Verified retries retain their proof even on shadow rollback. Compatibility then certifies new attempts while allowing legacy records; strict requires recorded Kubernetes incarnation. Already verified attempts cannot fall back in any mode. Direct static workers retain peer-based authority. Rollback after certification requires identity-aware readers. Existing bounded upload retries and reaper policy remain authoritative; no indefinite wait or blanket exemption.

Validation: all 316 internal API tests passed before extending the RPC matrix to shadow/compatibility/strict; the real RPC test exercises TokenReview, live GET failures, retired capture, relocation, preserved freshness, and replacement refusal. All 113 common authentication tests passed. Affected internal API, credential-manager, and snapshot-store all-target/all-feature Clippy passed. Public diff review and Gitleaks scan passed. Shadow rollout regression failed before correction and all three RPC mode cases now pass. All 13 generation-fence tests pass with transaction-specific lock synchronization. Workspace all-target/all-feature Clippy, affected aarch64 Linux musl compilation/lint, and isolated default/no-default builds passed. Linux/KVM live recovery remains unvalidated.

AI assistance: Codex contributed implementation, tests, and this description.
@zibi-pplx
zibi-pplx removed this pull request from stack #387 October 22, 2026 16:15
@zibi-pplx zibi-pplx changed the title Preserve reviewed Kubernetes caller identity for backup authorization Bind backup RPCs to verified historical worker identity Oct 22, 2026
@zibi-pplx
zibi-pplx changed the base branch from main to pr319 October 22, 2026 16:17
@zibi-pplx
zibi-pplx force-pushed the codex/verified-backup-caller branch from ad4d067 to ba0874e Compare October 22, 2026 16:17
@zibi-pplx
zibi-pplx added this pull request to stack #879 October 22, 2026 16:20
@zibi-pplx zibi-pplx closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants