Skip to content

Add schema for fenced backup ownership and retained recovery - #318

Closed
zibi-pplx wants to merge 1 commit into
mainfrom
pr318
Closed

zibi-pplx wants to merge 1 commit into
mainfrom
pr318

Conversation

@zibi-pplx

@zibi-pplx zibi-pplx commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Backup identity and retained-artifact recovery need durable provenance that survives retries and restarts. Add nullable execution/pool Node UIDs, materialization IDs, snapshot source identity, upload evidence, and constrained recovery approvals. Historical identity stays unknown; there is no backfill.

Deploy this schema before the dependent runtime PRs. DDL uses short lock timeouts; the new foreign key is added NOT VALID and validated separately. Approval history is retained independently of snapshot pruning, and the down migration refuses to erase issued approvals. Keep the additive schema on application rollback. The earlier draft migrations were never applied to a shared database, so these use fresh versions newer than main.

Validation: pinned Squawk passes all seven up migrations; fresh and main-to-new-schema PostgreSQL 17 runs produce identical schemas. Version and migration-isolation guards pass for each stack commit. PostgreSQL-backed RPC tests exercise the approval consistency constraints. Public diff review and Gitleaks passed.

Stack (land in order): #318 schema → #319 materialization ownership → #345 backup caller identity → #346 upload evidence → #283 retained-artifact recovery.

Rebuilt against d1abc66fe48bccaaa715e09593f875f7c6c4d11d. The schema is separate because current CI requires migration-only PRs. All existing PRs are reused.

Scope boundary: no file overlap with #828 (2a4988b6), #829 (1b696768), or #837 (58ceff04). Both compositions—#828+#829 and #837+#829—passed aarch64 Linux musl Clippy and all 98 selected snapshot/teardown tests each. These are component tests, not live Firecracker recovery evidence. Capture, encrypted sealing, teardown handoff, and source cleanup remain with those workstreams. Linux/KVM held-volume recovery, manager-restart/handoff failure, and live restore of the same artifact have not been validated here; issuance must remain disabled until that deployment evidence exists.

AI assistance: Codex contributed implementation, tests, and this description.

@zibi-pplx
zibi-pplx added this pull request to stack #320 October 12, 2026 20:16
zibi-pplx added a commit that referenced this pull request Oct 12, 2026
A worker may need a fresh backup attempt after its execution has retired.
Require an exact earlier registration from the same source incarnation for
that retry. Refuse first registration after retirement, even when inventory
and the current-artifact pin match; worker-reported inventory grants no authority.

First registration on active or target-only placement must match the active
execution's recorded Node UID. Persist that UID per attempt and check it again
on replay and finalize, holding sandbox and node locks through the catalog
write. Reject conflicting artifact kind, origin, or snapshot name. Retries
inherit the earliest verified registration timestamp so an older artifact
cannot become fresh restore evidence after relocation.

Depends on schema #318 and identity writers #319 in native GitHub stack #320. Deploy schema,
then both execution writers, then enforcement; drain old in-flight uploads first.
Historical attempts without UIDs cannot finalize under the new policy. Deploy
exact-Ready reuse from #281 before enforcement to avoid unnecessary reuploads
of restored artifacts. Keep this stack separate from approved #281 -> #285.

Registered-address authentication and credential vending remain separate
boundaries; credential-vending follow-up is #321. Registry convergence is required to detect node replacement; this
is not caller-carried incarnation authentication. No job reset, deployment, or
live recovery is included. This PR remains draft for architecture review.

Validation: 416 selected PostgreSQL/controller/storage tests passed with required
PostgreSQL and Redis, including the 24-case admission matrix, prior-registration
freshness, replay identity, terminal transitions, and pool adoption. Formatting
and repository policy checks passed. Affected-package all-target clippy passed with warnings denied. The final run
passed including the transaction-lock regression; nextest reported one child-output
leak in pooled-create case 1. Gitleaks and public-disclosure review had no findings in the changed scope.
The complete CI lint runner stops on macOS at its Linux-only Wild prerequisite;
Linux/Firecracker end-to-end behavior remains unverified.

AI assistance: Codex contributed implementation, tests, and description.
@zibi-pplx zibi-pplx changed the title Record Kubernetes incarnation fields for executions, pool slots, and backups Add schema for backup identity and bounded recovery approvals Oct 13, 2026
@zibi-pplx

Copy link
Copy Markdown
Collaborator Author

The schema prerequisite now also includes the per-materialization identity and bounded, audited recovery grants with attempt provenance. All columns are additive; historical worker identity remains unknown, with no backfill or enforcement. Migration-only isolation passed for this PR and all six stack commits, and required PostgreSQL fixtures exercised the schema. Runtime writers and policy changes remain in later draft PRs.

@shrikant-pplx shrikant-pplx left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate. This review was submitted twice after a GitHub API error; the findings are in #pullrequestreview-5508014453.

@shrikant-pplx shrikant-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed together with the rest of stack #345 → #318 → #319 → #346. No P0/P1 findings. One P2 and one P3 are inline.

Ran locally: scripts/check-migration-versions.sh (base ecd3f9a1d) and scripts/check-postgres-migrations-squawk.sh both pass.

  1. Fixes the stated problem and root cause? Partly. It supplies the execution, pool-slot and materialization columns that #319 uses. The recovery-grant schema has no consumer, and the schema doesn't enforce "bounded".
  2. Covers every affected caller and path? Partly. #319 consumes sandbox_k8s_executions.k8s_node_uid, kubernetes_pool_slots.k8s_node_uid and materialization_id. Nothing in the stack consumes snapshots.source_node_uid, snapshots.recovery_grant_id or backup_recovery_grants.
  3. Every change necessary, no scope creep? No. Three of the six migration pairs arrive before any writer exists (inline P2).
  4. Do the tests reproduce the failure and prove the fix and edge cases? Partly. There are no migration tests. The repo guards pass, and #319's Postgres tests exercise only the consumed columns.
  5. Regression? No. The DDL is additive and nullable, with short lock timeouts and a separate foreign-key validation.

Checked and found no problem:

  • Migration versions are newer than main's maximum.
  • Each ADD COLUMN is nullable with no default.
  • The foreign key is added NOT VALID and validated in a separate migration (SHARE UPDATE EXCLUSIVE lock).
  • Down migrations run in the correct order.
  • Having no index on snapshots.recovery_grant_id is fine as long as grants are never deleted.


-- Recovery authority is separate from historical ownership and survives attempt
-- retention as an audit record. It is never inferred from worker inventory.
CREATE TABLE backup_recovery_grants (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [func] Speculative schema: nothing in the stack reads or writes it

At #346, nothing outside db/ references backup_recovery_grants, snapshots.recovery_grant_id or snapshots.source_node_uid (the last one comes from 20261012193402_backup_source_node_identity). #346 records its proofs in the volume-manager job payload, not here.

Applied migrations are immutable (db/README.md). Merging this fixes in place:

  • a 19-column audit table
  • its index
  • a foreign key from snapshots

That happens before the code that defines the table's state machine has been reviewed, so any redesign needs follow-up migrations. The store guidance is also to add record columns together with their first caller.

Fix: keep 193400, 193401 and 220000, which #319 consumes. Move 193402, 223000 and 223001 into the migration PR that directly precedes their writer.

revoked_at TIMESTAMPTZ,
revoked_by UUID,
revoke_reason TEXT,
CHECK (captured_at <= approved_at AND expires_at > approved_at),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [check] The schema doesn't enforce "bounded"

The only time bound is expires_at > approved_at. There is:

  • no maximum approval window
  • no uniqueness on consumed_attempt_id
  • nothing tying revoke_reason to revoked_at
  • nothing stopping a grant from being both consumed and revoked

Adding these constraints is free while the table is empty. Later, they need a NOT VALID/VALIDATE pass or a concurrently built unique index.

Fix: when the table lands with its writer, add constraints that match the intended state machine. For example:

  • a maximum for expires_at - approved_at
  • (revoked_at IS NULL) = (revoke_reason IS NULL)
  • UNIQUE (consumed_attempt_id)

@ac-pplx
ac-pplx removed their request for review October 14, 2026 19:58
@kumarpallav-pplx
kumarpallav-pplx requested review from shrikant-pplx and removed request for nikhilbirmiwal-pplx October 16, 2026 17:56
Persist nullable execution/pool Node UIDs, materialization tokens, backup source identity, upload evidence, and bounded recovery approval provenance. Keep migration deployment separate from runtime rollout. Unknown historical identities stay unknown; no data backfill guesses ownership.

Apply before the dependent writers and readers. DDL uses short lock timeouts; the recovery foreign key is added NOT VALID and validated separately. Approval constraints enforce the 24-hour window, paired revocation fields, and exclusive consumption/revocation. Audit records outlive attempts, and the down migration refuses to erase issued approvals. Keep additive schema on application rollback. Draft migration versions were never applied to a shared database and use fresh versions newer than main.

Validation: pinned Squawk found no issues in seven up migrations. All 24 current migrations applied to fresh PostgreSQL 17 and to a main-schema database upgraded with these seven; resulting schemas match. PostgreSQL-backed RPC tests cover approval constraints and atomic consumption. Public diff review and Gitleaks scan passed.

AI assistance: Codex contributed implementation, tests, and this description.
@zibi-pplx
zibi-pplx removed this pull request from stack #387 October 22, 2026 16:15
@zibi-pplx zibi-pplx changed the title Add schema for backup identity and bounded recovery approvals Add schema for fenced backup ownership and retained recovery Oct 22, 2026
@zibi-pplx
zibi-pplx changed the base branch from codex/verified-backup-caller to main October 22, 2026 16:15
@zibi-pplx
zibi-pplx added this pull request to stack #879 October 22, 2026 16:20
@zibi-pplx zibi-pplx closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants