Repository navigation
Conversation
A worker may need a fresh backup attempt after its execution has retired. Require an exact earlier registration from the same source incarnation for that retry. Refuse first registration after retirement, even when inventory and the current-artifact pin match; worker-reported inventory grants no authority. First registration on active or target-only placement must match the active execution's recorded Node UID. Persist that UID per attempt and check it again on replay and finalize, holding sandbox and node locks through the catalog write. Reject conflicting artifact kind, origin, or snapshot name. Retries inherit the earliest verified registration timestamp so an older artifact cannot become fresh restore evidence after relocation. Depends on schema #318 and identity writers #319 in native GitHub stack #320. Deploy schema, then both execution writers, then enforcement; drain old in-flight uploads first. Historical attempts without UIDs cannot finalize under the new policy. Deploy exact-Ready reuse from #281 before enforcement to avoid unnecessary reuploads of restored artifacts. Keep this stack separate from approved #281 -> #285. Registered-address authentication and credential vending remain separate boundaries; credential-vending follow-up is #321. Registry convergence is required to detect node replacement; this is not caller-carried incarnation authentication. No job reset, deployment, or live recovery is included. This PR remains draft for architecture review. Validation: 416 selected PostgreSQL/controller/storage tests passed with required PostgreSQL and Redis, including the 24-case admission matrix, prior-registration freshness, replay identity, terminal transitions, and pool adoption. Formatting and repository policy checks passed. Affected-package all-target clippy passed with warnings denied. The final run passed including the transaction-lock regression; nextest reported one child-output leak in pooled-create case 1. Gitleaks and public-disclosure review had no findings in the changed scope. The complete CI lint runner stops on macOS at its Linux-only Wild prerequisite; Linux/Firecracker end-to-end behavior remains unverified. AI assistance: Codex contributed implementation, tests, and description.
|
The schema prerequisite now also includes the per-materialization identity and bounded, audited recovery grants with attempt provenance. All columns are additive; historical worker identity remains unknown, with no backfill or enforcement. Migration-only isolation passed for this PR and all six stack commits, and required PostgreSQL fixtures exercised the schema. Runtime writers and policy changes remain in later draft PRs. |
shrikant-pplx
left a comment
There was a problem hiding this comment.
Reviewed together with the rest of stack #345 → #318 → #319 → #346. No P0/P1 findings. One P2 and one P3 are inline.
Ran locally: scripts/check-migration-versions.sh (base ecd3f9a1d) and scripts/check-postgres-migrations-squawk.sh both pass.
- Fixes the stated problem and root cause? Partly. It supplies the execution, pool-slot and materialization columns that #319 uses. The recovery-grant schema has no consumer, and the schema doesn't enforce "bounded".
- Covers every affected caller and path? Partly. #319 consumes
sandbox_k8s_executions.k8s_node_uid,kubernetes_pool_slots.k8s_node_uidandmaterialization_id. Nothing in the stack consumessnapshots.source_node_uid,snapshots.recovery_grant_idorbackup_recovery_grants. - Every change necessary, no scope creep? No. Three of the six migration pairs arrive before any writer exists (inline P2).
- Do the tests reproduce the failure and prove the fix and edge cases? Partly. There are no migration tests. The repo guards pass, and #319's Postgres tests exercise only the consumed columns.
- Regression? No. The DDL is additive and nullable, with short lock timeouts and a separate foreign-key validation.
Checked and found no problem:
- Migration versions are newer than main's maximum.
- Each
ADD COLUMNis nullable with no default. - The foreign key is added
NOT VALIDand validated in a separate migration (SHARE UPDATE EXCLUSIVElock). - Down migrations run in the correct order.
- Having no index on
snapshots.recovery_grant_idis fine as long as grants are never deleted.
|
|
||
| -- Recovery authority is separate from historical ownership and survives attempt | ||
| -- retention as an audit record. It is never inferred from worker inventory. | ||
| CREATE TABLE backup_recovery_grants ( |
There was a problem hiding this comment.
P2 [func] Speculative schema: nothing in the stack reads or writes it
At #346, nothing outside db/ references backup_recovery_grants, snapshots.recovery_grant_id or snapshots.source_node_uid (the last one comes from 20261012193402_backup_source_node_identity). #346 records its proofs in the volume-manager job payload, not here.
Applied migrations are immutable (db/README.md). Merging this fixes in place:
- a 19-column audit table
- its index
- a foreign key from
snapshots
That happens before the code that defines the table's state machine has been reviewed, so any redesign needs follow-up migrations. The store guidance is also to add record columns together with their first caller.
Fix: keep 193400, 193401 and 220000, which #319 consumes. Move 193402, 223000 and 223001 into the migration PR that directly precedes their writer.
| revoked_at TIMESTAMPTZ, | ||
| revoked_by UUID, | ||
| revoke_reason TEXT, | ||
| CHECK (captured_at <= approved_at AND expires_at > approved_at), |
There was a problem hiding this comment.
P3 [check] The schema doesn't enforce "bounded"
The only time bound is expires_at > approved_at. There is:
- no maximum approval window
- no uniqueness on
consumed_attempt_id - nothing tying
revoke_reasontorevoked_at - nothing stopping a grant from being both consumed and revoked
Adding these constraints is free while the table is empty. Later, they need a NOT VALID/VALIDATE pass or a concurrently built unique index.
Fix: when the table lands with its writer, add constraints that match the intended state machine. For example:
- a maximum for
expires_at - approved_at (revoked_at IS NULL) = (revoke_reason IS NULL)UNIQUE (consumed_attempt_id)
Persist nullable execution/pool Node UIDs, materialization tokens, backup source identity, upload evidence, and bounded recovery approval provenance. Keep migration deployment separate from runtime rollout. Unknown historical identities stay unknown; no data backfill guesses ownership. Apply before the dependent writers and readers. DDL uses short lock timeouts; the recovery foreign key is added NOT VALID and validated separately. Approval constraints enforce the 24-hour window, paired revocation fields, and exclusive consumption/revocation. Audit records outlive attempts, and the down migration refuses to erase issued approvals. Keep additive schema on application rollback. Draft migration versions were never applied to a shared database and use fresh versions newer than main. Validation: pinned Squawk found no issues in seven up migrations. All 24 current migrations applied to fresh PostgreSQL 17 and to a main-schema database upgraded with these seven; resulting schemas match. PostgreSQL-backed RPC tests cover approval constraints and atomic consumption. Public diff review and Gitleaks scan passed. AI assistance: Codex contributed implementation, tests, and this description.
Backup identity and retained-artifact recovery need durable provenance that survives retries and restarts. Add nullable execution/pool Node UIDs, materialization IDs, snapshot source identity, upload evidence, and constrained recovery approvals. Historical identity stays unknown; there is no backfill.
Deploy this schema before the dependent runtime PRs. DDL uses short lock timeouts; the new foreign key is added NOT VALID and validated separately. Approval history is retained independently of snapshot pruning, and the down migration refuses to erase issued approvals. Keep the additive schema on application rollback. The earlier draft migrations were never applied to a shared database, so these use fresh versions newer than main.
Validation: pinned Squawk passes all seven up migrations; fresh and main-to-new-schema PostgreSQL 17 runs produce identical schemas. Version and migration-isolation guards pass for each stack commit. PostgreSQL-backed RPC tests exercise the approval consistency constraints. Public diff review and Gitleaks passed.
Stack (land in order): #318 schema → #319 materialization ownership → #345 backup caller identity → #346 upload evidence → #283 retained-artifact recovery.
Rebuilt against
d1abc66fe48bccaaa715e09593f875f7c6c4d11d. The schema is separate because current CI requires migration-only PRs. All existing PRs are reused.Scope boundary: no file overlap with #828 (
2a4988b6), #829 (1b696768), or #837 (58ceff04). Both compositions—#828+#829 and #837+#829—passed aarch64 Linux musl Clippy and all 98 selected snapshot/teardown tests each. These are component tests, not live Firecracker recovery evidence. Capture, encrypted sealing, teardown handoff, and source cleanup remain with those workstreams. Linux/KVM held-volume recovery, manager-restart/handoff failure, and live restore of the same artifact have not been validated here; issuance must remain disabled until that deployment evidence exists.AI assistance: Codex contributed implementation, tests, and this description.