Retry backup handoff after suspend publication by zibi-pplx · Pull Request #829 · perplexityai/SPACE

zibi-pplx · 2026-10-22T02:12:34Z

An encrypted suspend can publish a usable artifact and then quarantine solely because its backup handoff failed. Plaintext handoff failure can instead restore a database pointer to a predecessor that publication already retired. Preserve the published artifact and choose recovery according to runtime availability:

Case	Result
Live plaintext VM; handoff fails	Return to paused with the published artifact, retaining monitoring and wake availability.
Runtime gone; transient handoff failure	Retain MarkSuspended cleanup intent and retry before runtime cleanup.
Runtime gone; explicit permanent artifact refusal	Quarantine with the existing publication-failure reason, atomically clear cleanup intent and finish its cleanup job.

The shared suspend cleanup path also establishes backup ownership for startup and periodic recovery. Existing persisted transfer handles satisfy the handoff locally, so cleanup does not add a second remote registration dependency. Restore activation compensation keeps its existing rollback behavior, and teardown suspends still require remote durability. Unconfigured local-only backups remain supported. Transient retries use the existing cleanup executor and emit an error after five minutes pending.

Two guarded SQLite writers use existing columns: adopt the published artifact on live rollback, and terminate a permanently refused suspend cleanup. Both check execution ownership; refusal leaves all rows unchanged. There are no schema migrations or protocol changes. No node-manager deployment dependency is introduced. When downgrading sandbox-manager, settle pending suspend handoffs first: the older cleanup path lacks the backup-ownership gate.

Permanent refusals remain quarantines and do not count as prevented failures. Their existing delete/eviction retention limitations are tracked separately in #876; suspend ownership and capture deadlines are tracked in #875. This change targets post-publication handoff failure, with capture and seal ordering outside its scope.

Validation: 1,414 sandbox-manager library tests and 386 store tests; affected-package all-target Clippy on native macOS and aarch64 Linux musl; formatting. Regression checks cover live rollback, outage then retry using the same artifact/upload, permanent refusal, stale ownership, terminal job completion, persisted handoff reuse during control-plane outage, and restore compensation. Removing live rollback and atomic job completion makes their respective regressions fail. Tests use synthetic artifacts and injected services; no Firecracker restore was exercised.

AI assistance: implemented and reviewed with Codex.

A post-publication backup outage must not quarantine an otherwise recoverable suspend or roll a live plaintext VM back to a retired artifact. Live plaintext handoff failures return to paused while atomically adopting the published artifact. When the runtime is gone, the existing cleanup job owns handoff retries before resources are released. Keep artifact refusals typed. An explicit permanent refusal atomically quarantines this execution, clears its cleanup intent and finishes the matching job while retaining the artifact. Reuse persisted transfer handles locally; restore compensation and teardown durability keep their existing contracts. The guarded writers use existing columns, states and reason codes. There are no schema or protocol changes. Settle pending suspend handoffs before reverting to an older binary that lacks the cleanup backup-ownership gate. Validation: 1,414 sandbox-manager and 386 store library tests passed; affected all-target Clippy passed natively and for aarch64 Linux musl; formatting passed. Synthetic regressions cover outages, live rollback, permanent refusal, stale owners, job completion, persisted handoff reuse and restore compensation. Removing live rollback or atomic job completion makes the regressions fail. No Firecracker restore was exercised.

Retained full-suspend artifacts may lack ordinary historical publisher authority. Add managing-admin approvals bound to the sealed manifest, capture-time evidence, uploader incarnation, and current sandbox state. Begin permits one active attempt; Ready validates logical and object evidence and consumes the approval atomically. Expiry, revocation, supersession, or changed state refuses Ready while preserving failure reporting. Grant-backed attempts never become ordinary retry authority. Explicit approval takes precedence over an older failed registration; that registration cannot bypass the approval after consumption. Issuance defaults off until all backup readers and workers enforce this contract. Approvals do not seal captures, enqueue uploads, clear quarantine, or wake sandboxes. Preserve original retained bytes and the audit schema; grant-aware readers are the rollback floor after issuance. Existing capture and cleanup owners remain responsible for source retention. Validation: all 389 internal API nextest tests and all 228 admin tests pass. Real TokenReview/live-pod/RPC/PostgreSQL cases exercise approval consumption, expiry, revoke, proof mismatch, changed state, abandoned versus materialized successors, replay, and Ready/revoke concurrency. Compressed-versus-logical size regression reproduced before correction. Admin authorization and rollout gate tests and the refreshed OpenAPI snapshot pass. Full workspace all-target/all-feature Clippy passed. Fresh and upgrade PostgreSQL schemas match; Squawk found no issues in all seven migrations. Both compositions with the capture/cleanup work (#828+#829 and #837+#829) passed aarch64 Linux musl lint and all 98 selected snapshot/teardown tests each. Linux/KVM live recovery remains unvalidated. Documentation classification and six site build tests passed. Public diff review and Gitleaks scan passed. AI assistance: Codex contributed implementation, tests, and this description.

## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Restore compensation skips the gate; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The complete stack passes 1,833 sandbox-manager tests and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Affected-package all-target Clippy, formatting and architecture checks pass. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity and full CI qualification remain pending. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following #1762 and #1763. This incorporates the post-publication handoff scope of #829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

zibi-pplx · 2026-11-12T19:35:58Z

Closing as superseded by #1765, the third PR in stack #1764 (#1762 → #1763 → #1765). The replacement is still under review and has not landed.

Patch 3 covers this PR’s post-publication handoff recovery: retain the published artifact on live plaintext rollback, reuse persisted handoff evidence, and retry through the cleanup owner when the runtime is gone. It also bounds unresolved handoff retries to fifteen minutes from durable cleanup admission, then retains the capture in quarantine and atomically finishes cleanup. Active teardown continues to require remote durability. This consolidates the overlapping implementation in one stack; #829 does not need to merge separately.

## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Restore compensation skips the gate; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The complete stack passes 1,833 sandbox-manager tests and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Affected-package all-target Clippy, formatting and architecture checks pass. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity and full CI qualification remain pending. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following #1762 and #1763. This incorporates the post-publication handoff scope of #829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Restore compensation skips the gate; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. The final handoff evidence read and quarantine decision hold the same database mutex used by the upload worker, so a concurrent persisted handoff wins. Older upload payloads without an execution generation are accepted only with matching artifact/sandbox identity, staged metadata and persisted transfer handles; this never authorizes a new unfenced registration. Explicit generation mismatches remain refusals. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The concurrent-handoff regression fails before the final evidence recheck; the legacy-handoff case fails before its narrow compatibility exception. All 13 registration tests pass, including wrong artifact, wrong generation, missing metadata and unfinished legacy uploads. The complete stack passes 1,833 sandbox-manager tests and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Affected-package all-target Clippy, formatting and architecture checks pass. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity and full CI qualification remain pending. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following #1762 and #1763. This incorporates the post-publication handoff scope of #829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Restore compensation skips the gate; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. The final handoff evidence read and quarantine decision hold the same database mutex used by the upload worker, so a concurrent persisted handoff wins. Older upload payloads without an execution generation are accepted only with matching artifact/sandbox identity, staged metadata and persisted transfer handles; this never authorizes a new unfenced registration. Explicit generation mismatches remain refusals. Existing handoffs are checked before local sidecar reads; malformed metadata cannot block already-persisted transfer ownership. New uploads still require valid sidecar metadata. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The concurrent-handoff regression fails before the final evidence recheck; the legacy-handoff case fails before its narrow compatibility exception. The registration cases cover wrong artifact, wrong generation, missing metadata, unfinished legacy uploads and malformed sidecar after durable handoff; the malformed-sidecar case fails before moving the read behind the existing-upload check. The complete stack passes 1,834 sandbox-manager tests under repository-configured nextest and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Affected-package all-target Clippy, formatting and architecture checks pass. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity and full CI qualification remain pending. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following #1762 and #1763. This incorporates the post-publication handoff scope of #829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Age is refreshed after the awaited attempt, so crossing the deadline settles on that failure rather than adding another backoff. The gate uses the persisted suspending lifecycle, so both episode-backed restore compensation and row-intent restore retirement skip it across retries and restarts; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. The final handoff evidence read and quarantine decision hold the same database mutex used by the upload worker, so a concurrent persisted handoff wins. Older upload payloads without an execution generation are accepted only with matching artifact/sandbox identity, staged metadata and persisted transfer handles; this never authorizes a new unfenced registration. Explicit generation mismatches remain refusals. Existing handoffs are checked before local sidecar reads; malformed metadata cannot block already-persisted transfer ownership. New uploads still require valid sidecar metadata. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The concurrent-handoff regression fails before the final evidence recheck; the legacy-handoff case fails before its narrow compatibility exception. The registration cases cover wrong artifact, wrong generation, missing metadata, unfinished legacy uploads and malformed sidecar after durable handoff; the malformed-sidecar case fails before moving the read behind the existing-upload check. The restore-retirement regression reproduced a wrong-generation retained upload preventing node dispatch. An injected clock reproduced the deadline-crossing defect without wall-clock sleeps. Final-fix validation is pending in current-head CI; formatting passes. Earlier full validation passed 1,834 sandbox-manager tests with nextest and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity requires platform qualification. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following #1762 and #1763. This incorporates the post-publication handoff scope of #829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Age is refreshed after the awaited attempt, so crossing the deadline settles on that failure rather than adding another backoff. The gate uses the persisted suspending lifecycle, so both episode-backed restore compensation and row-intent restore retirement skip it across retries and restarts; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. The final handoff evidence read and quarantine decision hold the same database mutex used by the upload worker, so a concurrent persisted handoff wins. Older upload payloads without an execution generation are accepted only with matching artifact/sandbox identity, staged metadata and persisted transfer handles; this never authorizes a new unfenced registration. Explicit generation mismatches remain refusals. Existing handoffs are checked before local sidecar reads; malformed metadata cannot block already-persisted transfer ownership. New uploads still require valid sidecar metadata. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The removed `node_local_backup_fence_failure_preserves_runtime_and_rolls_back_to_paused` test exercised `require_node_local_backup_handoff`, which this patch replaces. Its `current_local_artifact_id.is_none()` assertion no longer holds: live rollback must retain the newly published capture. The replacement, `node_local_suspend_of_a_superseded_execution_retains_the_published_capture_on_live_rollback` in `registration_tests.rs`, checks the Internal error code, backup-fence metadata, preserved execution generation on both the sandbox row and upload payload, retained PID/artifact, paused state, and successful wake. The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The concurrent-handoff regression fails before the final evidence recheck; the legacy-handoff case fails before its narrow compatibility exception. The registration cases cover wrong artifact, wrong generation, missing metadata, unfinished legacy uploads and malformed sidecar after durable handoff; the malformed-sidecar case fails before moving the read behind the existing-upload check. The restore-retirement regression reproduced a wrong-generation retained upload preventing node dispatch. An injected clock reproduced the deadline-crossing defect without wall-clock sleeps. Final-fix validation is pending in current-head CI; formatting passes. Earlier full validation passed 1,834 sandbox-manager tests with nextest and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity requires platform qualification. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following #1762 and #1763. This incorporates the post-publication handoff scope of #829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

…tyai#1765) ## Summary Preserve a published suspend capture when backup handoff fails, and give that failure a bounded owner instead of leaving cleanup and wake waiting indefinitely. | Handoff outcome | Lifecycle result | | --- | --- | | Failure while a plaintext VM is still live | Roll back to paused with the newly published artifact, retaining wake availability. | | Transient failure after publication | The existing cleanup job retries the same artifact/upload before runtime cleanup. | | Terminal upload failure or expired retry window | Retain the artifact in quarantine and finish the cleanup job atomically. Wake refuses explicitly instead of waiting for cleanup forever. | | Transfer handles already persisted | Complete cleanup without another remote registration, including after restart or expiry. | Remote attempts are capped at 30 seconds. Pending handoff emits an error after five minutes and ends on the first failed attempt at or after fifteen minutes from the cleanup job's persisted creation time. Reconciliation does not renew that clock. Age is refreshed after the awaited attempt, so crossing the deadline settles on that failure rather than adding another backoff. The gate uses the persisted suspending lifecycle, so both episode-backed restore compensation and row-intent restore retirement skip it across retries and restarts; teardown suspends retain their existing remote-durability requirement. Node-local cleanup requires persisted transfer ownership, not a completed remote upload. The final handoff evidence read and quarantine decision hold the same database mutex used by the upload worker, so a concurrent persisted handoff wins. Older upload payloads without an execution generation are accepted only with matching artifact/sandbox identity, staged metadata and persisted transfer handles; this never authorizes a new unfenced registration. Explicit generation mismatches remain refusals. Existing handoffs are checked before local sidecar reads; malformed metadata cannot block already-persisted transfer ownership. New uploads still require valid sidecar metadata. Two guarded SQLite operations use existing columns: live rollback adopts the new artifact rather than its retired predecessor, and terminal settlement clears cleanup ownership and finishes its job in one transaction. Both reject stale execution/artifact owners without changing any rows. There is no additional migration, protobuf change, journal or worker. ## Validation The removed `node_local_backup_fence_failure_preserves_runtime_and_rolls_back_to_paused` test exercised `require_node_local_backup_handoff`, which this patch replaces. Its `current_local_artifact_id.is_none()` assertion no longer holds: live rollback must retain the newly published capture. The replacement, `node_local_suspend_of_a_superseded_execution_retains_the_published_capture_on_live_rollback` in `registration_tests.rs`, checks the Internal error code, backup-fence metadata, preserved execution generation on both the sandbox row and upload payload, retained PID/artifact, paused state, and successful wake. The terminal-handoff regression fails with the previous cleanup gate and passes with this change. Tests exercise the real cleanup orchestration and wake path across transient retry, terminal failure, expiry, already-persisted handoff and live rollback. They also verify stale-owner rejection and atomic job completion. The concurrent-handoff regression fails before the final evidence recheck; the legacy-handoff case fails before its narrow compatibility exception. The registration cases cover wrong artifact, wrong generation, missing metadata, unfinished legacy uploads and malformed sidecar after durable handoff; the malformed-sidecar case fails before moving the read behind the existing-upload check. The restore-retirement regression reproduced a wrong-generation retained upload preventing node dispatch. An injected clock reproduced the deadline-crossing defect without wall-clock sleeps. Final-fix validation is pending in current-head CI; formatting passes. Earlier full validation passed 1,834 sandbox-manager tests with nextest and 712 store tests. One unchanged non-UTF-8 filename test is excluded on macOS. Tests use synthetic services and artifacts; real Firecracker/btrfs restore integrity requires platform qualification. ## Severity and risk High: a valid capture can be quarantined after a transient handoff failure, or cleanup can retain wake indefinitely. A prolonged outage or terminal refusal still requires operator recovery; this change does not promise zero quarantines. The existing runtime and artifact retention rules remain in force. Settle pending handoffs before downgrading: older cleanup code lacks this retry and terminal policy. This patch adds no schema rollback requirement beyond its parent patch. ## Stack Patch 3 of 3, following perplexityai#1762 and perplexityai#1763. This incorporates the post-publication handoff scope of perplexityai#829 into the capture-recovery stack, with a bounded retry window and wake regression coverage. AI assistance: Codex drafted code, tests and this description.

zibi-pplx changed the title ~~Retry unobservable pause outcomes through lifecycle recovery~~ Retry backup handoff from durable suspend cleanup intent Oct 22, 2026

zibi-pplx force-pushed the pr829 branch 2 times, most recently from 35a4c1f to 0caba4b Compare October 22, 2026 02:57

zibi-pplx marked this pull request as draft October 22, 2026 13:49

zibi-pplx force-pushed the pr829 branch from 0caba4b to 1b69676 Compare October 22, 2026 15:29

zibi-pplx changed the title ~~Retry backup handoff from durable suspend cleanup intent~~ Retry backup handoff after suspend publication Oct 22, 2026

zibi-pplx marked this pull request as ready for review October 22, 2026 15:29

This was referenced Nov 12, 2026

suspend: recover the admitted capture from a producer-owned receipt #1763

Merged

suspend: bound backup handoff and retain published captures #1765

Merged

zibi-pplx closed this Nov 12, 2026

Navigation Menu

Repository navigation

Retry backup handoff after suspend publication - #829

Retry backup handoff after suspend publication#829
zibi-pplx wants to merge 1 commit into
mainfrom
pr829

zibi-pplx commented Oct 22, 2026 •

edited

Loading

Uh oh!

zibi-pplx commented Nov 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zibi-pplx commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zibi-pplx commented Nov 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zibi-pplx commented Oct 22, 2026 •

edited

Loading