[mabdelraouf-pplx]

Summary

A failed upload attempt must not permanently strand a generation suspend when its immutable local artifact is still usable.

This draft adds default-off automatic re-drive (SPACE_SUSPEND_BACKUP_REDRIVE_ENABLED=false):

• Failed full-suspend transfers exhaust the snapshot observer's bounded retries as retriable. The generation driver allocates a new attempt through the same submit_artifact_backup path as EnqueueArtifactBackup.
• Before submission, validate the retained source, metadata identity, and the current generation/artifact. Recheck ownership under the same local SQLite lock as job admission; reject locally retired/superseded generations and mismatched artifacts.
• Preserve a terminal result for missing/invalid artifacts, with explicit diagnostics. An unavailable inspector or I/O error is not proof of loss.
• Persist capped exponential generation-driver backoff (30 seconds to 15 minutes), and discover due jobs from periodic recovery even if the original caller disappears. Reuse the existing bounded driver pool.
• Emit snapshot.backup.redrive for re-drive submission attempts. No Datadog monitor or non-durable-age gauge is included.
• Recover historical permanent snapshot-job failures only after a typed live volume-manager observation proves a failed member transfer for the exact artifact. Do not classify historical error strings.

Severity: high. A transient storage dependency failure can otherwise become an indefinitely non-durable suspend.

Rollout and related changes — do not enable yet

Keep the flag off until #331's cluster admission fence has shipped and is integrated with this submission path. Main cannot protect the case where an execution has retired only in the control plane, but the old worker still holds an active local generation job and artifact. This PR does not reproduce or replace that remote fence. Local checks alone do not make enabling the flag safe.

• [backup] refuse teardown-suspend backups from retired executions and release their runtimes #331 overlaps the optional execution-generation plumbing in ArtifactBackupSubmission and the teardown driver. Preserve this PR's atomic local admission guard and [backup] refuse teardown-suspend backups from retired executions and release their runtimes #331's durable payload + remote admission fence when reconciling them.
• Approve bounded publication of validated retained suspends #283: ordinary retries must continue to satisfy exact prior registration and verified worker-incarnation admission. This PR grants no retained-artifact recovery approval and does not bypass a registration/finalization refusal.
• Add schema for fenced backup ownership and retained recovery #318 and Bind backup RPCs to verified historical worker identity #345 provide the schema/verified identity prerequisites for that policy. No schema or identity changes here.
• Carry verified upload evidence through backup finalization #346 changes successful-transfer proof collection. This PR changes failed-transfer handling, not proof production; retain its attempt/lease-fenced completion when combining.

The volume-manager time-budget extension is in the separate stacked draft #351. This PR alone still allows its existing five-attempt budget to expire; it repairs the next stage when an independent valid local artifact survives. It never restores missing bytes or marks a partial upload Ready.

Evidence and limitations

Synthetic loopback gRPC regression, on the real poll_artifact_job path:

cargo test --locked -p pplx-space-sbx-mgr exhausted_ --lib
Before failure-classification fix:
  exhausted_credential_transfer_can_be_redriven ... FAILED
  "must not permanently strand its suspend ... credential exchange failed"
  left: false, right: true
After failure-classification fix:
  3 passed; 0 failed

The test holds the remote transfer at Failed, reproducing the observed terminal boundary; it does not run a real credential authority or claim a complete credential-exhaustion-to-restored-VM test. The matching admission regression exercises the real generation backup method and persisted job submission with a test-owned unavailable cluster credential; a valid source gets a fresh job, while missing/invalid, stale-generation, retired, and wrong-artifact cases do not.

Validation:

• Common-jobs unit suite: 57 passed.
• Full sandbox-manager unit suite: 1,576 passed, 1 ignored (including the 52-test lifecycle module, persisted admission/backoff regressions, and transfer polling tests).
• Repository fmt stage: passed (includes generated shim environment and Rust error-policy checks).
• cargo arch: passed required checks; advisory findings remain.
• Repository full workspace clippy and feature-isolation checks: passed.
• Final exact-head CI, head 88a3af1925ef08cca734ea1c0b63d33f4982d379: all checks passed, including full Rust tests, clippy, fmt, gateway traffic, and the aggregate gate.
• The initial CI run caught an existing assertion expecting a stale caller to mutate durability; the follow-up test commit now asserts no mutation, matching the new local fence. The full local suite and final CI above include that correction. This stays draft as requested, with the rollout and end-to-end proof gaps below.
• No live cluster, cloud storage, or production changes. No Firecracker/Kind end-to-end restore claimed.

Deliberately deferred API signal

No observed-status schema change here. Proposed follow-up: additive local_suspend.backup_recovery = pending | terminal_artifact_unavailable | unknown, interpreted only with the exact existing generation + PodUID + local-artifact identity. Absent means unknown; transient inspection/RPC/I/O failure must never publish terminal loss. Consumers must stay conservative until an authoritative writer exists.

AI assistance: Perplexity contributed the implementation, tests, and this description.

Created with Perplexity