[mabdelraouf-pplx]

Summary

Stacked on #349. Pushes that fail because a storage dependency is unavailable should not spend their entire retry budget in five attempts.

• Add an executor-selected absolute retry deadline to the durable job worker. The deadline is reconstructed from persisted job creation time, so restarting does not replenish it. Existing jobs/executors keep their attempt-count policy.
• Volume-manager pushes only use a six-hour default for credential exchange outages and typed S3 timeout, dispatch, throttling, and server errors. Configure with SPACE_VOLMGR_DEPENDENCY_RETRY_BUDGET_SECS / --dependency-retry-budget-secs; zero restores the count-only policy.
• Preserve credential RPC status codes. Explicit input/auth/not-found refusals keep the ordinary count budget. Bound credential exchange to 30 seconds so a stalled authority is classified rather than spending the entire executor timeout.
• Carry the typed deadline through multi-file push aggregation and streaming push paths. Mixed local/non-dependency failures keep the original budget.
• Use the same exhaustion decision in staged-file cleanup and the worker, so attempt five cannot remove the source of a scheduled retry.

Severity: high. A dependency outage can otherwise cause durable uploads to become permanently failed before recovery.

Artifact retention and scope

The immutable sandbox-manager artifact and volume-manager's disposable staging copy are different files. While the deadline remains open, the volume-manager staging copy is retained for retries. At final expiry its existing cleanup policy still applies, as does restart orphan cleanup. Neither deletes the independent sandbox-manager source: #349 validates that exact source before allocating a new push. A regression checks both paths explicitly. This avoids leaking every permanently failed transfer's staging copy indefinitely.

No new schema, daemon, monitor, production configuration, or private composition change. Pulls, local codec/input errors, successful cleanup, and unrelated job executors keep their existing policies. Library callers constructing VolmgrRunConfig must supply the new duration when updating their pin.

Proof and validation

• Real durable-job failure handling: at attempt five an ordinary retry becomes Failed, while a dependency deadline leaves the persisted job Pending; expiration then becomes Failed. Common-jobs suite: 58 passed.
• Controlled red→green check: with only handle_failure's original count-only exhaustion decision restored, dependency_deadline_not_attempt_count_controls_durable_retry fails ("failed" vs expected "pending"). Restore the deadline-aware decision, unchanged test passes. This is a targeted policy-toggle baseline with the new error API present, not a claim that this new test compiles on pristine main.
• Real TransferExecutor::execute, synthetic object store: at five attempts / five hours, the staged copy survives; at seven hours it is removed; the independent source remains. Zero disables the extension and a configured eight-hour budget preserves the copy at seven hours. Volume-manager suite: 210 passed.
• Real AWS SDK requests to a loopback HTTP fixture: 503/429 preserve typed outage evidence; 403/400 do not. Existing timeout/stalled-stream tests now assert the typed dependency variant without changing their timing/retry assertions. Object-store suite: 76 passed.
• Repository full workspace clippy plus feature-isolation checks: passed. Repository fmt/generated-contract/error-policy stage: passed. cargo arch: no required findings (37 advisory).
• At head 7bd6ed9846cba5053f1189db35719dc1a222d0c4, a combined rerun of common-jobs, object-store, sandbox-manager, and volume-manager passed 58 + 76 + 1,576 + 210 tests (one pre-existing ignored sandbox-manager test). Exact-head CI: all checks passed, including full Rust tests, clippy, fmt, gateway traffic, and the aggregate gate. This remains draft as requested.
• These are synthetic local tests, not a live STS/credential-authority-to-VM end-to-end exercise. Retry failed generation-suspend backups behind an admission gate #349 records the red→green failed-transfer observer regression. No production or live cluster requests were made.

Rollout and integration

• Queue time counts toward the six hours; the budget starts at durable job creation, not the first error. Malformed historical creation timestamps fall back to the count budget rather than retrying forever.
• FailedPrecondition and Internal from credential exchange are not sufficient to distinguish every permanent authority configuration problem from an outage. They receive the bounded dependency budget; explicit auth/input rejections do not. This is intentionally not message-string matching.
• Synchronous waiter timeouts are unchanged and can expire before the durable retry deadline; timing out a waiter does not cancel the job.
• Retry failed generation-suspend backups behind an admission gate #349's automatic re-drive stays default off, pending integration of [backup] refuse teardown-suspend backups from retired executions and release their runtimes #331's remote generation admission fence. This stacked change does not supply that fence.
• Preserve Approve bounded publication of validated retained suspends #283/Add schema for fenced backup ownership and retained recovery #318/Bind backup RPCs to verified historical worker identity #345's verified registration/identity policy and Carry verified upload evidence through backup finalization #346's attempt-fenced completion and transfer-proof production when reconciling overlapping executor paths. This change does not authorize retained-artifact recovery or synthesize successful proof.

AI assistance: Perplexity contributed the implementation, tests, and this description.

Created with Perplexity