[mabdelraouf-pplx]

Summary

Bound reconnect recovery when the current execution has published a local suspend but its full artifact has not become durable. Return a typed 503 + Retry-After instead of claiming/joining recovery or polling the absent runtime to the readiness deadline.

Self-assessed severity: high — repeated reconnects can leave a session unavailable despite a potentially recoverable retained artifact. This change does not declare that artifact lost or discard it.

Behavior and source of truth

• Read the existing node_facet_observed_status.local_suspend observation together with the active execution and snapshot storage rows. Require the same source node, execution generation, and Pod UID; reject superseded activity. Match the observed artifact, not an older committed suspend marker.
• Treat creating, failed, and not-yet-registered uploads conservatively as pending. A Ready full snapshot with a non-orphaned storage location bypasses the guard.
• Check recovery admission, follower joins, selected-wake retries, and data-plane access before repairing placement. Do not add worker/API hops, enqueue uploads, change backup authorization, or alter teardown.
• Preserve 503 and Retry-After through ECW rather than converting publication waits into the SDK's 502 resume-and-retry timeout. Tag these outcomes wake_in_progress / restore / restore_source_pending.
• Document the client contract in docs/ecw.md.

Deliberately incomplete terminal-loss classification: failed upload jobs are node-local, and a failed/missing cluster snapshot alone does not prove retained bytes are gone. No authoritative terminal local-artifact-loss verdict is currently published to this API. This PR therefore adds 503-only handling for this state; the existing no-verified-restore-source 412 contract is unchanged. A future generation/Pod-UID/artifact-fenced terminal verdict is required before adding new 412 responses. Missing observations retain existing behavior.

Proof and validation

Baseline: 67e3ab347d68056edac89d2f550b9bcd85ec0282, with only the regression-test changes.

• The real Kubernetes wake-path regression, using PostgreSQL and the existing synthetic Kubernetes/worker harness, failed in all seven cases. Pending-upload cases crossed the test's two-second response bound with a 300-second requested readiness budget (Elapsed(())); failed/missing registration cases returned the generic Unavailable disposition rather than the typed publication wait.
• The ECW translation regression independently failed with actual 502, expected 503.
• After the implementation fix, the same seven wake regression cases passed, including recovery after durable publication. The same ECW translation test also passed (503 + retry hint; existing 412 and ordinary 502 cases remain intact).
• Tested head: 0005d59a10b46d298e2779d45030414f242e44fa.
• SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api -p pplx-space-db-test-fixtures --all-features --test-threads 8 --run-ignored all: 1,731 passed, 0 skipped, including the seven regression cases and the 14-case publication/identity/activity fencing matrix.
• ci/scripts/run-rust-checks.sh fmt: passed (generated environment contract, Rust error policy, workspace formatting).
• ci/scripts/run-rust-checks.sh clippy: passed (workspace/all-targets/all-features plus the runner's isolated-feature checks).
• cargo arch: passed, 0 required / 37 advisory findings.
• Gitleaks pre-commit hook passed; staged public-content review found no disclosures in this diff or PR description.
• GitHub CI is still running the complete repository suite. Local tests used CARGO_PROFILE_TEST_DEBUG=0 CARGO_PROFILE_DEV_DEBUG=0 CARGO_BUILD_JOBS=20; the full workspace test inventory is left to CI rather than claimed from the affected-package run.

This is synthetic path-level evidence, not a live Firecracker/Kind restore or production rollout. No live infrastructure was queried or changed.

Interactions

• Approve bounded publication of validated retained suspends #283: do not treat failed registration as irrecoverable or bypass verified retry/recovery authorization; retained artifacts may still be recovered under its policy.
• Add schema for fenced backup ownership and retained recovery #318 / Bind backup RPCs to verified historical worker identity #345 / Carry verified upload evidence through backup finalization #346: no schema, caller-identity, or upload-proof changes and no dependency on those drafts.
• [backup] refuse teardown-suspend backups from retired executions and release their runtimes #331: preserve its generation ownership and teardown fencing; this read-only API guard grants no authority to register or re-drive a backup.
• Automatic upload re-drive can ship independently. Re-drivable state remains 503; this PR does not enable retries or manufacture a terminal-loss signal.

AI assistance: Perplexity contributed implementation, tests, documentation, and this description. Kept draft as requested.

Created with Perplexity