[sbx-mgr] replace a backed-up stale suspended copy before a runtime upgrade stages its checkpoint

crates/l20_domain/events/README.md

@@ -69,7 +69,7 @@ context. Keep event names domain-level: prefer `pause_requested` or
 | `artifact_backup_queued`, `artifact_backup_completed`, `artifact_backup_failed` | Back up an immutable node-local artifact to durable storage. | Carries the exact `local_artifact_id` and `backup_attempt_id` that gate cross-node restore and node release. |
 | `local_snapshot_promotion_abandoned` | Node-drain gave up promoting a suspended sandbox's local snapshot (the on-node reupload was permanently refused because the sandbox was wedged in `restoring`) and adopted the current suspend cycle's ready remote snapshot so the node could finish draining. | Records the drain-progress decision: the sandbox stays `suspended` and recoverable from the adopted snapshot, abandoning the un-uploadable local state. Carries `source_node_id`. |
 | `sandbox_lost` | A reconciler determined no recoverable copy of a suspended sandbox's state exists anywhere — the local suspended artifact is gone (pruned, or the node's backing instance confirmed terminated) and no ready remote snapshot exists — and flipped the row to `failed`. | Explains a terminal `suspended → failed` flip on the sandbox timeline: the data was already lost before the flip, and the user's next wake/restore gets a clean lifecycle error instead of a snapshot-not-found. Carries `source_node_id`, from/to state, and a `reason` naming which give-up fired (`drain_sandbox_lost` on a draining node, `gone_node_unrecoverable` on a drained node whose EC2 instance is confirmed terminated). |
-| `local_eviction_completed`, `local_eviction_failed` | The node removed (or failed to remove) a sandbox's entire local presence — cache directory including the rootfs instance, plus node-local DB rows. The cold-sandbox eviction loop does it for a suspended sandbox whose state is durably restorable from a ready remote snapshot; node-ctl's facet sync does it for a sandbox the cluster reports deleted; the teardown driver does it for a teardown suspend whose backup the cluster refused because it retired the execution. A sandbox the cluster still has stays restorable from its own snapshots. | Carries `reason` (`latest_backed_idle` — every eviction requires proof tied to the exact bytes: the pinned artifact's confirmed backup, or exact generation equality for legacy unpinned rows; `cluster_row_absent` — the cluster has no row for the sandbox; `execution_retired` — the cluster refused the capture's backup for a retired execution, carrying the discarded `local_artifact_id` and `execution_generation` instead of a remote snapshot or `local_path`), the remote `snapshot_id` relied on where there is one, and the freed `local_path` (logical size only; CoW sharing makes physical reclaim lower); failures carry the error, and one after the local row was claimed leaves it in `deleting` for cleanup retry. |
+| `local_eviction_completed`, `local_eviction_failed` | The node removed (or failed to remove) a sandbox's entire local presence — cache directory including the rootfs instance, plus node-local DB rows. The cold-sandbox eviction loop does it for a suspended sandbox whose state is durably restorable from a ready remote snapshot; node-ctl's facet sync does it for a sandbox the cluster reports deleted; the teardown driver does it for a teardown suspend whose backup the cluster refused because it retired the execution; a runtime upgrade does it for a suspended copy of another checkpoint than the one it was decided from, before staging that checkpoint. A sandbox the cluster still has stays restorable from its own snapshots. | Carries `reason` (`latest_backed_idle` — every eviction requires proof tied to the exact bytes: the pinned artifact's confirmed backup, or exact generation equality for legacy unpinned rows; `cluster_row_absent` — the cluster has no row for the sandbox; `execution_retired` — the cluster refused the capture's backup for a retired execution, carrying the discarded `local_artifact_id` and `execution_generation` instead of a remote snapshot or `local_path`; `stale_upgrade_copy` — a runtime upgrade replaced a suspended copy whose pinned artifact was proven backed for its exact id, carrying that `local_artifact_id` instead of a remote snapshot or `local_path`), the remote `snapshot_id` relied on where there is one, and the freed `local_path` (logical size only; CoW sharing makes physical reclaim lower); failures carry the error, and one after the local row was claimed leaves it in `deleting` for cleanup retry. |
 | `rootfs_stream_reclaimed` | The local rootfs diff stream (`rootfs.btrfs-send`) of a suspended sandbox's current artifact was deleted after its backup reached the durable ready state. Same-node wake is preserved: the snapshot's `memory`/`vmstate` files and the `instance/` rootfs subvolume stay; only the upload-format stream, which no restore path reads locally, is removed. | Carries the exact `local_artifact_id`/`backup_attempt_id` the reclaim was proven against and the reclaimed `local_path`, so a later "where did the stream go" investigation lands on the durable backup that superseded it. |
 | `snapshot_pruned` | A retention sweep deleted an old durable snapshot of a still-live sandbox because it fell outside the newest-N `ready` full snapshots the cluster keeps per sandbox — its S3 prefix and DB rows were removed. The sandbox stays alive and restorable from a newer snapshot. | Carries the pruned `snapshot_id` and its `age_secs`; `source = cluster_controller`, `trigger = system`. Emitted by the `snapshot_retention_sweeper` reconciler. |
 | `scheduled_snapshots_blocked`, `scheduled_snapshots_unblocked` | A node's snapshot scheduler stopped, or resumed, capturing a sandbox's scheduled snapshots and full checkpoints because the unowned GC staging under its `snapshots/` reached, or fell back below, the scheduler's limit. | Explains a gap in a live sandbox's scheduled snapshots: carries `reason = gc_staging_backlog` and, on `blocked`, an `error` with the staging count and limit; `source = sbx_manager`, `trigger = system`. Emitted on the gate's edges, and again after a sandbox-manager restart or re-registration finds it still closed. The edge state is in memory, so a gate that reopens across a restart or re-registration has no `unblocked`. The `snapshot.scheduler.capture_skipped{reason:staging_backlog}` counter counts each blocked tick and the `snapshot.scheduler.unowned_staging` gauge sums the node's sandboxes; neither has a sandbox tag. |

crates/l30_storage/sbx/mgr/store/src/auto_lifecycle.rs

@@ -136,9 +136,36 @@ pub struct AutoLifecycleCoordinator {
     /// `suspended` row, so the test can move the row in between.
     #[cfg(any(test, feature = "test-support"))]
     pub suspended_stop_barrier: Arc<std::sync::Mutex<Option<Arc<tokio::sync::Barrier>>>>,
+    /// A test's two-phase rendezvous in backup admission, with the
+    /// sandbox's artifact lock held, just before the upload job is persisted.
+    #[cfg(any(test, feature = "test-support"))]
+    pub backup_admission_barrier: Arc<std::sync::Mutex<Option<Arc<tokio::sync::Barrier>>>>,
+    /// A test's two-phase rendezvous in a suspended row's eviction, with the
+    /// sandbox's artifact lock held, just before the claim.
+    #[cfg(any(test, feature = "test-support"))]
+    pub eviction_claim_barrier: Arc<std::sync::Mutex<Option<Arc<tokio::sync::Barrier>>>>,
+    /// Notified when a backup's admission is about to wait for the
+    /// sandbox's artifact lock, so a test knows the admission reached it.
+    #[cfg(any(test, feature = "test-support"))]
+    pub artifact_lock_wait_hook: Arc<std::sync::Mutex<Option<Arc<tokio::sync::Notify>>>>,
 }
 
 impl AutoLifecycleCoordinator {
+    /// A test's two-phase rendezvous at one of the barriers above: when
+    /// `slot` holds a barrier, wait on it twice, once to tell the test the
+    /// caller arrived and once for the test to let it go on.
+    #[cfg(any(test, feature = "test-support"))]
+    pub async fn rendezvous(slot: &std::sync::Mutex<Option<Arc<tokio::sync::Barrier>>>) {
+        let barrier = slot
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner)
+            .clone();
+        if let Some(barrier) = barrier {
+            barrier.wait().await;
+            barrier.wait().await;
+        }
+    }
+
     pub fn wake_lock(&self, sandbox_id: &str) -> Arc<LifecycleMutex> {
         lifecycle_lock(&self.wake_locks, sandbox_id)
     }

crates/l30_storage/sbx/mgr/store/src/db.rs

@@ -131,13 +131,15 @@ pub use pplx_space_common_core::types::QuarantineRecoveryNoProgressReason;
 pub mod read {
     mod cleanup_job;
     mod counts;
+    mod eviction;
     mod local_artifact;
     mod runtime_upgrade;
     mod sandbox;
     mod sandbox_view;
     mod state_page;
     pub use cleanup_job::*;
     pub use counts::*;
+    pub use eviction::*;
     pub use local_artifact::*;
     pub use runtime_upgrade::*;
     pub use sandbox::*;