[ac-pplx]

Additive, read-only telemetry. A new stuck-sandbox gauge. Unlike the nine moves below it in the stack, this adds behaviour (a leader-only loop and new metrics); it writes nothing. Part of the wave-2 stack (10 of 10, on #1328).

Summary

SPACE had no gauge of how many sandboxes are stuck in an in-progress lifecycle, or for how long. After this change, the cluster controller's elected leader counts them from the sandboxes rows once a minute and emits:

• sandbox.transient_lifecycle.rows, tags lifecycle and age (lt_1m, 1m_10m, 10m_1h, 1h_6h, 6h_1d, ge_1d)
• sandbox.transient_lifecycle.oldest_age_seconds, tag lifecycle (0 when there are no rows)
• sandbox.transient_lifecycle.loop_latency_ms, the time each tick takes

In Prometheus: space_sandbox_transient_lifecycle_{rows,oldest_age_seconds,loop_latency_milliseconds}.

• Age: NOW() - lifecycle_changed_at on the database clock. The lifecycle trigger sets that column only when the lifecycle changes, so a restoring -> restoring retry or an updated_at bump doesn't reset it.
• What the age cannot see: any lifecycle change restarts it, so a restore that keeps rolling back to suspended and retrying looks young on every attempt. A wake refused before it changes the row, and a row stuck in a settled lifecycle such as started, are not counted.
• Lifecycles counted: SandboxLifecycle::all() filtered by is_transient(), including deleting. Non-canonical lifecycle strings are not counted.
• Zeros: every (lifecycle, age) pair is emitted on every tick, zeros included.
• Failed read: logs one warn and emits neither gauge for that tick (loop_latency_ms still times it). It never reports zeros it didn't read.
• Replicas: combine replicas with max, not sum: after a leader change the previous leader's last values can still be queried.
• Tags: never a sandbox, node or org.

Pieces:

• New pplx-space-sandbox-store module stuck. transient_lifecycle_ages is one executor-generic statement with a sql! locator. It binds the lifecycle list and bucket bounds and reads an index on lifecycle, not the table. Typed result, small thiserror error.
• New metrics module pplx_space_metrics::sandbox::transient_lifecycle.
• A loop spawned from run_reconcilers on a child token. It is lease-gated when leader election is on (with it off, the controller assumes a single replica), runs on both substrates, and doesn't depend on SPACE_TRANSIENT_LIFECYCLE_SWEEPER_ENABLED. It runs every 60 s; there is no setting.
• Documented in docs/observability.md.

No migration, no write, no setting. sandbox_writes and the archcheck baseline are unchanged.

Validation

At this head:

• cargo fmt --all -- --check, scripts/check-rust-error-policy.sh, cargo clippy --workspace --locked --all-targets --all-features -- -D warnings: clean.
• cargo arch: 0 required findings. The 110 advisory findings all predate this PR and none touches these files.
• SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --all-features -p pplx-space-sandbox-store -p pplx-space-metrics -p pplx-space-cluster-controller-server: 1263 passed, 0 skipped.
• Store Postgres tests: rows land in the right bucket; same-lifecycle writes keep their age; a lifecycle change resets it; settled, deleted and non-canonical rows are excluded; unordered bounds are refused.
• EXPLAIN test on 20k settled rows with distinct lifecycle_changed_at: the plan is Bitmap Index Scan on idx_sandboxes_lifecycle with no Seq Scan. The test accepts either index on lifecycle. Mutation: lower(lifecycle) = ANY($1) gives a Seq Scan and fails the test.
• Mutation: using updated_at instead of lifecycle_changed_at fails same_lifecycle_writes_keep_the_age and rows_land_in_their_age_bucket.
• Metrics unit tests: every pair is emitted for an empty and a partial census. Mutation: skipping zero values fails 2 tests.
• Controller tests: a tick against Postgres emits all 54 + 9 gauge series; a failed read emits neither gauge.

Kind, fresh cluster, SPACE_KIND_TELEMETRY_MODE=otel (bring-up 602 s, 2 controller replicas):

1. After bring-up: 54 rows and 9 oldest_age_seconds series, all 0, all from the lease holder. The standby emitted none.
2. A row forced into restoring by SQL showed in restoring/lt_1m (oldest 29.8 s), then 1m_10m (89.8 s → 209.8 s). It dropped to 0 one tick after the row was restored. The sandbox was then deleted.
3. After the leader pod was deleted, the new leader sent its first sample about 2.6 s after the old leader's last one. No two pods ever sent samples at the same time.
4. SKIP_BOOTSTRAP=1 scripts/tests/smoke-kind-typed.sh all, 3831 s: ECW 89 passed, 0 failed, 2 inconclusive; native 35 passed. The inconclusive ecw.cold_create_environment and ecw.warm_pool_environment need CI's admin-token file and cold-receipts directory; both passed on rerun with those supplied. The census logged no failed reads.

The Kind runs above used a build whose controller and store code matches this head; the later changes touch only docs, comments and one test.

Not validated: the Direct substrate lane and the Datadog/DogStatsD path.

Severity and risk

Severity: does not apply (new visibility only).
Risk: one indexed aggregate a minute on the leader, plus 63 new gauge series and one latency histogram. Behaviour is unchanged. Roll back by reverting the commit.

Rollout

Old and new controllers can run side by side safely. Only the lease holder runs the census, so during a rollout the series come from whichever version holds the lease. Older versions emit nothing.

AI assistance: an AI agent (Claude Code) wrote the code, the tests and this description, and ran the validation.

After the wave-2 restack

This PR is one layer of a 10-PR stack, restacked onto main c5fdf0c7 (which includes #826) and then rebased mechanically onto 42a99002, which changed only baseline context lines, in order #1329 → #1336 → #1338 → #1339 → #1342 → #1347 → #1333 → #1335 → #1328 → #1332. The validation earlier in this description ran at pre-restack heads; this layer's head is 77fbae1e.

• [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826: Main's [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 and the commits since the previous restack touched nothing this layer moves, so the commit is unchanged apart from its new base (git range-diff shows =). For every file [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 and this stack share, each SQL line [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 added was checked to be present verbatim at the top head, in place or in the store module it moved to.
• At this head: cargo fmt --all -- --check, cargo check --workspace --all-targets --all-features and cargo arch (0 required findings, with main's layers and table-owner rules), and the ratchet and caps this layer changes are recounted exactly.
• At the top of the stack (e90aa284 before that rebase; nextest passed 4370 tests again at 77fbae1e): cargo clippy --workspace --locked --all-targets --all-features -- -D warnings is clean, and SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --all-features passed 4355 tests across the store, api, internal-api server, cluster-controller server, metrics, db-test-fixtures, common-core, common-svc and portability crates, [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's suites included.
• Kind at the top of the stack, fresh clusters with the otel telemetry mode and CI's cold-receipt directory: scripts/tests/smoke-kind-typed.sh all gave ECW 91/91 and native 42/42, including [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826's native.forced_drain_re_places_stopped_sandbox and its re-drain and restarted variants. The Direct substrate lane's all gave ECW 51/51 and native 27/27. No flakes and no reruns.
• Equivalence: apart from the [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826 port named above, the PR's own +/- lines and its SQL string literals match its reviewed head (range-diff plus a literal multiset compare).

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.