Skip to content

metrics: split worker lifecycle populations by runtime version - #1581

Merged
nikhilbirmiwal-pplx merged 2 commits into
mainfrom
feat/sandbox-runtime-lifecycle-metric
Nov 5, 2026
Merged

nikhilbirmiwal-pplx merged 2 commits into
mainfrom
feat/sandbox-runtime-lifecycle-metric

Conversation

@nikhilbirmiwal-pplx

Copy link
Copy Markdown
Member

Summary

Extend the existing sandbox.worker.lifecycle.count gauge with runtime and runtime_version so worker-local lifecycle populations can be broken down by their recorded execution runtime. This does not add a competing population metric.

The SQLite census groups durable rows using their persisted runtime identity, normalizes runtime aliases, and preserves rows with absent or malformed metadata as unknown rather than assuming a version. The sampler tracks the complete state/runtime/version bucket when emitting a final zero for a disappearing population. No sandbox, template, or execution identifier becomes a metric dimension.

Validation

  • Affected-package cargo check --all-targets and cargo clippy --all-targets -- -D warnings passed for pplx-space-sbx-mgr-store, pplx-space-sbx-mgr, and pplx-space-metrics.
  • All 673 storage library tests and 11 metrics library tests passed. Storage coverage exercises mixed runtime versions, aliases, suspended rows, malformed and missing metadata, count preservation, and removal of a runtime bucket.
  • The sandbox-manager library suite reported 1700 passed, 2 failed, and 1 ignored. Both failures also reproduce on the unchanged baseline 3458cc6ac7eebcddab6e68a5dc47571eb44d3db6: a_restart_suspended_teardown_capture_never_registers_unfenced and lapsed_supersession_registers_the_capture_again. A baseline run of the retired-teardown group reported 53 passed and those same 2 failures. The full suite is not green.
  • Formatting, whitespace checks, and the Gitleaks pre-commit scan passed.
  • Live metric ingestion and deployment have not been validated.

Severity and risk

Severity is low: the existing total remains available, but it cannot currently distinguish runtime versions. This changes observability, not sandbox placement or execution policy.

Risk is increased series cardinality and incorrect grouping if persisted runtime metadata is not decoded correctly. Legacy rows remain explicitly unknown, and summing runtime buckets preserves the per-state population. The census represents locally recorded lifecycle state, not an independent VM process-liveness check.

Rollout

Metric backends with explicit tag-retention allowlists must retain runtime and runtime_version alongside state. During a mixed-version worker rollout, old emitters lack these dimensions, so complete version coverage requires updating the intended worker cohort. Reverting the emitter restores the previous state-only metric.

AI assistance: Perplexity Computer contributed the code, tests, documentation, validation, and this description.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Explain Change →

Automatic approval withheld

Assessment Current risk Maximum for approval
Merge risk Moderate Low

Why: The lifecycle metric now depends on a changed SQLite population query. Its metadata format matches the writer, but the query change warrants validation before merge under the repository’s risk guidance.

Commit: 74a750f139e4.

No actionable comments were generated in the recent review.

Walkthrough

Sandbox lifecycle counts now include optional runtime kind and runtime version. The storage query derives runtime identity from execution metadata, falls back to the stored runtime kind, and retains rows with unavailable metadata. The metrics pipeline emits runtime and version dimensions, using "unknown" for missing values. Tests cover grouping, metadata cases, and deletion. The observability documentation describes the metric and its sampling behavior.

Possibly related PRs

  • perplexityai/SPACE#1555: Changes the lifecycle state column used by the count query that this change extends with runtime and version grouping.
Merge Risk: Moderate · up to 74a75

The lifecycle metric now depends on a changed SQLite population query. Its metadata format matches the writer, but the query change warrants validation before merge under the repository’s risk guidance.

Architecture risk: Low · up to 74a75

The change affects 2 systems.

Changed systems: crates, docs

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — crates (service) was modified; 7 changed files map to changed impact.
  • observed — docs (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in crates/l30_storage/sbx/mgr/store/src/db.rs: The db module’s common-core type imports now include RuntimeIdentity and RuntimeVersion.
  • observed — Modified behavior in crates/l40_api/sbx/mgr/server/src/lifecycle_metrics.rs: Imports RuntimeKind and RuntimeVersion for lifecycle bucket metadata.
  • observed — Modified behavior in crates/l40_api/sbx/mgr/server/src/lifecycle_metrics.rs: LifecycleBucket adds optional runtime kind and runtime version fields, populated from each SandboxLifecycleCount; buckets therefore distinguish lifecycle counts by state and runtime metadata.
  • observed — Modified behavior in crates/l40_api/sbx/mgr/server/src/lifecycle_metrics.rs: Lifecycle metric emission now passes the bucket’s optional runtime kind and runtime version, in addition to count and state.
Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title describes the lifecycle metric change and meets the specified format.
Description Follows The Pr Template Passed The description includes a populated Summary, Validation, and Severity and risk section. It also includes an AI assistance line that names the tool and its contributions.
Bug Fixes Name A Regression Test Passed This PR adds runtime dimensions to an existing metric. The base census grouped only by state. The metadata decoding fix corrects code introduced within this PR, so the PR is not an application bug fix…
Weakened Safeguards Are Justified Passed The diff does not remove or weaken a test, assertion, lint, CI check, or repository guard. Existing test assertions retain their expected counts and include the new fields.
Public-Safe Content Passed The reviewed diff, title, and description contain no content that matches the specified failure conditions. The added query is an example, not production output.
Full details: Bug Fixes Name A Regression Test

Explanation

This PR adds runtime dimensions to an existing metric. The base census grouped only by state. The metadata decoding fix corrects code introduced within this PR, so the PR is not an application bug fix against the base.

Recent review info

Automatic approval policy

Commit: 74a750f139e4.

Setting Effective value Source
Automatic approvals Enabled Global configuration
Merge risk Low Override 2
Security risk Not set Global configuration

Matching overrides (numbered as in the resolved configuration):

  • Override 2 — any author; paths: **, !ui/**; merge risk: Low.

Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between f3e3be1 and 74a750f.

Files selected for processing: 3

  • crates/l30_storage/sbx/mgr/store/Cargo.toml
  • crates/l30_storage/sbx/mgr/store/src/db/sandboxes.rs
  • crates/l30_storage/sbx/mgr/store/src/db/tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

@nikhilbirmiwal-pplx
nikhilbirmiwal-pplx added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit c2ab2f9 Nov 5, 2026
29 checks passed
@nikhilbirmiwal-pplx
nikhilbirmiwal-pplx deleted the feat/sandbox-runtime-lifecycle-metric branch November 5, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants