Repository navigation
api: record requested size for activations and resumes - #1511
Closed
gilalon-pplx wants to merge 17 commits into
Closed
gilalon-pplx wants to merge 17 commits into
gilalon-pplx wants to merge 17 commits into
Conversation
|
|
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true |
Add the sandbox.api.requested.vcpu and sandbox.api.requested.memory_bytes distributions and their bounded tag vocabulary: kind (create, activate, resume), surface, admission, template, client_id and, on creates, defaulted. admission takes its accepted and rejected values from the existing AdmissionDecision the cluster admission limiter reports. memory_bytes is the guest memory in MiB converted to bytes. Nothing records them yet; later changes record creates, activations and resumes. The module documentation defines each tag and value.
Clarify that paused runtime recovery can require admission and be rejected. Remove the telemetry test module declaration in accordance with the repository test policy; the test file is removed in the following commit.
Remove metric-value and tag assertion tests to comply with the repository test-economy policy. The test module declaration was removed in the preceding commit.
Keep the metric units, tags, recording boundaries and runtime-recovery caveat while removing redundant detail. Documentation only; emitter behavior is unchanged.
A cluster activation admission refusal reaches ControllerError two ways: from the request-entry admission check, and from SubstrateError::AdmissionRejected when the substrate charges a restore claim later. Each built the 429 by hand. Both now use ControllerError::cluster_admission_rejected, so a later change can mark every admission refusal in one place. Behavior is unchanged. Only a rate-limited refusal carries the ClusterActivationAdmission retry cause; an empty-cluster refusal still omits it, so the async dispatcher keeps giving it a bounded retry. A new test pins the status, retry header and cause for all four cases.
Record sandbox.api.requested.vcpu and sandbox.api.requested.memory_bytes with kind:create for every cold and warm-pool create, using the size the sandbox will run with after template and warm-pool defaults are applied. A cold create records at its cluster admission decision, so a create the admission limiter refuses still reports the size it asked for, as admission:rejected. A warm-pool claim records admission:accepted when it is adopted. defaulted is false only when the request named both vCPU and memory; warm-pool claims are always defaulted. A fork, whose size comes from its source snapshot, and a create whose size the API did not resolve record nothing. To tell admission refusals from other 429s, ControllerError gains a ClusterAdmissionRejection kind, set by the shared admission refusal constructor. Retry causes are unchanged.
gilalon-pplx
force-pushed
the
gil/requested-size-4-activation
branch
from
November 5, 2026 17:37
84ccb33 to
c53084b
Compare
A cold create's resolved size reaches the requested-size distributions at its admission decision, before the insert converts each value to the sandbox row's i32 column. CreateSandboxRequest accepts any u32, so a request for 2147483648 vCPUs with valid memory was recorded as an accepted size even though stored_resource then refused it with a 400. The recorder now passes the create's size through the same stored_resource bound before emitting, so a size the create cannot store enters neither distribution. Every size the recorder reads is then the stored i32, so the positive-size check takes that type directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extend requested vCPU and memory recording to lifecycle activations and resumes, using the sandbox row’s effective size. Record successful actions and explicit admission rejections; exclude callers joining another activation, unrelated failures, and unresolved or invalid sizes ([incremental diff](gil/requested-size-3-create...gil/requested-size-4-activation)).
Document admission during resume recovery, eligible implicit wakes, and distribution counts representing eligible recorded calls rather than total incoming requests. Reuse the existing metric definitions and admission classification without adding telemetry-assertion tests ([incremental diff](gil/requested-size-3-create...gil/requested-size-4-activation)).
Validation
cargo test -p pplx-space-metrics -p pplx-space-api --all-features: passed locally.cargo fmt --all -- --check: passed locally.pre-commit run --all-files: passed.git diff --check: passed.Runtime metric ingestion has not been validated.
Severity and risk
Severity does not apply: this adds observability rather than fixing application behavior. The main risk is incorrect sample counting or classification; admission decisions and lifecycle behavior are intended to remain unchanged.
Rollback: revert this PR’s lifecycle recording changes. The parent metric definitions, admission classification, and create recording can remain.
Stack
Requested-size metrics stack; merge from bottom to top:
gil/requested-size-3-create.AI assistance: Perplexity Computer reviewed and updated the implementation, restacked the branch, and drafted this description.