Skip to content

Pin runsc fix for checkpoint restore compaction stalls - #1969

Merged
luke-lombardi merged 1 commit into
mainfrom
codex/restore-latency
Oct 7, 2026
Merged

luke-lombardi merged 1 commit into
mainfrom
codex/restore-latency

Conversation

@luke-lombardi

@luke-lombardi luke-lombardi commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

VoxCPM2 checkpoints intermittently spent tens of seconds inside runsc restore even when the image and checkpoint were already local. Pin the exact node-tested runsc hotfix from beam-cloud/gvisor#3: honor the existing host huge-page policy to avoid synchronous compaction, and reduce shared-memory contention between checkpoint readers.

The artifact preserves the deployed checkpoint compatibility version (ebbaed3f9dd5). Serialized state is unchanged; strict version checks and the worker compatibility key remain stable. Its new source revision, release tag and SHA512 identify the hotfix separately.

Validation used this exact binary and the real beta9 runtime on reserved RTX 4090 GPUs. An unchanged VoxCPM2 checkpoint captured on .22 restored on .19 in 13.56–14.34 s and .22 in 11.64–13.24 s, retaining its nonce and identical audio. The deployed runtime reproduced a 77.36 s managed restore; a paired private .22 control took 19.31 s. Registered host memory and GPU state also passed, with restore improving from 12.05 to 7.23 s. The published asset was downloaded through the Dockerfile URL and verified byte-identical to the tested binary.

Some warm deployed restores were faster; these tests establish removal of a compaction path, not a guaranteed seven-second restore or production p99. No scheduler-free 5090 GPU was available for this artifact's GPU test. The runtime dependency is approved and merged; the artifact remains a prerelease and production has not been changed.

Canonical build · Pinned artifact

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

View guided diff | Turn on auto-fix | Re-trigger cubic

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pin bump to release-20260714.0-beam.10 looks right: the release targets f5056750 (head of beam-cloud/gvisor#3), and the SHA512 in the Dockerfile matches the published runsc.x86_64 asset byte for byte. Non-blocking: that release is still a prerelease built from the unmerged gvisor#3 branch, so worth merging gvisor#3 (or noting it) before this ships, so the pinned commit stays reachable from main.

@luke-lombardi

Copy link
Copy Markdown
Contributor Author

The approved runtime PR is now merged with merge commit 0eb6353ae8705f7463f641d57f274901f6280ddf. The exact tested source f5056750a788c74f5aabcb328dd4973a56b56f42 remains an ancestor of master and the release target. The artifact and pin are unchanged; production has not been deployed.

@luke-lombardi
luke-lombardi merged commit cd1ac77 into main Oct 7, 2026
5 checks passed
@luke-lombardi
luke-lombardi deleted the codex/restore-latency branch October 7, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants