Skip to content

Avoid compaction stalls during checkpoint restore - #3

Merged
luke-lombardi merged 3 commits into
masterfrom
codex/cuda-restore-thp-policy
Oct 7, 2026
Merged

luke-lombardi merged 3 commits into
masterfrom
codex/cuda-restore-thp-policy

Conversation

@luke-lombardi

Copy link
Copy Markdown

Restoring CUDA checkpoints advised huge pages even when the destination host policy had disabled them. On the affected host this caused direct compaction while loading and pinning restored memory. Honor the existing immutable ExpectHugepages policy at both advice sites, retaining eager page loading, writable prefaulting and CUDA pinning.

Use eight 4 MiB page reads instead of 128 256 KiB reads, keeping 32 MiB in flight. Writer defaults, read limits, short-read/error handling and serialized checkpoint fields remain unchanged.

The optional artifact-build input preserves ebbaed3f9dd5 for this state-compatible hotfix. Both strict checkpoint-version checks remain enabled, and the worker's runtime-version fingerprint input is unchanged. Identify the artifact by its new source revision, immutable release tag and SHA512; default builds retain their existing git-described version behavior.

Validation: the canonical Linux artifact restored the original, unmodified owned VoxCPM2 checkpoint with retained nonce and identical audio. Final runtime durations were 13.56–14.34s on the affected host and 11.64–13.24s on the control host; one corresponding deployed control took 19.31s with 12.28s in prefault versus 4.03–4.65s for the candidate. Registered host memory, GPU allocations, module and stream state also passed. Profiling demonstrated direct compaction in deployed restores and none in the policy-guard cases.

These private runtime measurements exclude scheduling and checkpoint transfer. Some warmed deployed restores were faster, and the reader change's independent full-model benefit is not established; this addresses compaction stalls rather than guaranteeing every restore is faster. No production runtime was replaced during validation.

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right: the ExpectHugepages guard in adviseHugepageRanges covers both callers (AdviseHugepageRanges and PrefaultRanges), the reader default change only affects restore reads (writer and NewPagesFileFDReader callers untouched), and the checkpoint_version input is passed via env and regex-checked before reaching the shell, so no injection. Non-blocking: guarding on ExpectHugepages rather than AdviseHugepage means a host that expects THP but has advice disabled will still madvise here; fine if that's intended, maybe worth a comment.

@luke-lombardi

Copy link
Copy Markdown
Author

Using ExpectHugepages is intentional. It is the loader's decision whether huge pages are permitted; AdviseHugepage controls advice for ordinary allocations and remains false for shmem_enabled=always, within_size and force. This guard preserves the previous CUDA preparation on those permitted policies and changes only hosts where the loader selected small pages. I kept the patch limited to that policy correction.

@luke-lombardi
luke-lombardi merged commit 0eb6353 into master Oct 7, 2026
1 of 2 checks passed
luke-lombardi added a commit to beam-cloud/beta9 that referenced this pull request Oct 7, 2026
VoxCPM2 checkpoints intermittently spent tens of seconds inside runsc
restore even when the image and checkpoint were already local. Pin the
exact node-tested runsc hotfix from
[beam-cloud/gvisor#3](beam-cloud/gvisor#3):
honor the existing host huge-page policy to avoid synchronous
compaction, and reduce shared-memory contention between checkpoint
readers.

The artifact preserves the deployed checkpoint compatibility version
(`ebbaed3f9dd5`). Serialized state is unchanged; strict version checks
and the worker compatibility key remain stable. Its new source revision,
release tag and SHA512 identify the hotfix separately.

Validation used this exact binary and the real beta9 runtime on reserved
RTX 4090 GPUs. An unchanged VoxCPM2 checkpoint captured on .22 restored
on .19 in 13.56–14.34 s and .22 in 11.64–13.24 s, retaining its nonce
and identical audio. The deployed runtime reproduced a 77.36 s managed
restore; a paired private .22 control took 19.31 s. Registered host
memory and GPU state also passed, with restore improving from 12.05 to
7.23 s. The published asset was downloaded through the Dockerfile URL
and verified byte-identical to the tested binary.

Some warm deployed restores were faster; these tests establish removal
of a compaction path, not a guaranteed seven-second restore or
production p99. No scheduler-free 5090 GPU was available for this
artifact's GPU test. The runtime dependency is approved and merged; the
artifact remains a prerelease and production has not been changed.

[Canonical
build](https://github.com/beam-cloud/gvisor/actions/runs/37680251173) ·
[Pinned
artifact](https://github.com/beam-cloud/gvisor/releases/tag/release-20260714.0-beam.10)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants