Skip to content

fix(doctor): budget boot stages on guest work, not host steal - #269

Merged
ebursztein merged 3 commits into
mainfrom
fix/boot-budget-steal-time
Sep 28, 2026
Merged

ebursztein merged 3 commits into
mainfrom
fix/boot-budget-steal-time

Conversation

@ebursztein

Copy link
Copy Markdown
Collaborator

Release run 36351667646 failed test_boot_stages_within_budget on
profile_root_seed at 510ms; the stage copies sixteen small files and
measured 40ms on the previous run. On a shared nested-virt runner the
vCPU was descheduled for most of it.

capsem-init's boot_mark now records steal_ms per stage: the largest
per-vCPU delta of the /proc/stat steal column (USER_HZ ticks), so vCPUs
waiting in parallel cannot add up to more than one vCPU's wall time.
One awk reads /proc/uptime and /proc/stat per mark (busybox-compatible;
the kernel line goes through the same mark).

assess_boot_timing budgets duration_ms minus steal_ms, capped at the
duration. A missing field means 0, and anything that is not a finite
non-negative integer discounts nothing, so a hostile or corrupt line can
only make the budget stricter. Slow stages are reported raw, with their
steal. The field stays in the guest: BootStage and the IPC schema hash
are unchanged, and the agent ignores it.

Blocked the 0.6.4 stable release: run 36351667646, x86_64 KVM on a shared 4-vCPU nested runner with the ironbank suite running 4 VMs, failed only test_boot_stages_within_budget: profile_root_seed 510 ms > 500 ms. The same code measured it at 40 ms in run 36290484745; the stage copies 16 files (100K). The release process refuses an unchanged retry, so this is the fix forward.

  • Steal is exposed: the guest has CONFIG_PARAVIRT + CONFIG_KVM_GUEST; the host's KVM leaf 0x40000001 has the steal-time bit and setup_cpuid passes it through (new capsem-core test pins it).
  • Steal stays in the guest (doctor reads the file); no IPC schema change. The agent ignores the new field (tested with hostile values).
  • Largest per-vCPU steal delta, not the sum, so parallel waits can't hide a regression.
  • Not proven on a real VM here (asset rebuild needs disk this host didn't have; this GCP host also reports zero steal).

Tests: Python doctor/initrd suites 454 passed; citadel 1255; capsem-agent musl tests; clippy agent (musl+host) and capsem-core; capsem-proto and guest_report tests.

🤖 Generated with Claude Code

The parser's eleven tests were spread over three places in the agent's
1280-line tests.rs, which sat at its size ratchet. They move to
boot_timing/tests.rs and the ratchet drops to 1115.

Adds a test that a line carrying capsem-init's new steal_ms field, sane
or hostile, still yields its stage: the field is for the in-guest doctor
and BootStage does not carry it.
The guest only accounts steal time when KVM advertises it in CPUID
0x40000001. setup_cpuid passes KVM_GET_SUPPORTED_CPUID through and only
rewrites topology leaves; this pins that the paravirt leaves survive, so
the boot budget's steal subtraction keeps a source on x86_64.
Release run 36351667646 failed test_boot_stages_within_budget on
profile_root_seed at 510ms; the stage copies sixteen small files and
measured 40ms on the previous run. On a shared nested-virt runner the
vCPU was descheduled for most of it.

capsem-init's boot_mark now records steal_ms per stage: the largest
per-vCPU delta of the /proc/stat steal column (USER_HZ ticks), so vCPUs
waiting in parallel cannot add up to more than one vCPU's wall time.
One awk reads /proc/uptime and /proc/stat per mark (busybox-compatible;
the kernel line goes through the same mark).

assess_boot_timing budgets duration_ms minus steal_ms, capped at the
duration. A missing field means 0, and anything that is not a finite
non-negative integer discounts nothing, so a hostile or corrupt line can
only make the budget stricter. Slow stages are reported raw, with their
steal. The field stays in the guest: BootStage and the IPC schema hash
are unchanged, and the agent ignores it.
@codecov-commenter

codecov-commenter commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.3%. Comparing base (9ed6941) to head (3e3d72c).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##             main     #269     +/-   ##
=========================================
- Coverage    65.4%    65.3%   -0.2%     
=========================================
  Files        1450     1452      +2     
  Lines      127617   127936    +319     
  Branches    91731    91872    +141     
=========================================
+ Hits        83554    83604     +50     
- Misses      39120    39336    +216     
- Partials     4943     4996     +53     
Flag Coverage Δ
integration 17.2% <ø> (ø)
linux-unit 70.5% <ø> (-0.1%) ⬇️
mcp-server 93.8% <ø> (ø)
python-sdk 98.7% <ø> (ø)
typescript-sdk 97.8% <ø> (ø)
unit 63.5% <ø> (-0.2%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
TypeScript SDK 97.8% <ø> (ø)
Python SDK 98.7% <ø> (ø)
Network 84.4% <ø> (ø)
Security 81.6% <ø> (ø)
Tooling 88.6% <ø> (ø)
Monitoring 87.5% <ø> (-0.1%) ⬇️
Virtualization 64.4% <ø> (+<0.1%) ⬆️
Confined Port Router 76.7% <ø> (-0.2%) ⬇️
Private Network 78.9% <ø> (-0.4%) ⬇️
Assets 80.8% <ø> (ø)
Gateway API 96.7% <ø> (ø)
Rust SDK 96.3% <ø> (ø)
Configuration 86.8% <ø> (ø)
Credentials 80.7% <ø> (ø)
Host Foundation 76.3% <ø> (ø)
Core Platform 55.8% <ø> (ø)
Runtime 60.7% <ø> (+<0.1%) ⬆️
Daemon 41.6% <ø> (ø)
Service 71.5% <ø> (ø)
Process 48.5% <ø> (ø)
Admin 63.7% <ø> (ø)
CLI 47.9% <ø> (ø)
MCP Server 93.8% <ø> (ø)
MCP Aggregator 61.8% <ø> (ø)
MCP Builtin 57.4% <ø> (ø)
Gateway 78.9% <ø> (ø)
TUI 68.5% <ø> (ø)
System Tray 53.2% <ø> (ø)
Guard 92.2% <ø> (ø)
UI 86.6% <ø> (ø)
Release Site 15.0% <ø> (∅)
Builder 43.5% <ø> (ø)
Mock Server 59.0% <ø> (ø)
Bench 47.8% <ø> (+<0.1%) ⬆️
Files with missing lines Coverage Δ
crates/capsem-agent/src/boot_timing.rs 100.0% <ø> (ø)

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ebursztein
ebursztein merged commit f7da631 into main Sep 28, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants