feat(scripts): local A/B boot benchmark harness - #253
Open
gmaclennan wants to merge 1 commit into
Open
Conversation
gmaclennan
force-pushed
the
feat/boot-benchmark-harness
branch
from
August 22, 2026 14:26
5460adf to
2a8b8e7
Compare
`scripts/benchmark-boot.sh --ab before.apk after.apk` runs interleaved cold boots of `apps/integration`, alternating which build goes first each round, discarding the warm-up boot after each install and wiping the compile cache before every boot so neither build inherits the other's. It prints medians, deltas, Mann–Whitney p-values and per-round medians — the last being the check that matters on a machine that is also doing other work. Peak RSS, RSS and the heap breakdown come from the `[comapeo.memory] boot` line the backend already logs once per launch, so no root is needed and the harness works unmodified on a production device; `adb root` only adds a 50 ms `/proc` trajectory on top. Parsing and statistics live in a pure `scripts/lib/benchmark-core.mjs` covered by `npm run test:scripts`; the shell wrapper is only device orchestration. Usage, the flags, how to read the output and the ways a run can mislead you are in docs/BENCHMARKING.md.
gmaclennan
force-pushed
the
feat/boot-benchmark-harness
branch
from
August 24, 2026 13:50
2a8b8e7 to
c87f3d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #249, which adds
backend/lib/memory-snapshot.jsand the[comapeo.memory] bootlog line this harness reads. Review that one first; the base branch here isfeat/backend-memory-telemetry.Answering "is build B lighter than build A?" has so far meant a throwaway script each time. This makes it one command:
It runs interleaved cold boots of
apps/integration, alternating which build goes first each round, discarding the warm-up boot after each install and wiping the compile cache before every boot so neither build inherits the other's. It prints medians (not means — one scheduling stall drags a mean around), deltas, two-sided Mann–Whitney p-values, and per-round medians.The per-round table is the part worth reading first. A development machine is doing other things and host load drifts over the twenty-odd minutes a full run takes, so the question is not whether the two medians differ but whether they separate the same way in every round. If the rounds disagree, you measured the machine.
Peak RSS, RSS and the heap breakdown come from the backend logging its own reading once per launch, so no root is needed and the harness works unmodified on a production device;
adb rootonly adds an optional 50 ms/proctrajectory on top. With no--apk/--abit buildsapps/integrationin Release first — benchmark against Release, never debug, wherestartForegroundalone is about ten times slower.Parsing and statistics live in a pure
scripts/lib/benchmark-core.mjswith no device or filesystem dependency, covered bynpm run test:scripts; the shell wrapper is only device orchestration. Output lands inbenchmark-results/(gitignored): one JSON per label plus the raw per-boot logcat captures.docs/BENCHMARKING.mdgains the usage sections — the flags, how to read the output, and the ways a run can mislead you (steady-state RSS is bimodal,VmSizeis not a footprint).Validation
Used to re-run the V8 pointer-compression A/B end to end — 24 boots, one command — reproducing an earlier throwaway-script measurement independently: peak RSS −12.4 MB, live V8 heap −37.9%, per-round medians non-overlapping.
npm run lint,npm test,npm run backend:test(113) andnpm run test:scripts(40) all pass.Depends on #248 for the default build path (
--apk/--abwork without it).