Linux/C++ Performance Analyzer

A Linux/C++ CLI tool for running benchmark suites, capturing wall time, CPU time, peak RSS, and hardware/software performance counters, and comparing runs against configurable regression thresholds -- built for CI-driven performance regression detection, not as a full profiler replacement.

Quick start

Requires a Linux host (fork/exec, sched_setaffinity, wait4, and perf are all Linux-specific), CMake 3.24+, Ninja, and a C++20 compiler. Dependencies are managed via vcpkg (manifest mode, vcpkg.json).

git clone https://github.com/czhao-dev/linux-performance-analyzer.git
cd linux-performance-analyzer

git clone https://github.com/microsoft/vcpkg.git
./vcpkg/bootstrap-vcpkg.sh -disableMetrics

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build build
ctest --test-dir build --output-on-failure

A Docker dev container (Dockerfile.dev) is available if you're not on Linux natively.

Project structure

app/            CLI entry point (main.cpp)
src/, include/  Core library (cpp_perf_lab_core): config, model, execution,
                runner, statistics, comparison, reporting, metrics, system
benchmarks/     Google Benchmark microbenchmarks + the scalability harness
examples/       Portfolio demo workloads/configs, validation experiment
                workloads/configs/templates, and a committed demo-output
                snapshot
scripts/        Experiment drivers (bash) and analysis/plotting (Python)
tests/          Unit, integration, and golden-file tests (Catch2)
docs/           Architecture reference and full validation results

Architecture

cpp-perf-lab is built from scratch on the measurement path -- no external benchmarking framework drives process execution, timing, or comparison. External libraries are used only for plumbing: CLI11 (argument parsing), yaml-cpp (config), nlohmann-json (result serialization), OpenSSL (SHA-256 benchmark-definition digests), fmt/spdlog (formatting/logging), and Catch2/Google Benchmark for testing. The library itself (cpp_perf_lab_core) is a set of narrow, single-responsibility modules with dependencies flowing one direction only, linked by a thin CLI (app).

Two things fall out of that design directly: run is a measurement pipeline (config in, samples out, no comparison logic involved at all), and compare/check is an independent analysis pipeline that only ever reads already-measured ResultFile JSON -- it has no idea how the numbers were produced. This is why report can regenerate output from a stored comparison with zero dependency on the original processes ever having run, and why swapping in a new report format or a new metric collector doesn't require touching the other pipeline at all.

Pipelines

============================== run ==============================

      suite.yaml
           |
           v
+---------------------+      +---------------------+
|       config        |----->|       runner        |
+---------------------+      |      warm-up +      |
                             |    measured reps    |
                             +---------------------+
                                        |
                                        v
                       +----------------+---------------+
                       |                                |
            +---------------------+          +---------------------+
            |      execution      |          |       metrics       |
            |    fork/execvpe,    |          | wraps `perf stat`,  |
            |   wait4, affinity   |          |parses -x ';' output |
            +---------------------+          +---------------------+
                       |                                |
                       +----------------+---------------+
                                        |
                                        v
                             +---------------------+
                             |     statistics      |
                             |percentiles, stddev, |
                             | per-metric summary  |
                             +---------------------+
                                        |
                                        v
                             +---------------------+
                             |        model        |      +---------------+
                             |    ResultFile /     |----->|  result.json  |
                             |   BenchmarkResult   |      +---------------+
                             +---------------------+
                                        ^
                                        |
                             +---------------------+
                             |       system        |
                             |    /proc, uname,    |
                             |    cpu governor     |
                             +---------------------+


========================= compare / check =========================

+-------------------+
|   baseline.json   |---+    +-----------------------+
+-------------------+   |    |   comparison engine   |
                        |    | thresholds, severity, |      +-------------------+
                        +--->|     noise-CV cap,     |----->| ComparisonReport  |
                        |    |     bootstrap CI      |      +-------------------+
+-------------------+   |    +-----------------------+                |
|   current.json    |---+                                             v
+-------------------+                                 +---------------+---------------+
                                                      |               |               |
                                                +----------+    +----------+    +----------+
                                                | terminal |    | markdown |    |   json   |
                                                +----------+    +----------+    +----------+

check reads suite.comparison from its config into the comparison engine above; compare always uses the built-in default policy instead (see "Known limitations" in docs/architecture.md).

• config -- YAML suite parsing into SuiteConfig/BenchmarkConfig/ ComparisonConfig, plus a SHA-256 digest of each benchmark's command/environment/affinity, used to flag when a benchmark's definition changed between the baseline and current run.
• runner -- drives warm-up and measured repetitions, merges suite-level and per-benchmark overrides, and decides per repetition whether to go through execution directly or wrap it via metrics.
• execution -- low-level Linux process control: fork/execvpe (never a shell), process groups, SIGTERM-then-SIGKILL timeout escalation, wait4-based resource accounting, CPU affinity.
• metrics -- wraps a benchmark's command with perf stat, parses its machine-readable delimited (-x ';') output, and probes availability.
• system -- machine/OS metadata (/proc/cpuinfo, uname, CPU governor) attached to every result file for later environment-diff checks.
• statistics -- percentiles (linear interpolation, matching NumPy's default), N-1 sample standard deviation, and a fixed-seed bootstrap confidence interval for the median difference between two sample sets.
• model -- the ResultFile/BenchmarkResult/Sample/MetricSummary data model with full JSON serialization; the one format every other module produces or consumes.
• comparison -- the regression-detection core: combined relative+absolute threshold evaluation, severity classification (Informational/Warning/Regression/Critical), noise-CV capping, badness-direction normalization, environment-compatibility checks.
• reporting -- renders a ComparisonReport as a terminal table or a self-contained Markdown document (JSON is just the report's own serialization).
• app -- the five CLI subcommands and the documented exit-code contract.

Full module reference, extension points, and testing strategy are in docs/architecture.md.

Commands

Command	Purpose
run --config suite.yaml --output result.json	Execute a benchmark suite, write a result file
compare --baseline a.json --current b.json --markdown report.md	Compare two existing result files (fixed built-in threshold policy)
check --config suite.yaml --baseline a.json --output b.json --fail-on-regression	Run + compare in one step, using the suite's own comparison thresholds
report --input comparison.json --markdown report.md	Regenerate terminal/Markdown output from a stored comparison JSON
doctor	Report environment suitability and perf availability

Exit codes: 0 success, 1 invalid CLI/config, 2 benchmark execution failed, 3 required collector unavailable, 4 result file/schema error, 5 regression threshold exceeded, 6 environment incompatible. See docs/architecture.md for the full reference, including why compare and check behave differently with respect to custom thresholds.

Live demo

examples/workloads/vector_processing/ contains a baseline and a regressed version of the same benchmark, differing by exactly one line of code (diff -u baseline.cpp regressed.cpp also shows a header-comment-only hunk, omitted below for clarity):

 int main() {
     std::vector<double> values;
-    values.reserve(kSize);
     for (std::size_t i = 0; i < kSize; ++i) {
         values.push_back(static_cast<double>(i));
     }

Reproduce it:

cmake --build build --target vector_bench_baseline vector_bench_regressed

./build/app/cpp-perf-lab run --config examples/configs/baseline.yaml \
  --output artifacts/baseline.json
./build/app/cpp-perf-lab run --config examples/configs/regressed.yaml \
  --output artifacts/regressed.json
./build/app/cpp-perf-lab compare --baseline artifacts/baseline.json \
  --current artifacts/regressed.json --markdown artifacts/report.md

Real, captured output from this exact sequence (examples/demo-output/):

Benchmark           Wall time   Peak RSS   IPC   Result
-------------------------------------------------------
vector_processing   +71.1%      +66.4%     n/a   REGRESSION

1 regression, 0 passes, 0 execution failures

Peak RSS rose almost as much as wall time (+66.4%) -- not just a speed regression: repeatedly reallocating and copying the vector as it grows without reserve() also leaves more transient memory outstanding during the run. IPC shows n/a because hardware performance counters weren't available on the validation machine (a real, honestly-reported limitation of that specific host -- see below); the tool never fabricates a value it can't measure.

The full Markdown report is in examples/demo-output/comparison.md.

Key metrics and methodology

• Wall time, user/system CPU time, peak RSS: captured directly via wait4 resource accounting on the benchmark's own process (spec's "basic collector").
• Hardware/software performance counters: via perf stat's machine-readable delimited output (never perf's human-oriented format), restricted to a bounded, validated event set. Derived metrics (IPC, branch/cache miss rates) are computed once per benchmark from summary-level counter aggregates, not per-sample.
• Percentiles: linear interpolation (matching NumPy's default method='linear'). Standard deviation: N-1 (Bessel-corrected sample stddev). Both cross-checked directly against NumPy -- see below.
• Regression severity: combined relative + absolute thresholds (AND logic) per metric, with a configurable noise-CV ceiling that caps severity at Warning when a measurement's coefficient of variation is too high to trust a hard Regression verdict.
• Bootstrap confidence intervals (fixed-seed, reproducible) are reported alongside every regression for context, but are informational only -- they do not themselves gate pass/fail.

Benchmarking guide

Everything below is reproducible from a clean build; the actual numbers in the next section were measured once, on a dedicated GCP c2-standard-4 VM (4 vCPU, no CPU steal, provisioned solely for this validation pass and torn down afterward), because the noise-sensitive experiments need a quiet, dedicated core -- not a shared CI runner or a laptop with power-management jitter. The scripts detect and report a missing prerequisite (binary not built, hyperfine//usr/bin/time not installed) rather than failing silently.

1. Build with the example workloads and microbenchmarks enabled (both default ON, but shown explicitly):

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake \
  -DCPL_BUILD_EXAMPLE_WORKLOADS=ON -DCPL_BUILD_MICROBENCHMARKS=ON
cmake --build build

pip install -r scripts/analysis/requirements.txt   # matplotlib, numpy

2. Not noise-sensitive -- safe to run anywhere, including this build directory:

# spec 17.1 -- internal microbenchmarks (Google Benchmark)
./build/benchmarks/cpp_perf_lab_microbenchmarks

# spec 17.3 -- scalability harness: one fresh process per size point,
# since getrusage(RUSAGE_SELF, ...).ru_maxrss is a whole-process-lifetime
# high-water mark and would otherwise contaminate smaller cases
./scripts/experiments/run_scalability_experiment.sh
python3 scripts/analysis/scalability_report.py

3. Noise-sensitive -- run on dedicated hardware, all driven by the same cpp-perf-lab binary they're validating (each writes raw JSON under the gitignored artifacts/, then a matching analysis script aggregates/plots into docs/validation-results/):

# spec 17.2 -- process-launch overhead (direct / basic / perf_stat arms)
./scripts/experiments/run_process_overhead_experiment.sh
python3 scripts/analysis/process_overhead_report.py

# spec 17.4 -- repeatability: 30 reps x {cpu-bound, mem-bandwidth,
# multithreaded} x {pinned, unpinned} x {background load off/on}
# (takes on the order of hours; this is the long-pole experiment)
./scripts/experiments/run_repeatability_experiment.sh
python3 scripts/analysis/repeatability_report.py

# spec 17.5 -- regression detection: 7 named variants x 20 trials each,
# plus one-factor-at-a-time sweeps over repetition count / relative
# threshold / absolute threshold
./scripts/experiments/run_regression_detection_experiment.sh
python3 scripts/analysis/regression_detection_report.py

# spec 17.6 -- cross-checks against hyperfine, /usr/bin/time -v, direct
# `perf stat`, and NumPy
./scripts/experiments/run_cross_checks_experiment.sh
python3 scripts/analysis/cross_check_report.py

If your build directory isn't <repo>/build, set CPL_BUILD_DIR before running any experiment script. run_process_overhead_experiment.sh needs hyperfine on PATH; run_cross_checks_experiment.sh needs /usr/bin/time (the real binary, not the shell builtin -- apt install time).

Validation results

Full experiment methodology, raw findings, and four real bugs found (and fixed) purely through running this on real hardware are in docs/validation-results.md. All numbers below were measured on the same GCP c2-standard-4 VM described above; hardware performance counters were unavailable there (confirmed directly, not assumed -- see the doc), so every measurement uses the software-tracked event set (context-switches, cpu-migrations, page-faults).

Process-launch overhead (spec 17.2)

Basic-collection overhead shrinks from +9.0% at a 1ms workload to effectively +0.0% by 1s, as expected -- orchestration cost becomes negligible once the workload itself dominates. perf_stat overhead is different in kind: it's large and essentially fixed at ~112.5ms per invocation regardless of workload duration (+7570% at 1ms, still +11.2% at 1s). Read the flat perf_stat line, not the falling basic-collection line, as the operative constraint: enable perf_stat only for benchmarks whose duration is large relative to ~100ms.

Scalability (spec 17.3)

File size and load time scale roughly linearly with benchmark count, and report rendering stays fast even at 10,000 benchmarks (14ms) since it doesn't involve resampling. Comparison time is the outlier: it does not meet the spec's original "under one second for 1,000 benchmarks" target -- measured directly at ~26s for 1,000x100 and over 4 minutes at 10,000x100. The internal microbenchmarks below isolate why: BM_Comparison scales ~linearly at ~6.5ms per benchmark, matching this curve almost exactly, which points at bootstrap resampling (2,000 resamples/metric/benchmark by default) as the dominant cost, not JSON I/O. This is reported as a real, open limitation rather than walked back quietly -- see docs/architecture.md for the natural next steps (fewer resamples, optional/lazy bootstrap CI).

Repeatability (spec 17.4)

CPU-bound and memory-bandwidth-bound workloads hold a low, stable 0.4-1.4% CV regardless of pinning or background load on this lightly-loaded 4-vCPU machine -- read that as "noise is dominated by the workload's own character, not by scheduling, when there's no core contention." The multithreaded workload tells the more interesting story: pinning it to a single core roughly doubles its wall time (394ms vs. 192ms unpinned) -- four threads now time-share one core instead of running in parallel, a clean demonstration of a real resource constraint, not noise. Background load leaves the pinned case almost untouched (CV 0.08% -> 0.84%, since the competing process lands on one of the other three cores most of the time) but makes the unpinned case much noisier (CV 0.06% -> 7.41%), because now both the workload and the background load are actually contending for the same four cores. Pinning trades average throughput for measurement stability -- which one matters depends on what you're trying to measure.

Regression detection (spec 17.5)

False-positive rate is 0% across 20 baseline-vs-baseline trials. The headline finding is in the threshold sweep: the same slower_5pct configuration is detected 100% of the time at a 1% relative threshold and 0% of the time at 3%, 5%, or 10% -- because the workload's real measured effect is only ~4.1% wall-time increase (fixed process-launch/memory-touch overhead dilutes the nominal "+10% more iterations"), and that effect sits just below the default 5% threshold. Read the chart as: detection rate is a step function of threshold vs. true effect size, not of sample count -- the companion repetition-count sweep (5/10/20/50 reps, all 0% detected) confirms that more repetitions sharpen the estimate but can't move it past a threshold set above the real effect. The same_median_higher_variance variant (45% detection despite an unchanged median) is the deliberate counterexample: high intrinsic variance turns detection into something closer to a coin flip than a reliable signal, exactly as the spec intends to demonstrate.

Cross-checks (spec 17.6)

Wall time agrees with hyperfine within +9.0% at the 1ms case (both tools flag that duration as near their own measurement precision limit) down to +0.0% at 1s. Peak RSS matches /usr/bin/time -v exactly (159,104 KiB both). page-faults matches direct perf stat exactly (39,181 both); context-switches differs by one count (2 vs. 1), expected noise at that small an integer. Descriptive statistics (mean, median, N-1 stddev, p90/p95 via linear interpolation) match NumPy to displayed precision. This directly validates the conventions documented in statistics/descriptive_statistics.hpp against independent reference tools, not just against this project's own tests.

Three of the four real bugs this validation pass found (config working_directory resolution, a memory-footprint confound in the regression-detection workload design, and compare silently ignoring per-suite thresholds) and a fourth found via the cross-check above (the perf_stat availability probe using an unparseable event) are written up in full, with root cause and fix, in docs/validation-results.md.

References

• perf-stat(1) and perf_event_open(2) -- the Linux performance-counter subsystem this project wraps.
• Efron & Tibshirani, An Introduction to the Bootstrap (1993) -- methodology for the bootstrap confidence interval on the median difference between two sample sets.
• NumPy percentile and std -- the reference conventions this project's percentile (linear interpolation) and standard deviation (N-1) implementations match, cross-checked directly (see Validation results above).
• Google Benchmark -- drives the internal microbenchmark suite (benchmarks/).
• hyperfine -- independent reference tool for the wall-time cross-check.
• CLI11, yaml-cpp, nlohmann/json, fmt, spdlog, Catch2, OpenSSL -- plumbing libraries listed in vcpkg.json; see Architecture above for what each is used for.
• vcpkg -- manifest-mode dependency management for the build.

License

MIT -- see LICENSE.