Skip to content

Clean up and streamline the README a bit #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cfallin merged 1 commit into from
Oct 23, 2026
Merged

Clean up and streamline the README a bit #349

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 121 additions & 161 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<h1><code>sightglass</code></h1>

<p>
<strong>A benchmarking suite and tooling for Wasmtime and Cranelift</strong>
<strong>Benchmarking for Wasmtime and Cranelift</strong>
</p>

<strong>A <a href="https://bytecodealliance.org/">Bytecode Alliance</a> project</strong>
Expand All @@ -22,208 +22,153 @@

## About

This repository contains benchmarking infrastructure for Wasmtime and Cranelift,
as described in [this
RFC](https://github.com/bytecodealliance/rfcs/blob/main/accepted/benchmark-suite.md).
In particular, it has
This repository contains benchmarking infrastructure for Wasmtime and
Cranelift. In particular, it has

* a benchmark suite of Wasm applications in `benchmarks/*`, and
* a suite of Wasm benchmark programs in `benchmarks/*`, and

* a benchmark runner CLI tool to record, analyze, and display benchmark results
in `crates/cli/*`.
* a CLI tool to record, analyze, compare, and display benchmark results in
`crates/cli/*`.

We plan to implement a server that periodically runs benchmarks as new commits
are pushed to Wasmtime and display the history of those benchmark results,
similar to Firefox's [Are We Fast Yet?](https://arewefastyet.com). However, this
work is not completed yet. See [issue
93](https://github.com/bytecodealliance/sightglass/issues/93) for details.
Continuous benchmarking, showing historical performance over time and filing
issues for performance regressions, is built on top of this infrastructure in
the [`bytecodealliance/are-we-fast-yet`] repository.

[`bytecodealliance/are-we-fast-yet`]: https://github.com/bytecodealliance/are-we-fast-yet

Analyses use sound and rigorous statistical methods: Mann-Whitney U-tests for
significance (because benchmark results are not normally distributed), reporting
effect sizes only for statistically significant results, etc...

Results are always broken down by *phase* &mdash; compilation vs. instantiation
vs. execution &mdash; for each program in the suite. This allows us to reason
about, for example, compiler performance separately from its generated code
quality. How all this works together:

- each __benchmark__ is compiled to a `benchmark.wasm` module that calls two host functions,
`bench.start` and `bench.end`, to notify Sightglass of the portion of the execution to measure
(see the [benchmarks README])
- we build an __engine__ (e.g., Wasmtime) as a shared library that implements the [bench API]; the
Sightglass infrastructure uses this to measure each phase (see an [engine README])
- the __`sightglass-cli`__ tool runs __benchmarks__ using the __engines__ and emits __measurements__
for each phase; this is configurable, e.g., by various measurement mechanisms, various output
formats, different aggregations, etc.

[benchmarks README]: benchmarks/README.md
[engine README]: engines/wasmtime/README.md

## This is *NOT* a General-Purpose WebAssembly Benchmark Suite

This benchmark suite and tooling is specifically designed for Wasmtime and
Cranelift, as explained in [the benchmarking suite
RFC](https://github.com/bytecodealliance/rfcs/blob/main/accepted/benchmark-suite.md#nongoal-creating-a-general-purpose-webassembly-benchmark-suite):

> It is also worth mentioning this explicit non-goal: we do not intend to
> develop a general-purpose WebAssembly benchmark suite, used to compare between
> different WebAssembly compilers and runtimes. We don't intend to trigger a
> WebAssembly benchmarking war, reminiscent of JavaScript benchmarking wars in
> Web browsers. Doing so would make the benchmark suite's design high stakes,
> because engineers would be incentivized to game the benchmarks, and would
> additionally impose cross-engine portability constraints on the benchmark
> runner. We only intend to compare the performance of various versions of
> Wasmtime and Cranelift, where we don't need the cross-engine portability in
> the benchmark runner, and where gaming the benchmarks isn't incentivized.
>
> Furthermore, general-purpose WebAssembly benchmarking must include WebAssembly
> on the Web. Doing that well requires including interactions with the rest of
> the Web browser: JavaScript, rendering, and the DOM. Building and integrating
> a full Web browser is overkill for our purposes, and represents significant
> additional complexity that we would prefer to avoid.

Even if someone did manage to get other Wasm engines hooked into this
benchmarking infrastructure, comparing results across engines would likely be
invalid. The `wasmtime-bench-api` intentionally does things that will likely
hurt its absolute performance numbers but which help us more easily get
statistically meaningful results, like randomizing the locations of heap
allocations. Without taking great care to level the playing field with respect
to these sorts of tweaks, as well as keeping an eye on all engine specific
configuration options, you'll end up comparing apples and oranges.
quality.

## Usage
More details:

You can always see all subcommands and options via
* [`benchmarks/README.md`]: Wasm benchmark programs and adding new ones.
* [`engines/wasmtime/README.md`]: Building Wasmtime for benchmarking with Sightglass.

```
cargo run -- help
[`benchmarks/README.md`]: ./benchmarks/README.md
[`engines/wasmtime/README.md`]: ./engines/wasmtime/README.md

## Quickstart

Compare the performance between a baseline commit and a feature commit:

```bash
rustc engines/wasmtime/build.rs

# Build the baseline engine.
mkdir baseline
REPOSITORY=/path/to/my/wasmtime REVISION=my-baseline-branch-or-sha ./build baseline/

# Build the feature engine.
mkdir feature
REPOSITORY=/path/to/my/wasmtime REVISION=my-feature-branch-or-sha ./build feature/

# Run the benchmarks in both engines and compare their performance!
cargo run -- benchmark \
-e baseline/libengine.so \ # or `.dylib` on macos; `.dll` on windows
-e feature/libengine.so
```

There are flags to control how many iterations of each benchmark we measure
(each in its own process), etc...
## Usage

That said, here are a couple of typical usage scenarios.
You can always see all usage options via:

### Building the Runtime Engine for Wasmtime
```
$ cd engines/wasmtime && rustc build.rs && ./build && cd ../../
cargo run -- --help
```

### Running the Default Benchmark Suite

```
$ cargo run -- benchmark --engine engines/wasmtime/libengine.so
cargo run -- benchmark --engine engines/wasmtime/libengine.so
```

This runs all benchmarks listed in [`default.suite`](benchmarks/default.suite).
The output will be a summary of each benchmark program's compilation,
instantiation, and execution times.

### Running a Single Wasm Benchmark
When one engine is given, the output will be a summary of each benchmark's
compilation, instantiation, and execution times. When multiple engines are
given, each pair of engines' results are compared and tested for statistical
significance.

### Running Particular Wasm Benchmarks

```
$ cargo run -- benchmark --engine engines/wasmtime/libengine.so -- path/to/benchmark.wasm
cargo run -- benchmark --engine engines/wasmtime/libengine.so -- path/to/benchmark.wasm
```

Append multiple `*.wasm` paths to the end of that command to run multiple
benchmarks.

### Running All Benchmarks
See also "Which Benchmark Programs Should I Run?" in [`benchmarks/README.md`].

### Running Alternative Benchmark Suites

```
$ cargo run -- benchmark --engine engines/wasmtime/libengine.so -- benchmarks/all.suite
$ cargo run -- benchmark --engine engines/wasmtime/libengine.so -- path/to/the.suite
```

`*.suite` files contain relative paths of a list of benchmarks to run. This is a
convenience for organizing benchmarks but is functionally equivalent to listing
all `*.wasm` paths at the end of the `benchmark` command.
all `*.wasm` paths at the end of the `benchmark` command. There are a handful of
alternative suites defined in `./benchmarks/*.suite` files.

See also "Which Benchmark Programs Should I Run?" in [`benchmarks/README.md`].

### Comparing a Feature Branch to Main

First, build `libwasmtime_bench_api.so` (or `.dylib` or `.dll` depending on your
OS) for the latest `main` branch:
Build one `libengine.so` for `main` and another for your `my-feature` branch and
pass both to `cargo run -- benchmark`:

```
$ cd ~/wasmtime
$ git checkout main
$ cargo build --release -p wasmtime-bench-api
$ cp target/release/libwasmtime_bench_api.so /tmp/wasmtime_main.so
```
rustc engines/wasmtime/build.rs

Then, checkout your feature branch and build its `libwasmtime_bench_api.so`:
# Build the engine for `main`.
mkdir main
REPOSITORY=/path/to/my/wasmtime REVISION=main ./build main/

```
$ git checkout my-feature
$ cargo build --release -p wasmtime-bench-api
```

Finally, run the benchmarks and supply both versions of
`libwasmtime_bench_api.so` via repeated use of the `--engine` flag:
# Build the engine for the `my-feature` branch.
mkdir my-feature
REPOSITORY=/path/to/my/wasmtime REVISION=my-feature ./build my-feature/

# Pass all the engines to `benchmark` to compare their performance.
cargo run -- benchmark \
--engine main/libengine.so \ # or `.dylib` on macos; `.dll` on windows
--engine my-feature/libengine.so
```
$ cd ~/sightglass
$ cargo run -- \
benchmark \
--engine /tmp/wasmtime_main.so \
--engine ~/wasmtime/target/release/libwasmtime_bench_api.so \
-- \
benchmarks/all.suite
```

The output will show a comparison between the `main` branch's results and your
feature branch's results, giving you an effect size and confidence interval
(i.e. "we are 99% confident that `my-feature` is 1.32x to 1.37x faster than
`main`" or "there is no statistically significant difference in performance
between `my-feature` and `main`") for each benchmark Wasm program in the suite.
You may pass in a third or fourth or etc... `--engine` and each pair of engines
will be compared.

### Collecting Different Measurements

As you make further changes to your `my-feature` branch, you can execute this
command whenever you want new, updated benchmark results:
You can change what measures are recorded via the `-m`/`--measure` CLI flag:

```
$ cargo build --manifest-path ~/wasmtime/Cargo.toml --release -p wasmtime-bench-api && \
cargo run --manifest-path ~/sightglass/Cargo.toml -- \
benchmark \
--engine /tmp/wasmtime_main.so \
--engine ~/wasmtime/target/release/libwasmtime_bench_api.so \
-- \
benchmarks/all.suite
cargo run -- benchmark -e path/to/libengine.so -m insts-retired
```

### Collecting Different Kinds of Results
Available measures:

Sightglass comes enabled with several different kinds of measurement mechanisms
&mdash; a _measure_. The default _measure_ is `cycles`, which simply measures
the elapsed duration of CPU cycles for each phase (e.g., using `RDTSC`). The
accuracy of this _measure_ is documented [here](crates/recorder/README.md) but
note that measuring using CPU cycles alone can be problematic (e.g., CPU
frequency changes, context switches, etc.).
* `cycles`: CPU cycles elapsed. This is the default.

Several _measures_ can be configured using the `--measure` option:
* `insts-retired`: Instructions retired.

- `cycles`: The number of CPU cycles elapsed. On macOS this is the kernel's
per-process count from the hardware performance monitor; elsewhere it is a
user-space counter such as `RDTSC`, which ticks at a constant rate rather than
at the core's actual clock speed.
* `perf-counters`: A selection of common `perf` counters (CPU cycles,
instructions retired, cache accesses, cache misses). Only available on Linux.

- `perf-counters`: A selection of common `perf` counters (CPU cycles,
instructions retired, cache accesses, cache misses); only available on Linux.

- `callgrind`: Uses Valgrind's Callgrind to count instructions retired and
* `callgrind`: Uses Valgrind's Callgrind to count instructions retired and
simulate caches and branch prediction. Mostly deterministic and very low
noise. Only available on Linux and when built with `--features callgrind`.

- `vtune`: Record each phase as a VTune task for analysis; see [this help
documentation](docs/vtune.md) for more details.

- `noop`: No measurement is performed.

For example, run:

```
$ cargo run -- benchmark --measure perf-counters ...
```
* `vtune`: Record each phase as a VTune task for analysis; see
[`docs/vtune.md`](./docs/vtune.md) for details.

For `callgrind`, Sightglass runs benchmark children under `setarch -R valgrind`
with a fixed cache model and forces single-threaded Wasmtime compilation with
`RAYON_NUM_THREADS=1` so results stay stable across machines. Use the same
Valgrind version when comparing data recorded on different machines for best
results.
See also [`crates/recorder/README.md`](crates/recorder/README.md).

### Re-analyzing a Previous Run's Results

Expand All @@ -248,21 +193,21 @@ to do with the raw results yourself.
### Getting Raw JSON or CSV Results

If you don't want the results to be summarized and displayed in a human-readable
format, you can get raw JSON or CSV via the `--raw` flag:
format, you can get raw JSON or CSV via the `--raw` and `--output-format` CLI
flags:

```
$ cargo run -- benchmark --raw --output-format json -- benchmark.wasm
```

Then you can use your own R/Python/spreadsheets/etc. to analyze and visualize
Then you can use your own R/Python/spreadsheets/etc... to analyze and visualize
the benchmark results.

### HTML Report Generation

For raw results collected in the JSON format, sightglass includes a built-in `report`
subcommand that generates comprehensive HTML reports with statistical analysis and
interactive visualizations. This approach provides rigorous statistical significance
testing following the Bytecode Alliance RFC standards.
For raw results collected in the JSON format, sightglass includes a built-in
`report` subcommand that generates comprehensive HTML reports with statistical
analysis and interactive visualizations:

```bash
# Generate HTML report comparing multiple measurement files
Expand All @@ -272,19 +217,34 @@ $ cargo run -- report --output-file report.html baseline.json experiment1.json e
$ cargo run -- report --baseline-engine wasmtime-baseline --output-file report.html baseline.json experiment.json
```

The generated HTML report includes:
- Statistical significance testing with confidence intervals
- Interactive charts showing performance distributions
- Effect size analysis with practical significance assessment
- Filtering and sorting capabilities for large result sets
### Building the Engine(s) to Run Benchmarks Within

```shell-session
rustc engines/wasmtime/build.rs
./build
```

The resulting engine will be at `libengine.so` (or `libengine.dll` on windows
and `libengine.dylib` on macos). You may pass a directory path as an argument to
`./build` and the engine will be placed there instead of the current directory.

By default this builds the latest commit on `main` from the canonical
`bytecodealliance/wasmtime` repository.

See `cargo run -- report --help` for all available options.
You can set the following environment variables while running `./build` to
configure the engine's build:

### Adding a New Benchmark
* `REPOSITORY`: the URL or file path of the Wasmtime repository to be built. If
the version of Wasmtime you'd like to benchmark is not merged upstream yet,
you can set this to your fork. Defaults to
`https://github.com/bytecodealliance/wasmtime.git`.
* `REVISION`: the git branch, commit, tag, etc... that should be built. Defaults
to `main`.

Add a Dockerfile under `benchmarks/<your benchmark>` building a Wasm file that
brackets the work to measure with the `bench.start` and `bench.end` host calls.
See the [benchmarks README] for a fuller set of requirements and the
[`build.sh`] script for building this file.
For example:

[`build.sh`]: benchmarks/build.sh
```
REPOSITORY=https://github.com/fitzgen/wasmtime.git \
REVISION=add-cool-new-optimization \
./build
```
Loading