From af321c138ff17ac9444775c108250f577633aab1 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Thu, 24 Sep 2026 11:38:37 +0200 Subject: [PATCH 01/15] Add aspnet-benchmark skill for running TechEmpower benchmarks locally Documents the workflow used to validate runtime changes (e.g. the io_uring socket path) end-to-end under wrk/TechEmpower PlatformBenchmarks load: cloning aspnet/Benchmarks, publishing against the repo's own SDK, overlaying the SDK's shared Microsoft.NETCore.App with a locally-built testhost so Microsoft.AspNetCore.App and the local runtime can be combined, running wrk against /json, and a dedicated perfcollect profiling section with symbol-resolution caveats. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 165 +++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 .github/skills/aspnet-benchmark/SKILL.md diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md new file mode 100644 index 00000000000000..a491dcfda232dc --- /dev/null +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -0,0 +1,165 @@ +--- +name: aspnet-benchmark +description: Run the TechEmpower ASP.NET Core benchmarks (PlatformBenchmarks) locally against a locally-built dotnet/runtime, to validate the end-to-end impact of a runtime change (e.g. sockets, io_uring, GC, JIT) under realistic HTTP load. Use this when asked to benchmark, load-test, or profile an ASP.NET Core / Kestrel scenario, or to compare an env-var-gated feature (e.g. `DOTNET_USE_IO_URING`) end-to-end. +--- + +# Running the TechEmpower (PlatformBenchmarks) Benchmark Locally + +When a runtime change needs to be validated under realistic HTTP server load (not just a BenchmarkDotNet microbenchmark — see the `performance-benchmark` skill for that), use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, driven by `wrk` (or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows, since `wrk` is Linux/macOS-only), against a locally-built runtime. + +This is the right tool when the change affects the socket/IO stack, GC, JIT, or anything else only observable under many concurrent connections and real request/response processing. + +## Prerequisites + +- A locally-built dotnet/runtime with the change under test (see the `build-and-test` skill). Build **everything** (runtime and libraries) in Release with `-c release`, not just `-rc release` — the latter only forces the runtime to Release and leaves libraries at their default (`Debug`), producing a mismatched testhost: + + ```bash + ./build.sh clr+libs -c release + ``` + +- A load-generation tool: on Linux, `wrk` (`sudo apt install wrk` or build from source — ). `wrk` doesn't build/run on Windows; use [`bombardier`](https://github.com/codesenberg/bombardier) instead (`go install github.com/codesenberg/bombardier@latest`, or download a prebuilt binary from its releases page) — the examples below use `wrk` syntax, with the `bombardier` equivalent noted alongside. +- The [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, which contains the TechEmpower and other benchmark apps. + +## Step 1: Clone the Benchmarks Repository + +```bash +git clone https://github.com/aspnet/Benchmarks.git +``` + +The relevant app is `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). It targets the latest `net*.0` TFM and does **not** require a database for the `/json` and `/plaintext` endpoints, which is normally all you need. + +## Step 2: Prefer the `/json` Endpoint, Not `/plaintext` + +Use `/json` for a "realistic max throughput" measurement. `/plaintext` conventionally uses HTTP pipelining (16 requests batched per connection in the official TechEmpower harness/`wrk` scripts), which does not reflect normal request/response traffic and inflates throughput numbers in a way that isn't representative. `/json` sends one request per round trip and is the more realistic comparison point. + +## Step 3: Publish the App Against the Repo's Own SDK + +Build/publish the app with the dotnet/runtime repo's own SDK (`/.dotnet/dotnet`), not a system-wide SDK, so the app targets the same TFM as your local build: + +```bash +cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks +/.dotnet/dotnet build -c Release +``` + +This produces `bin/Release//PlatformBenchmarks.dll`. + +## Step 4: Make the Repo's SDK Run Against Your Local Runtime Build + +`PlatformBenchmarks` needs `Microsoft.AspNetCore.App`, which the freshly-built `artifacts/bin/testhost` does **not** contain (it only has `Microsoft.NETCore.App`) — running it directly via `testhost`'s `corerun`/`dotnet` fails with a "no framework found" error. The repo SDK's own `dotnet` (under `/.dotnet`) already has `Microsoft.AspNetCore.App`, so the simplest way to combine "the ASP.NET Core framework" with "your locally-built `Microsoft.NETCore.App`" is to temporarily overlay the SDK's shared `Microsoft.NETCore.App/` folder with the testhost's freshly-built one: + +```bash +SDK_FX=$(find /.dotnet/shared/Microsoft.NETCore.App -maxdepth 1 -type d | tail -1) +TESTHOST_FX="/artifacts/bin/testhost/--Release-/shared/Microsoft.NETCore.App/" + +# ALWAYS back up the original SDK framework files first, so they can be restored exactly. +mkdir -p /tmp/netcoreapp_sdk_backup +rsync -a --delete "$SDK_FX"/ /tmp/netcoreapp_sdk_backup/ + +# Overlay with the local build. +rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ +``` + +**This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. + +## Step 5: Run the App and Verify It's Serving Requests + +```bash +cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks/bin/Release/ +/.dotnet/dotnet exec PlatformBenchmarks.dll --urls http://127.0.0.1:5000 & +sleep 5 +curl -sS http://127.0.0.1:5000/json +``` + +Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. + +## Step 6: Run `wrk` (or `bombardier` on Windows) Against It + +```bash +wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/json +``` + +- `-t`: number of `wrk` threads — a good default is the machine's core count. +- `-c`: number of concurrent connections — 256 is a reasonable default for a many-concurrent-connections scenario. +- `-d15s`: run duration; 15s is normally enough to get a stable number. +- `--latency`: also reports the latency percentile breakdown, not just aggregate throughput. + +On Windows, use `bombardier` instead, with equivalent options: + +```powershell +bombardier -c 256 -d 15s -l http://127.0.0.1:5000/json +``` + +- `-c`: concurrent connections (same meaning as `wrk -c`). +- `-d`: run duration (same meaning as `wrk -d`). +- `-l`: print the latency percentile breakdown (equivalent to `wrk --latency`). +- `bombardier` has no direct equivalent to `wrk -t`; it manages its own worker goroutines internally. + +Run **at least two** runs per configuration (there's meaningful run-to-run variance) and report both, along with p50/p99 latency, not just a single throughput number. Present the results in a table. + +Repeat Steps 5-6 for each configuration being compared (e.g. once with the env var on, once with it off), reusing the same overlay — only the running process needs to be restarted between configurations, not the overlay. + +## Step 7: Clean Up + +1. Kill the server process(es) — find the actual `dotnet exec ... PlatformBenchmarks.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` and `kill `. +2. **Restore the SDK's shared framework from the backup** and verify it via checksum before considering the machine clean: + + ```bash + rsync -a --delete /tmp/netcoreapp_sdk_backup/ "$SDK_FX"/ + md5sum "$SDK_FX"/System.Private.CoreLib.dll # compare against the value recorded before overlaying + ``` + +3. Remove the temporary backup and any log files once restoration is verified. + +## Profiling a Benchmark Run with `perfcollect` + +If the load-test numbers show a difference (or don't, and you need to know why) and you need native + managed call stacks, use `perfcollect` rather than a raw `perf record` — it also captures LTTng CLR events and can resolve JIT-generated code, not just precompiled/R2R code. `perfcollect` itself is Linux-only (it relies on LTTng/perf), regardless of which platform you used to drive the load test. + +- Tool and full instructions: . Also see the repo's own [Linux Performance Tracing](../../../docs/project/linux-performance-tracing.md) doc. +- Basic flow (needs two terminals — one to drive `perfcollect`, one to run the server): + + ```bash + curl -OL https://aka.ms/perfcollect + chmod +x perfcollect + sudo ./perfcollect install # one-time, installs LTTng/perf prerequisites + + # [App terminal] enable perf maps for JIT-generated code before starting the process: + export DOTNET_PerfMapEnabled=1 + export DOTNET_EnableEventLog=1 + /.dotnet/dotnet exec PlatformBenchmarks.dll --urls http://127.0.0.1:5000 & + + # [Trace terminal] start collection, then drive load with wrk from a third terminal: + sudo ./perfcollect collect webapp_trace + + # Ctrl+C to stop once wrk has run long enough (a few seconds under load is enough for a CPU investigation). + ``` + + This produces `webapp_trace.trace.zip`, analyzable with PerfView () or `perf report`/`perf script` directly on the underlying `.perf.data`. + +- Raising `kernel.perf_event_paranoid` may be required for an unprivileged `perf record`/`perfcollect` to work at all: + + ```bash + sudo sysctl -w kernel.perf_event_paranoid=-1 # temporary; restore the original value afterward + ``` + + Always restore the original value once profiling is done — don't leave a relaxed `perf_event_paranoid` on a shared machine. + +### ⚠️ Symbol Resolution Warnings + +- **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/crossgen2_publish///crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 4) before collecting: + + ```bash + cp artifacts/bin/crossgen2_publish/x64/Release/crossgen2 "$SDK_FX"/ + ``` + + For a runtime you didn't build yourself, see the "Resolving Framework Symbols" section of [linux-performance-tracing.md](../../../docs/project/linux-performance-tracing.md) instead (it walks through obtaining a matching `crossgen2` via a self-contained publish). +- **Native runtime frames (`libcoreclr.so`, `libclrjit.so`, etc.) resolve out of the box, as long as you don't move or discard the build's `.dbg` files.** A Release native build ships each `.so` *stripped*, but the matching, un-stripped `libXyz.so.dbg` is written right alongside it in the same output directory (e.g. `artifacts/bin/coreclr/./`), linked via its `.gnu_debuglink` section and a matching Build ID. `perf`/`perfcollect` follow that link automatically as long as the `.dbg` file stays next to its `.so` — don't clean up or selectively copy only the `.so` files into a deployment layout, or you'll silently lose native symbolication. +- **JIT-generated (Tier0/Tier1) symbols require `DOTNET_PerfMapEnabled=1` to be set *before* the process starts.** If you forget this and only realize partway through a run, the trace from that run cannot be fixed after the fact — kill the process, re-export the env var, and restart it before collecting again. +- Even after doing all of the above, expect *some* residual unresolved frames from components you didn't build locally (e.g. the OS's own libraries, or a `crossgen2`/`.dbg` mismatch if you mixed binaries from different build configurations) — this doesn't mean the whole trace is useless; managed app-level and JIT-emitted frames still resolve correctly via the perf map regardless. +- `DOTNET_PerfMapEnabled=1` has a real (if usually small) overhead of its own — don't leave it set for the throughput (`wrk`/`bombardier`) runs themselves, only for the dedicated profiling run. + +## Common Pitfalls + +- **Running via `artifacts/bin/testhost` directly fails** with "no framework found" — that layout only has `Microsoft.NETCore.App`, not `Microsoft.AspNetCore.App`. Use the SDK-overlay approach in Step 4 instead. +- **Forgetting to set the env var before starting the process** — most feature switches (like `DOTNET_USE_IO_URING`) are read once at startup, so changing it and re-`curl`-ing the same running process has no effect. +- **Comparing a single run per configuration** — throughput varies run to run; always do at least two runs per configuration. +- **Leaving the SDK shared framework overlaid** — always restore it (Step 7) and verify via checksum; a stale/mismatched overlay silently breaks unrelated work on the same machine later. From b83a77b677a3efa05202cbee854250acffddc550 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Thu, 24 Sep 2026 12:03:28 +0200 Subject: [PATCH 02/15] Rename performance-benchmark skill to microbenchmark The old name was too broad for a skill scoped specifically to BenchmarkDotNet-based microbenchmarks; renamed to disambiguate from the new aspnet-benchmark skill, which covers end-to-end HTTP server load testing instead. Updated the skill's own frontmatter (name, description) and every cross-reference to it across other skills and .github/copilot-instructions.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- .github/skills/api-proposal/SKILL.md | 2 +- .github/skills/aspnet-benchmark/SKILL.md | 2 +- .github/skills/build-and-test/SKILL.md | 2 +- .github/skills/issue-triage/SKILL.md | 2 +- .github/skills/jit-regression-test/SKILL.md | 2 +- .../{performance-benchmark => microbenchmark}/SKILL.md | 6 +++--- .github/skills/vectorization/SKILL.md | 4 ++-- 8 files changed, 11 insertions(+), 11 deletions(-) rename .github/skills/{performance-benchmark => microbenchmark}/SKILL.md (95%) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 035f22463f5c27..14625ae418f647 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -16,7 +16,7 @@ Use the `code-review` skill when reviewing pull requests, and — when running u When starting work in an unfamiliar directory, search for `README.md` files in it and its parents up to the repository root. Read any you find — they contain conventions, patterns, and architectural context relevant to your work. -If the changes are intended to improve performance, or if they could negatively impact performance, use the `performance-benchmark` skill to validate the impact before completing. +If the changes are intended to improve performance, or if they could negatively impact performance, use the `microbenchmark` skill to validate the impact before completing. When writing or reviewing SIMD / hardware-intrinsics code (anything using `Vector128`/`Vector256`/`Vector512`, `Vector`, or the platform intrinsics in `System.Runtime.Intrinsics.*`), use the `vectorization` skill. diff --git a/.github/skills/api-proposal/SKILL.md b/.github/skills/api-proposal/SKILL.md index 6949afab51a0da..7038d55df6e26f 100644 --- a/.github/skills/api-proposal/SKILL.md +++ b/.github/skills/api-proposal/SKILL.md @@ -177,7 +177,7 @@ This: 2. **All errors and warnings must be fixed** before proceeding to the draft phase. -3. If the API change could affect performance (hot paths, allocations, new collection types), suggest running the **performance-benchmark** skill. +3. If the API change could affect performance (hot paths, allocations, new collection types), suggest running the **microbenchmark** skill. 4. Re-run tests after any review-driven changes to confirm nothing regressed. diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index a491dcfda232dc..1bc1e92d9087fe 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -5,7 +5,7 @@ description: Run the TechEmpower ASP.NET Core benchmarks (PlatformBenchmarks) lo # Running the TechEmpower (PlatformBenchmarks) Benchmark Locally -When a runtime change needs to be validated under realistic HTTP server load (not just a BenchmarkDotNet microbenchmark — see the `performance-benchmark` skill for that), use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, driven by `wrk` (or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows, since `wrk` is Linux/macOS-only), against a locally-built runtime. +When a runtime change needs to be validated under realistic HTTP server load (not just a BenchmarkDotNet microbenchmark — see the `microbenchmark` skill for that), use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, driven by `wrk` (or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows, since `wrk` is Linux/macOS-only), against a locally-built runtime. This is the right tool when the change affects the socket/IO stack, GC, JIT, or anything else only observable under many concurrent connections and real request/response processing. diff --git a/.github/skills/build-and-test/SKILL.md b/.github/skills/build-and-test/SKILL.md index 9f3603f3555d29..67d49534c2d69d 100644 --- a/.github/skills/build-and-test/SKILL.md +++ b/.github/skills/build-and-test/SKILL.md @@ -5,7 +5,7 @@ description: > per-component build and test commands, Windows/PowerShell equivalents, running individual tests, and troubleshooting. USE FOR: any build or test invocation in this repo, and before making code changes under CCA. DO NOT USE FOR: CI pipeline - triage (use ci-pipeline-monitor) or benchmarking (use performance-benchmark). + triage (use ci-pipeline-monitor) or benchmarking (use microbenchmark). --- # Building & Testing in dotnet/runtime diff --git a/.github/skills/issue-triage/SKILL.md b/.github/skills/issue-triage/SKILL.md index 38a2434cae9075..540c4ad7879d34 100644 --- a/.github/skills/issue-triage/SKILL.md +++ b/.github/skills/issue-triage/SKILL.md @@ -483,5 +483,5 @@ After triage, these skills can help with next steps: - **api-proposal** -- if an API proposal is recommended as KEEP, offer to draft a formal proposal with working prototype. - **jit-regression-test** -- if a bug with root cause is JIT-related, offer to create a regression test. -- **performance-benchmark** -- if a performance regression is confirmed, offer to validate with ad hoc benchmarks. +- **microbenchmark** -- if a performance regression is confirmed, offer to validate with ad hoc benchmarks. - **code-review** -- if a fix PR is linked, offer to review for correctness and consistency. diff --git a/.github/skills/jit-regression-test/SKILL.md b/.github/skills/jit-regression-test/SKILL.md index 01835c82602a6c..ca50fb9dc19709 100644 --- a/.github/skills/jit-regression-test/SKILL.md +++ b/.github/skills/jit-regression-test/SKILL.md @@ -7,7 +7,7 @@ description: > bug", "create a regression test for issue #NNNNN", converting issue repro to xunit test. DO NOT USE FOR: non-JIT tests (use standard test patterns), debugging JIT issues without a known repro, performance benchmarks (use - performance-benchmark skill). + microbenchmark skill). --- # JIT Regression Test Extraction diff --git a/.github/skills/performance-benchmark/SKILL.md b/.github/skills/microbenchmark/SKILL.md similarity index 95% rename from .github/skills/performance-benchmark/SKILL.md rename to .github/skills/microbenchmark/SKILL.md index dc1bc34a78279c..198ff021442d5e 100644 --- a/.github/skills/performance-benchmark/SKILL.md +++ b/.github/skills/microbenchmark/SKILL.md @@ -1,9 +1,9 @@ --- -name: performance-benchmark -description: Generate and run ad hoc performance benchmarks to validate code changes. Use this when asked to benchmark, profile, or validate the performance impact of a code change in dotnet/runtime. +name: microbenchmark +description: Generate and run ad hoc BenchmarkDotNet microbenchmarks to validate the performance impact of a code change. Use this when asked to benchmark, profile, or validate the performance impact of a code change in dotnet/runtime at the method/API level. For end-to-end ASP.NET Core / HTTP server load testing, use the aspnet-benchmark skill instead. --- -# Ad Hoc Performance Benchmarking Locally (or with @EgorBot) +# Ad Hoc Microbenchmarking Locally (or with @EgorBot) When you need to validate the performance impact of a code change, follow this process to write a BenchmarkDotNet benchmark and compare local baseline and changed builds. diff --git a/.github/skills/vectorization/SKILL.md b/.github/skills/vectorization/SKILL.md index 5dce6175ecaea7..fbb8aa174f1fc5 100644 --- a/.github/skills/vectorization/SKILL.md +++ b/.github/skills/vectorization/SKILL.md @@ -6,7 +6,7 @@ description: > code that uses Vector128/Vector256/Vector512, Vector, or the platform intrinsics in System.Runtime.Intrinsics.X86/Arm/Wasm, and validating remainder handling, load/store safety, and hardware-acceleration fallbacks. DO NOT USE - FOR: general performance work unrelated to SIMD (use performance-benchmark), + FOR: general performance work unrelated to SIMD (use microbenchmark), or non-vectorized code review (use code-review). --- @@ -93,7 +93,7 @@ and the same `DOTNET_Enable*` variables to compare scalar / `Vector128` / `Vecto in mind: larger inputs benefit more (small buffers can be *slower* due to setup), speedups are rarely the theoretical multiple (memory throughput, alignment, and latency all factor in), and randomized allocation alignment adds noise — allocate aligned memory or enable BenchmarkDotNet's randomization for -stable/observable results. For non-trivial changes, use the `performance-benchmark` skill. +stable/observable results. For non-trivial changes, use the `microbenchmark` skill. ## Review checklist From 49e680b223d3ebf500bf3b357ef2b1df0f18103e Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Thu, 24 Sep 2026 12:38:56 +0200 Subject: [PATCH 03/15] Document external ASP.NET benchmarking with Crank Preserve the local SDK-overlay workflow and document minimal runtime payloads, symbols, environment settings, and Linux trace collection on external infrastructure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 208 +++++++++++++++++++++-- 1 file changed, 193 insertions(+), 15 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 1bc1e92d9087fe..01272ef1e67bf7 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -1,15 +1,21 @@ --- name: aspnet-benchmark -description: Run the TechEmpower ASP.NET Core benchmarks (PlatformBenchmarks) locally against a locally-built dotnet/runtime, to validate the end-to-end impact of a runtime change (e.g. sockets, io_uring, GC, JIT) under realistic HTTP load. Use this when asked to benchmark, load-test, or profile an ASP.NET Core / Kestrel scenario, or to compare an env-var-gated feature (e.g. `DOTNET_USE_IO_URING`) end-to-end. +description: Run the TechEmpower ASP.NET Core benchmarks (PlatformBenchmarks) against a locally-built dotnet/runtime, either locally with an SDK overlay or on external infrastructure using Crank, to validate the end-to-end impact of a runtime change (e.g. sockets, io_uring, GC, JIT) under realistic HTTP load. Use this when asked to benchmark, load-test, or profile an ASP.NET Core / Kestrel scenario, or to compare an env-var-gated feature (e.g. `DOTNET_USE_IO_URING`) end-to-end. --- -# Running the TechEmpower (PlatformBenchmarks) Benchmark Locally +# Running the TechEmpower (PlatformBenchmarks) Benchmarks -When a runtime change needs to be validated under realistic HTTP server load (not just a BenchmarkDotNet microbenchmark — see the `microbenchmark` skill for that), use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, driven by `wrk` (or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows, since `wrk` is Linux/macOS-only), against a locally-built runtime. +Use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository to validate runtime changes under realistic HTTP server load. Microbenchmarks can isolate an operation's cost, but end-to-end ASP.NET benchmarks show how a change affects throughput, latency, and CPU usage under concurrent request/response processing. -This is the right tool when the change affects the socket/IO stack, GC, JIT, or anything else only observable under many concurrent connections and real request/response processing. +These benchmarks are useful for changes to the socket/IO stack, GC, JIT, and other components whose impact depends on application behavior. Both workflows below run the benchmark against a locally-built runtime: start with local measurements to validate ideas, then use external infrastructure for final validation when access is available. -## Prerequisites +## Benchmarking a Local Runtime Build Locally + +**Always validate ideas locally first.** This workflow provides quick verification on the development machine and requires no VPN or access to external infrastructure. Drive the application with `wrk`, or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows because `wrk` is Linux/macOS-only. + +Treat local results as an initial signal, not a definitive performance measurement: the application, load generator, and other processes share the same machine and compete for CPU, memory, and other resources. This contention can distort throughput and latency, so local results may not predict performance on dedicated benchmark machines. + +### Prerequisites - A locally-built dotnet/runtime with the change under test (see the `build-and-test` skill). Build **everything** (runtime and libraries) in Release with `-c release`, not just `-rc release` — the latter only forces the runtime to Release and leaves libraries at their default (`Debug`), producing a mismatched testhost: @@ -20,7 +26,7 @@ This is the right tool when the change affects the socket/IO stack, GC, JIT, or - A load-generation tool: on Linux, `wrk` (`sudo apt install wrk` or build from source — ). `wrk` doesn't build/run on Windows; use [`bombardier`](https://github.com/codesenberg/bombardier) instead (`go install github.com/codesenberg/bombardier@latest`, or download a prebuilt binary from its releases page) — the examples below use `wrk` syntax, with the `bombardier` equivalent noted alongside. - The [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, which contains the TechEmpower and other benchmark apps. -## Step 1: Clone the Benchmarks Repository +### Step 1: Clone the Benchmarks Repository ```bash git clone https://github.com/aspnet/Benchmarks.git @@ -28,11 +34,11 @@ git clone https://github.com/aspnet/Benchmarks.git The relevant app is `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). It targets the latest `net*.0` TFM and does **not** require a database for the `/json` and `/plaintext` endpoints, which is normally all you need. -## Step 2: Prefer the `/json` Endpoint, Not `/plaintext` +### Step 2: Prefer the `/json` Endpoint, Not `/plaintext` Use `/json` for a "realistic max throughput" measurement. `/plaintext` conventionally uses HTTP pipelining (16 requests batched per connection in the official TechEmpower harness/`wrk` scripts), which does not reflect normal request/response traffic and inflates throughput numbers in a way that isn't representative. `/json` sends one request per round trip and is the more realistic comparison point. -## Step 3: Publish the App Against the Repo's Own SDK +### Step 3: Publish the App Against the Repo's Own SDK Build/publish the app with the dotnet/runtime repo's own SDK (`/.dotnet/dotnet`), not a system-wide SDK, so the app targets the same TFM as your local build: @@ -43,7 +49,7 @@ cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks This produces `bin/Release//PlatformBenchmarks.dll`. -## Step 4: Make the Repo's SDK Run Against Your Local Runtime Build +### Step 4: Make the Repo's SDK Run Against Your Local Runtime Build `PlatformBenchmarks` needs `Microsoft.AspNetCore.App`, which the freshly-built `artifacts/bin/testhost` does **not** contain (it only has `Microsoft.NETCore.App`) — running it directly via `testhost`'s `corerun`/`dotnet` fails with a "no framework found" error. The repo SDK's own `dotnet` (under `/.dotnet`) already has `Microsoft.AspNetCore.App`, so the simplest way to combine "the ASP.NET Core framework" with "your locally-built `Microsoft.NETCore.App`" is to temporarily overlay the SDK's shared `Microsoft.NETCore.App/` folder with the testhost's freshly-built one: @@ -61,7 +67,7 @@ rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ **This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. -## Step 5: Run the App and Verify It's Serving Requests +### Step 5: Run the App and Verify It's Serving Requests ```bash cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks/bin/Release/ @@ -72,7 +78,7 @@ curl -sS http://127.0.0.1:5000/json Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. -## Step 6: Run `wrk` (or `bombardier` on Windows) Against It +### Step 6: Run `wrk` (or `bombardier` on Windows) Against It ```bash wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/json @@ -98,7 +104,7 @@ Run **at least two** runs per configuration (there's meaningful run-to-run varia Repeat Steps 5-6 for each configuration being compared (e.g. once with the env var on, once with it off), reusing the same overlay — only the running process needs to be restarted between configurations, not the overlay. -## Step 7: Clean Up +### Step 7: Clean Up 1. Kill the server process(es) — find the actual `dotnet exec ... PlatformBenchmarks.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` and `kill `. 2. **Restore the SDK's shared framework from the backup** and verify it via checksum before considering the machine clean: @@ -110,7 +116,7 @@ Repeat Steps 5-6 for each configuration being compared (e.g. once with the env v 3. Remove the temporary backup and any log files once restoration is verified. -## Profiling a Benchmark Run with `perfcollect` +### Profiling a Benchmark Run with `perfcollect` If the load-test numbers show a difference (or don't, and you need to know why) and you need native + managed call stacks, use `perfcollect` rather than a raw `perf record` — it also captures LTTng CLR events and can resolve JIT-generated code, not just precompiled/R2R code. `perfcollect` itself is Linux-only (it relies on LTTng/perf), regardless of which platform you used to drive the load test. @@ -143,7 +149,7 @@ If the load-test numbers show a difference (or don't, and you need to know why) Always restore the original value once profiling is done — don't leave a relaxed `perf_event_paranoid` on a shared machine. -### ⚠️ Symbol Resolution Warnings +#### ⚠️ Symbol Resolution Warnings - **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/crossgen2_publish///crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 4) before collecting: @@ -157,9 +163,181 @@ If the load-test numbers show a difference (or don't, and you need to know why) - Even after doing all of the above, expect *some* residual unresolved frames from components you didn't build locally (e.g. the OS's own libraries, or a `crossgen2`/`.dbg` mismatch if you mixed binaries from different build configurations) — this doesn't mean the whole trace is useless; managed app-level and JIT-emitted frames still resolve correctly via the perf map regardless. - `DOTNET_PerfMapEnabled=1` has a real (if usually small) overhead of its own — don't leave it set for the throughput (`wrk`/`bombardier`) runs themselves, only for the dedicated profiling run. -## Common Pitfalls +### Common Pitfalls - **Running via `artifacts/bin/testhost` directly fails** with "no framework found" — that layout only has `Microsoft.NETCore.App`, not `Microsoft.AspNetCore.App`. Use the SDK-overlay approach in Step 4 instead. - **Forgetting to set the env var before starting the process** — most feature switches (like `DOTNET_USE_IO_URING`) are read once at startup, so changing it and re-`curl`-ing the same running process has no effect. - **Comparing a single run per configuration** — throughput varies run to run; always do at least two runs per configuration. - **Leaving the SDK shared framework overlaid** — always restore it (Step 7) and verify via checksum; a stale/mismatched overlay silently breaks unrelated work on the same machine later. + +## Benchmarking a Local Runtime Build on External Infrastructure with Crank + +**Use external infrastructure for final validation after testing locally.** The .NET benchmarking infrastructure described here requires VPN access and is unavailable to external contributors. Runs take substantially longer than local checks, and the infrastructure is not always available; do not depend on it for the initial iteration loop. + +Use [Crank](https://github.com/dotnet/crank) to deploy `PlatformBenchmarks` with selected locally-built runtime binaries to a remote application agent; a separate load agent drives HTTP requests without competing for the application's machine resources. These access restrictions apply to the shared infrastructure, not to Crank itself: external contributors can use Crank with their own agents. + +Prefer `/json` for representative maximum-throughput comparisons. The standard `/plaintext` scenario pipelines 16 requests per connection; its throughput is not directly comparable to non-pipelined JSON traffic. Neither scenario needs a database. + +References: + +- [Crank documentation](https://github.com/dotnet/crank/blob/main/docs/README.md) and [getting started](https://github.com/dotnet/crank/blob/main/docs/getting_started.md): controller installation, agents, scenarios, and profiles. +- [Selecting .NET versions](https://github.com/dotnet/crank/blob/main/docs/dotnet_versions.md): framework, SDK, runtime, and ASP.NET version overrides. +- [PlatformBenchmarks configuration](https://github.com/aspnet/Benchmarks/blob/main/scenarios/platform.benchmarks.yml): `json` scenario, `application` and `load` jobs, and load variables. +- [Linux performance tracing](../../../docs/project/linux-performance-tracing.md): native and managed symbol resolution. + +### 1. Establish the Inputs + +- Use an existing, validated **Release** build for the agent's OS and architecture. Both native runtime and managed libraries must be Release. Consult the `build-and-test` skill if a build is needed; do not rebuild merely to stage binaries, and never build when the user requested documentation or explicitly prohibited builds. +- Confirm the artifacts correspond to the intended source revision. Switching Git branches does not update existing build outputs. Record source commits, any uncommitted changes, configuration, and binary hashes. +- Use an installed Crank controller and reachable application/load agents. Prefer separate machines for throughput measurements so load generation does not compete with the server for CPU. +- Select a PlatformBenchmarks configuration and an authorized machine profile. The public configuration includes profiles for particular infrastructure; choose or define one for the actual agents rather than assuming those machines are accessible. The getting-started guide shows how profiles assign `application` and `load` endpoints. +- Pin the benchmark source revision and exact SDK, runtime, and ASP.NET versions for comparisons. Do not allow `main`, `latest`, or `edge` to change dependencies between runs. Ensure the benchmark TFM and selected versions are compatible with the local runtime overlay. + +Keep stock ASP.NET binaries unless the experiment explicitly includes a local ASP.NET build. A runtime-only comparison must not accidentally include a modified Kestrel transport. + +### 2. Choose the Smallest Coherent Binary Subset + +**Do not upload the entire testhost, SDK, or artifacts directory.** Start with the changed component, then add only dependencies required for its implementation, managed/native ABI, or runtime compatibility. This reduces upload time and makes the experiment attributable. + +Inspect the source diff and relevant project outputs before choosing files: + +| Change under test | Files to consider for a Linux overlay | +|---|---| +| Managed library only, such as `System.Net.Sockets` | The changed implementation `.dll` and matching `.pdb`; additional implementation dependencies only if needed. | +| Native PAL or interop contract | The affected native library, such as `libSystem.Native.so`, its matching `.so.dbg`, and any managed assembly changed with that ABI. | +| CoreLib, including ThreadPool | `System.Private.CoreLib.dll` and `.pdb`, with compatible VM/JIT binaries. When using a locally-built CoreLib against a different published runtime, use the matching `libcoreclr.so` and `libclrjit.so` plus their `.dbg` files rather than assuming internal contracts match. | +| JIT only | `libclrjit.so` and `.so.dbg` if compatible with the selected VM's JIT/EE interface; otherwise include the matching runtime components. | +| VM or GC | `libcoreclr.so` and `.so.dbg`, plus matching CoreLib/JIT when their internal contracts require it. | +| Added APIs or type forwarders | Any additional deployable implementation/forwarder assemblies needed by consumers, such as `System.Runtime.dll` or `System.Threading.ThreadPool.dll`, with their symbols. **Do not deploy reference assemblies from `ref/`.** | +| Explicitly requested local ASP.NET change | The affected ASP.NET implementation assemblies and symbols, for example `Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll` and `.pdb`, plus required dependencies. | + +These are selection rules, not a universal file list. A Socket-only change may need just two files; an experiment spanning Socket, CoreLib, and native code needs a larger coherent set. Do not omit a required dependency merely to minimize file count. + +The prepared shared-framework directory is a useful source of coherent runtime files: + +```text +artifacts/bin/testhost/--Release-/shared/Microsoft.NETCore.App// +``` + +Determine the actual path from the build; do not assume the directory version equals the benchmark TFM. For incremental component builds, verify that the testhost contains the updated files, or stage the verified component outputs directly. + +#### Stage an immutable overlay + +For a managed Socket-only experiment with no other changed dependencies: + +```bash +SHARED='/absolute/path/to/the/validated/shared/Microsoft.NETCore.App/' +OVERLAY=/absolute/path/to/runtime/artifacts/tmp/crank-sockets-candidate + +# A new directory for each binary variant; fail if it already exists. +mkdir "$OVERLAY" +cp "$SHARED/System.Net.Sockets.dll" "$SHARED/System.Net.Sockets.pdb" "$OVERLAY/" + +# Keep the manifest outside the upload directory. +(cd "$OVERLAY" && sha256sum *) > "$OVERLAY.sha256" +``` + +Create the parent directory first if necessary. Extend the explicit copy list according to the table; never replace it with a wildcard over the full testhost. Preserve each overlay and its symbols until the comparison and trace analysis are complete. Do not overwrite an overlay while a run uses it. + +#### Include symbols from the exact same build + +- Managed assemblies: retain their matching portable `.pdb` files. +- Linux native libraries: keep each matching `.so.dbg` alongside its `.so`, preserving names. A Release `.so` is normally stripped; the matching debug file supplies native symbols. Windows builds need the corresponding native PDBs instead. +- A PDB or `.dbg` from another build is not interchangeable, even if the assembly version or filename is identical. Preserve hashes and native build IDs for later analysis. +- ReadyToRun framework frames can require additional matching symbol-generation tooling. Follow the Linux tracing guide when frames remain unresolved; do not upload all build tools speculatively or assume managed PDBs alone resolve all native code. + +### 3. Upload the Overlay and Run JSON + +Crank's `--application.options.outputFiles` uploads local files into the application's published output after its build. It is different from uploading source files or build inputs. Leave the controller's SDK/shared frameworks untouched; no in-place SDK replacement is needed. + +Set the following variables to the chosen configuration, profile, and exact compatible versions. `CONFIG` can be a local configuration file or a URL; a pinned/local copy makes comparisons reproducible. The example uses Bash syntax: + +```bash +CONFIG=/absolute/path/to/platform.benchmarks.yml +PROFILE='' +TFM='' +SDK_VERSION='' +RUNTIME_VERSION='' +ASPNET_VERSION='' +RUN=/absolute/path/to/results/json-candidate-a + +crank \ + --config "$CONFIG" --scenario json --profile "$PROFILE" \ + --application.framework "$TFM" \ + --application.sdkVersion "$SDK_VERSION" \ + --application.runtimeVersion "$RUNTIME_VERSION" \ + --application.aspNetCoreVersion "$ASPNET_VERSION" \ + --application.options.outputFiles "$OVERLAY/*" \ + --application.environmentVariables DOTNET_EnableEventPipe=0 \ + --application.beforeScript '/bin/sh -c "sha256sum published/System.Net.Sockets.dll published/System.Net.Sockets.pdb"' \ + --application.options.downloadOutput true \ + --application.options.downloadOutputOutput "$RUN-output" \ + --application.options.downloadBuildLog true \ + --application.options.downloadBuildLogOutput "$RUN-build" \ + --load.variables.connections 512 \ + --load.variables.warmup 90 \ + --load.variables.duration 60 \ + --json "$RUN.json" +``` + +Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded output with the local manifest. + +The example's `published/` paths and `/bin/sh` command are for the standard Linux PlatformBenchmarks job; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. + +#### Windows controller with Linux binaries in WSL + +The payload must match the **agent**, not the controller OS. A Windows controller can upload Linux binaries directly from a WSL UNC path, without copying an entire build onto a Windows drive: + +```powershell +crank.exe --config C:\benchmarks\platform.yml --scenario json --profile ` + --application.options.outputFiles '\\wsl.localhost\Ubuntu\home\user\runtime\artifacts\tmp\crank-sockets-candidate\*' +``` + +This illustrates only the path syntax; retain the version, load, environment, and result options from the full command. Use paths accessible to the controller for configurations, uploads, and downloaded traces. From WSL, `wslpath -w ` converts a path for `crank.exe`. + +### 4. Set Environment Variables on the Application Job + +Repeat `--application.environmentVariables NAME=VALUE` for each variable. Setting a variable only in the controller's shell does **not** configure the remote application: + +```text +--application.environmentVariables DOTNET_USE_IO_URING=1 +--application.environmentVariables DOTNET_IORING_THREAD_COUNT=8 +--application.environmentVariables DOTNET_ThreadPool_ForceMinWorkerThreads=0x10 +``` + +These are examples for an io_uring experiment, not required defaults for every benchmark. Set only the controls relevant to the change. Check their parsing semantics: `DOTNET_ThreadPool_ForceMinWorkerThreads=0x10` means **16 decimal**; `=16` is parsed as hexadecimal and means 22. A minimum is not a cap on active workers. + +Restart the application for every configuration because runtime switches are generally read at startup. For a flag-only comparison, reuse identical binaries and change only that flag. Do not silently change ring counts, worker settings, ASP.NET binaries, or load parameters at the same time. + +### 5. Collect a Linux Trace with Crank + +Use a **separate profiling run**, retaining the same binary overlay and workload. Add these options to the complete benchmark command: + +```text +--application.dotnetTrace true +--application.dotnetTraceCollectMode collect-linux +--application.options.traceOutput /absolute/controller/path/json-candidate.nettrace +``` + +Choose a new `RUN` name for its JSON and logs as well. Crank starts `dotnet-trace collect-linux` on the agent and downloads the resulting trace to the controller. This mode captures Linux CPU samples, including native and kernel frames, rather than only managed EventPipe sampling. Use Crank's built-in collection instead of adding a startup hook or custom collector to the application. + +- Use controller/agent versions supporting `dotnetTraceCollectMode` and an agent with a supported `dotnet-trace` tool. The current Crank implementation requires Linux, effective UID 0, and kernel 6.4 or newer for this mode. Container perf permissions can still prevent collection. Consult the agent's error rather than silently substituting a different trace mode or changing shared-machine security settings. +- Keep `--application.environmentVariables DOTNET_EnableEventPipe=0` for this native CPU profiling run to avoid EventPipe noise. This is **not** appropriate for a separate EventPipe-only `collect` run, which needs EventPipe enabled. +- Upload and retain the matching PDB/`.dbg` files with the binaries. Keep the overlay/manifest beside the collected evidence for offline symbol lookup; do not assume the downloaded trace embeds every symbol. +- Allow collection to finalize. For large traces, increase the wait budgets if needed: + + ```text + --application.dotnetTraceStopTimeoutSec 600 + --application.collectTimeout 00:12:00 + ``` + +- Check the downloaded trace exists, is nonempty, and opens with useful samples and resolved target frames in a compatible viewer such as PerfView. Review collector errors and finalization warnings in the logs. A successful benchmark or controller exit code alone does not prove trace collection succeeded; a partial trace may still have been downloaded. +- Do not use traced throughput as the headline performance result. Trace collection changes overhead, and its window may include warmup; select the steady-state interval when analyzing CPU costs. + +### 6. Compare and Preserve Results + +Run at least two independent application launches per configuration, preferably in reverse-order pairs such as baseline/candidate/candidate/baseline. Keep source, binaries, machine profile, versions, connection count, warmup, measurement duration, and unrelated environment variables fixed. + +Report results in a table with throughput, relative change, CPU, p99 latency, errors, launch count, and run-to-run variation. Distinguish peak CPU from mean CPU over the measured window; an average of per-run p99 values is not a pooled percentile. Do not call a small single-run difference an established improvement. + +Preserve the exact command/configuration, source provenance, binary/symbol hashes, Crank JSON, application/build logs, and any trace files. Verify zero bad responses/socket errors and investigate failures rather than accepting misleading throughput. Keep failed profiling evidence separate from successful untraced measurements. From 06d7ed66994690dcd9fd2b1d3bd56e6693b6f71c Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Thu, 24 Sep 2026 13:02:24 +0200 Subject: [PATCH 04/15] Consolidate ASP.NET benchmark prerequisites and scenario guidance Share Release-build prerequisites and JSON scenario guidance across local and Crank workflows, clarify database requirements, and update local step references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 56 +++++++++++++----------- 1 file changed, 30 insertions(+), 26 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 01272ef1e67bf7..b4b80f7fb2035c 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -9,21 +9,33 @@ Use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](http These benchmarks are useful for changes to the socket/IO stack, GC, JIT, and other components whose impact depends on application behavior. Both workflows below run the benchmark against a locally-built runtime: start with local measurements to validate ideas, then use external infrastructure for final validation when access is available. +## Common Prerequisites + +Both workflows require a validated local dotnet/runtime build containing the change under test, targeting the benchmark machine's OS and architecture: the development machine for local runs, or the application agent for Crank runs. **Both the native runtime and managed libraries must be built in Release.** + +Reuse existing validated artifacts when available; do not rebuild merely to stage binaries. Consult the `build-and-test` skill before building, and never build when the user requested documentation or explicitly prohibited builds. If a build is needed, build runtime and libraries in Release with `-c release`, not just `-rc release`: the latter only forces the runtime to Release and leaves libraries at their default (`Debug`), producing a mismatched testhost. + +```bash +./build.sh clr+libs -c release +``` + +Confirm the artifacts correspond to the intended source revision. Switching Git branches does not update existing build outputs. Record source commits, any uncommitted changes, configuration, and binary hashes. + +## Choosing a Benchmark + +For both workflows, prefer `/json` for representative maximum-throughput comparisons. The standard TechEmpower `/plaintext` scenario uses HTTP pipelining (16 requests batched per connection), whereas `/json` sends one request per round trip and better represents normal request/response traffic. Their throughput numbers are not directly comparable. + +Both `/json` and `/plaintext` can run locally without a database. Other benchmarks, such as fortunes, single/multiple queries, and updates, require setting up a database. These can also run locally, but database setup is outside the scope of this document. + ## Benchmarking a Local Runtime Build Locally -**Always validate ideas locally first.** This workflow provides quick verification on the development machine and requires no VPN or access to external infrastructure. Drive the application with `wrk`, or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows because `wrk` is Linux/macOS-only. +**Always validate ideas locally first.** This workflow provides quick verification on the development machine and requires no VPN or access to external infrastructure. Drive the application with `wrk`, or [`bombardier`](https://github.com/codesenberg/bombardier) on Windows (because `wrk` is Linux/macOS-only). Treat local results as an initial signal, not a definitive performance measurement: the application, load generator, and other processes share the same machine and compete for CPU, memory, and other resources. This contention can distort throughput and latency, so local results may not predict performance on dedicated benchmark machines. ### Prerequisites -- A locally-built dotnet/runtime with the change under test (see the `build-and-test` skill). Build **everything** (runtime and libraries) in Release with `-c release`, not just `-rc release` — the latter only forces the runtime to Release and leaves libraries at their default (`Debug`), producing a mismatched testhost: - - ```bash - ./build.sh clr+libs -c release - ``` - -- A load-generation tool: on Linux, `wrk` (`sudo apt install wrk` or build from source — ). `wrk` doesn't build/run on Windows; use [`bombardier`](https://github.com/codesenberg/bombardier) instead (`go install github.com/codesenberg/bombardier@latest`, or download a prebuilt binary from its releases page) — the examples below use `wrk` syntax, with the `bombardier` equivalent noted alongside. +- A load-generation tool: on Linux, `wrk` (`sudo apt install wrk` or build from [source](https://github.com/wg/wrk)). `wrk` doesn't build/run on Windows; use [`bombardier`](https://github.com/codesenberg/bombardier) instead (`go install github.com/codesenberg/bombardier@latest`, or download a prebuilt binary from its [releases page](https://github.com/codesenberg/bombardier/releases)) - the examples below use `wrk` syntax, with the `bombardier` equivalent noted alongside. - The [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository, which contains the TechEmpower and other benchmark apps. ### Step 1: Clone the Benchmarks Repository @@ -32,13 +44,9 @@ Treat local results as an initial signal, not a definitive performance measureme git clone https://github.com/aspnet/Benchmarks.git ``` -The relevant app is `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). It targets the latest `net*.0` TFM and does **not** require a database for the `/json` and `/plaintext` endpoints, which is normally all you need. - -### Step 2: Prefer the `/json` Endpoint, Not `/plaintext` +The relevant app is `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). It targets the latest `net*.0` TFM. -Use `/json` for a "realistic max throughput" measurement. `/plaintext` conventionally uses HTTP pipelining (16 requests batched per connection in the official TechEmpower harness/`wrk` scripts), which does not reflect normal request/response traffic and inflates throughput numbers in a way that isn't representative. `/json` sends one request per round trip and is the more realistic comparison point. - -### Step 3: Publish the App Against the Repo's Own SDK +### Step 2: Publish the App Against the Repo's Own SDK Build/publish the app with the dotnet/runtime repo's own SDK (`/.dotnet/dotnet`), not a system-wide SDK, so the app targets the same TFM as your local build: @@ -49,7 +57,7 @@ cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks This produces `bin/Release//PlatformBenchmarks.dll`. -### Step 4: Make the Repo's SDK Run Against Your Local Runtime Build +### Step 3: Make the Repo's SDK Run Against Your Local Runtime Build `PlatformBenchmarks` needs `Microsoft.AspNetCore.App`, which the freshly-built `artifacts/bin/testhost` does **not** contain (it only has `Microsoft.NETCore.App`) — running it directly via `testhost`'s `corerun`/`dotnet` fails with a "no framework found" error. The repo SDK's own `dotnet` (under `/.dotnet`) already has `Microsoft.AspNetCore.App`, so the simplest way to combine "the ASP.NET Core framework" with "your locally-built `Microsoft.NETCore.App`" is to temporarily overlay the SDK's shared `Microsoft.NETCore.App/` folder with the testhost's freshly-built one: @@ -67,7 +75,7 @@ rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ **This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. -### Step 5: Run the App and Verify It's Serving Requests +### Step 4: Run the App and Verify It's Serving Requests ```bash cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks/bin/Release/ @@ -78,7 +86,7 @@ curl -sS http://127.0.0.1:5000/json Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. -### Step 6: Run `wrk` (or `bombardier` on Windows) Against It +### Step 5: Run `wrk` (or `bombardier` on Windows) Against It ```bash wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/json @@ -102,9 +110,9 @@ bombardier -c 256 -d 15s -l http://127.0.0.1:5000/json Run **at least two** runs per configuration (there's meaningful run-to-run variance) and report both, along with p50/p99 latency, not just a single throughput number. Present the results in a table. -Repeat Steps 5-6 for each configuration being compared (e.g. once with the env var on, once with it off), reusing the same overlay — only the running process needs to be restarted between configurations, not the overlay. +Repeat Steps 4-5 for each configuration being compared (e.g. once with the env var on, once with it off), reusing the same overlay — only the running process needs to be restarted between configurations, not the overlay. -### Step 7: Clean Up +### Step 6: Clean Up 1. Kill the server process(es) — find the actual `dotnet exec ... PlatformBenchmarks.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` and `kill `. 2. **Restore the SDK's shared framework from the backup** and verify it via checksum before considering the machine clean: @@ -151,7 +159,7 @@ If the load-test numbers show a difference (or don't, and you need to know why) #### ⚠️ Symbol Resolution Warnings -- **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/crossgen2_publish///crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 4) before collecting: +- **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/crossgen2_publish///crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 3) before collecting: ```bash cp artifacts/bin/crossgen2_publish/x64/Release/crossgen2 "$SDK_FX"/ @@ -165,10 +173,10 @@ If the load-test numbers show a difference (or don't, and you need to know why) ### Common Pitfalls -- **Running via `artifacts/bin/testhost` directly fails** with "no framework found" — that layout only has `Microsoft.NETCore.App`, not `Microsoft.AspNetCore.App`. Use the SDK-overlay approach in Step 4 instead. +- **Running via `artifacts/bin/testhost` directly fails** with "no framework found" — that layout only has `Microsoft.NETCore.App`, not `Microsoft.AspNetCore.App`. Use the SDK-overlay approach in Step 3 instead. - **Forgetting to set the env var before starting the process** — most feature switches (like `DOTNET_USE_IO_URING`) are read once at startup, so changing it and re-`curl`-ing the same running process has no effect. - **Comparing a single run per configuration** — throughput varies run to run; always do at least two runs per configuration. -- **Leaving the SDK shared framework overlaid** — always restore it (Step 7) and verify via checksum; a stale/mismatched overlay silently breaks unrelated work on the same machine later. +- **Leaving the SDK shared framework overlaid** — always restore it (Step 6) and verify via checksum; a stale/mismatched overlay silently breaks unrelated work on the same machine later. ## Benchmarking a Local Runtime Build on External Infrastructure with Crank @@ -176,8 +184,6 @@ If the load-test numbers show a difference (or don't, and you need to know why) Use [Crank](https://github.com/dotnet/crank) to deploy `PlatformBenchmarks` with selected locally-built runtime binaries to a remote application agent; a separate load agent drives HTTP requests without competing for the application's machine resources. These access restrictions apply to the shared infrastructure, not to Crank itself: external contributors can use Crank with their own agents. -Prefer `/json` for representative maximum-throughput comparisons. The standard `/plaintext` scenario pipelines 16 requests per connection; its throughput is not directly comparable to non-pipelined JSON traffic. Neither scenario needs a database. - References: - [Crank documentation](https://github.com/dotnet/crank/blob/main/docs/README.md) and [getting started](https://github.com/dotnet/crank/blob/main/docs/getting_started.md): controller installation, agents, scenarios, and profiles. @@ -187,8 +193,6 @@ References: ### 1. Establish the Inputs -- Use an existing, validated **Release** build for the agent's OS and architecture. Both native runtime and managed libraries must be Release. Consult the `build-and-test` skill if a build is needed; do not rebuild merely to stage binaries, and never build when the user requested documentation or explicitly prohibited builds. -- Confirm the artifacts correspond to the intended source revision. Switching Git branches does not update existing build outputs. Record source commits, any uncommitted changes, configuration, and binary hashes. - Use an installed Crank controller and reachable application/load agents. Prefer separate machines for throughput measurements so load generation does not compete with the server for CPU. - Select a PlatformBenchmarks configuration and an authorized machine profile. The public configuration includes profiles for particular infrastructure; choose or define one for the actual agents rather than assuming those machines are accessible. The getting-started guide shows how profiles assign `application` and `load` endpoints. - Pin the benchmark source revision and exact SDK, runtime, and ASP.NET versions for comparisons. Do not allow `main`, `latest`, or `edge` to change dependencies between runs. Ensure the benchmark TFM and selected versions are compatible with the local runtime overlay. From 921137454ef10075eb4f62513f9f3e4ffa8f830f Mon Sep 17 00:00:00 2001 From: Adam Sitnik Date: Thu, 24 Sep 2026 13:17:24 +0200 Subject: [PATCH 05/15] apply suggestions Updated performance benchmarking instructions for clarity and added specific skills for method/API-level and end-to-end testing. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- .github/skills/build-and-test/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 14625ae418f647..6b72e4aa39266c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -16,7 +16,7 @@ Use the `code-review` skill when reviewing pull requests, and — when running u When starting work in an unfamiliar directory, search for `README.md` files in it and its parents up to the repository root. Read any you find — they contain conventions, patterns, and architectural context relevant to your work. -If the changes are intended to improve performance, or if they could negatively impact performance, use the `microbenchmark` skill to validate the impact before completing. +If the changes are intended to improve performance, or if they could negatively impact performance, use the `microbenchmark` skill for method/API-level measurements or the `aspnet-benchmark` skill for end-to-end ASP.NET Core/HTTP server load testing. When writing or reviewing SIMD / hardware-intrinsics code (anything using `Vector128`/`Vector256`/`Vector512`, `Vector`, or the platform intrinsics in `System.Runtime.Intrinsics.*`), use the `vectorization` skill. diff --git a/.github/skills/build-and-test/SKILL.md b/.github/skills/build-and-test/SKILL.md index 67d49534c2d69d..ed39b3a12d024c 100644 --- a/.github/skills/build-and-test/SKILL.md +++ b/.github/skills/build-and-test/SKILL.md @@ -5,7 +5,7 @@ description: > per-component build and test commands, Windows/PowerShell equivalents, running individual tests, and troubleshooting. USE FOR: any build or test invocation in this repo, and before making code changes under CCA. DO NOT USE FOR: CI pipeline - triage (use ci-pipeline-monitor) or benchmarking (use microbenchmark). + triage (use ci-pipeline-monitor) or benchmarking (use microbenchmark for method/API-level measurements or aspnet-benchmark for end-to-end ASP.NET Core/HTTP load testing). --- # Building & Testing in dotnet/runtime From f66bacd338c7901bd995e7d8a93f2e19cd71d0ee Mon Sep 17 00:00:00 2001 From: Adam Sitnik Date: Thu, 24 Sep 2026 13:42:51 +0200 Subject: [PATCH 06/15] Fix framework issue by overlaying SDK and testhost Updated instructions to overlay SDK's Microsoft.NETCore.App with testhost's version to resolve framework issues. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index b4b80f7fb2035c..7721be7c820976 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -62,8 +62,9 @@ This produces `bin/Release//PlatformBenchmarks.dll`. `PlatformBenchmarks` needs `Microsoft.AspNetCore.App`, which the freshly-built `artifacts/bin/testhost` does **not** contain (it only has `Microsoft.NETCore.App`) — running it directly via `testhost`'s `corerun`/`dotnet` fails with a "no framework found" error. The repo SDK's own `dotnet` (under `/.dotnet`) already has `Microsoft.AspNetCore.App`, so the simplest way to combine "the ASP.NET Core framework" with "your locally-built `Microsoft.NETCore.App`" is to temporarily overlay the SDK's shared `Microsoft.NETCore.App/` folder with the testhost's freshly-built one: ```bash -SDK_FX=$(find /.dotnet/shared/Microsoft.NETCore.App -maxdepth 1 -type d | tail -1) -TESTHOST_FX="/artifacts/bin/testhost/--Release-/shared/Microsoft.NETCore.App/" +SDK_VERSION='' +SDK_FX="/.dotnet/shared/Microsoft.NETCore.App/$SDK_VERSION" +TESTHOST_FX="/artifacts/bin/testhost/--Release-/shared/Microsoft.NETCore.App/$SDK_VERSION" # ALWAYS back up the original SDK framework files first, so they can be restored exactly. mkdir -p /tmp/netcoreapp_sdk_backup From 539142e3cf7c96f17c7dee29bd5b79f90f238bea Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Thu, 24 Sep 2026 16:13:48 +0200 Subject: [PATCH 07/15] Address remaining Copilot review feedback on aspnet-benchmark skill - Use a unique, non-reused SDK backup directory (PID-suffixed) instead of a fixed /tmp path, so an interrupted prior run can't cause the overlay step to clobber the only pristine backup. - Record the pre-overlay System.Private.CoreLib.dll checksum in Step 3 so Step 6's restore verification actually has something to compare against. - Call out that Step 4 leaves the terminal cwd inside the PlatformBenchmarks output directory, and anchor the crossgen2 copy command to instead of a bare relative path that silently resolved under the wrong directory. - Document that PlatformBenchmarks publishes framework-dependent by default, so Crank's --application.options.outputFiles has no effect on framework assemblies (CoreLib, VM, GC, JIT, System.Net.Sockets, etc.) in that mode; add --application.selfContained true to the Crank command so the local runtime overlay actually lands in the published output it overwrites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 34 +++++++++++++++++------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 7721be7c820976..4a4391d39039bc 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -67,14 +67,21 @@ SDK_FX="/.dotnet/shared/Microsoft.NETCore.App/$SDK_VERSION" TESTHOST_FX="/artifacts/bin/testhost/--Release-/shared/Microsoft.NETCore.App/$SDK_VERSION" # ALWAYS back up the original SDK framework files first, so they can be restored exactly. -mkdir -p /tmp/netcoreapp_sdk_backup -rsync -a --delete "$SDK_FX"/ /tmp/netcoreapp_sdk_backup/ +# Use a fresh, uniquely-named backup directory every time: a fixed, reused path is unsafe +# if a previous run was interrupted before restoring — `mkdir -p` would silently succeed +# and the overlay step below would overwrite the only pristine backup with mutated files. +BACKUP_DIR="/tmp/netcoreapp_sdk_backup.$$" +mkdir "$BACKUP_DIR" # fails loudly (no -p) if this exact path somehow already exists +rsync -a --delete "$SDK_FX"/ "$BACKUP_DIR"/ + +# Record the pre-overlay checksum so Step 6's restore can be verified against it. +md5sum "$SDK_FX"/System.Private.CoreLib.dll # Overlay with the local build. rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ ``` -**This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. +**This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. Keep `$BACKUP_DIR` and the recorded checksum around until Step 6 has verified the restore. ### Step 4: Run the App and Verify It's Serving Requests @@ -85,6 +92,8 @@ sleep 5 curl -sS http://127.0.0.1:5000/json ``` +This leaves the terminal's working directory inside `PlatformBenchmarks/bin/Release/` for the rest of the session. Any later step that references a path relative to `` (such as copying `crossgen2` below) must either `cd` back to `` first or use an absolute/``-anchored path — a bare relative path resolves under this benchmark output directory instead and the copy silently fails or copies nothing. + Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. ### Step 5: Run `wrk` (or `bombardier` on Windows) Against It @@ -119,11 +128,11 @@ Repeat Steps 4-5 for each configuration being compared (e.g. once with the env v 2. **Restore the SDK's shared framework from the backup** and verify it via checksum before considering the machine clean: ```bash - rsync -a --delete /tmp/netcoreapp_sdk_backup/ "$SDK_FX"/ - md5sum "$SDK_FX"/System.Private.CoreLib.dll # compare against the value recorded before overlaying + rsync -a --delete "$BACKUP_DIR"/ "$SDK_FX"/ + md5sum "$SDK_FX"/System.Private.CoreLib.dll # must match the checksum recorded in Step 3 before overlaying ``` -3. Remove the temporary backup and any log files once restoration is verified. +3. Remove `$BACKUP_DIR` and any log files once restoration is verified. ### Profiling a Benchmark Run with `perfcollect` @@ -163,7 +172,9 @@ If the load-test numbers show a difference (or don't, and you need to know why) - **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/crossgen2_publish///crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 3) before collecting: ```bash - cp artifacts/bin/crossgen2_publish/x64/Release/crossgen2 "$SDK_FX"/ + # Use a path anchored to (or `cd` back there first) — a bare relative + # path resolves under the PlatformBenchmarks output directory left by Step 4 instead. + cp /artifacts/bin/crossgen2_publish/x64/Release/crossgen2 "$SDK_FX"/ ``` For a runtime you didn't build yourself, see the "Resolving Framework Symbols" section of [linux-performance-tracing.md](../../../docs/project/linux-performance-tracing.md) instead (it walks through obtaining a matching `crossgen2` via a self-contained publish). @@ -253,9 +264,11 @@ Create the parent directory first if necessary. Extend the explicit copy list ac ### 3. Upload the Overlay and Run JSON -Crank's `--application.options.outputFiles` uploads local files into the application's published output after its build. It is different from uploading source files or build inputs. Leave the controller's SDK/shared frameworks untouched; no in-place SDK replacement is needed. +Crank's `--application.options.outputFiles` uploads local files into the application's published output (the `published/` folder produced by `dotnet publish` on the agent) *after* that build completes. It is different from uploading source files or build inputs. Leave the controller's SDK/shared frameworks untouched; no in-place SDK replacement is needed. -Set the following variables to the chosen configuration, profile, and exact compatible versions. `CONFIG` can be a local configuration file or a URL; a pinned/local copy makes comparisons reproducible. The example uses Bash syntax: +**`PlatformBenchmarks` publishes framework-dependent by default, and `outputFiles` alone does not override the shared framework for a framework-dependent app.** A framework-dependent publish's own `published/` output does not contain a private copy of `Microsoft.NETCore.App`/`Microsoft.AspNetCore.App` at all — the host resolves `System.Private.CoreLib.dll`, `libcoreclr.so`, `libclrjit.so`, and other framework assemblies (including `System.Net.Sockets.dll`) exclusively from the shared framework directory selected by `--application.runtimeVersion`/`--application.aspNetCoreVersion`, which Crank provisions separately and does not touch when applying `outputFiles`. Dropping same-named files into `published/` in this mode has no effect: the run silently benchmarks the stock shared-framework binaries instead of the uploaded ones, for every row in the table above except application-local, non-framework assemblies. + +To make the overlay actually take effect, publish the job **self-contained** instead, so the local runtime bits become part of `published/` itself and the later `outputFiles` copy overwrites them in place: ```bash CONFIG=/absolute/path/to/platform.benchmarks.yml @@ -269,6 +282,7 @@ RUN=/absolute/path/to/results/json-candidate-a crank \ --config "$CONFIG" --scenario json --profile "$PROFILE" \ --application.framework "$TFM" \ + --application.selfContained true \ --application.sdkVersion "$SDK_VERSION" \ --application.runtimeVersion "$RUNTIME_VERSION" \ --application.aspNetCoreVersion "$ASPNET_VERSION" \ @@ -287,6 +301,8 @@ crank \ Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded output with the local manifest. +`--application.selfContained true` still needs `--application.runtimeVersion`/`--application.aspNetCoreVersion` pinned to the versions matching your overlay's ABI, since the self-contained publish step is what brings those shared-framework files into `published/` in the first place — `outputFiles` only replaces specific files afterward, it doesn't provision the rest of the runtime. Confirm the `beforeScript`'s `sha256sum` output in the build log matches the overlay's own manifest to prove the substitution actually landed, and check the downloaded build log for the `--self-contained` publish flag. + The example's `published/` paths and `/bin/sh` command are for the standard Linux PlatformBenchmarks job; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. #### Windows controller with Linux binaries in WSL From fe7c4c8f51a6398e357c4252b802a3b574901ee2 Mon Sep 17 00:00:00 2001 From: Adam Sitnik Date: Mon, 28 Sep 2026 11:50:47 +0200 Subject: [PATCH 08/15] Address ASP.NET benchmark review feedback Document both PlatformBenchmarks and Minimal APIs validation, preserve scenario-owned load settings, and link the Crank controller options directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 28 ++++++++++++++++-------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 4a4391d39039bc..7decc31afc5d12 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -1,14 +1,21 @@ --- name: aspnet-benchmark -description: Run the TechEmpower ASP.NET Core benchmarks (PlatformBenchmarks) against a locally-built dotnet/runtime, either locally with an SDK overlay or on external infrastructure using Crank, to validate the end-to-end impact of a runtime change (e.g. sockets, io_uring, GC, JIT) under realistic HTTP load. Use this when asked to benchmark, load-test, or profile an ASP.NET Core / Kestrel scenario, or to compare an env-var-gated feature (e.g. `DOTNET_USE_IO_URING`) end-to-end. +description: Run the TechEmpower ASP.NET Core benchmarks against a locally-built dotnet/runtime, either locally with an SDK overlay or on external infrastructure using Crank, to validate the end-to-end impact of a runtime change (e.g. sockets, io_uring, GC, JIT) under realistic HTTP load. Use this when asked to benchmark, load-test, or profile an ASP.NET Core / Kestrel scenario, or to compare an env-var-gated feature (e.g. `DOTNET_USE_IO_URING`) end-to-end. --- -# Running the TechEmpower (PlatformBenchmarks) Benchmarks +# Running ASP.NET Core Benchmarks -Use the TechEmpower `PlatformBenchmarks` app from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository to validate runtime changes under realistic HTTP server load. Microbenchmarks can isolate an operation's cost, but end-to-end ASP.NET benchmarks show how a change affects throughput, latency, and CPU usage under concurrent request/response processing. +Use the benchmark apps from the [`aspnet/Benchmarks`](https://github.com/aspnet/Benchmarks) repository to validate runtime changes under realistic HTTP server load. Microbenchmarks can isolate an operation's cost, but end-to-end ASP.NET benchmarks show how a change affects throughput, latency, and CPU usage under concurrent request/response processing. These benchmarks are useful for changes to the socket/IO stack, GC, JIT, and other components whose impact depends on application behavior. Both workflows below run the benchmark against a locally-built runtime: start with local measurements to validate ideas, then use external infrastructure for final validation when access is available. +Use both of these implementations when assessing a runtime change: + +- `PlatformBenchmarks` exercises a highly optimized raw Kestrel `HttpApplication`. +- `BasicMinimalApi` exercises the ASP.NET Core Minimal APIs stack. + +A final comparison must show that neither implementation regresses. An improvement in one does not compensate for a regression in the other. + ## Common Prerequisites Both workflows require a validated local dotnet/runtime build containing the change under test, targeting the benchmark machine's OS and architecture: the development machine for local runs, or the application agent for Crank runs. **Both the native runtime and managed libraries must be built in Release.** @@ -199,8 +206,10 @@ Use [Crank](https://github.com/dotnet/crank) to deploy `PlatformBenchmarks` with References: - [Crank documentation](https://github.com/dotnet/crank/blob/main/docs/README.md) and [getting started](https://github.com/dotnet/crank/blob/main/docs/getting_started.md): controller installation, agents, scenarios, and profiles. +- [Crank controller command-line arguments](https://github.com/dotnet/crank/blob/main/src/Microsoft.Crank.Controller/README.md): complete controller and per-job option reference. - [Selecting .NET versions](https://github.com/dotnet/crank/blob/main/docs/dotnet_versions.md): framework, SDK, runtime, and ASP.NET version overrides. - [PlatformBenchmarks configuration](https://github.com/aspnet/Benchmarks/blob/main/scenarios/platform.benchmarks.yml): `json` scenario, `application` and `load` jobs, and load variables. +- [Minimal APIs configuration](https://github.com/aspnet/Benchmarks/blob/main/scenarios/goldilocks.benchmarks.yml): `basicminimalapivanilla` and deployment-oriented Minimal APIs scenarios. - [Linux performance tracing](../../../docs/project/linux-performance-tracing.md): native and managed symbol resolution. ### 1. Establish the Inputs @@ -262,7 +271,7 @@ Create the parent directory first if necessary. Extend the explicit copy list ac - A PDB or `.dbg` from another build is not interchangeable, even if the assembly version or filename is identical. Preserve hashes and native build IDs for later analysis. - ReadyToRun framework frames can require additional matching symbol-generation tooling. Follow the Linux tracing guide when frames remain unresolved; do not upload all build tools speculatively or assume managed PDBs alone resolve all native code. -### 3. Upload the Overlay and Run JSON +### 3. Upload the Overlay and Run the Scenarios Crank's `--application.options.outputFiles` uploads local files into the application's published output (the `published/` folder produced by `dotnet publish` on the agent) *after* that build completes. It is different from uploading source files or build inputs. Leave the controller's SDK/shared frameworks untouched; no in-place SDK replacement is needed. @@ -271,7 +280,7 @@ Crank's `--application.options.outputFiles` uploads local files into the applica To make the overlay actually take effect, publish the job **self-contained** instead, so the local runtime bits become part of `published/` itself and the later `outputFiles` copy overwrites them in place: ```bash -CONFIG=/absolute/path/to/platform.benchmarks.yml +CONFIG=https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml PROFILE='' TFM='' SDK_VERSION='' @@ -293,18 +302,19 @@ crank \ --application.options.downloadOutputOutput "$RUN-output" \ --application.options.downloadBuildLog true \ --application.options.downloadBuildLogOutput "$RUN-build" \ - --load.variables.connections 512 \ - --load.variables.warmup 90 \ - --load.variables.duration 60 \ --json "$RUN.json" ``` -Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded output with the local manifest. +Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded output with the local manifest. Keep the scenario's configured connection count, warmup, and duration unless the experiment specifically requires changing them; the scenario owners tune these defaults and may update them over time. `--application.selfContained true` still needs `--application.runtimeVersion`/`--application.aspNetCoreVersion` pinned to the versions matching your overlay's ABI, since the self-contained publish step is what brings those shared-framework files into `published/` in the first place — `outputFiles` only replaces specific files afterward, it doesn't provision the rest of the runtime. Confirm the `beforeScript`'s `sha256sum` output in the build log matches the overlay's own manifest to prove the substitution actually landed, and check the downloaded build log for the `--self-contained` publish flag. The example's `published/` paths and `/bin/sh` command are for the standard Linux PlatformBenchmarks job; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. +Repeat the comparison with the Minimal APIs scenario. Reuse the same command and options, but set `CONFIG` to `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml`, select `--scenario basicminimalapivanilla`, and use a distinct `RUN` name. + +Do not accept the runtime change based on only one implementation. Compare baseline and candidate results for both `json` and `basicminimalapivanilla`, investigating any regression before concluding that the change is beneficial. + #### Windows controller with Linux binaries in WSL The payload must match the **agent**, not the controller OS. A Windows controller can upload Linux binaries directly from a WSL UNC path, without copying an entire build onto a Windows drive: From d58f229606643fc8578e8fcf92520753c6c08d85 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Tue, 29 Sep 2026 10:01:55 +0200 Subject: [PATCH 09/15] Document Minimal API benchmark workflows Add complete local and Crank instructions for comparing BasicMinimalApi with PlatformBenchmarks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 65 +++++++++++++++++------- 1 file changed, 47 insertions(+), 18 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 7decc31afc5d12..1af792f7bb24d4 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -51,18 +51,26 @@ Treat local results as an initial signal, not a definitive performance measureme git clone https://github.com/aspnet/Benchmarks.git ``` -The relevant app is `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). It targets the latest `net*.0` TFM. +The relevant apps are: -### Step 2: Publish the App Against the Repo's Own SDK +- `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). +- `src/BenchmarksApps/BasicMinimalApi` — an ASP.NET Core Minimal APIs application whose `/todos` endpoint returns JSON. -Build/publish the app with the dotnet/runtime repo's own SDK (`/.dotnet/dotnet`), not a system-wide SDK, so the app targets the same TFM as your local build: +### Step 2: Build the Apps Against the Repo's Own SDK + +Build both apps with the dotnet/runtime repo's own SDK (`/.dotnet/dotnet`), not a system-wide SDK, so they target the same TFM as your local build: ```bash -cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks -/.dotnet/dotnet build -c Release +TFM='' + +cd /src/BenchmarksApps/TechEmpower/PlatformBenchmarks +/.dotnet/dotnet build -c Release -p:TargetFrameworks="$TFM" + +cd /src/BenchmarksApps/BasicMinimalApi +/.dotnet/dotnet build -c Release -p:TargetFramework="$TFM" -p:TargetFrameworks="$TFM" ``` -This produces `bin/Release//PlatformBenchmarks.dll`. +This produces `bin/Release//PlatformBenchmarks.dll` and `bin/Release//BasicMinimalApi.dll` under their respective project directories. ### Step 3: Make the Repo's SDK Run Against Your Local Runtime Build @@ -90,25 +98,39 @@ rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ **This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. Keep `$BACKUP_DIR` and the recorded checksum around until Step 6 has verified the restore. -### Step 4: Run the App and Verify It's Serving Requests +### Step 4: Run Each App and Verify It's Serving Requests + +Run one app at a time because both commands below listen on port 5000. Start with `PlatformBenchmarks`: ```bash -cd Benchmarks/src/BenchmarksApps/TechEmpower/PlatformBenchmarks/bin/Release/ +cd /src/BenchmarksApps/TechEmpower/PlatformBenchmarks/bin/Release/"$TFM" /.dotnet/dotnet exec PlatformBenchmarks.dll --urls http://127.0.0.1:5000 & sleep 5 curl -sS http://127.0.0.1:5000/json ``` -This leaves the terminal's working directory inside `PlatformBenchmarks/bin/Release/` for the rest of the session. Any later step that references a path relative to `` (such as copying `crossgen2` below) must either `cd` back to `` first or use an absolute/``-anchored path — a bare relative path resolves under this benchmark output directory instead and the copy silently fails or copies nothing. +After running its load tests, stop `PlatformBenchmarks`, then run `BasicMinimalApi`: + +```bash +cd /src/BenchmarksApps/BasicMinimalApi/bin/Release/"$TFM" +/.dotnet/dotnet exec BasicMinimalApi.dll --urls http://127.0.0.1:5000 & +sleep 5 +curl -sS http://127.0.0.1:5000/todos +``` + +These commands leave the terminal's working directory inside the selected app's `bin/Release/` directory. Any later step that references a path relative to `` (such as copying `crossgen2` below) must either `cd` back to `` first or use an absolute/``-anchored path — a bare relative path resolves under the benchmark output directory instead and the copy silently fails or copies nothing. Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. -### Step 5: Run `wrk` (or `bombardier` on Windows) Against It +### Step 5: Run `wrk` (or `bombardier` on Windows) Against Each App ```bash wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/json +wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/todos ``` +Run the `/json` command while `PlatformBenchmarks` is running and the `/todos` command while `BasicMinimalApi` is running. + - `-t`: number of `wrk` threads — a good default is the machine's core count. - `-c`: number of concurrent connections — 256 is a reasonable default for a many-concurrent-connections scenario. - `-d15s`: run duration; 15s is normally enough to get a stable number. @@ -118,6 +140,7 @@ On Windows, use `bombardier` instead, with equivalent options: ```powershell bombardier -c 256 -d 15s -l http://127.0.0.1:5000/json +bombardier -c 256 -d 15s -l http://127.0.0.1:5000/todos ``` - `-c`: concurrent connections (same meaning as `wrk -c`). @@ -127,11 +150,11 @@ bombardier -c 256 -d 15s -l http://127.0.0.1:5000/json Run **at least two** runs per configuration (there's meaningful run-to-run variance) and report both, along with p50/p99 latency, not just a single throughput number. Present the results in a table. -Repeat Steps 4-5 for each configuration being compared (e.g. once with the env var on, once with it off), reusing the same overlay — only the running process needs to be restarted between configurations, not the overlay. +Repeat Steps 4-5 for each configuration being compared (e.g. once with the env var on, once with it off) and for both apps, reusing the same overlay — only the running process needs to be restarted between configurations, not the overlay. Do not conclude that a change is beneficial unless neither app regresses. ### Step 6: Clean Up -1. Kill the server process(es) — find the actual `dotnet exec ... PlatformBenchmarks.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` and `kill `. +1. Kill the server process — find the actual `dotnet exec ... PlatformBenchmarks.dll` or `dotnet exec ... BasicMinimalApi.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` or `pgrep -af BasicMinimalApi`, then use `kill `. 2. **Restore the SDK's shared framework from the backup** and verify it via checksum before considering the machine clean: ```bash @@ -280,16 +303,17 @@ Crank's `--application.options.outputFiles` uploads local files into the applica To make the overlay actually take effect, publish the job **self-contained** instead, so the local runtime bits become part of `published/` itself and the later `outputFiles` copy overwrites them in place: ```bash -CONFIG=https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml +CONFIG='' +SCENARIO='' PROFILE='' TFM='' SDK_VERSION='' RUNTIME_VERSION='' ASPNET_VERSION='' -RUN=/absolute/path/to/results/json-candidate-a +RUN='/absolute/path/to/results/-candidate-a' crank \ - --config "$CONFIG" --scenario json --profile "$PROFILE" \ + --config "$CONFIG" --scenario "$SCENARIO" --profile "$PROFILE" \ --application.framework "$TFM" \ --application.selfContained true \ --application.sdkVersion "$SDK_VERSION" \ @@ -305,13 +329,18 @@ crank \ --json "$RUN.json" ``` +Run the command once for each implementation, setting these values: + +| Implementation | `CONFIG` | `SCENARIO` | Example `RUN` suffix | +|---|---|---|---| +| PlatformBenchmarks | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml` | `json` | `json-candidate-a` | +| Minimal APIs | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml` | `basicminimalapivanilla` | `basicminimalapivanilla-candidate-a` | + Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded output with the local manifest. Keep the scenario's configured connection count, warmup, and duration unless the experiment specifically requires changing them; the scenario owners tune these defaults and may update them over time. `--application.selfContained true` still needs `--application.runtimeVersion`/`--application.aspNetCoreVersion` pinned to the versions matching your overlay's ABI, since the self-contained publish step is what brings those shared-framework files into `published/` in the first place — `outputFiles` only replaces specific files afterward, it doesn't provision the rest of the runtime. Confirm the `beforeScript`'s `sha256sum` output in the build log matches the overlay's own manifest to prove the substitution actually landed, and check the downloaded build log for the `--self-contained` publish flag. -The example's `published/` paths and `/bin/sh` command are for the standard Linux PlatformBenchmarks job; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. - -Repeat the comparison with the Minimal APIs scenario. Reuse the same command and options, but set `CONFIG` to `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml`, select `--scenario basicminimalapivanilla`, and use a distinct `RUN` name. +The example's `published/` paths and `/bin/sh` command are for the standard Linux jobs used by these scenarios; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. Do not accept the runtime change based on only one implementation. Compare baseline and candidate results for both `json` and `basicminimalapivanilla`, investigating any regression before concluding that the change is beneficial. From a3cae2c4b44015dd1ca02ea5f2901aefd9090921 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Tue, 29 Sep 2026 11:12:25 +0200 Subject: [PATCH 10/15] Address benchmark skill review feedback Verify the full SDK framework restore and use the target architecture when locating crossgen2. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 1af792f7bb24d4..3e491d9ed45add 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -89,9 +89,6 @@ BACKUP_DIR="/tmp/netcoreapp_sdk_backup.$$" mkdir "$BACKUP_DIR" # fails loudly (no -p) if this exact path somehow already exists rsync -a --delete "$SDK_FX"/ "$BACKUP_DIR"/ -# Record the pre-overlay checksum so Step 6's restore can be verified against it. -md5sum "$SDK_FX"/System.Private.CoreLib.dll - # Overlay with the local build. rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ ``` @@ -159,10 +156,10 @@ Repeat Steps 4-5 for each configuration being compared (e.g. once with the env v ```bash rsync -a --delete "$BACKUP_DIR"/ "$SDK_FX"/ - md5sum "$SDK_FX"/System.Private.CoreLib.dll # must match the checksum recorded in Step 3 before overlaying + diff --recursive --brief "$BACKUP_DIR" "$SDK_FX" # no output and exit code 0 means an exact content restore ``` -3. Remove `$BACKUP_DIR` and any log files once restoration is verified. +3. Remove `$BACKUP_DIR` and any log files only after the recursive comparison succeeds. ### Profiling a Benchmark Run with `perfcollect` @@ -204,7 +201,7 @@ If the load-test numbers show a difference (or don't, and you need to know why) ```bash # Use a path anchored to (or `cd` back there first) — a bare relative # path resolves under the PlatformBenchmarks output directory left by Step 4 instead. - cp /artifacts/bin/crossgen2_publish/x64/Release/crossgen2 "$SDK_FX"/ + cp /artifacts/bin/crossgen2_publish//Release/crossgen2 "$SDK_FX"/ ``` For a runtime you didn't build yourself, see the "Resolving Framework Symbols" section of [linux-performance-tracing.md](../../../docs/project/linux-performance-tracing.md) instead (it walks through obtaining a matching `crossgen2` via a self-contained publish). From 3f7f5e84b503dec34b450e8bd3fac0bdb07808df Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Tue, 29 Sep 2026 14:19:46 +0200 Subject: [PATCH 11/15] Correct crossgen2 profiling path Point the benchmark profiling workflow at the published CoreCLR crossgen2 artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 3e491d9ed45add..7409932edd640e 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -196,12 +196,12 @@ If the load-test numbers show a difference (or don't, and you need to know why) #### ⚠️ Symbol Resolution Warnings -- **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/crossgen2_publish///crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 3) before collecting: +- **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/coreclr/../crossgen2-published/crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 3) before collecting: ```bash # Use a path anchored to (or `cd` back there first) — a bare relative # path resolves under the PlatformBenchmarks output directory left by Step 4 instead. - cp /artifacts/bin/crossgen2_publish//Release/crossgen2 "$SDK_FX"/ + cp /artifacts/bin/coreclr/..Release/crossgen2-published/crossgen2 "$SDK_FX"/ ``` For a runtime you didn't build yourself, see the "Resolving Framework Symbols" section of [linux-performance-tracing.md](../../../docs/project/linux-performance-tracing.md) instead (it walks through obtaining a matching `crossgen2` via a self-contained publish). From 93d297ce07805d0f0a46c4ade1850501d13dc884 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Tue, 29 Sep 2026 15:15:26 +0200 Subject: [PATCH 12/15] Correct ASP.NET profiling guidance Document perfcollect's current symbol flow and identify the Crank log that contains overlay hashes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 7409932edd640e..f9c8375e78f7ef 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -115,8 +115,6 @@ sleep 5 curl -sS http://127.0.0.1:5000/todos ``` -These commands leave the terminal's working directory inside the selected app's `bin/Release/` directory. Any later step that references a path relative to `` (such as copying `crossgen2` below) must either `cd` back to `` first or use an absolute/``-anchored path — a bare relative path resolves under the benchmark output directory instead and the copy silently fails or copies nothing. - Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. ### Step 5: Run `wrk` (or `bombardier` on Windows) Against Each App @@ -196,18 +194,9 @@ If the load-test numbers show a difference (or don't, and you need to know why) #### ⚠️ Symbol Resolution Warnings -- **Precompiled (R2R/crossgen) framework symbols are not resolved automatically.** `perfcollect` needs a `crossgen2` tool matching the exact runtime build to map native framework code back to method names; without it, framework frames show up unresolved/hex-only in the trace. When profiling a **locally-built** runtime you don't need to hunt one down — your own build already produced the exact matching binary at `artifacts/bin/coreclr/../crossgen2-published/crossgen2`. Copy (or symlink) it next to `libcoreclr.so` in the directory you're actually running from (e.g. the testhost/SDK-overlay shared framework folder from Step 3) before collecting: - - ```bash - # Use a path anchored to (or `cd` back there first) — a bare relative - # path resolves under the PlatformBenchmarks output directory left by Step 4 instead. - cp /artifacts/bin/coreclr/..Release/crossgen2-published/crossgen2 "$SDK_FX"/ - ``` - - For a runtime you didn't build yourself, see the "Resolving Framework Symbols" section of [linux-performance-tracing.md](../../../docs/project/linux-performance-tracing.md) instead (it walks through obtaining a matching `crossgen2` via a self-contained publish). +- **JIT-generated (Tier0/Tier1) and precompiled R2R framework symbols require `DOTNET_PerfMapEnabled=1` to be set *before* the process starts.** This makes the runtime emit the perf map and JIT dump that `perfcollect` uses for symbol resolution; no separate `crossgen2` copy is required. If you forget this and only realize partway through a run, the trace from that run cannot be fixed after the fact — kill the process, re-export the env var, and restart it before collecting again. - **Native runtime frames (`libcoreclr.so`, `libclrjit.so`, etc.) resolve out of the box, as long as you don't move or discard the build's `.dbg` files.** A Release native build ships each `.so` *stripped*, but the matching, un-stripped `libXyz.so.dbg` is written right alongside it in the same output directory (e.g. `artifacts/bin/coreclr/./`), linked via its `.gnu_debuglink` section and a matching Build ID. `perf`/`perfcollect` follow that link automatically as long as the `.dbg` file stays next to its `.so` — don't clean up or selectively copy only the `.so` files into a deployment layout, or you'll silently lose native symbolication. -- **JIT-generated (Tier0/Tier1) symbols require `DOTNET_PerfMapEnabled=1` to be set *before* the process starts.** If you forget this and only realize partway through a run, the trace from that run cannot be fixed after the fact — kill the process, re-export the env var, and restart it before collecting again. -- Even after doing all of the above, expect *some* residual unresolved frames from components you didn't build locally (e.g. the OS's own libraries, or a `crossgen2`/`.dbg` mismatch if you mixed binaries from different build configurations) — this doesn't mean the whole trace is useless; managed app-level and JIT-emitted frames still resolve correctly via the perf map regardless. +- Even after doing all of the above, expect *some* residual unresolved frames from components you didn't build locally (e.g. the OS's own libraries, or a missing/mismatched `.dbg` file if you mixed binaries from different build configurations) — this doesn't mean the whole trace is useless; managed app-level, R2R, and JIT-emitted frames still resolve correctly via the generated symbol data. - `DOTNET_PerfMapEnabled=1` has a real (if usually small) overhead of its own — don't leave it set for the throughput (`wrk`/`bombardier`) runs themselves, only for the dedicated profiling run. ### Common Pitfalls @@ -333,9 +322,9 @@ Run the command once for each implementation, setting these values: | PlatformBenchmarks | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml` | `json` | `json-candidate-a` | | Minimal APIs | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml` | `basicminimalapivanilla` | `basicminimalapivanilla-candidate-a` | -Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded output with the local manifest. Keep the scenario's configured connection count, warmup, and duration unless the experiment specifically requires changing them; the scenario owners tune these defaults and may update them over time. +Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded job output log with the local manifest. Keep the scenario's configured connection count, warmup, and duration unless the experiment specifically requires changing them; the scenario owners tune these defaults and may update them over time. -`--application.selfContained true` still needs `--application.runtimeVersion`/`--application.aspNetCoreVersion` pinned to the versions matching your overlay's ABI, since the self-contained publish step is what brings those shared-framework files into `published/` in the first place — `outputFiles` only replaces specific files afterward, it doesn't provision the rest of the runtime. Confirm the `beforeScript`'s `sha256sum` output in the build log matches the overlay's own manifest to prove the substitution actually landed, and check the downloaded build log for the `--self-contained` publish flag. +`--application.selfContained true` still needs `--application.runtimeVersion`/`--application.aspNetCoreVersion` pinned to the versions matching your overlay's ABI, since the self-contained publish step is what brings those shared-framework files into `published/` in the first place — `outputFiles` only replaces specific files afterward, it doesn't provision the rest of the runtime. Confirm the `beforeScript`'s `sha256sum` output in the downloaded job output log matches the overlay's own manifest to prove the substitution actually landed, and check the downloaded build log for the `--self-contained` publish flag. The example's `published/` paths and `/bin/sh` command are for the standard Linux jobs used by these scenarios; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. From eaa2e27c9c54616e770e9283b75e6dc3bb6f23b7 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Tue, 29 Sep 2026 20:20:09 +0200 Subject: [PATCH 13/15] Address ASP.NET benchmark review feedback Make endpoint health checks fail on HTTP errors and describe Crank's self-contained default accurately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index f9c8375e78f7ef..79376ed38ff7c0 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -103,7 +103,7 @@ Run one app at a time because both commands below listen on port 5000. Start wit cd /src/BenchmarksApps/TechEmpower/PlatformBenchmarks/bin/Release/"$TFM" /.dotnet/dotnet exec PlatformBenchmarks.dll --urls http://127.0.0.1:5000 & sleep 5 -curl -sS http://127.0.0.1:5000/json +curl --fail-with-body -sS http://127.0.0.1:5000/json ``` After running its load tests, stop `PlatformBenchmarks`, then run `BasicMinimalApi`: @@ -112,7 +112,7 @@ After running its load tests, stop `PlatformBenchmarks`, then run `BasicMinimalA cd /src/BenchmarksApps/BasicMinimalApi/bin/Release/"$TFM" /.dotnet/dotnet exec BasicMinimalApi.dll --urls http://127.0.0.1:5000 & sleep 5 -curl -sS http://127.0.0.1:5000/todos +curl --fail-with-body -sS http://127.0.0.1:5000/todos ``` Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. @@ -284,9 +284,9 @@ Create the parent directory first if necessary. Extend the explicit copy list ac Crank's `--application.options.outputFiles` uploads local files into the application's published output (the `published/` folder produced by `dotnet publish` on the agent) *after* that build completes. It is different from uploading source files or build inputs. Leave the controller's SDK/shared frameworks untouched; no in-place SDK replacement is needed. -**`PlatformBenchmarks` publishes framework-dependent by default, and `outputFiles` alone does not override the shared framework for a framework-dependent app.** A framework-dependent publish's own `published/` output does not contain a private copy of `Microsoft.NETCore.App`/`Microsoft.AspNetCore.App` at all — the host resolves `System.Private.CoreLib.dll`, `libcoreclr.so`, `libclrjit.so`, and other framework assemblies (including `System.Net.Sockets.dll`) exclusively from the shared framework directory selected by `--application.runtimeVersion`/`--application.aspNetCoreVersion`, which Crank provisions separately and does not touch when applying `outputFiles`. Dropping same-named files into `published/` in this mode has no effect: the run silently benchmarks the stock shared-framework binaries instead of the uploaded ones, for every row in the table above except application-local, non-framework assemblies. +Crank currently makes .NET jobs self-contained by default. If a configuration or command-line override explicitly sets `selfContained=false`, however, `outputFiles` cannot replace shared-framework binaries: a framework-dependent publish's `published/` directory has no private copy of `Microsoft.NETCore.App`/`Microsoft.AspNetCore.App`, and the host resolves `System.Private.CoreLib.dll`, `libcoreclr.so`, `libclrjit.so`, and other framework assemblies (including `System.Net.Sockets.dll`) from the separately provisioned shared framework. -To make the overlay actually take effect, publish the job **self-contained** instead, so the local runtime bits become part of `published/` itself and the later `outputFiles` copy overwrites them in place: +Keep the job explicitly **self-contained** so the local runtime bits are part of `published/` and the later `outputFiles` copy overwrites them in place, regardless of configuration overrides: ```bash CONFIG='' From 1b5444d19d654134c7654eac004d628318612d33 Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Tue, 29 Sep 2026 20:43:40 +0200 Subject: [PATCH 14/15] Clarify SDK restore verification Describe the recursive content comparison used by the cleanup workflow instead of referring to a checksum. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 79376ed38ff7c0..1a9a3c172dee1d 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -93,7 +93,7 @@ rsync -a --delete "$SDK_FX"/ "$BACKUP_DIR"/ rsync -a --delete "$TESTHOST_FX"/ "$SDK_FX"/ ``` -**This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. Keep `$BACKUP_DIR` and the recorded checksum around until Step 6 has verified the restore. +**This mutates the repo's own SDK shared framework in place.** Never skip the backup step, and always restore it when done (Step 6) — leaving it mutated will silently break every other use of that SDK on the machine. Keep `$BACKUP_DIR` until Step 6 has verified the restored contents. ### Step 4: Run Each App and Verify It's Serving Requests @@ -150,7 +150,7 @@ Repeat Steps 4-5 for each configuration being compared (e.g. once with the env v ### Step 6: Clean Up 1. Kill the server process — find the actual `dotnet exec ... PlatformBenchmarks.dll` or `dotnet exec ... BasicMinimalApi.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` or `pgrep -af BasicMinimalApi`, then use `kill `. -2. **Restore the SDK's shared framework from the backup** and verify it via checksum before considering the machine clean: +2. **Restore the SDK's shared framework from the backup** and verify it with a recursive content comparison before considering the machine clean: ```bash rsync -a --delete "$BACKUP_DIR"/ "$SDK_FX"/ @@ -204,7 +204,7 @@ If the load-test numbers show a difference (or don't, and you need to know why) - **Running via `artifacts/bin/testhost` directly fails** with "no framework found" — that layout only has `Microsoft.NETCore.App`, not `Microsoft.AspNetCore.App`. Use the SDK-overlay approach in Step 3 instead. - **Forgetting to set the env var before starting the process** — most feature switches (like `DOTNET_USE_IO_URING`) are read once at startup, so changing it and re-`curl`-ing the same running process has no effect. - **Comparing a single run per configuration** — throughput varies run to run; always do at least two runs per configuration. -- **Leaving the SDK shared framework overlaid** — always restore it (Step 6) and verify via checksum; a stale/mismatched overlay silently breaks unrelated work on the same machine later. +- **Leaving the SDK shared framework overlaid** — always restore it (Step 6) and verify with a recursive content comparison; a stale/mismatched overlay silently breaks unrelated work on the same machine later. ## Benchmarking a Local Runtime Build on External Infrastructure with Crank From 3e1e99fb428c1bca14892d4e1994622c647013be Mon Sep 17 00:00:00 2001 From: adamsitnik Date: Wed, 30 Sep 2026 16:58:14 +0200 Subject: [PATCH 15/15] aspnet-benchmark skill: use TechEmpower/Minimal instead of BasicMinimalApi Addresses review feedback: BasicMinimalApi lacks a /json endpoint and is only otherwise used by the goldilocks Crank scenario for AOT/trimming validation. Switch the local benchmarking workflow to src/BenchmarksApps/TechEmpower/Minimal, which implements the full TechEmpower suite (including /json) alongside PlatformBenchmarks. For the Crank workflow, use the Minimal project's own minimal.benchmarks.yml (colocated with the project, not under the top-level scenarios/ folder), which defines a matching `json` scenario, instead of the unrelated goldilocks.benchmarks.yml / BasicMinimalApi scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/aspnet-benchmark/SKILL.md | 32 ++++++++++++------------ 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/.github/skills/aspnet-benchmark/SKILL.md b/.github/skills/aspnet-benchmark/SKILL.md index 1a9a3c172dee1d..fd0e72dfcc7154 100644 --- a/.github/skills/aspnet-benchmark/SKILL.md +++ b/.github/skills/aspnet-benchmark/SKILL.md @@ -12,7 +12,7 @@ These benchmarks are useful for changes to the socket/IO stack, GC, JIT, and oth Use both of these implementations when assessing a runtime change: - `PlatformBenchmarks` exercises a highly optimized raw Kestrel `HttpApplication`. -- `BasicMinimalApi` exercises the ASP.NET Core Minimal APIs stack. +- `Minimal` exercises the ASP.NET Core Minimal APIs stack. A final comparison must show that neither implementation regresses. An improvement in one does not compensate for a regression in the other. @@ -54,7 +54,7 @@ git clone https://github.com/aspnet/Benchmarks.git The relevant apps are: - `src/BenchmarksApps/TechEmpower/PlatformBenchmarks` — a raw Kestrel `HttpApplication` implementation of the TechEmpower benchmark suite (JSON serialization, plaintext, fortunes, single/multiple queries, updates). -- `src/BenchmarksApps/BasicMinimalApi` — an ASP.NET Core Minimal APIs application whose `/todos` endpoint returns JSON. +- `src/BenchmarksApps/TechEmpower/Minimal` — an ASP.NET Core Minimal APIs implementation of the same TechEmpower benchmark suite. ### Step 2: Build the Apps Against the Repo's Own SDK @@ -66,11 +66,11 @@ TFM='' cd /src/BenchmarksApps/TechEmpower/PlatformBenchmarks /.dotnet/dotnet build -c Release -p:TargetFrameworks="$TFM" -cd /src/BenchmarksApps/BasicMinimalApi +cd /src/BenchmarksApps/TechEmpower/Minimal /.dotnet/dotnet build -c Release -p:TargetFramework="$TFM" -p:TargetFrameworks="$TFM" ``` -This produces `bin/Release//PlatformBenchmarks.dll` and `bin/Release//BasicMinimalApi.dll` under their respective project directories. +This produces `bin/Release//PlatformBenchmarks.dll` and `bin/Release//Minimal.dll` under their respective project directories. ### Step 3: Make the Repo's SDK Run Against Your Local Runtime Build @@ -106,13 +106,13 @@ sleep 5 curl --fail-with-body -sS http://127.0.0.1:5000/json ``` -After running its load tests, stop `PlatformBenchmarks`, then run `BasicMinimalApi`: +After running its load tests, stop `PlatformBenchmarks`, then run `Minimal`: ```bash -cd /src/BenchmarksApps/BasicMinimalApi/bin/Release/"$TFM" -/.dotnet/dotnet exec BasicMinimalApi.dll --urls http://127.0.0.1:5000 & +cd /src/BenchmarksApps/TechEmpower/Minimal/bin/Release/"$TFM" +/.dotnet/dotnet exec Minimal.dll --urls http://127.0.0.1:5000 & sleep 5 -curl --fail-with-body -sS http://127.0.0.1:5000/todos +curl --fail-with-body -sS http://127.0.0.1:5000/json ``` Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_URING=1`/`=0`) *before* starting the process — it's read once at startup. @@ -121,21 +121,21 @@ Set whatever env var/`AppContext` switch you're comparing (e.g. `DOTNET_USE_IO_U ```bash wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/json -wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/todos +wrk -t12 -c256 -d15s --latency http://127.0.0.1:5000/json ``` -Run the `/json` command while `PlatformBenchmarks` is running and the `/todos` command while `BasicMinimalApi` is running. +Run the first `/json` command while `PlatformBenchmarks` is running and the second while `Minimal` is running. - `-t`: number of `wrk` threads — a good default is the machine's core count. - `-c`: number of concurrent connections — 256 is a reasonable default for a many-concurrent-connections scenario. - `-d15s`: run duration; 15s is normally enough to get a stable number. - `--latency`: also reports the latency percentile breakdown, not just aggregate throughput. -On Windows, use `bombardier` instead, with equivalent options: +On Windows, use `bombardier` instead, with equivalent options (run the first command against `PlatformBenchmarks` and the second against `Minimal`): ```powershell bombardier -c 256 -d 15s -l http://127.0.0.1:5000/json -bombardier -c 256 -d 15s -l http://127.0.0.1:5000/todos +bombardier -c 256 -d 15s -l http://127.0.0.1:5000/json ``` - `-c`: concurrent connections (same meaning as `wrk -c`). @@ -149,7 +149,7 @@ Repeat Steps 4-5 for each configuration being compared (e.g. once with the env v ### Step 6: Clean Up -1. Kill the server process — find the actual `dotnet exec ... PlatformBenchmarks.dll` or `dotnet exec ... BasicMinimalApi.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` or `pgrep -af BasicMinimalApi`, then use `kill `. +1. Kill the server process — find the actual `dotnet exec ... PlatformBenchmarks.dll` or `dotnet exec ... Minimal.dll` PID (not the shell that launched it) with `pgrep -af PlatformBenchmarks` or `pgrep -af Minimal.dll`, then use `kill `. 2. **Restore the SDK's shared framework from the backup** and verify it with a recursive content comparison before considering the machine clean: ```bash @@ -218,7 +218,7 @@ References: - [Crank controller command-line arguments](https://github.com/dotnet/crank/blob/main/src/Microsoft.Crank.Controller/README.md): complete controller and per-job option reference. - [Selecting .NET versions](https://github.com/dotnet/crank/blob/main/docs/dotnet_versions.md): framework, SDK, runtime, and ASP.NET version overrides. - [PlatformBenchmarks configuration](https://github.com/aspnet/Benchmarks/blob/main/scenarios/platform.benchmarks.yml): `json` scenario, `application` and `load` jobs, and load variables. -- [Minimal APIs configuration](https://github.com/aspnet/Benchmarks/blob/main/scenarios/goldilocks.benchmarks.yml): `basicminimalapivanilla` and deployment-oriented Minimal APIs scenarios. +- [Minimal APIs configuration](https://github.com/aspnet/Benchmarks/blob/main/src/BenchmarksApps/TechEmpower/Minimal/minimal.benchmarks.yml): `json`, `plaintext`, `fortunes`, and other scenarios for the same `Minimal` project used for local runs above. Unlike `PlatformBenchmarks`'s config, this one lives alongside the project rather than under the top-level `scenarios/` folder. - [Linux performance tracing](../../../docs/project/linux-performance-tracing.md): native and managed symbol resolution. ### 1. Establish the Inputs @@ -320,7 +320,7 @@ Run the command once for each implementation, setting these values: | Implementation | `CONFIG` | `SCENARIO` | Example `RUN` suffix | |---|---|---|---| | PlatformBenchmarks | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml` | `json` | `json-candidate-a` | -| Minimal APIs | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml` | `basicminimalapivanilla` | `basicminimalapivanilla-candidate-a` | +| Minimal APIs | `https://raw.githubusercontent.com/aspnet/Benchmarks/main/src/BenchmarksApps/TechEmpower/Minimal/minimal.benchmarks.yml` | `json` | `minimal-json-candidate-a` | Create the results directory beforehand and use a unique run name for every launch. Replace angle-bracket placeholders before executing. Quote `"$OVERLAY/*"` so Crank, not the shell, expands the upload pattern. Extend the `sha256sum` list to cover the complete selected payload and compare the downloaded job output log with the local manifest. Keep the scenario's configured connection count, warmup, and duration unless the experiment specifically requires changing them; the scenario owners tune these defaults and may update them over time. @@ -328,7 +328,7 @@ Create the results directory beforehand and use a unique run name for every laun The example's `published/` paths and `/bin/sh` command are for the standard Linux jobs used by these scenarios; adapt them if the job uses a different layout or OS. Confirm the build logs show the intended framework versions and that the application uses the deployed replacements, not an incompatible or separately located shared framework. File presence alone is not proof that a module was loaded; use module paths/build IDs from a diagnostic trace when investigating binding. -Do not accept the runtime change based on only one implementation. Compare baseline and candidate results for both `json` and `basicminimalapivanilla`, investigating any regression before concluding that the change is beneficial. +Do not accept the runtime change based on only one implementation. Compare baseline and candidate results for both implementations' `json` scenarios, investigating any regression before concluding that the change is beneficial. #### Windows controller with Linux binaries in WSL