Skip to content

Add Basic Blocking Locust perf test - #8158

Merged
Amaury Chamayou (achamayou) merged 10 commits into
mainfrom
locust-blocking-perf
Aug 19, 2026
Merged

Add Basic Blocking Locust perf test#8158
Amaury Chamayou (achamayou) merged 10 commits into
mainfrom
locust-blocking-perf

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Adds pi_basic_blocking_locust (perf label Basic Blocking Locust), which measures blocking writes (PUT /records/blocking/{key}) with locust instead of piccolo. It is intended to replace pi_basic_blocking for blocking workloads, where the piccolo client materially limits the figure it reports.

Locust is already a test dependency and is already used in tests/infra/service_load.py. No new dependencies.

All numbers below are from the bench-ab job on the CI runner, not from a developer machine.

Rebased onto current main. The radar-chart handling for benchmarks with no main history already landed in #8171 and is no longer part of this PR.

Why not piccolo for blocking workloads

The piccolo client limits the result. submit polls a non-blocking socket without ever sleeping (src/clients/tls_client.h loops on BIO_read while BIO_should_retry). For pipelined tests a client is rarely idle, so this costs little. For blocking writes a client spends nearly all of its time waiting for commit, and spins throughout.

Both tests run in the same CI job, on the same machine, against the same build. A blocking client sends one request at a time and each returns on commit, so with a signature every 100ms the closed-loop prediction is simply the client count divided by 0.1s:

test clients tx/s predicted achieved
Basic Blocking (piccolo) 128 1,008 1,280 79%
Basic Blocking Locust 128 1,244 1,280 97%
Basic Blocking Locust 320 3,052 3,200 95%

At the same client count on the same machine, locust reports 23% more throughput and tracks the model to within 3%. The remaining 21% that piccolo leaves on the table is client overhead, not the service.

Its headline metric includes client startup. average_throughput_tx/s divides total requests by wall-clock from first send to last receive, so the staggered startup of 128 client processes counts as measured time. basicperf.py already records the honest figure alongside it as all_clients_active_average_throughput_tx/s. The distortion grows as a run gets faster, so the metric understates improvements, which is the wrong way round for regression detection.

Concurrency is fixed at generation time. Piccolo takes one pre-generated parquet file per client, so changing the client count means regenerating the workload. Locust takes it as an argument, which is what makes the sweep below possible.

What it runs

setting value why
clients 320 each holds one blocking write in flight
locust processes 10 a locust process drives all of its users from one thread, so a single process becomes the limit before the service does
signature intervals 2ms, 20ms, 100ms each against its own network; see below
snapshot interval 10000 transactions matches add_piccolo_test; add_e2e_test passes nothing and e2e_args defaults to 10, which fsyncs a ~213KB snapshot every 10 transactions and measures the disk
measurement window 20s, after all clients have spawned

FastHttpUser (geventhttpclient) is used rather than HttpUser (requests), which cannot drive enough requests per second to saturate the service. mTLS is configured through ssl_context_factory.

The window is timed from locust's spawning_complete, and --reset-stats discards everything recorded during the ramp. Locust's own --run-time cannot be used, because it starts counting when locust starts and so includes the ramp: with a slow spawn rate a run would end mid-ramp and report a plausible looking figure for a client count it never reached. It is passed only as a backstop. The run fails, rather than reporting, if it ends before all clients have spawned, if the window achieved is short, or if any request failed.

Results

From the CI run at f23fce1, before the 20ms point was added:

signature interval tx/s p50 p99 clients / interval
2ms 24,354 11ms 21ms -
100ms 3,049 99ms 100ms 3,200

The points measure two different things, which is why they are swept. At 100ms the workload is latency-bound: throughput is the client count divided by the interval, to within 5%, and p50 sits on the interval, so this detects regressions in the commit path. At 2ms it leaves that regime, since p50 is 11ms rather than 2ms, so the node itself is the limit and the figure measures capacity. A regression in request handling shows up there and nowhere else.

20ms is the new middle point, to see whether it is still interval-bound (p50 ~ 20ms, ~16,000 tx/s) or already at the capacity knee. The 1000ms point was dropped: it only restated the latency-bound regime already measured at 100ms.

The capacity point is genuinely saturated rather than merely faster. The previous CI run measured the same configuration with a 5ms interval instead of 2ms and reported 24,388 tx/s against 24,354, a difference of 0.1%. Shortening the interval further changes nothing because the node, not the timer, is already the constraint.

CI reproduces all of this closely. Across consecutive runs, the 100ms point gave 1,241 / 1,245 / 1,247 tx/s at 128 clients and 3,049 / 3,055 at 320, and the capacity point gave 17,338 / 17,368 tx/s at 128 clients. Every one of those pairs agrees to within 0.5%, which is what makes the benchmark usable for regression detection.

Raising the client count from 128 to 320 moves the capacity point from 17,338 to 24,354 tx/s, a 41% gain for 2.5x the clients, while p50 rises from 6ms to 11ms. The service is therefore past its knee at 320 clients, which is where a capacity benchmark should sit.

Structure

  • tests/infra/basicperf_locustfile.py defines the load: one blocking write in flight per user, and the run's stop timing.
  • tests/basicperf_locust.py owns the network. Per interval it starts a node, pre-populates the key space, runs locust, and converts locust's statistics into bencher metrics.
  • tests/infra/key_space.py holds the key space helper previously private to basicperf.py. It has to move: basicperf.py imports piccolo, which only resolves when tests/infra is on sys.path, so it cannot be imported from a script in tests/.

Bencher output

Registered with add_e2e_test(... LABEL perf CONFIGURATIONS perf), following the commit_latency precedent for a perf test that is not piccolo-driven. bencher-ab.yml needs no change: it runs ./tests.sh -VV -L perf -C perf. One key per interval, each carrying throughput, latency and memory:

"Basic Blocking Locust (sig_ms_interval=100ms)": {
    "throughput": { "value": 3049.0 },
    "latency": { "value": 99.0, "high_value": 100.0, "low_value": 71.8 },
    "memory": { "value": 95260672, "high_value": 95260672 }
}

The locust master binds a free port per run rather than the default 5557, so concurrent locust runs on one machine do not collide.

Follow-up work

#8161 ("Isolate node and clients on separate CPUs for the locust perf test") is stacked on this branch and should merge after it. It pins the node and the load generator to disjoint CPU sets. Its measured benefit at the 100ms operating point is zero, because the node uses about a quarter of a core there. The case for it is insurance against co-tenant load on shared CI runners, which no single measurement can observe.

commit_latency and historical_query_perf_test are also registered via add_e2e_test and so also snapshot every 10 transactions. Not changed here, since correcting them would shift their published baselines, but their current figures partly measure snapshot I/O.

Testing

  • ctest -R pi_basic_blocking_locust -C perf passes.
  • ctest -R pi_basic_blocking -C perf still passes, covering the key_space.py refactor.
  • black, ruff, mypy, cmake-format, copyright and ASCII checks pass.

Draft while the defaults settle: the measurement window (20s), the process count (10), and whether the client ramp should eventually become a locust LoadTestShape with staged counts in one run.

@achamayou
Amaury Chamayou (achamayou) requested a balanced review from Copilot August 14, 2026 13:22
Copilot AI reviewed Aug 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Locust-driven blocking-write performance benchmark with configurable concurrency.

Changes:

  • Adds the FastHttpUser workload and benchmark runner.
  • Publishes throughput, latency, and memory metrics to Bencher.
  • Extracts shared key-space setup and registers the perf test.

Custom instructions used

  • .github/copilot-instructions.md

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
CMakeLists.txt Registers the new performance test.
tests/basicperf_locust.py Runs Locust and publishes metrics.
tests/infra/basicperf_locustfile.py Defines the blocking-write workload.
tests/infra/key_space.py Provides shared key-space setup.
tests/infra/basicperf.py Uses the extracted helper.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/basicperf_locust.py Outdated
@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Description

Comparing 5 available runs from this branch (#8158) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(9){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(10){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 71,159 tx/s ▲ 5%"]
  axis b1["Basic Blocking: 1,019 tx/s ▬ 0%"]
  axis b2["Basic Blocking Locust 100ms: 3,054 tx/s ▬ 0%"]
  axis b3["Basic Blocking Locust 20ms: 15,154 tx/s ▬ 0%"]
  axis b4["Basic Blocking Locust 2ms: 33,383 tx/s ▲ 4%"]
  axis b5["Basic JS: 4,734 tx/s ▬ -1%"]
  axis b6["Basic Multi-Threaded: 87,869 tx/s ▬ +1%"]
  axis b7["Historical Queries: 214,565 tx/s ▬ +4%"]
  axis b8["Logging: 64,901 tx/s ▲ 4%"]
  axis b9["Logging JWT: 10,038 tx/s ▬ -1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{103.97, 102.97, 100.15, 100.05, 104.35, 102.97, 104.17, 107.89, 106.81, 104.07}
  curve stddev1_high["main EWMA + 1 std dev"]{101.98, 101.49, 100.08, 100.02, 102.17, 101.48, 102.08, 103.94, 103.40, 102.03}
  curve stddev1_low["main EWMA - 1 std dev"]{98.02, 98.51, 99.92, 99.98, 97.83, 98.52, 97.92, 96.06, 96.60, 97.97}
  curve stddev2_low["main EWMA - 2 std dev"]{96.03, 97.03, 99.85, 99.95, 95.65, 97.03, 95.83, 92.11, 93.19, 95.93}
  curve branch_2["#8158 (2 runs earlier)"]{98.72, 102.17, 99.76, 100.00, 97.34, 101.62, 97.09, 97.12, 99.12, 96.49}
  curve branch_3["#8158 (1 run earlier)"]{99.85, 98.80, 99.86, 100.01, 101.06, 98.87, 97.94, 95.25, 100.27, 98.51}
  curve branch_4["#8158"]{105.09, 100.38, 99.90, 99.95, 104.05, 99.18, 101.23, 103.51, 103.73, 99.08}
  graticule polygon
  max 114
  min 86
  ticks 0
  showLegend false
Loading

Latency (ms)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking Locust 100ms: 99 ms ▬ 0%"]
  axis b1["Basic Blocking Locust 20ms: 19 ms ▬ 0%"]
  axis b2["Basic Blocking Locust 2ms: 8 ms ▼ 11%"]
  axis b3["Commit Latency 16ms: 5.69 ms ▬ +8%"]
  axis b4["Commit Latency 1ms: 1.94 ms ▬ +1%"]
  axis b5["Commit Latency 256ms: 203 ms ▬ 0%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 108.89, 161.26, 107.43, 101.02}
  curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 104.44, 130.63, 103.72, 100.51}
  curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 95.56, 69.37, 96.28, 99.49}
  curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 91.11, 38.74, 92.57, 98.98}
  curve branch_2["#8158 (2 runs earlier)"]{100.00, 100.00, 100.00, 96.42, 103.51, 99.89}
  curve branch_3["#8158 (1 run earlier)"]{100.00, 100.00, 100.00, 103.73, 100.18, 100.16}
  curve branch_4["#8158"]{100.00, 100.00, 88.89, 108.26, 101.07, 99.56}
  graticule polygon
  max 205
  ticks 0
  showLegend false
Loading

Memory (bytes)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(9){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 84.6 MiB ▬ -2%"]
  axis b1["Basic Blocking: 71.7 MiB ▬ 0%"]
  axis b2["Basic Blocking Locust 100ms: 90.2 MiB ▬ 0%"]
  axis b3["Basic Blocking Locust 20ms: 89.7 MiB ▲ 1%"]
  axis b4["Basic Blocking Locust 2ms: 93.4 MiB ▬ +1%"]
  axis b5["Basic JS: 69.6 MiB ▼ 2%"]
  axis b6["Basic Multi-Threaded: 88.2 MiB ▬ -1%"]
  axis b7["Logging: 76.6 MiB ▲ 1%"]
  axis b8["Logging JWT: 70.4 MiB ▲ 2%"]
  curve stddev2_high["main EWMA + 2 std dev"]{106.88, 100.60, 101.12, 100.63, 101.72, 103.57, 102.71, 101.54, 101.41}
  curve stddev1_high["main EWMA + 1 std dev"]{103.44, 100.30, 100.56, 100.31, 100.86, 101.79, 101.35, 100.77, 100.71}
  curve stddev1_low["main EWMA - 1 std dev"]{96.56, 99.70, 99.44, 99.69, 99.14, 98.21, 98.65, 99.23, 99.29}
  curve stddev2_low["main EWMA - 2 std dev"]{93.12, 99.40, 98.88, 99.37, 98.28, 96.43, 97.29, 98.46, 98.59}
  curve branch_2["#8158 (2 runs earlier)"]{102.55, 100.11, 98.30, 100.00, 100.09, 98.32, 99.69, 100.17, 100.30}
  curve branch_3["#8158 (1 run earlier)"]{102.64, 100.43, 99.41, 100.64, 101.71, 98.24, 99.82, 99.94, 99.30}
  curve branch_4["#8158"]{98.28, 100.15, 99.57, 100.68, 100.86, 97.77, 99.47, 100.87, 102.27}
  graticule polygon
  max 112
  min 88
  ticks 0
  showLegend false
Loading

Rate (ops/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(6){fill:#2DA44E!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["CHAMP get: 38,017,449 ops/s ▬ 0%"]
  axis b1["CHAMP put: 5,393,746 ops/s ▼ 2%"]
  axis b2["KV deserialisation: 1,626,281 ops/s ▬ 0%"]
  axis b3["KV serialisation: 1,449,275 ops/s ▬ 0%"]
  axis b4["KV s…t deserialisation: 4,247 ops/s ▲ 2%"]
  axis b5["KV snapshot serialisation: 4,830 ops/s ▲ 7%"]
  curve stddev2_high["main EWMA + 2 std dev"]{101.32, 102.03, 102.40, 102.10, 102.26, 109.31}
  curve stddev1_high["main EWMA + 1 std dev"]{100.66, 101.02, 101.20, 101.05, 101.13, 104.65}
  curve stddev1_low["main EWMA - 1 std dev"]{99.34, 98.98, 98.80, 98.95, 98.87, 95.35}
  curve stddev2_low["main EWMA - 2 std dev"]{98.68, 97.97, 97.60, 97.90, 97.74, 90.69}
  curve branch_0["#8158 (4 runs earlier)"]{99.34, 101.08, 101.91, 100.03, 101.75, 99.78}
  curve branch_1["#8158 (3 runs earlier)"]{99.70, 101.76, 100.76, 100.48, 101.55, 107.77}
  curve branch_2["#8158 (2 runs earlier)"]{100.13, 99.66, 99.60, 99.60, 100.08, 95.06}
  curve branch_3["#8158 (1 run earlier)"]{99.91, 100.96, 100.93, 99.90, 101.08, 98.74}
  curve branch_4["#8158"]{99.83, 98.27, 100.11, 100.19, 101.99, 106.98}
  graticule polygon
  max 116
  min 84
  ticks 0
  showLegend false
Loading

@eddyashton Eddy Ashton (eddyashton) mentioned this pull request Aug 14, 2026
Closed
@achamayou Amaury Chamayou (achamayou) mentioned this pull request Aug 14, 2026
Draft
@achamayou
Amaury Chamayou (achamayou) requested a balanced review from Copilot August 14, 2026 18:13
Copilot AI reviewed Aug 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@achamayou
Amaury Chamayou (achamayou) marked this pull request as ready for review August 14, 2026 20:31
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner August 14, 2026 20:31
@achamayou Amaury Chamayou (achamayou) mentioned this pull request Aug 18, 2026
Merged
@achamayou
Add Basic Blocking Locust perf test
c42c11f 
Adds pi_basic_blocking_locust, which measures the same blocking-write
workload as pi_basic_blocking but drives it with locust rather than
piccolo, so the number of concurrent clients can be varied.

The load is defined in tests/infra/basicperf_locustfile.py and uses
FastHttpUser, since HttpUser cannot drive enough requests per second to
saturate the service. tests/basicperf_locust.py owns the network, runs
locust against it, and converts locust statistics into bencher metrics
(throughput, latency, memory).

The key space helper shared with basicperf.py moves to
tests/infra/key_space.py, since basicperf.py can only be imported from
tests/infra.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Normalise new test files to LF line endings
b95bec8 
The three files added by this branch were committed with CRLF, unlike
every other Python file under tests/. No functional change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure for a fixed window after users have spawned
c77ad78 
Locust --run-time starts counting when locust starts, so it includes the
ramp, and --reset-stats discards the statistics gathered during the ramp
without extending the deadline. The measurement window was therefore
shorter than requested, and shrank as the spawn rate was lowered, until
it disappeared entirely.

This mattered because varying the client count is the point of this
test: at --users 128 --spawn-rate 4 the run ended mid-ramp and reported
127 tx/s instead of the ~1250 tx/s that 128 users actually sustain, and
did so without failing.

Start the shutdown timer from locust spawning_complete instead, so the
window is the same length whatever the spawn rate is, and rename
--run-time-s to --measure-time-s to describe what it now does.
--run-time is kept as a backstop against a run which never finishes
spawning.

Also fail, rather than report, when a run ends without having spawned
all users, or when the window measured is shorter than the one asked
for. Both produce plausible looking figures that do not describe steady
state.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Bind the locust master to a free port
8fff211 
Locust workers reach the master on port 5557 by default, so a second
locust run anywhere on the same machine fails to bind. Pick a free port
per run instead, via the existing infra.net helper.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Fail with a diagnosis when locust reports N/A
872b74a 
Locust writes N/A rather than a number in the statistics CSV when it has
too few samples to compute a percentile. float() then raised a bare
ValueError, after the network had already been stopped, losing the run
with no indication of what had gone wrong.

Read the numeric columns through a helper which reports the column and
value, and says that the run did not gather enough data. Found by a run
which produced almost no samples.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Snapshot as rarely as the piccolo perf tests do
499780c 
add_piccolo_test passes --snapshot-tx-interval 10000 for every piccolo
perf test. add_e2e_test does not pass it at all, and e2e_args defaults
it to 10, which is sensible for functional tests but not for a
benchmark.

This test was therefore writing and fsyncing a ~213KB snapshot every 10
transactions for the whole run, which measures the disk rather than the
service, and makes the figure incomparable with Basic Blocking.

Found by the vegeta comparison work, which hit the same defect.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure at 5ms, 100ms and 1s signature intervals
6b085f0 
Blocking writes return once their transaction commits, and commit cannot
outpace the signature interval, so a single interval only measures one
regime. Sweeping three separates them: at 1s and 100ms the workload is
latency-bound and throughput is simply the client count divided by the
interval, while at 5ms the node becomes the limit and the benchmark
measures capacity instead.

Each interval gets its own network, since the interval is fixed in the
node configuration at startup. consensus_update_timeout_ms moves with
it, as in commit_latency.py, because commit cannot be observed faster
than the primary sends updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Drive 320 clients from 10 locust processes
9c60c8d 
The blocking workload is latency-bound at the longer signature intervals,
so throughput there is set by the client count: 320 clients raises the
100ms point from ~1260 to ~3100 tx/s and the 1s point from ~130 to ~314,
both within a few percent of clients divided by interval.

Ten sending processes rather than four keeps locust from becoming the
limit while driving that many clients, since each process drives all of
its users from a single thread.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Measure the fastest signatures at 2ms rather than 5ms
76f6d93 
The shortest interval is the point at which the node, rather than the
signature timer, becomes the limit, so it is the one which measures
capacity. Shortening it to 2ms pushes further past the latency-bound
regime. The node ticks every 1ms in these tests, so a 2ms interval is
representable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9
@achamayou
Sweep 2ms, 20ms and 100ms signature intervals
f6723fd 
Drop the 1000ms point, which only restates the latency-bound regime
already measured at 100ms, and insert 20ms to find whether that is
still interval-bound or already at the capacity knee.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 569d1ede-c4d3-420e-828e-78fef51d205f
@achamayou
Amaury Chamayou (achamayou) merged commit 2ef3c22 into main Aug 19, 2026
18 checks passed
@achamayou
Amaury Chamayou (achamayou) deleted the locust-blocking-perf branch August 19, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@eddyashton Eddy Ashton (eddyashton) eddyashton approved these changes

Assignees

No one assigned

Labels

bench-ab

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@achamayou @eddyashton