Skip to content

bench: v4 experiments: replay on the real broker, aging isolation, baselines, sensitivity, simulator validation - #55

Merged
divyanshusingh2903 merged 20 commits into
mainfrom
bench/v4-replay-sensitivity
Oct 18, 2026
Merged

divyanshusingh2903 merged 20 commits into
mainfrom
bench/v4-replay-sensitivity

Conversation

@divyanshusingh2903

@divyanshusingh2903 divyanshusingh2903 commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

What this is

The v4 experiments for the paper, driven by the review critique and by the simulator-validation failure (the simulator overstated the production flow's gain by about 40%): the registered runs, their pre-registrations and result records, and the tooling behind them. Twenty commits, in four groups. The paper update is a separate PR (stacked on this one, see below).

All five registrations were frozen before their registered runs, and the results are recorded in benchmarks/results/, each directory with a README stating its conclusion. Raw per-message files (about 8 GB) are not committed; their SHA-256 hashes and every run's run.json are.

Results

Experiment Where Result
v4-simvalidation-1 local Gate failed on effect size (11.4 and 11.8 points off a 10-point limit); directions and policy order right.
v4-cloud-reruns (E5, E6) AWS, one host The aging and size-bin gates fail on AWS as they did locally; every verdict unchanged.
v4-baselines-1 AWS, one host, 110 runs Predictive −32.1% vs FIFO again. A queue per job type with no prediction beats it on mean by 42% (P99 worse). Aging off cuts the mean a further 37.4%, to −57.6% vs FIFO, the RabbitMQ sidecar's gain. No predictor component shows a benefit.
v4-sensitivity-1 local simulator Gate B passes in 18/18 (Huawei) and 16/18 (Azure) variants; gate A as built sits at its bar on Huawei and never passes on Azure; the aging threshold dominates. With 28 windows and a block bootstrap two cells flip to fail (gate A 0/6, gate B 4/6). Part C (a real-broker confirmation) was not run, and the README says why.
v4-replay-1 AWS, two pairs, 76 cells Gate A 0 of 6, gate B 4 of 6 on the real broker: B passes at loads 0.8 and 0.95 on Azure and Huawei, fails at 0.5 (Huawei −37.0% and −47.1%, Azure −15.3% and −17.2%). That is the simulator's 28-window verdict exactly. The simulator matches the broker to about a point on both traces. On Huawei the live predictor does not beat a static map from the same history.
v4-replay-aging-1 AWS, 18 cells, smaller hosts Aging costs gate A on Huawei at both loads: aging off passes (−37.4%, −45.4%), default aging fails (−11.0%, −8.5%).

What is in the branch

  • Simulator validation and cloud reruns: the replay-mode simulator validation and its result; the AWS runbook, the E5/E6 configs and their results.
  • Sensitivity tooling: the predictor-override option in trace_sim, the shared trace loader (tools/trace_io.hpp), a moving-block bootstrap and a sensitivity report in sim_report.py, and a variant generator.
  • Replay harness: harbinger_replay (prepare and run, roles all, broker and client, a +no_aging arm flag), replay_report.py, the runner run_replay.sh (sharding, retry-once, registered-mode refusals), the smoke kit, the Docker image update, and 25 tests.
  • Baselines: the predictive_noaging arm beside the baseline and ablation arms, the ten-seed config, readings in prodflow_report.py, and an e7 entry in the AWS runner.
  • Pre-registrations: v4-replay, v4-replay-aging, v4-baselines, v4-sensitivity, v4-cloud-reruns, and v4-simvalidation, each with its draft status, decisions and what had been seen before freezing.

Review notes

  • Bugs the smoke runs and analyses caught, all fixed with tests: Azure's job keys are 129 bytes and the broker silently ignores keys over 128, so unmodified every Azure message would have been unpredicted (keys now travel as k<index>); a broker-PID bug in the split runner that would have hung every split run; a sensitivity report that evaluated criteria on arms a variant does not run; two replay_report.py bugs found while analysing (the simulator's compress entry treated as an arm, a replay-only gate shown as a simulator failure).
  • Two Huawei replay cells lost the broker within seconds of starting (one per pair). The runner kept each as .failed and re-ran it once, as registered; both retries are valid. The cause is unknown (no kernel or Docker log trace; the broker containers' logs were lost). It may be a rare startup defect in the broker and deserves a reproduction loop.
  • The replay's handlers sleep and do no CPU work. The simulator's agreement with the broker therefore tests the queueing model, not an application with CPU-bound handlers.
  • Deviations from the registrations are recorded in each README: Azure ran the gate arms only at loads 0.5 and 0.95; DAS2 and LPC were dropped from the replay; the aging replay used smaller hosts and carries its own FIFO and predictive arms; sensitivity Part C was not run.
  • Tested: unit and integration tests for the harness, the runner (embedded, split over a stand-in ssh, retry, shards, refusals), the report and the smoke kit. The replay ctest needs -DHARBINGER_BUILD_BENCHMARKS=ON.
  • All AWS infrastructure used for this work has been deleted (instances, volumes, placement groups, the security group and the key pair).

Paper

The paper update is in the stacked PR (paper/v4-results, based on this branch), so it can be reviewed apart from the experiments. This PR leaves paper/ untouched.

🤖 Generated with Claude Code

…lidation

Pre-register the simulator validation (docs/v4-simvalidation-preregistration.md,
benchmarks/configs/v4-simvalidation.json) before any registered run: V1 effect
agreement within 10 points, V2 per-phase direction, V3 group order, on the local
and AWS production-flow runs.

trace_sim gains a replay mode, an oracle_bounds arm, named static maps and an
optional per-message dump; configs without them reproduce the published results
(Huawei v3-traces-1 re-run and compared). analysis/trace_format.py gains a
prodflow converter. benchmarks/sim_validation.py implements the analysis and is
tested on synthetic runs only; the registered runs have not been read.
The registered run used freeze commit cdcc20b and changed nothing afterwards.
V2 (phase direction) and V3 (group order) passed in the local and AWS sets; V1
(effect size) failed in both: the simulator overstates predictive routing's
mean-latency reduction (37.0% against 25.6% locally, 42.7% against 30.8% on AWS)
and its gaps for predictive and the oracle are 11.4 to 13.1 points against a
limit of 10. Absolute latencies are 0.31 to 0.41 of the measured ones.
…baseline and ablation arms

Draft, not frozen. benchmarks/cloud/RUNBOOK-v4.md gives the single-host lane for the
aging (E5) and size-bin (E6) reruns on AWS and the quota layout. v4-cloud-aging.json and
v4-cloud-sizebins.json copy the frozen v3 configs unchanged apart from version,
environment and pointer fields, with frozen_on null; docs/v4-cloud-reruns-preregistration.md
is their draft pre-registration. run_all.sh gains e5 and e6, which refuse unfrozen configs
in registered mode.

prodflow gains the arms for the v4-baselines experiment: predictive_mean, per_type_fair
(one level per job type, equal weights, no aging) and the ablate_decay, ablate_spread,
ablate_stale and ablate_nmin arms. Each ran a 20 s smoke test on seed 7 with no lost work.
Freeze the pre-registration and set frozen_on in v4-cloud-aging.json and
v4-cloud-sizebins.json, before any registered run. The configs are the frozen v3
aging and size-bin configs unchanged. The seed-7 debug pass ran clean on a
c8i.2xlarge on 2026-10-14 (all four runs completed 7066 of 7066 messages, no lost
work); its latencies were not read.
…tstrap, variant generator)

trace_sim reads an optional `predictor` block (min_samples, max_spread_ratio, decay, time_half_life_ms,
stale_after_ms); configs without it reproduce v3-traces-1 on Azure and Huawei exactly, apart from timing fields. The
trace loader moves to tools/trace_io.hpp so the replay harness reads traces the same way. boundaries_ms now lists one
bound per level boundary, which two levels would have read out of range.

sim_report gains a moving-block bootstrap, `--blocks/--primary-block/--compare` for part B and the verdict-change list,
and a `sensitivity` command with the R1 to R3 readings for part A. make_sensitivity_configs.py writes the 18 variants
and the 28-window config per trace, so none is written by hand. No variant result has been read; the draft
pre-registration records what is done and the generator choices to confirm before freezing.
…-replay, draft)

harbinger_replay replays a trace's evaluation half through the real broker: producers submit at the scaled arrival
deadline, 64 consumers sleep for the scaled duration, and the broker's time constants are divided by the time scale. A
`prepare` step builds the tier boundaries, per-load compress factors and warm snapshots from the history half with the
broker's own predictor. Roles all, broker and client as in harbinger_prodflow.

Trace keys travel on the wire as k<index>: Azure's keys are 129 bytes and the broker silently ignores job keys over 128,
so unmodified every Azure message would have been unpredicted.

replay_report.py applies the registered validity rules and gates with the block bootstrap (seven-window interval
alongside), and compares with the simulator. run_replay.sh lays runs out for it, shards a matrix across pairs, keeps a
run that does not count as .failed and repeats it once, and refuses overrides in registered mode. make_replay_smoke.py
writes the synthetic trace and config for the AWS smoke run. The image now builds harbinger_replay.

Tested with the real binary on synthetic traces (embedded, split over a stand-in ssh, retry, shards, refusals); the
Docker path is first exercised on AWS. Both configs are drafts (frozen_on null). No registered trace has been replayed.
…raft

Runbook: two pairs, Azure at loads 0.8 and 0.95 and Huawei at three loads, sharded across the pairs, about 38 h each; the
synthetic smoke run, freezing, registered runs, resuming after a stop, and teardown. Pre-registration draft: what is built
and tested, the Azure load deviation, the key-length finding, and the choices to confirm before freezing.
…on AWS)

The registered run used freeze commit d9e21cf on a c8i.2xlarge, 2026-10-14 22:22 to
2026-10-15 01:19 UTC, 35 runs, all completed with no lost work. Both gates came out as
in the local quiet-machine re-run (not passed), and every criterion's pass or fail
matches: E5's all-message P99 against predictive routing is 1.100 [1.091, 1.107] against
a limit of 1.1; E6's mean against predictive routing is -1.5% [-4.5%, +0.4%] against a
bar of -3%. The raw messages.tsv files are not committed; each SHA256SUMS hashes them.
Repeats are pooled, and a Huawei gate cell whose single-repeat verdicts disagree is reported as unstable and not counted
as a pass. The lateness bound is in real time, with lateness in trace time also reported. Latency stays measured from the
scheduled arrival, with the difference from measuring at the actual submit reported as a sensitivity check.

Azure now replays loads 0.5, 0.8 and 0.95: gate B passed at all three in the simulator but at 0.5 by the narrowest margin
(-15.4% against the -15% bar), so that cell is the one most likely to change and dropping it would have removed the most
informative result. The gate arms run at 0.5 and 0.95 and all five at 0.8, via `arms_by_load` in the config and the
runner (61.2 h across the matrix). DAS2 and LPC leave this registration. The runbook's numbers, shard assignment per pair,
spend cap and cost are updated.
…broker)

Frozen before any registered run. The scope is the two traces; DAS2 and LPC leave this version because their 1 to 17 ms
aging constants need a calibration run first. The smoke runs were on synthetic traces only (locally, with the Docker image,
and on one AWS pair over real ssh), and what was read from them is recorded in the pre-registration. The configs' frozen_on
is set; registered runs refuse anything else.
…tion), draft for AWS

Adds the predictive_noaging arm (predictive routing with aging off) beside the baseline and ablation arms already in
prodflow.cpp, the ten-seed v4-baselines.json (11 arms), and `e7` in the AWS runner. prodflow_report gains readings
(up / tied / down / inconclusive against a 3% margin), report-only criteria and groups that combine an ablation's overall
and shifted-phase readings, with tests. The draft pre-registration moves to AWS, adds the aging-isolation part with its
reading fixed before any run, replaces "worse otherwise" with "inconclusive", and points the quiet check at the AWS
production-flow results. Still a draft: the AWS debug pass and the freeze come next. The only runs of the new arms were
8-second local checks on seed 7 that read only that they completed.
…olation on AWS)

Frozen before any registered run. The AWS debug pass (all 11 arms, seed 7, 60 s each, commit 97a10b0) completed every run
with no dead letter or submit error; only completion was read. The readings, the 3% margin and the aging reading were
fixed before any run of a new arm at a registered seed.
…ntervals on the simulator)

Frozen before any registered run, with the generated configs (18 variants and the 28-window config for Azure and Huawei,
frozen_on set). The generator choices (aging interval at a tenth of the threshold, weights (8,1) and (8,4,2,1), fixed class
bounds) are confirmed. The pre-registration now says the simulator's effects are overstated by about 40%, so the sweep is
read for whether verdicts change, and adds the conditional stage-2 rule for confirming flips on the real broker.

A synthetic-trace smoke of the whole pipeline found that the sensitivity report evaluated criteria on arms the variant
runs do not have; fixed with a regression test.
…es; two trace cells change under block bootstrap)

Part A, simulator, load 0.8: gate B passes in 18 of 18 variants on Huawei and 16 of 18 on Azure (failing at n_min 100 and at
two levels); gate A passes in none on Azure and 10 of 18 on Huawei, where the default is exactly at the -15% bar and the
aging threshold dominates (-3.3% at 1 s, -15.0% at 5 s, -40.7% at 30 s). Part B, 28 windows with block-4 bootstrap: Azure
load 0.5 gate B and Huawei load 0.8 gate A change from pass to fail, so gate A passes 0 of 6 cells and gate B 4 of 6. All
simulator output, which overstates effects by about 40%. Part C is triggered; v4-sensitivity-2 is to be registered before
any replay.
…he mean; aging explains the sidecar gap)

110 runs on AWS (11 arms, 10 seeds), all complete. Predictive routing reproduces its gain (-32.1% against FIFO) and beats the
hand-tuned static map (-15.5%), but a no-prediction queue per job type with equal worker-time shares beats it on mean latency
(predictive is 74% worse) while losing all-message P99 (+52%). Predictive routing with aging off cuts the mean a further 37.4%
to -57.6% against FIFO, the size of the RabbitMQ sidecar's gain, at a cost of 10 to 14% in P99 and long-class wait. The mean
and p75 summaries are 3.7% worse than the median. No predictor component shows a benefit on this workload (overall within
+-3%, shifted-phase intervals inconclusive). Raw files are not committed; their hashes and every run.json are.
…off on Huawei, real broker)

The replay binary accepts a `+no_aging` arm flag (strict priority, no aging thread; refused with weights and pausing), with a
test. The draft registration runs fifo, predictive_warm and predictive_warm+no_aging on Huawei at loads 0.8 and 0.95, three
repeats, on two c8i.xlarge hosts (the quota left beside the two v4-replay-1 pairs), so it carries its own fifo and
predictive_warm arms. Readings are fixed before any run. Still a draft until frozen.
… Huawei, real broker)

Frozen before any registered run. The smoke run on a synthetic trace over two c8i.xlarge hosts completed every message with
producer lateness p99 0.12 ms, which shows the smaller client keeps up; only those facts were read. No cell of v4-replay-1
has been read.
…wei, at both loads)

18 runs on AWS, all valid. Predictive routing with default aging fails gate A at loads 0.8 and 0.95; with aging switched off it
passes gate N at both (mean -37.4% and -45.4% against FIFO, 28 windows, block bootstrap), P99 within 1.07x to 1.12x and the
long-class wait within 1.14x to 1.16x of FIFO, the three repeats agreeing. Whole-run mean latency falls from 17.7 s to 1.7 s
at 0.8 and from 41.8 s to 7.2 s at 0.95. On this trace the simulator matches the real broker to about a point, including an
unregistered simulator run of the no-aging arm; the production flow's 40% overstatement does not show here, where the handlers
do no CPU work.

Also fixes two replay_report bugs found while analysing: the simulator's per-load compress entry was treated as an arm, and a
replay-only gate was shown as a simulator failure. Raw files are not committed; their hashes and every run.json are.
… the real broker)

76 cells on AWS, all counted: Huawei R5 and Azure 2021 through the real broker. Predictive routing as built (gate A) fails in every
cell on both traces. Weights and pausing aging (gate B) passes at loads 0.8 and 0.95 on both and fails at 0.5 (Huawei -37.0% and
-47.1%, Azure -15.3% and -17.2% against FIFO), with P99 within 1.12x and long wait within 1.14x of FIFO. That is the simulator's
verdict with 28 windows and a block bootstrap, and two cells fewer than its original seven-window verdict. On Huawei the live
predictor does not beat a static map built from the same history. The simulator matches the real broker to about a point on both
traces (the 40% overstatement was specific to the production flow). Two Huawei cells lost the broker and were re-run once as
registered; the cause is unknown. Raw files are not committed; their hashes and every run.json are.
@divyanshusingh2903 divyanshusingh2903 changed the title bench: v4 cloud prep: simulator validation, E5/E6 on AWS, replay harness, sensitivity tooling bench: v4 experiments: replay on the real broker, aging isolation, baselines, sensitivity, simulator validation Oct 18, 2026
@divyanshusingh2903
divyanshusingh2903 merged commit d047aca into main Oct 18, 2026
6 checks passed
@divyanshusingh2903
divyanshusingh2903 deleted the bench/v4-replay-sensitivity branch October 18, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant