Repository navigation
bench: v4 experiments: replay on the real broker, aging isolation, baselines, sensitivity, simulator validation - #55
Merged
Conversation
…lidation Pre-register the simulator validation (docs/v4-simvalidation-preregistration.md, benchmarks/configs/v4-simvalidation.json) before any registered run: V1 effect agreement within 10 points, V2 per-phase direction, V3 group order, on the local and AWS production-flow runs. trace_sim gains a replay mode, an oracle_bounds arm, named static maps and an optional per-message dump; configs without them reproduce the published results (Huawei v3-traces-1 re-run and compared). analysis/trace_format.py gains a prodflow converter. benchmarks/sim_validation.py implements the analysis and is tested on synthetic runs only; the registered runs have not been read.
The registered run used freeze commit cdcc20b and changed nothing afterwards. V2 (phase direction) and V3 (group order) passed in the local and AWS sets; V1 (effect size) failed in both: the simulator overstates predictive routing's mean-latency reduction (37.0% against 25.6% locally, 42.7% against 30.8% on AWS) and its gaps for predictive and the oracle are 11.4 to 13.1 points against a limit of 10. Absolute latencies are 0.31 to 0.41 of the measured ones.
…baseline and ablation arms Draft, not frozen. benchmarks/cloud/RUNBOOK-v4.md gives the single-host lane for the aging (E5) and size-bin (E6) reruns on AWS and the quota layout. v4-cloud-aging.json and v4-cloud-sizebins.json copy the frozen v3 configs unchanged apart from version, environment and pointer fields, with frozen_on null; docs/v4-cloud-reruns-preregistration.md is their draft pre-registration. run_all.sh gains e5 and e6, which refuse unfrozen configs in registered mode. prodflow gains the arms for the v4-baselines experiment: predictive_mean, per_type_fair (one level per job type, equal weights, no aging) and the ablate_decay, ablate_spread, ablate_stale and ablate_nmin arms. Each ran a 20 s smoke test on seed 7 with no lost work.
Freeze the pre-registration and set frozen_on in v4-cloud-aging.json and v4-cloud-sizebins.json, before any registered run. The configs are the frozen v3 aging and size-bin configs unchanged. The seed-7 debug pass ran clean on a c8i.2xlarge on 2026-10-14 (all four runs completed 7066 of 7066 messages, no lost work); its latencies were not read.
…tstrap, variant generator) trace_sim reads an optional `predictor` block (min_samples, max_spread_ratio, decay, time_half_life_ms, stale_after_ms); configs without it reproduce v3-traces-1 on Azure and Huawei exactly, apart from timing fields. The trace loader moves to tools/trace_io.hpp so the replay harness reads traces the same way. boundaries_ms now lists one bound per level boundary, which two levels would have read out of range. sim_report gains a moving-block bootstrap, `--blocks/--primary-block/--compare` for part B and the verdict-change list, and a `sensitivity` command with the R1 to R3 readings for part A. make_sensitivity_configs.py writes the 18 variants and the 28-window config per trace, so none is written by hand. No variant result has been read; the draft pre-registration records what is done and the generator choices to confirm before freezing.
…-replay, draft) harbinger_replay replays a trace's evaluation half through the real broker: producers submit at the scaled arrival deadline, 64 consumers sleep for the scaled duration, and the broker's time constants are divided by the time scale. A `prepare` step builds the tier boundaries, per-load compress factors and warm snapshots from the history half with the broker's own predictor. Roles all, broker and client as in harbinger_prodflow. Trace keys travel on the wire as k<index>: Azure's keys are 129 bytes and the broker silently ignores job keys over 128, so unmodified every Azure message would have been unpredicted. replay_report.py applies the registered validity rules and gates with the block bootstrap (seven-window interval alongside), and compares with the simulator. run_replay.sh lays runs out for it, shards a matrix across pairs, keeps a run that does not count as .failed and repeats it once, and refuses overrides in registered mode. make_replay_smoke.py writes the synthetic trace and config for the AWS smoke run. The image now builds harbinger_replay. Tested with the real binary on synthetic traces (embedded, split over a stand-in ssh, retry, shards, refusals); the Docker path is first exercised on AWS. Both configs are drafts (frozen_on null). No registered trace has been replayed.
…raft Runbook: two pairs, Azure at loads 0.8 and 0.95 and Huawei at three loads, sharded across the pairs, about 38 h each; the synthetic smoke run, freezing, registered runs, resuming after a stop, and teardown. Pre-registration draft: what is built and tested, the Azure load deviation, the key-length finding, and the choices to confirm before freezing.
…on AWS) The registered run used freeze commit d9e21cf on a c8i.2xlarge, 2026-10-14 22:22 to 2026-10-15 01:19 UTC, 35 runs, all completed with no lost work. Both gates came out as in the local quiet-machine re-run (not passed), and every criterion's pass or fail matches: E5's all-message P99 against predictive routing is 1.100 [1.091, 1.107] against a limit of 1.1; E6's mean against predictive routing is -1.5% [-4.5%, +0.4%] against a bar of -3%. The raw messages.tsv files are not committed; each SHA256SUMS hashes them.
Repeats are pooled, and a Huawei gate cell whose single-repeat verdicts disagree is reported as unstable and not counted as a pass. The lateness bound is in real time, with lateness in trace time also reported. Latency stays measured from the scheduled arrival, with the difference from measuring at the actual submit reported as a sensitivity check. Azure now replays loads 0.5, 0.8 and 0.95: gate B passed at all three in the simulator but at 0.5 by the narrowest margin (-15.4% against the -15% bar), so that cell is the one most likely to change and dropping it would have removed the most informative result. The gate arms run at 0.5 and 0.95 and all five at 0.8, via `arms_by_load` in the config and the runner (61.2 h across the matrix). DAS2 and LPC leave this registration. The runbook's numbers, shard assignment per pair, spend cap and cost are updated.
…broker) Frozen before any registered run. The scope is the two traces; DAS2 and LPC leave this version because their 1 to 17 ms aging constants need a calibration run first. The smoke runs were on synthetic traces only (locally, with the Docker image, and on one AWS pair over real ssh), and what was read from them is recorded in the pre-registration. The configs' frozen_on is set; registered runs refuse anything else.
…tion), draft for AWS Adds the predictive_noaging arm (predictive routing with aging off) beside the baseline and ablation arms already in prodflow.cpp, the ten-seed v4-baselines.json (11 arms), and `e7` in the AWS runner. prodflow_report gains readings (up / tied / down / inconclusive against a 3% margin), report-only criteria and groups that combine an ablation's overall and shifted-phase readings, with tests. The draft pre-registration moves to AWS, adds the aging-isolation part with its reading fixed before any run, replaces "worse otherwise" with "inconclusive", and points the quiet check at the AWS production-flow results. Still a draft: the AWS debug pass and the freeze come next. The only runs of the new arms were 8-second local checks on seed 7 that read only that they completed.
…olation on AWS) Frozen before any registered run. The AWS debug pass (all 11 arms, seed 7, 60 s each, commit 97a10b0) completed every run with no dead letter or submit error; only completion was read. The readings, the 3% margin and the aging reading were fixed before any run of a new arm at a registered seed.
…ntervals on the simulator) Frozen before any registered run, with the generated configs (18 variants and the 28-window config for Azure and Huawei, frozen_on set). The generator choices (aging interval at a tenth of the threshold, weights (8,1) and (8,4,2,1), fixed class bounds) are confirmed. The pre-registration now says the simulator's effects are overstated by about 40%, so the sweep is read for whether verdicts change, and adds the conditional stage-2 rule for confirming flips on the real broker. A synthetic-trace smoke of the whole pipeline found that the sensitivity report evaluated criteria on arms the variant runs do not have; fixed with a regression test.
…es; two trace cells change under block bootstrap) Part A, simulator, load 0.8: gate B passes in 18 of 18 variants on Huawei and 16 of 18 on Azure (failing at n_min 100 and at two levels); gate A passes in none on Azure and 10 of 18 on Huawei, where the default is exactly at the -15% bar and the aging threshold dominates (-3.3% at 1 s, -15.0% at 5 s, -40.7% at 30 s). Part B, 28 windows with block-4 bootstrap: Azure load 0.5 gate B and Huawei load 0.8 gate A change from pass to fail, so gate A passes 0 of 6 cells and gate B 4 of 6. All simulator output, which overstates effects by about 40%. Part C is triggered; v4-sensitivity-2 is to be registered before any replay.
…he mean; aging explains the sidecar gap) 110 runs on AWS (11 arms, 10 seeds), all complete. Predictive routing reproduces its gain (-32.1% against FIFO) and beats the hand-tuned static map (-15.5%), but a no-prediction queue per job type with equal worker-time shares beats it on mean latency (predictive is 74% worse) while losing all-message P99 (+52%). Predictive routing with aging off cuts the mean a further 37.4% to -57.6% against FIFO, the size of the RabbitMQ sidecar's gain, at a cost of 10 to 14% in P99 and long-class wait. The mean and p75 summaries are 3.7% worse than the median. No predictor component shows a benefit on this workload (overall within +-3%, shifted-phase intervals inconclusive). Raw files are not committed; their hashes and every run.json are.
…off on Huawei, real broker) The replay binary accepts a `+no_aging` arm flag (strict priority, no aging thread; refused with weights and pausing), with a test. The draft registration runs fifo, predictive_warm and predictive_warm+no_aging on Huawei at loads 0.8 and 0.95, three repeats, on two c8i.xlarge hosts (the quota left beside the two v4-replay-1 pairs), so it carries its own fifo and predictive_warm arms. Readings are fixed before any run. Still a draft until frozen.
… Huawei, real broker) Frozen before any registered run. The smoke run on a synthetic trace over two c8i.xlarge hosts completed every message with producer lateness p99 0.12 ms, which shows the smaller client keeps up; only those facts were read. No cell of v4-replay-1 has been read.
…wei, at both loads) 18 runs on AWS, all valid. Predictive routing with default aging fails gate A at loads 0.8 and 0.95; with aging switched off it passes gate N at both (mean -37.4% and -45.4% against FIFO, 28 windows, block bootstrap), P99 within 1.07x to 1.12x and the long-class wait within 1.14x to 1.16x of FIFO, the three repeats agreeing. Whole-run mean latency falls from 17.7 s to 1.7 s at 0.8 and from 41.8 s to 7.2 s at 0.95. On this trace the simulator matches the real broker to about a point, including an unregistered simulator run of the no-aging arm; the production flow's 40% overstatement does not show here, where the handlers do no CPU work. Also fixes two replay_report bugs found while analysing: the simulator's per-load compress entry was treated as an arm, and a replay-only gate was shown as a simulator failure. Raw files are not committed; their hashes and every run.json are.
… the real broker) 76 cells on AWS, all counted: Huawei R5 and Azure 2021 through the real broker. Predictive routing as built (gate A) fails in every cell on both traces. Weights and pausing aging (gate B) passes at loads 0.8 and 0.95 on both and fails at 0.5 (Huawei -37.0% and -47.1%, Azure -15.3% and -17.2% against FIFO), with P99 within 1.12x and long wait within 1.14x of FIFO. That is the simulator's verdict with 28 windows and a block bootstrap, and two cells fewer than its original seven-window verdict. On Huawei the live predictor does not beat a static map built from the same history. The simulator matches the real broker to about a point on both traces (the 40% overstatement was specific to the production flow). Two Huawei cells lost the broker and were re-run once as registered; the cause is unknown. Raw files are not committed; their hashes and every run.json are.
…ing variants) was not run, and why
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The v4 experiments for the paper, driven by the review critique and by the simulator-validation failure (the simulator overstated the production flow's gain by about 40%): the registered runs, their pre-registrations and result records, and the tooling behind them. Twenty commits, in four groups. The paper update is a separate PR (stacked on this one, see below).
All five registrations were frozen before their registered runs, and the results are recorded in
benchmarks/results/, each directory with a README stating its conclusion. Raw per-message files (about 8 GB) are not committed; their SHA-256 hashes and every run'srun.jsonare.Results
v4-simvalidation-1v4-cloud-reruns(E5, E6)v4-baselines-1v4-sensitivity-1v4-replay-1v4-replay-aging-1What is in the branch
trace_sim, the shared trace loader (tools/trace_io.hpp), a moving-block bootstrap and asensitivityreport insim_report.py, and a variant generator.harbinger_replay(prepareandrun, roles all, broker and client, a+no_agingarm flag),replay_report.py, the runnerrun_replay.sh(sharding, retry-once, registered-mode refusals), the smoke kit, the Docker image update, and 25 tests.predictive_noagingarm beside the baseline and ablation arms, the ten-seed config, readings inprodflow_report.py, and ane7entry in the AWS runner.v4-replay,v4-replay-aging,v4-baselines,v4-sensitivity,v4-cloud-reruns, andv4-simvalidation, each with its draft status, decisions and what had been seen before freezing.Review notes
k<index>); a broker-PID bug in the split runner that would have hung every split run; a sensitivity report that evaluated criteria on arms a variant does not run; tworeplay_report.pybugs found while analysing (the simulator'scompressentry treated as an arm, a replay-only gate shown as a simulator failure)..failedand re-ran it once, as registered; both retries are valid. The cause is unknown (no kernel or Docker log trace; the broker containers' logs were lost). It may be a rare startup defect in the broker and deserves a reproduction loop.-DHARBINGER_BUILD_BENCHMARKS=ON.Paper
The paper update is in the stacked PR (
paper/v4-results, based on this branch), so it can be reviewed apart from the experiments. This PR leavespaper/untouched.🤖 Generated with Claude Code