[dmihalcik-virtru]

Found while reviewing #669. This is a pre-existing bug on main, so this PR is based on main and is independent of the stage 2/3 stacks.

The bug

pytest_sessionfinish promises "Artifacts are written first and unconditionally". In fact it computes the gate (recorder.gate()) before writing anything, and write_json needs the gate too. Any exception from analyze(), the statistics or to_dict therefore ended the hook with nothing on disk, and a 30-minute run's per-round samples were lost exactly when someone needed them to debug the failure. The error itself was loud, so nothing went green; only the data was gone.

#669 makes the report index contrast keys directly, so a wrongly built key raises instead of being dropped. That makes this window wider.

What it does

• The hook catches the failure, writes the raw samples, then re-raises. If computing the gate or the JSON raises, it writes <name>.raw.json, adds a note naming that file, and re-raises the original exception. The session still fails on the same error.

• The raw document holds what re-analysis needs:

  • analysis_error (the exception's type and message);
  • metadata;
  • config;
  • skipped;
  • every CellResult field by field, samples included.

  It is serialized with dataclasses.asdict, not through to_dict, which needs a gate and may be what raised. It also follows CellResult when chore(xtest): measure benchmark cells over a tuple of arms #669 changes its shape.

• It goes under its own name rather than a marker, so nothing that opens <name>.json can take a run without statistics for a complete one. It has no schema key, so a reader of the normal artifact handed it by mistake fails at its first check. The upload step's *.json glob already collects it.

• A run that analyses cleanly is unchanged: the same <name>.json byte for byte, the same step summary, and no extra file. The normal artifact stays at schema: 1.

• Docs: the hook docstring and the perf README (the artifact table, and "Cells record; the session gates") describe the new path.

Limits

• Ctrl-C during analysis still loses the samples. Only Exception is caught, so other BaseExceptions, Ctrl-C included, are not covered.
• Stale files can sit side by side locally. When --bench-out persists between runs, a stale .raw.json can sit next to a fresh .json, or the reverse. CI runners start clean.

Validation

• uv run pytest test_bench_*.py: 231 passed. ruff, ruff format and pyright are clean.
• TestSessionFinish runs the real conftest.pytest_sessionfinish with stubbed metadata, so no platform is needed.

  • A failed analysis still writes the raw samples. This is tested twice, with analyze raising and with to_dict raising. It checks:

    • the error propagates with the note;
    • only benchmarks.raw.json exists, with no schema key;
    • every cell rebuilds exactly from it.

    Both cases fail on main.

  • A clean analysis writes exactly what it always did. This pins <name>.json and the step summary to the existing renderers.

Interaction with #669

#669 was rebased onto this change in a scratch repository:

• Clean: report.py, conftest.py and the README.
• test_bench_report.py: conflicts only on its import lines and at end-of-file; keep both sides. Then replace this PR's _measured helper with chore(xtest): measure benchmark cells over a tuple of arms #669's _cell, and rebuild cells with CellResult(**c | {"arm_ids": tuple(c["arm_ids"])}).
• Result: 277 passed.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

X-Test Results

✅ java@main-v0.29.0
✅ js@main-main
✅ java@v0.19.1-v0.29.0
✅ java@main-main
✅ java@v0.19.1-main
✅ js@v0.23.0-main
✅ js@main-v0.29.0
✅ go@main-main
✅ go@v0.39.0-main
✅ go@main-v0.29.0
✅ go@v0.39.0-v0.29.0
✅ js@v0.23.0-v0.29.0