feat(xtest): benchmark payloads up to 1 GiB, with per-dispatch budget by dmihalcik-virtru · Pull Request #622 · opentdf/tests

dmihalcik-virtru · 2026-09-23T17:17:30Z

Split out of #583, based on main.

At the default sizes (1KiB, 1MiB, 32MiB), startup is ~86% of a go encrypt. The 1.15x gate therefore misses any throughput regression smaller than about 2x. This PR lets a manual dispatch measure sizes up to 1 GiB, where the gate can catch one. DSPX-5232 needs this.

--bench-payloads (bench-payloads input): comma-separated sizes, e.g. 1KiB,1GiB. Validated in pytest_configure, and in resolve-versions before any runner starts the platform. The A/A control stays at 1 MiB.
bench-budget-seconds input: the bench job's timeout is derived from it. Scheduled runs keep 1500 s and 45 min.
Disk:
- Payloads are written in chunks.
- A run that won't fit is refused up front.
- --bench-prune-scratch, set in CI, deletes each file once no cell needs it, so a 1 GiB run peaks at ~5 GiB.
- A cell that fails on a full disk stops the run instead of reading as a broken build.

Follow-ups:

DSPX-5253: smarter disk scheduling.
DSPX-5260: the nightly bench measures nothing when a release is main's HEAD; also covers what each cron run should compare.

Validation

test_bench*.py: 228 passed. ruff and pyright are clean.
Dispatch with go and 1MiB: all 3 cells passed, and no scratch files were left behind.
Not yet run: a 1GiB dispatch.

Summary by CodeRabbit

New Features
- Added configurable benchmark payload sizes for manual and reusable workflow runs.
- Added benchmark time budgets with automatic timeout handling.
- Added optional scratch-file cleanup to reduce disk usage during benchmarks.
- Added validation for payload settings, available disk space, and budget limits.
- Benchmark results now report remaining temporary files and handle disk exhaustion clearly.
Documentation
- Updated benchmark guidance for payload configuration, throughput regression analysis, cleanup options, and storage requirements.

coderabbitai · 2026-09-23T17:17:36Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 6cebeab6-7e87-455e-8555-358386409f49

📥 Commits

Reviewing files that changed from the base of the PR and between 74bab11 and 2dd146f.

📒 Files selected for processing: 8

.github/workflows/xtest.yml
xtest/conftest.py
xtest/fixtures/bench.py
xtest/perf/README.md
xtest/perf/cells.py
xtest/test_bench_arms.py
xtest/test_bench_config.py
xtest/test_benchmarks.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 01afeba1-8f9c-4047-adf4-7e930b765910

📥 Commits

Reviewing files that changed from the base of the PR and between e17816b and 74bab11.

📒 Files selected for processing: 8

.github/workflows/xtest.yml
xtest/conftest.py
xtest/fixtures/bench.py
xtest/perf/README.md
xtest/perf/cells.py
xtest/test_bench_arms.py
xtest/test_bench_config.py
xtest/test_benchmarks.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The benchmark harness now supports configurable payload sizes, validates scratch-disk capacity, and can prune temporary benchmark files. The workflow accepts payload and time-budget inputs, calculates a job timeout, and passes the resolved settings to pytest.

Changes: Configurable benchmark payloads

Layer / File(s)	Summary
Payload selection and benchmark cells `xtest/perf/cells.py`, `xtest/conftest.py`, `xtest/fixtures/bench.py`, `xtest/test_bench_arms.py`, `xtest/test_bench_config.py`, `xtest/perf/README.md`	Payload specifications are parsed, normalized, deduplicated, and sorted. Benchmark cells use the selected payloads; the 1 MiB control payload remains fixed. Tests and documentation cover the payload options and cell matrix.
Payload generation and disk validation `xtest/conftest.py`, `xtest/fixtures/bench.py`, `xtest/test_bench_arms.py`, `xtest/test_bench_config.py`	Payload files are generated in deterministic chunks. Configuration checks required disk space before tests begin, and the harness reports when free space is below its exhaustion threshold.
Scratch cleanup and benchmark failures `xtest/fixtures/bench.py`, `xtest/test_bench_arms.py`, `xtest/test_benchmarks.py`, `xtest/conftest.py`	Optional pruning removes tracked cell files and releases payload files after their final use. Benchmark measurement and subprocess failures trigger a failed session when disk exhaustion is detected; other failures are re-raised.
Workflow budget and benchmark invocation `.github/workflows/xtest.yml`	The workflow validates payload and budget inputs, calculates a timeout with 20 minutes of overhead, and passes the resolved settings to pytest. An always-run step reports filesystem capacity and remaining benchmark scratch files.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Actions as GitHub Actions
  participant Resolve as resolve-versions
  participant Parser as xtest/perf/cells.py
  participant Pytest as benchmark pytest run
  Actions->>Resolve: pass payload and budget inputs
  Resolve->>Parser: parse payload specification
  Resolve->>Actions: publish normalized payloads and timeout
  Actions->>Pytest: pass payloads, budget, and pruning option
  Pytest->>Actions: return benchmark result

Suggested reviewers: c-r33d

Merge Risk: ⚪ Minimal · up to 74bab

This change adds configurable benchmark payload sizes, disk-space checks, optional scratch cleanup and a configurable workflow time budget, while scheduled runs keep their default settings. The change is limited to the benchmark harness and its CI workflow, and no outstanding defects remain. A 1 GiB dispatch has not yet been run, but no known issue blocks merging.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 34.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 67 functions across 6 files. (2 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely summarizes the main changes: configurable benchmark payloads up to 1 GiB and a per-dispatch time budget.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 34.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 67 functions across 6 files. (2 skipped: 2 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 2

📝 Generate docstrings: 💡

Commit to this branch
Create a new PR

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{A rabbit checks each payload size,
Then hops through tests beneath the skies.
The scratch files vanish when they’re done,
The clock allows each benchmark run.
One-megabyte control stays in place,
While carrots wait in a tidy space.}

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-23T17:22:50Z

X-Test Failure Report

server-logs-js@main-main
server-logs-go@main-v0.27.0
server-logs-java@main-v0.27.0
server-logs-java@main-main
server-logs-go@main-main
server-logs-java@v0.19.0-v0.27.0
server-logs-go@v0.38.0-main
server-logs-js@main-v0.27.0
server-logs-java@v0.19.0-main
server-logs-js@v0.4.0-main
server-logs-js@v0.4.0-v0.27.0
server-logs-go@v0.38.0-v0.27.0

At the default sizes (1 KiB, 1 MiB, 32 MiB) this harness cannot fail a build on throughput: on a 4-core runner a go encrypt costs ~450ms of fixed startup against ~72ms that scales with the payload, so the 1.15x gate is wider than the whole payload-dependent part. --bench-payloads lets a manual dispatch ask a throughput question the nightly cannot afford to ask every night. - perf/cells.py: Payload/parse_payload/parse_payloads replace the fixed 3-entry PAYLOADS tuple, so a spec like '1KiB,1GiB' builds an arbitrary cell matrix. CONTROL_PAYLOAD is a fixed 1 MiB literal rather than PAYLOADS[1] -- same value, but no longer coupled to whatever the selection happens to be, since the control's CI width is the run's noise floor and it must not move with --bench-payloads. - fixtures/bench.py: --bench-payloads wiring, chunked payload generation (a 1 GiB file is written in 8 MiB chunks rather than held in memory), and an up-front disk-space check -- a run holds roughly twice the payload total plus one live output per arm of the largest, and running out mid-benchmark otherwise surfaces as a non-zero exit that reads as 'this build is broken'. - conftest.py: the payload spec is validated in pytest_configure with the other benchmark options, before any session setup. - xtest.yml: bench-payloads and bench-budget-seconds dispatch inputs, validated in resolve-versions with the harness's own parser before any bench runner starts the platform. The bench job's timeout is derived from the budget instead of a fixed 45 minutes; scheduled runs keep the current sizes, 1500 s budget, and 45-minute timeout. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

- --bench-prune-scratch (set by the bench job) deletes each cell's outputs and ciphertext when it ends, and each plaintext after its last cell. A 1GiB run peaks at ~5 GiB of scratch instead of ~7. - The disk check moves to pytest_configure, where a shortfall is one usage error (exit 4) rather than an error on every cell. Its estimate now matches what is actually written, with and without pruning. - A cell that fails with the scratch disk nearly full is recorded as skipped and ends the session, instead of reading as a broken build. - Payload labels are canonical by unit (1024MiB -> 1GiB); the workflow falls back to DEFAULT_PAYLOAD_SPEC rather than a copy of it. - Comments, README and tests brought in line with the code. Smarter disk scheduling is DSPX-5253. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-10-20T12:27:03Z

X-Test Failure Report

github-actions · 2026-10-20T12:46:46Z

X-Test Results

✅ go@main-main
✅ go@main-v0.29.0
✅ js@main-main
✅ js@main-v0.29.0
✅ java@v0.19.1-v0.29.0
✅ java@main-main
✅ js@v0.4.0-main
✅ java@v0.19.1-main
✅ java@main-v0.29.0
✅ js@v0.4.0-v0.29.0

sonarqubecloud · 2026-10-20T17:05:14Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-10-20T18:10:26Z

X-Test Failure Report

✅ go@main-main
✅ js@main-main
✅ java@v0.19.1-main
✅ java@v0.19.1-v0.29.0
✅ java@main-main
✅ go@main-v0.29.0
✅ java@main-v0.29.0
✅ js@v0.4.0-v0.29.0
✅ js@v0.4.0-main
✅ go@v0.39.0-main
✅ go@v0.39.0-v0.29.0

Stages are milestones, not pull requests. Records the stack each stage lands as, with branch names; runs stages 2 and 3 as parallel tracks from main, since they share no imports; and adds a size budget and refactor-first rule to the merge discipline. Also brings the plan up to date: the work that landed as separate fixes (#638, #640, #641, #647), #622's payload sizes, item 7's move into stage 3, the neutral-nightly gap (DSPX-5260), and scratch-disk scheduling (DSPX-5253). Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

This was referenced Sep 23, 2026

feat(xtest): cross-SDK benchmark roll-up and decision-first summaries #623

Draft

feat(xtest): benchmark two named refs via workflow_dispatch #583

Closed

dmihalcik-virtru mentioned this pull request Sep 28, 2026

feat(xtest): compare up to four benchmark arms in one run #621

Closed

dmihalcik-virtru mentioned this pull request Oct 19, 2026

fix(sdk): native CRC-32 on Node, slice-by-16 elsewhere (DSPX-5232) opentdf/web-sdk#1096

Merged

dmihalcik-virtru force-pushed the DSPX-4372-03-payload-sizes branch from 65e13ab to fffd648 Compare October 19, 2026 17:28

dmihalcik-virtru changed the base branch from DSPX-4372-02-karm-core to main October 19, 2026 17:29

This was referenced Oct 19, 2026

ci(xtest): run benchmarks through the local preparation path #660

Draft

docs(xtest): six-stage delivery plan for multi-arm SDK benchmarks #657

Draft

dmihalcik-virtru added 2 commits October 20, 2026 08:10

ci(xtest): report leftover bench scratch files after the run

74bab11

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

dmihalcik-virtru marked this pull request as ready for review October 20, 2026 16:10

dmihalcik-virtru requested review from a team as code owners October 20, 2026 16:10

dmihalcik-virtru mentioned this pull request Oct 20, 2026

chore(xtest): otdf-sdk-mgr install benchmark to resolve, install, and record benchmark arms #626

Merged

pflynn-virtru approved these changes Oct 20, 2026

View reviewed changes

Merge branch 'main' into DSPX-4372-03-payload-sizes

2dd146f

dmihalcik-virtru merged commit d231569 into main Oct 20, 2026
20 of 21 checks passed

dmihalcik-virtru deleted the DSPX-4372-03-payload-sizes branch October 20, 2026 17:38

dmihalcik-virtru mentioned this pull request Oct 21, 2026

chore(xtest): measure benchmark cells over a tuple of arms #669

Draft

Navigation Menu

Repository navigation

feat(xtest): benchmark payloads up to 1 GiB, with per-dispatch budget - #622

feat(xtest): benchmark payloads up to 1 GiB, with per-dispatch budget#622
dmihalcik-virtru merged 4 commits into
mainfrom
DSPX-4372-03-payload-sizes

dmihalcik-virtru commented Sep 23, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 23, 2026 •

edited

Loading

Review limit reached

Sequence Diagram(s)

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

sonarqubecloud Bot commented Oct 20, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dmihalcik-virtru commented Sep 23, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Validation

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Sequence Diagram(s)

Uh oh!

github-actions Bot commented Sep 23, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Oct 20, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Oct 20, 2026

X-Test Results

Uh oh!

sonarqubecloud Bot commented Oct 20, 2026

Quality Gate passed

Uh oh!

Uh oh!

github-actions Bot commented Oct 20, 2026

X-Test Failure Report

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dmihalcik-virtru commented Sep 23, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 23, 2026 •

edited

Loading