Add: benchmark host_build_graph runtime

[doraemonmj]

Summary

• Add host_build_graph support to tools/benchmark_rounds.sh while keeping
  tensormap_and_ringbuffer as the default runtime.
• Define independent benchmark corpora for all four
  {a2a3, a5} × {tensormap_and_ringbuffer, host_build_graph} quadrants.
• Include each runtime-specific Qwen case on both architectures and mark all
  four Qwen cases as manual, keeping them in daily rather than per-PR CI.
• Report only metrics backed by each runtime trace: TMR reports Host / Device /
  Effective / Orch / Sched, while HBG reports Host / Device.
• Skip golden validation during benchmark rounds, fail fast when a configured
  workload is missing, and propagate failures through task-submit, tee, and
  parallel baseline/current comparisons.

Benchmark matrix

Architecture	TMR	HBG
A2/A3	8 invocations, including Qwen	8 invocations, including Qwen
A5	8 invocations, including Qwen	8 invocations, including Qwen

The Qwen entries use StressBatch16Seq3500 for TMR and
GraphExecutionBatch16Seq3500 for HBG. Tests assert those runtime-specific
case names in all four quadrants and verify every Qwen benchmark case is
manual.

The shared corpus covers alternating_matmul_add, benchmark_bgemm,
paged_attention_unroll, paged_attention_unroll_manual_scope, and
batch_paged_attention. spmd_paged_attention remains outside the benchmark
corpus because of a pre-existing onboard stall reproduced on the baseline.

Timing scope

This PR provides the HBG regression surface supported by markers available
today. A follow-up performance change can add HBG-specific record-vs-replay
reporting and finer host stages such as graph build, relocation, H2D upload,
and dispatch without coupling that instrumentation design to the harness.

Testing

• Targeted benchmark/manual/timing unit tests: 56 passed.
• A5 Qwen collection: both cases deselected with the per-PR default and both
  collected with the daily --manual include mode.
• Pre-commit hooks passed for every file changed from the latest main,
  including markdownlint, Ruff, formatting, and pyright.
• Routing tests cover the exact corpus for all four architecture/runtime
  quadrants, runtime-specific Qwen selection, --skip-golden, manual-case
  inclusion, missing-workload failure, and Bash 3.2 compatibility.

Part of #1727.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 34a23235-1edd-49d7-a63e-8d5957d8b899

📥 Commits

Reviewing files that changed from the base of the PR and between 50c0660 and 2362637.

📒 Files selected for processing (11)

• .claude/skills/benchmark/SKILL.md
• .claude/skills/perf-example-device/SKILL.md
• .claude/skills/perf-runtime-device/SKILL.md
• docs/dfx/host-trace.md
• docs/dfx/l2-timing.md
• simpler_setup/tools/README.md
• simpler_setup/tools/strace_timing.py
• tests/ut/py/test_benchmark_rounds.py
• tests/ut/py/test_strace_timing.py
• tools/README.md
• tools/benchmark_rounds.sh

---

📝 Walkthrough

Walkthrough

The benchmark workflow adds host_build_graph support, task-based device allocation, architecture prechecks, runtime-specific execution modes, and runtime-specific timing reports. Documentation and tests cover HBG corpus selection, TMR compatibility, marker parsing, and conditional report columns.

Changes

Benchmark runtime and execution

Layer / File(s)	Summary
Runtime selection and invocation tools/benchmark_rounds.sh, tests/ut/py/test_benchmark_rounds.py	The harness selects TMR or HBG corpora and default modes. HBG rejects serial orchestration flags. Tests cover corpus selection, defaults, validation, and help output.
Shared-host task allocation .claude/skills/benchmark/SKILL.md	Single and compare workflows use one task-submit allocation for architecture checks, preparation, device ownership, and benchmark execution.
Runtime-specific timing reporting .claude/skills/..., docs/dfx/..., simpler_setup/tools/..., tools/README.md, tests/ut/py/test_strace_timing.py	TMR reports Host, Device, Effective, Orch, and Sched. HBG reports Host and Device. Unsupported columns are omitted from rounds tables and documentation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• [Feature] hbg: Add benchmark cases and harness/skill support (default stays tmr) #1727 — The issue covers the HBG benchmark harness, corpus selection, runtime selection, and runtime-specific metrics implemented here.

Possibly related PRs

• hw-native-sys/simpler#1575 — Both changes extend tools/benchmark_rounds.sh for runtime-specific benchmark execution.
• hw-native-sys/simpler#1730 — Both changes modify runtime-specific metric parsing and reporting in strace_timing.py.
• hw-native-sys/simpler#1771 — This change adds benchmark support for the HBG benchmark introduced there.

Poem

A rabbit checks the task queue bright,
Then claims one device for the night.
TMR tracks each phase in flight,
HBG keeps Host and Device right.
Clean markers make the table light.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: adding benchmark support for the host_build_graph runtime.
Description check	✅ Passed	The description directly explains the runtime support, benchmark matrix, metrics, validation, documentation, and testing changes.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ChaoZheng109]

Recommend holding this until the benchmark matrix covers all four arch × runtime quadrants.

The harness should guarantee the full {a2a3, a5} × {tensormap_and_ringbuffer, host_build_graph} matrix, not just light up a2a3-HBG. Current state:

quadrant	status
a2a3 × TMR	full (default)
a2a3 × HBG	this PR
a5 × TMR	partial — benchmark_bgemm / qwen3_14b_decode have no a5 example and the list isn't arch-aware, so they SKIP/FAIL on -p a5
a5 × HBG	no benchmark corpus (only basic functional tests); runtime still a temp a2a3 copy

Since we're reworking the benchmark story anyway, please close a5 for both runtimes (in this PR or a companion that lands together) so the matrix is symmetric, rather than leaving a5 gated behind an error.

[ChaoZheng109]

Review 反馈

整体实现质量不错(空列隐藏逻辑 + 对应单测、EXTRA_ARGS_COUNT 对 macOS Bash 3.2 set -u 的修复、HBG 前置校验都到位),in-tree 文档也准确同步了 "TMR 有 orch/sched 细分、HBG 只有 device_wall"。但有一个功能缺口和一处 body/代码不一致,建议合并前处理。

🔴 1. HBG 语料漏了 qwen(且被单测锁死)

a2a3 上 qwen 两个 runtime 都有示例:

runtime	示例	case
TMR	examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/	StressBatch16Seq3500
HBG	examples/a2a3/host_build_graph/qwen3_14b_decode/	GraphExecutionBatch16Seq3500

但 HBG_EXAMPLE_CASES 只有 7 条、不含 qwen —— 这是漏加,不是 "HBG 不支持"。结果 a2a3 HBG 实际是 7 条,而 PR body 矩阵写的是 "8 including Qwen"。

由于 HBG runtime 已被 gate 成 a2a3-only,直接把 qwen 加进 HBG_EXAMPLE_CASES 即可,天然只在 a2a3 生效、不会漏到 a5(注意 case 名与 TMR 不同):

HBG_EXAMPLE_CASES=(
    "alternating_matmul_add=Case1"
    "benchmark_bgemm=Case0"
    "paged_attention_unroll=Case1,Case2"
    "paged_attention_unroll_manual_scope=Case1,Case2"
    "batch_paged_attention=Case1"
    "qwen3_14b_decode=GraphExecutionBatch16Seq3500"
)

test_host_build_graph_runs_its_matching_benchmark_corpus 现在把 7 条列表写死成断言,所以它"绿"恰好锁死了这个缺口 —— 需同步改成 8 条并补上 qwen 断言。

🟠 2. PR body 与代码/实现有多处不一致

body 描述的是一个比代码更宏大的设计(疑似 squash 简化后 body 未更新):

• "四象限独立语料 / arch-aware lists" —— 代码实际只有 2 张按 runtime 的扁平表,无 arch 拆分。
• "A2/A3 HBG = 8 含 Qwen" —— 见上,实际 7 无 Qwen。
• "A5 HBG = 7 条" —— 代码对 HBG 非 a2a3 直接报错退出,并不会跑。
• "mark 两个 A2/A3 Qwen 为 manual + 每-PR CI 排除" —— 本 PR diff 未包含这些改动。

建议把 body 改为匹配实际实现(简化版),或明确说明哪些属于 #1727 后续 PR。

🟡 3. (pre-existing,非本 PR 引入)a5 + TMR 会因 qwen 失败

TMR 表无 arch 闸且无条件含 qwen,benchmark_rounds.sh -p a5 跑到 qwen 会 SKIP → FAIL → 非零退出(基线即如此)。若想彻底兑现 body 的 "arch-aware 避免 A5 skip/fail" 说法,可把语料改成按 arch × runtime 拆表,顺带修掉这个老问题;否则至少把 body 里相关声称去掉。

---

小结:功能上补 qwen + 改单测即可闭合缺口;文档/body 层面让描述与代码对齐。

[doraemonmj]

@ChaoZheng109 Addressed:

• Rebased the single PR commit onto the latest main.
• Kept independent corpora for all four architecture/runtime quadrants.
• Added the newly merged A5 TMR/HBG Qwen scenes to their matching corpora; all four quadrants now run eight invocations including the runtime-specific Qwen case.
• Marked the A5 TMR/HBG Qwen cases manual, matching A2/A3, so per-PR CI excludes all four and Daily includes them.
• Updated the PR body and regression assertions to match the final matrix.