[Performance] host_build_graph: shrink ready-queue capacity 65536 -> 8192

[SergioMartin86]

Human Summary

The 65k ready queue capacity is excessive; it is filled with the initial set of ready tasks and then during runtime as scheduler threads detect ready-to-go tasks, and; emptied, as tasks are ultimately scheduled for execution. The capacity of this queue then obeys to a runtime amplitude between how fast schedulers can add them and how AI cores can execute them.

The queue use amplitude never seems to get anywhere near the maximum here, which seems to be a couple orders of magnitude larger than it needs to be, at least. This PR safely reduces the size of the queue, reducing its memory footprint, but also the computational need to move this array from host to device before kernel execution.

AI Summary

PR notes — host_build_graph: shrink ready-queue capacity 65536 → 8192

• Branch: hbg-ready-queue-8192 (commit 29e4738c, off upstream/main cf0fbc06)
• Diff: 2 files, +25 / −19 — the capacity constant plus a graph-queue overflow guardrail
• Arch/runtime: a2a3 host_build_graph only

TL;DR

The per-shape ready queue is a Vyukov ring buffer whose capacity bounds peak
concurrent occupancy, not total task count. At 65536 the nine ready queues
occupy ~13.5 MB of the per-run runtime arena, which is rebuilt and uploaded H2D
on every bind. Measured peak occupancy across the whole HBG onboard suite is
≤ 106 (paged_attention = 1, bgemm = 64), so 65536 was ~600–65000× oversized.
Dropping to 8192 keeps 77–128× headroom and cuts host bind wall by −41.6% on
paged_attention (8627 → 5037 µs), golden-clean, with device time unchanged.

Why the constant was oversized

PTO2ReadyQueue is a Vyukov MPMC ring (slots[capacity], enqueue_pos,
dequeue_pos, mask = capacity − 1). Two facts:

1. Capacity bounds concurrency, not throughput. The ring recycles slots, so
   a queue of capacity C processes unlimited tasks as long as no more than C
   are ever simultaneously enqueued-but-unconsumed (enqueue_pos − dequeue_pos).
   65536 is therefore a worst-case peak-occupancy bound, not one-slot-per-task.
2. The queues dominate the per-run arena. PTO2SchedulerState::reserve_layout
   allocates 9 full-size queues — 3 ready + 3 ready-sync + dummy + graph_ready
   • graph_prepare — each capacity × sizeof(PTO2ReadyQueueSlot) (24 B). At 65536
     that is 9 × 65536 × 24 B ≈ 13.5 MB. This region is re-seeded (arena_build,
     the sequence = i loop over every slot) and copied H2D (arena_upload) on
     every bind. After PR [Performance] HBG: up to 99% host-side overhead reduction #1659 skipped the ~8.5 MB orchestrator block from the
     upload, these queues are the single largest remaining bind cost.

So the queues were being rebuilt and re-shipped at 65536-slot size every run to
hold, in practice, ≤ 106 live entries.

The change

// Per-shape ready-queue capacity (power of two). This is a ring buffer that
// bounds peak CONCURRENT occupancy (enqueue_pos - dequeue_pos), not total task
// count: slots recycle, so capacity need only exceed the most tasks ever
// simultaneously ready in any one queue. Overflow on the ready/sync/dummy queues
// latches PTO2_ERROR_READY_QUEUE_OVERFLOW (safe-fail), so it must exceed the
// worst-case ready burst with margin.
#define PTO2_READY_QUEUE_SIZE 8192   // was 65536

8192 is a power of two (required — index is pos & mask) and gives 77×
headroom over the observed suite max (106) and 128× over the steady per-run
max (bgemm 64).

Graph-queue guardrail (second half of the diff)

All nine queues share PTO2_READY_QUEUE_SIZE, so the shared cut also has to make
the graph_ready path safe-fail. push_ready_routed previously ignored the
return of graph_ready_queue.push(), so a full graph_ready_queue would silently
drop the task and stall the run — unlike ready/sync/dummy, which latch
PTO2_ERROR_READY_QUEUE_OVERFLOW. The graph task is now routed through the same
checked path, so every queue that shares the constant reports overflow as a
named error rather than an anonymous forward-progress timeout.

graph_prepare_queue pushes (scheduler_cold_path.cpp, scheduler_dispatch.cpp)
are already while (!push_tagged(...)) spin-retries that wait for space instead
of dropping, so they are left unchanged.

Measurements

All on a2a3 onboard, paged_attention, median of 50 rounds, from the
[STRACE] bind span tree. The three capacities were measured back-to-back on
one locked device to control for contention.

Bind-phase breakdown (µs)

Phase	65536	8192	2048
arena_upload	2667.1	360.4	221.5
arena_build	2622.7	1490.3	1337.9
orch_reinit	1246.2	1290.9	1241.3
host_orch (rest)	363.0	336.8	352.0
bind subtotal	7126.0	3695.5	3365.8
runner_run (host)	637.3	572.7	568.5
validate	106.3	109.0	96.3

Delta subtotals — Host / Device / Total

Subtotal	65536	8192 (Δ)	2048 (Δ)
Host (total − device)	8556.1	4960.7 (−42.0%)	4620.1 (−46.0%)
Device (device_wall)	71.1	75.8 (flat)	70.5 (flat)
Total (host wall)	8627.2	5036.5 (−41.6%)	4690.6 (−45.6%)

Why 8192 and not lower: 65536 → 8192 captures 91% of the achievable win
(−3591 µs); 8192 → 2048 adds only −346 µs (−6.9%, within host-wall noise) while
cutting overflow headroom 4×. The queue is no longer the bottleneck at 8192 —
arena_upload is already down to 360 µs. Device wall is flat throughout (no
on-NPU regression).

Correctness & peak-occupancy evidence

Peak occupancy is measured with a high-water mark on enqueue_pos − dequeue_pos
sampled at each push (a near-upper-bound: occupancy only rises at an enqueue, so
its max is always caught at a push instant). Suite results:

Workload	peak occupancy	golden @ 8192
qwen3 3-layer graph	106	(measured 65536)
bgemm	64	✓ pass
graph_execution (2 cases)	4	✓ pass
matmul / vector / predicated	≤ 2	—
paged_attention	1	✓ pass
graph_mix_spmd (wide-SPMD + graph)	0–1	✓ pass

At both 8192 and 2048: golden clean on paged_attention / bgemm / graph_execution
/ graph_mix_spmd, no READY_QUEUE_OVERFLOW, no 507018.

Risk analysis

What happens if the cap is ever too low (documented for reviewers):

• ready / ready_sync / dummy queues — safe-fail. push_tagged returns
  false on a full ring (never overwrites an occupied slot → no corruption);
  push_ready_routed latches PTO2_ERROR_READY_QUEUE_OVERFLOW; the scheduler's
  idle/exit check sees it and emergency_shutdowns → the run aborts promptly
  with a named error, not a hang, not a wrong answer.
• graph_ready — now guardrailed by this PR. graph_ready_queue.push() used
  to ignore its return (silent drop → stall); it now routes through the same
  checked path and latches PTO2_ERROR_READY_QUEUE_OVERFLOW. So all queues that
  share the constant are uniformly safe-fail.
• graph_prepare — already safe. Its pushes are while (!push_tagged(...))
  spin-retries that wait for space rather than dropping.

Scope

• a2a3 host_build_graph only. a5 HBG carries the same 65536 constant but is
  left unchanged pending an a5 peak-occupancy measurement — a5 has different
  core counts, hence a different boot ready-burst.
• tensormap_and_ringbuffer is out of scope — a separate runtime that uses
  the queue differently; not analyzed here.

Follow-ups (not in this PR)

1. wide_dispatch coverage — the sim-only SPMD-spill stress
   (tests/st/host_build_graph_wide_dispatch/) is the designated wide-SPMD
   worst case; its kernels don't compile in our container (_Float16 on host
   gcc). If CI's sim lane exercises it, that closes the last worst-case gap.
2. a5 HBG — repeat the occupancy measurement and apply the matching cut.
3. orch_reinit (~1.25 ms) — now the biggest single bind sub-phase and
   queue-independent (the orchestrator is initialized twice per bind: device-SM
   in arena_build, then re-initialized against host-SM in
   run_host_orchestration). The next lever after this PR.

Reproduce

# A/B one workload (baseline vs this branch) — needs a2a3 silicon + a device lock:
python tests/st/a2a3/host_build_graph/paged_attention/test_paged_attention.py \
    -p a2a3 -d <dev> --rounds 50 --skip-golden
# Read the bind span tree (Host wall + bind sub-phases):
python -m simpler_setup.tools.strace_timing <captured-stderr> --tree

# Rebuild the HBG libs after the constant change:
cmake --build build/cache/a2a3/onboard/host_build_graph/host  -j"$(nproc)"
cmake --build build/cache/a2a3/onboard/host_build_graph/aicpu -j"$(nproc)"

[coderabbitai]

📝 Walkthrough

Walkthrough

The per-shape ready-queue capacity changed from 65536 to 8192. Comments now define the capacity as a power-of-two bound based on peak concurrent occupancy. Queue overflow continues to use PTO2_ERROR_READY_QUEUE_OVERFLOW.

Changes

Ready Queue Capacity

Layer / File(s)	Summary
Capacity and overflow contract src/a2a3/runtime/host_build_graph/runtime/pto_runtime2_types.h	PTO2_READY_QUEUE_SIZE is reduced to 8192. Comments document per-shape occupancy and the existing safe-fail overflow latch.

Estimated code review effort: 2 (Simple) | ~5 minutes

Possibly related issues

• [Code Health] hbg: Port host-overhead reduction (#1659) to a5 #1716 — This PR changes the same ready-queue configuration area, but does not include the issue’s broader host-overhead reductions.

Possibly related PRs

• hw-native-sys/simpler#1759 — This PR changes the same ready-queue capacity mechanism and relates to its overflow handling.

Poem

A rabbit trims the queue with care,
From sixty-five thousand to eight thousand there.
The ring stays safe when bounds are crossed,
The error latch records the cost.
Hop, hop—the shape is neat!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: reducing the a2a3 host_build_graph ready-queue capacity.
Description check	✅ Passed	The description directly explains the queue-capacity reduction, performance impact, safety behavior, measurements, and scope.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/a2a3/runtime/host_build_graph/runtime/pto_runtime2_types.h`:
- Around line 97-103: Protect graph queues before applying the shared
PTO2_READY_QUEUE_SIZE capacity: either validate a separate capacity for
graph_ready_queue and graph_prepare_queue, or add overflow detection and
PTO2_ERROR_READY_QUEUE_OVERFLOW handling in the graph scheduling path around
graph_ready_queue.push(slot_state). Ensure graph-queue occupancy is covered by
the capacity sizing so failed pushes cannot silently drop tasks or stall
execution.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3dd7cc12-d96e-491d-a28f-13b450450f7d

📥 Commits

Reviewing files that changed from the base of the PR and between bd5ecb5 and 791f4e3.

📒 Files selected for processing (1)

• src/a2a3/runtime/host_build_graph/runtime/pto_runtime2_types.h