Add: polling completion scheduler for host_build_graph (a2a3) - #1435
Merged
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe runtime now uses inline producer local IDs, shared-memory completion flags, wake lists, and a completed watermark for polling-based readiness and reclamation. Dependency pools, fanout wiring, and deferred producer-release paths are removed across orchestration, scheduling, memory layout, and host runtime preparation. ChangesPolling scheduler port
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp (1)
359-418: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winStale doc header contradicts the narrowed relocation.
The block comment immediately above
relocate_host_orch_image(and the file namespace comment) still describes the pre-polling model — relocatingfanin_inline_slot_states[],fanout_head, thedep_pool/dep-entry/ready-queue web across two deltas, and aPTO2_FANIN_INLINE_CAPspill-fatal path. The implementation now relocates onlyss->task/ss->payload, and the new inline comment at Line 409 already states this. Please trim the header so the two don't disagree and mislead future maintainers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp` around lines 359 - 418, Update the documentation immediately above relocate_host_orch_image and the related file namespace comment to describe only the current relocation behavior: relocating each slot’s task and payload pointers across the SM or arena windows. Remove references to fanin_inline_slot_states, fanout_head, dep_pool/dep-entry, ready queues, dual-delta traversal, and PTO2_FANIN_INLINE_CAP spill handling, while leaving the implementation unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/a2a3/runtime/host_build_graph/runtime/scheduler/pto_scheduler.h`:
- Around line 577-586: Update the watermark-advancement loop in the surrounding
completion handler to remove the my_id-based upper bound and continue folding
consecutive set completion_flags entries until the first unset flag. Preserve
the existing atomic compare_exchange_weak progression and rely on the zeroed
unsubmitted slots to terminate traversal without wrapping.
---
Nitpick comments:
In `@src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp`:
- Around line 359-418: Update the documentation immediately above
relocate_host_orch_image and the related file namespace comment to describe only
the current relocation behavior: relocating each slot’s task and payload
pointers across the SM or arena windows. Remove references to
fanin_inline_slot_states, fanout_head, dep_pool/dep-entry, ready queues,
dual-delta traversal, and PTO2_FANIN_INLINE_CAP spill handling, while leaving
the implementation unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1f142d85-4728-4ea1-bf04-40de02595875
📒 Files selected for processing (19)
src/a2a3/runtime/host_build_graph/docs/polling-port-plan.mdsrc/a2a3/runtime/host_build_graph/host/runtime_maker.cppsrc/a2a3/runtime/host_build_graph/runtime/orchestrator_core/pto_orchestrator.cppsrc/a2a3/runtime/host_build_graph/runtime/orchestrator_core/pto_ring_buffer.cppsrc/a2a3/runtime/host_build_graph/runtime/orchestrator_core/pto_runtime2.cppsrc/a2a3/runtime/host_build_graph/runtime/pto_async_wait.hsrc/a2a3/runtime/host_build_graph/runtime/pto_orchestrator.hsrc/a2a3/runtime/host_build_graph/runtime/pto_ring_buffer.hsrc/a2a3/runtime/host_build_graph/runtime/pto_runtime2.hsrc/a2a3/runtime/host_build_graph/runtime/pto_runtime2_types.hsrc/a2a3/runtime/host_build_graph/runtime/pto_shared_memory.hsrc/a2a3/runtime/host_build_graph/runtime/scheduler/pto_scheduler.cppsrc/a2a3/runtime/host_build_graph/runtime/scheduler/pto_scheduler.hsrc/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cppsrc/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_completion.cppsrc/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.hsrc/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cppsrc/a2a3/runtime/host_build_graph/runtime/shared/pto_runtime2_init.cppsrc/a2a3/runtime/host_build_graph/runtime/shared/pto_shared_memory.cpp
💤 Files with no reviewable changes (1)
- src/a2a3/runtime/host_build_graph/runtime/scheduler/pto_scheduler.cpp
This was referenced Jul 22, 2026
ChaoWao
changed the title
6 tasks
ChaoWao
force-pushed
the
hbg-polling
branch
2 times, most recently
from
July 22, 2026 13:01
9177eb5 to
eeeb8b9
Compare
Replace host_build_graph's wiring dependency subsystem with the polling completion scheduler (introduced for tensormap_and_ringbuffer in hw-native-sys#1137). Removed: dep_pool / PTO2FaninPool / PTO2DepListPool, fanin_refcount / fanout_refcount / fanout_head / fanout_lock, the two-phase deferred producer-release, and most of relocate_host_orch_image. Added: per-slot completion_flags byte + monotonic completed_watermark + intrusive wake_list; position-independent inline integer fanin (fanin_local_ids) with last_consumer_local_id; a device boot initial-classify pass. Fits hbg's single-ring, whole-graph-resident, host-orchestrated model: integer fanin collapses the host->device pointer relocation to task/payload only. Only the dependency subsystem changes — dispatch, core management, sync_start, and the async/deferred-completion mailbox are preserved (deferred tasks reach the same on_mixed_task_complete endpoint, so they publish completion_flags too). Early-dispatch is stubbed for this milestone. Host-orch specifics: task_state is kept as the host-visible completion mirror (host polls it in wait_for_tensor_ready, the allocator deadlock detector, and the cold-path dump); completed_watermark is load-bearing — wait_for_consumers gates on watermark >= producer.last_consumer_local_id; hidden-alloc producers completed on the host pre-set their completion_flags in the H2D image. completed_watermark advances over the full contiguous completed prefix (bounded by current_task_index, not the completing task's id): a my_id cap would make the final value completion-order-dependent and hang wait_for_consumers when a low-id task completes last. Reconciles with predicated dispatch (hw-native-sys#1434): a predicate-failing task routes to dummy_ready_queue and retires inline, evaluated at push_ready_routed. docs: replace the migration plan with an updated RUNTIME_LOGIC.md describing the polling dependency/completion model. Validated: onboard + sim builds (host/aicpu/aicore); full hbg st suite passes on a2a3sim (11) and onboard a2a3 (11 pass / 1 skip), including predicated_dispatch; a2a3sim dfx-off build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Sergio Martin <sergio.miguel.martin@huawei.com>
ChaoZheng109
added a commit
to ChaoZheng109/simpler
that referenced
this pull request
Jul 23, 2026
Trim low-signal LOG_* call sites across the runtime and platform layers, keeping the diagnostics that trace program flow or failures: - Flow / lifecycle / summary logs kept at INFO; per-item (per-tensor, per-arg) detail demoted to LOG_DEBUG; failure paths stay LOG_ERROR / LOG_WARN. Loader / device-runner load-path diagnostics kept at their original levels. - AICPU hot paths (per-task / per-scope dispatch loops) carry no logs (codestyle rule 7); DFX collectors keep their buffer-level init/flush diagnostics and telemetry counters. - STRACE, LOG_RUNTIME_FAILURE and ACL logging left untouched. - Log-only support removed with its logs (bookkeeping counters, dump scaffolding); genuine out-of-scope strip collateral restored (unused-parameter removals, blank-line deletions). Rebased onto upstream/main; dense dependency diagnostics follow upstream's debug-only / removed decisions (hw-native-sys#1448), completion follows the polling mechanism (hw-native-sys#1435).
ChaoZheng109
added a commit
to ChaoZheng109/simpler
that referenced
this pull request
Jul 27, 2026
Trim low-signal LOG_* call sites across the runtime and platform layers, keeping the diagnostics that trace program flow or failures: - Flow / lifecycle / summary logs kept at INFO; per-item (per-tensor, per-arg) detail demoted to LOG_DEBUG; failure paths stay LOG_ERROR / LOG_WARN. Loader / device-runner load-path diagnostics kept at their original levels. - AICPU hot paths (per-task / per-scope dispatch loops) carry no logs (codestyle rule 7); DFX collectors keep their buffer-level init/flush diagnostics and telemetry counters. - STRACE, LOG_RUNTIME_FAILURE and ACL logging left untouched. - Log-only support removed with its logs (bookkeeping counters, dump scaffolding); genuine out-of-scope strip collateral restored (unused-parameter removals, blank-line deletions). Rebased onto upstream/main; dense dependency diagnostics follow upstream's debug-only / removed decisions (hw-native-sys#1448), completion follows the polling mechanism (hw-native-sys#1435). Co-Authored-By: vegetabledoww <114059112+vegetabledoww@users.noreply.github.com>
ChaoZheng109
added a commit
to ChaoZheng109/simpler
that referenced
this pull request
Jul 27, 2026
Trim low-signal LOG_* call sites across the runtime and platform layers, keeping the diagnostics that trace program flow or failures: - Flow / lifecycle / summary logs kept at INFO; per-item (per-tensor, per-arg) detail demoted to LOG_DEBUG; failure paths stay LOG_ERROR / LOG_WARN. Loader / device-runner load-path diagnostics kept at their original levels. - AICPU hot paths (per-task / per-scope dispatch loops) carry no logs (codestyle rule 7); DFX collectors keep their buffer-level init/flush diagnostics and telemetry counters. - STRACE, LOG_RUNTIME_FAILURE and ACL logging left untouched. - Log-only support removed with its logs (bookkeeping counters, dump scaffolding); genuine out-of-scope strip collateral restored (unused-parameter removals, blank-line deletions). Rebased onto upstream/main; dense dependency diagnostics follow upstream's debug-only / removed decisions (hw-native-sys#1448), completion follows the polling mechanism (hw-native-sys#1435). Co-Authored-By: vegetabledoww <114059112+vegetabledoww@users.noreply.github.com>
ChaoZheng109
added a commit
that referenced
this pull request
Jul 27, 2026
Trim low-signal LOG_* call sites across the runtime and platform layers, keeping the diagnostics that trace program flow or failures: - Flow / lifecycle / summary logs kept at INFO; per-item (per-tensor, per-arg) detail demoted to LOG_DEBUG; failure paths stay LOG_ERROR / LOG_WARN. Loader / device-runner load-path diagnostics kept at their original levels. - AICPU hot paths (per-task / per-scope dispatch loops) carry no logs (codestyle rule 7); DFX collectors keep their buffer-level init/flush diagnostics and telemetry counters. - STRACE, LOG_RUNTIME_FAILURE and ACL logging left untouched. - Log-only support removed with its logs (bookkeeping counters, dump scaffolding); genuine out-of-scope strip collateral restored (unused-parameter removals, blank-line deletions). Rebased onto upstream/main; dense dependency diagnostics follow upstream's debug-only / removed decisions (#1448), completion follows the polling mechanism (#1435). Co-authored-by: vegetabledoww <114059112+vegetabledoww@users.noreply.github.com>
This was referenced Jul 28, 2026
This was referenced Aug 6, 2026
SergioMartin86
added a commit
to huawei-csl/simpler
that referenced
this pull request
Aug 12, 2026
The on-device Graph scheduler expanded a GRAPH task into all its nodes before dispatching any of them: graph_execution_materialize_slice reached PREPARED only after the last node, and activate_prepared_graph routed the roots only at that point. For a large graph the AICores idle through the whole expansion — on the qwen3-14B 3-layer decode that front-loaded ramp is ~530 us per layer during which every core waits. Route roots (and any node whose producers have all completed) as soon as they materialize, once the outer Graph task's external-dependency gate has opened, so compute overlaps the remaining expansion. This is possible only because of the polling-completion scheduler: the per-slot completion_flags byte + task_state completion mirror introduced for tensormap_and_ringbuffer in hw-native-sys#1137 and adapted to host_build_graph in hw-native-sys#1435. That flag lets a consumer discover a producer that completed mid-expansion by reading its state instead of relying solely on the producer's wake list. Under the older pure wake-list model, dispatching before full materialization would have lost wakes if a producer finished before its consumer registered — which is exactly why the whole-graph PREPARED gate existed. - Materialization-time registration moves from register_initial_graph_waiter (a raw wake-list store that assumed no producer could complete during expansion) to register_graph_wake, which resolves a producer that completed mid-expansion through its task_state completion flag (the hw-native-sys#1137 / hw-native-sys#1435 mechanism) instead of losing the wake. The graph's topological node order guarantees every producer a node references is already constructed. - execution.nodes is published at MATERIALIZING (under materialize_busy) so the wake path can read producer slots while the graph is still expanding. - Roots reach the ready queue through route_cursor, a monotonic per-execution cursor that makes routing idempotent across the per-slice calls and the final call at the activation meet, so each root is pushed exactly once. Non-roots reach the queue only through their producers' wake list, unchanged. Correctness: the three graph_execution scene tests (2D replay, AIC+AIV, MIX-SPMD) and the qwen3-14B 3-layer graph-execution golden pass, including a 3-round replay and a 45-run repeat. Effect (qwen3-14B 3-layer, a2a3 onboard, same-device A/B, 15 rounds): device_wall 3397.8 -> 3247.1 us (-4.4%); the chip swimlane shows compute begins overlapping graph expansion instead of waiting for it. The first-compute start is still bounded by the outer task's external-dependency latency, so the recovered time is the expansion tail, not an earlier start. Applies the identical change to the a5 host_build_graph sibling (same graph scheduler; a5 is compile-verified, not perf-measured on this a2a3 box).
SergioMartin86
added a commit
to huawei-csl/simpler
that referenced
this pull request
Aug 12, 2026
The on-device Graph scheduler expanded a GRAPH task into all its nodes before dispatching any of them: graph_execution_materialize_slice reached PREPARED only after the last node, and activate_prepared_graph routed the roots only at that point. For a large graph the AICores idle through the whole expansion — on the qwen3-14B 3-layer decode that front-loaded ramp is ~530 us per layer during which every core waits. Route roots (and any node whose producers have all completed) as soon as they materialize, once the outer Graph task's external-dependency gate has opened, so compute overlaps the remaining expansion. This is possible only because of the polling-completion scheduler: the per-slot completion_flags byte + task_state completion mirror introduced for tensormap_and_ringbuffer in hw-native-sys#1137 and adapted to host_build_graph in hw-native-sys#1435. That flag lets a consumer discover a producer that completed mid-expansion by reading its state instead of relying solely on the producer's wake list. Under the older pure wake-list model, dispatching before full materialization would have lost wakes if a producer finished before its consumer registered — which is exactly why the whole-graph PREPARED gate existed. - Materialization-time registration moves from register_initial_graph_waiter (a raw wake-list store that assumed no producer could complete during expansion) to register_graph_wake, which resolves a producer that completed mid-expansion through its task_state completion flag (the hw-native-sys#1137 / hw-native-sys#1435 mechanism) instead of losing the wake. The graph's topological node order guarantees every producer a node references is already constructed. - execution.nodes is published at MATERIALIZING (under materialize_busy) so the wake path can read producer slots while the graph is still expanding. - Roots reach the ready queue through route_cursor, a monotonic per-execution cursor that makes routing idempotent across the per-slice calls and the final call at the activation meet, so each root is pushed exactly once. Non-roots reach the queue only through their producers' wake list, unchanged. Correctness: the three graph_execution scene tests (2D replay, AIC+AIV, MIX-SPMD) and the qwen3-14B 3-layer graph-execution golden pass, including a 3-round replay and a 45-run repeat. Effect (qwen3-14B 3-layer, a2a3 onboard, same-device A/B, 15 rounds): device_wall 3397.8 -> 3247.1 us (-4.4%); the chip swimlane shows compute begins overlapping graph expansion instead of waiting for it. The first-compute start is still bounded by the outer task's external-dependency latency, so the recovered time is the expansion tail, not an earlier start. Applies the identical change to the a5 host_build_graph sibling (same graph scheduler; a5 is compile-verified, not perf-measured on this a2a3 box).
SergioMartin86
added a commit
to huawei-csl/simpler
that referenced
this pull request
Aug 13, 2026
The on-device Graph scheduler expanded a GRAPH task into all its nodes before dispatching any of them: graph_execution_materialize_slice reached PREPARED only after the last node, and activate_prepared_graph routed the roots only at that point. For a large graph the AICores idle through the whole expansion — on the qwen3-14B 3-layer decode that front-loaded ramp is ~530 us per layer during which every core waits. Route roots (and any node whose producers have all completed) as soon as they materialize, once the outer Graph task's external-dependency gate has opened, so compute overlaps the remaining expansion. This is possible only because of the polling-completion scheduler: the per-slot completion_flags byte + task_state completion mirror introduced for tensormap_and_ringbuffer in hw-native-sys#1137 and adapted to host_build_graph in hw-native-sys#1435. That flag lets a consumer discover a producer that completed mid-expansion by reading its state instead of relying solely on the producer's wake list. Under the older pure wake-list model, dispatching before full materialization would have lost wakes if a producer finished before its consumer registered — which is exactly why the whole-graph PREPARED gate existed. - Materialization-time registration moves from register_initial_graph_waiter (a raw wake-list store that assumed no producer could complete during expansion) to register_graph_wake, which resolves a producer that completed mid-expansion through its task_state completion flag (the hw-native-sys#1137 / hw-native-sys#1435 mechanism) instead of losing the wake. The graph's topological node order guarantees every producer a node references is already constructed. - execution.nodes is published at MATERIALIZING (under materialize_busy) so the wake path can read producer slots while the graph is still expanding. - Roots reach the ready queue through route_cursor, a monotonic per-execution cursor that makes routing idempotent across the per-slice calls and the final call at the activation meet, so each root is pushed exactly once. Non-roots reach the queue only through their producers' wake list, unchanged. Correctness: the three graph_execution scene tests (2D replay, AIC+AIV, MIX-SPMD) and the qwen3-14B 3-layer graph-execution golden pass, including a 3-round replay and a 45-run repeat. Two host-side unit tests (tests/ut/cpp/{a2a3,a5}/test_graph_activation.cpp) deterministically force the producer-completes-before-consumer-registers interleaving that the scene tests hit only probabilistically, asserting register_graph_wake and graph_incremental_publish route the consumer through the task_state re-scan rather than losing it on an already-drained wake list. Effect (qwen3-14B 3-layer, a2a3 onboard, same-device A/B, 15 rounds): device_wall 3397.8 -> 3247.1 us (-4.4%); the chip swimlane shows compute begins overlapping graph expansion instead of waiting for it. The first-compute start is still bounded by the outer task's external-dependency latency, so the recovered time is the expansion tail, not an earlier start. Applies the identical change to the a5 host_build_graph sibling (same graph scheduler; a5 is compile-verified, not perf-measured on this a2a3 box).
SergioMartin86
added a commit
to huawei-csl/simpler
that referenced
this pull request
Aug 13, 2026
The on-device Graph scheduler expanded a GRAPH task into all its nodes before dispatching any of them: graph_execution_materialize_slice reached PREPARED only after the last node, and activate_prepared_graph routed the roots only at that point. For a large graph the AICores idle through the whole expansion — on the qwen3-14B 3-layer decode that front-loaded ramp is ~530 us per layer during which every core waits. Route roots (and any node whose producers have all completed) as soon as they materialize, once the outer Graph task's external-dependency gate has opened, so compute overlaps the remaining expansion. This is possible only because of the polling-completion scheduler: the per-slot completion_flags byte + task_state completion mirror introduced for tensormap_and_ringbuffer in hw-native-sys#1137 and adapted to host_build_graph in hw-native-sys#1435. That flag lets a consumer discover a producer that completed mid-expansion by reading its state instead of relying solely on the producer's wake list. Under the older pure wake-list model, dispatching before full materialization would have lost wakes if a producer finished before its consumer registered — which is exactly why the whole-graph PREPARED gate existed. - Materialization-time registration moves from register_initial_graph_waiter (a raw wake-list store that assumed no producer could complete during expansion) to register_graph_wake, which resolves a producer that completed mid-expansion through its task_state completion flag (the hw-native-sys#1137 / hw-native-sys#1435 mechanism) instead of losing the wake. The graph's topological node order guarantees every producer a node references is already constructed. - execution.nodes is published at MATERIALIZING (under materialize_busy) so the wake path can read producer slots while the graph is still expanding. - Roots reach the ready queue through route_cursor, a monotonic per-execution cursor that makes routing idempotent across the per-slice calls and the final call at the activation meet, so each root is pushed exactly once. Non-roots reach the queue only through their producers' wake list, unchanged. Because a node now dispatches before the graph reaches ACTIVE, it can complete while the graph is still MATERIALIZING or PREPARED. hw-native-sys#1767's complete_task rejects a node completion unless the graph is ACTIVE; that guard now accepts MATERIALIZING and PREPARED as well, rejecting only SUBMITTED (execution not yet localized) and COMPLETED (already retired). Without this, the largest graphs (longest materialization tail) fault with sched_error_code=5 INVALID_ARGS while the small graph_execution scene tests, which reach ACTIVE before any node completes, pass. Correctness: the three graph_execution scene tests (2D replay, AIC+AIV, MIX-SPMD) and the qwen3-14B 3-layer graph-execution golden pass, and the qwen3-14B 40-layer host-graph decode (the largest graph) passes 6/6 where it faulted 0/6 without the completion-state fix. Three host-side unit tests (tests/ut/cpp/{a2a3,a5}/test_graph_activation.cpp) cover the interleavings deterministically: a consumer registered after its producer completed and drained routes through the task_state re-scan; graph_incremental_publish routes all-producers-complete consumers and wake-chains the rest; and complete_task accepts a node completion in MATERIALIZING/PREPARED/ACTIVE and rejects it in SUBMITTED/COMPLETED. Applies the identical change to the a5 host_build_graph sibling (same graph scheduler; a5 is compile-verified, not perf-measured on this a2a3 box).
ChaoWao
pushed a commit
that referenced
this pull request
Aug 14, 2026
…ize (#1804) The on-device Graph scheduler expanded a GRAPH task into all its nodes before dispatching any of them: graph_execution_materialize_slice reached PREPARED only after the last node, and activate_prepared_graph routed the roots only at that point. For a large graph the AICores idle through the whole expansion — on the qwen3-14B 3-layer decode that front-loaded ramp is ~530 us per layer during which every core waits. Route roots (and any node whose producers have all completed) as soon as they materialize, once the outer Graph task's external-dependency gate has opened, so compute overlaps the remaining expansion. This is possible only because of the polling-completion scheduler: the per-slot completion_flags byte + task_state completion mirror introduced for tensormap_and_ringbuffer in #1137 and adapted to host_build_graph in #1435. That flag lets a consumer discover a producer that completed mid-expansion by reading its state instead of relying solely on the producer's wake list. Under the older pure wake-list model, dispatching before full materialization would have lost wakes if a producer finished before its consumer registered — which is exactly why the whole-graph PREPARED gate existed. - Materialization-time registration moves from register_initial_graph_waiter (a raw wake-list store that assumed no producer could complete during expansion) to register_graph_wake, which resolves a producer that completed mid-expansion through its task_state completion flag (the #1137 / #1435 mechanism) instead of losing the wake. The graph's topological node order guarantees every producer a node references is already constructed. - execution.nodes is published at MATERIALIZING (under materialize_busy) so the wake path can read producer slots while the graph is still expanding. - Roots reach the ready queue through route_cursor, a monotonic per-execution cursor that makes routing idempotent across the per-slice calls and the final call at the activation meet, so each root is pushed exactly once. Non-roots reach the queue only through their producers' wake list, unchanged. Because a node now dispatches before the graph reaches ACTIVE, it can complete while the graph is still MATERIALIZING or PREPARED. #1767's complete_task rejects a node completion unless the graph is ACTIVE; that guard now accepts MATERIALIZING and PREPARED as well, rejecting only SUBMITTED (execution not yet localized) and COMPLETED (already retired). Without this, the largest graphs (longest materialization tail) fault with sched_error_code=5 INVALID_ARGS while the small graph_execution scene tests, which reach ACTIVE before any node completes, pass. Correctness: the three graph_execution scene tests (2D replay, AIC+AIV, MIX-SPMD) and the qwen3-14B 3-layer graph-execution golden pass, and the qwen3-14B 40-layer host-graph decode (the largest graph) passes 6/6 where it faulted 0/6 without the completion-state fix. Three host-side unit tests (tests/ut/cpp/{a2a3,a5}/test_graph_activation.cpp) cover the interleavings deterministically: a consumer registered after its producer completed and drained routes through the task_state re-scan; graph_incremental_publish routes all-producers-complete consumers and wake-chains the rest; and complete_task accepts a node completion in MATERIALIZING/PREPARED/ACTIVE and rejects it in SUBMITTED/COMPLETED. Applies the identical change to the a5 host_build_graph sibling (same graph scheduler; a5 is compile-verified, not perf-measured on this a2a3 box).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the polling completion scheduler (introduced for
tensormap_and_ringbufferin #1137) into thehost_build_graph(hbg) runtime, replacing hbg's wiring dependency subsystem. This is Milestone 1 (core polling; predicated early-dispatch is stubbed,sync_startpreserved).Wiring removed:
dep_pool/PTO2FaninPool/PTO2DepListPool,fanin_refcount/fanout_refcount/fanout_head/fanout_lock, the two-phase deferred producer-release, and most ofrelocate_host_orch_image.Polling in: per-slot
completion_flagsbyte + monotoniccompleted_watermark+ intrusivewake_list; position-independent inline integer fanin (fanin_local_ids); a device boot initial-classify pass.Why it's a good fit for hbg
hbg is single-ring (
PTO2_MAX_RING_DEPTH==1), whole-graph-resident, and host-orchestrated (host builds the whole graph → relocates → H2Ds; AICPU boots scheduler-only). Polling makes the dependency representation position-independent integers, which collapses the fragile host→device pointer relocation (relocate_host_orch_image) to justtask/payload— the single-ring model also removes the cross-ring reclaim gap entirely.Only the dependency subsystem is swapped; dispatch / core-management /
sync_start/ async-mailbox are preserved.Three host-orch-specific correctness points
task_stateis kept — the host still polls it (wait_for_tensor_ready, allocator deadlock detector, cold-path dump);on_mixed_task_completesetscompletion_flags(device readiness) and mirrorstask_state(host).completed_watermarkis load-bearing — the hostwait_for_consumersnow blocks onwatermark >= producer.last_consumer_local_id(wasfanout_refcount == fanout_count).completion_flagsin the H2D image, so device consumers don'tregister_wakeon a producer that never runs on device.Validation
pytest tests/st/a2a3/host_build_graph→ 10 passed, 1 skipped (matmul, bgemm, vector_example, dump_args, paged_attention [multi-producer fanin + wake-list drain], prepared_callable [hidden-alloc]).tensormap_and_ringbuffer).Notes for review
src/a2a3/runtime/host_build_graph/docs/polling-port-plan.md(this PR).publish_flagsearly-dispatch that also fixes tmr'spredicated_dispatch) is a follow-up.🤖 Generated with Claude Code