Fix: guard tensor producer resolution on the returned slot pointer

[ChaoWao]

Summary

Follow-ups from the review of #1674, which merged as e041083. No behavior change
on any path that was already correct; each item removes a latent hazard or a
claim the code does not support.

• wait_for_tensor_ready's resolve_producer returns nullptr for a rejected
  producer and both call sites dereference the result immediately, but they
  branched on the failed flag rather than the pointer. That is safe only
  because every rejection path happens to latch failed first — a coupling
  invisible at the call site, and one early-return without the flag turns it
  into a null deref inside the fatal-error path. Branch on the pointer.
• submit_task aborted via always_assert on an empty MixedKernels active
  mask while the adjacent block_num check reports INVALID_ARGS. Both are
  caller input errors; report them the same way. submit_dummy_task is
  deliberately untouched — it calls submit_task_common directly and an empty
  mask is exactly what routes it to the DUMMY bucket.
• Document ThreadCompletionGate's contract: the release/acquire pair that
  makes the finalizer's writes visible to the cleanup claimant, that
  thread_count is a per-round constant rather than a property of one arrival,
  and that reset() is unsynchronized and may only run outside a round.
• Correct the wide-dispatch scene test's docstring, which claimed coverage
  "across the high half of the 128-bit core mask". It sizes cohorts from
  rt_available_cluster_count(), which is SIM_AUTO_BLOCKDIM (8) on the sim
  platforms it is marked for — 24 core-state bits, so it cannot reach a cluster
  offset above bit 63. test_hbg_core_tracker.cpp carries that coverage by
  driving CoreTracker directly at MAX_CLUSTERS.

Both arch trees stay byte-identical for the two .cpp files touched.

Testing

• thread_completion_gate.h compiled standalone with -Wall -Wextra and
  exercised with the blocked-finalizer and reset-reuse cases from the existing
  UT: passes, and passes again under -fsanitize=thread.
• Both edited .cpp files: clang++ -std=c++17 -fsyntax-only clean with the
  UT target's include paths.
• pre-commit on the changed files: clang-format, cpplint, ruff check, ruff
  format, pyright, and the repo header/English/platform-literal hooks all pass.

Not run, and why: this host has neither the scikit-build-core/nanobind build
environment nor an NPU (npu-smi absent), so I could not do an editable rebuild
or run the C++ UTs, the sim sweeps, or any onboard test. The three C++ edits are
a branch-condition swap, an assert-to-error-return conversion, and comments; the
Python edit is docstring and comment only. CI is the first real build of this
branch.

One unrelated environment defect found: the clang-tidy pre-commit hook aborts
with FileNotFoundError on a stale worktree path in PYTHONPATH
(tests/lint/clang_tidy.py:33). It reproduces on a clean tree with all local
changes stashed, so it is not caused by this branch, and CI is unaffected since
it builds a fresh environment.

🤖 Generated with Claude Code

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR replaces assertion-based failures with runtime fatal error reporting for invalid kernel masks. It also uses null producer slots for immediate failure handling and adds documentation for thread completion and wide-dispatch behavior.

Changes

Orchestrator error handling

Layer / File(s)	Summary
Runtime validation for empty active masks src/a2a3/.../pto_orchestrator.cpp, src/a5/.../pto_orchestrator.cpp	submit_task() reports PTO2_ERROR_INVALID_ARGS and returns empty task outputs when no kernel slot is active.
Null-based producer failure handling src/a2a3/.../pto_runtime2.cpp, src/a5/.../pto_runtime2.cpp	wait_for_tensor_ready() checks the nullptr result from resolve_producer() before dereferencing producer slots.
Concurrency and dispatch documentation src/common/utils/thread_completion_gate.h, tests/st/host_build_graph_wide_dispatch/test_host_build_graph_wide_dispatch.py	Comments document completion ordering, reset requirements, dispatch coverage, and buffer capacity bounds.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• hw-native-sys/simpler#1441: Both modify submit_task() in pto_orchestrator.cpp.
• hw-native-sys/simpler#1444: Both modify orchestrator and runtime graph-execution code.
• hw-native-sys/simpler#1674: Both modify producer and consumer error handling in pto_runtime2.cpp.

Poem

A rabbit checked each kernel slot,
And found the empty mask was caught.
No crash, just errors clearly shown,
Null slots stop unsafe paths alone.
The gate and tests now tell their tale.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the primary tensor producer resolution fix in the changeset.
Description check	✅ Passed	The description accurately explains the code fixes, documentation updates, testing, and known test limitations.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/common/utils/thread_completion_gate.h`:
- Around line 32-34: Update the reset() precondition documentation and
enforcement so reset occurs only after the previous round’s cleanup claim has
completed, not merely after the finalizer returns; alternatively synchronize
reset() with any pending claim_cleanup() calls. Ensure cleanup_ready_ cannot be
cleared before claim_cleanup() finishes.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ef00c447-e6de-4828-90be-c0b44d0d2333

📥 Commits

Reviewing files that changed from the base of the PR and between e041083 and 7c83594.

📒 Files selected for processing (6)

• src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/pto_orchestrator.cpp
• src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/pto_runtime2.cpp
• src/a5/runtime/host_build_graph/runtime/orchestrator_core/pto_orchestrator.cpp
• src/a5/runtime/host_build_graph/runtime/orchestrator_core/pto_runtime2.cpp
• src/common/utils/thread_completion_gate.h
• tests/st/host_build_graph_wide_dispatch/test_host_build_graph_wide_dispatch.py

[ChaoWao]

CI triage: st-onboard-a5 failure is a broken NPU card on the runner, not this change

task-submit refused the submission before pytest started:

错误: NPU 卡 7 已损坏，无法使用（本机损坏卡: 7）
请改用其他卡号，或使用 --device auto 自动分配空闲卡

("NPU card 7 is damaged and unusable — use a different card, or --device auto.")

The wheel built fine (~35 s) and the job died at the very next step; total runtime 68 s,
so no scene test ever executed.

Why this is not attributable to the diff:

• The card failed partway through today. st-onboard-a5 passed on this same runner for
  fix/parallelize-a5-dummy-task-resolution (10:27 UTC), worker-async-b6a-enqueue-drain
  (11:03) and l0_0727 (11:18), then failed here at 11:56.
• DEVICE_RANGE is set in the self-hosted runner's own environment;
  .github/workflows/_st-npu.yml only consumes it. This PR touches no file under
  .github/, so nothing here can influence device selection.
• The failure is a submission refusal, not a test assertion, timeout, or 507018.

This needs a maintainer with runner access — either drop card 7 from the a5 runner's
DEVICE_RANGE, switch that step to --device auto, or replace the card. Every a5 onboard
job on this runner will fail until then; it is not specific to this PR.

I could not gh run rerun --failed while sibling jobs are still in flight; I'll retry once
the run settles.

Meanwhile ut-a2a3 has passed, which is the check that actually exercises this diff — it
builds and runs test_thread_completion_gate and test_hbg_core_tracker against the edited
thread_completion_gate.h.

[ChaoWao]

Update: card-7 fault confirmed on an unrelated branch — re-run would be pointless

worker-async-b6a-enqueue-drain (run 30905804059, job 91984193720) hit the identical
failure at 12:01 UTC, five minutes after this PR's:

错误: NPU 卡 7 已损坏，无法使用（本机损坏卡: 7）

Same runner, same refusal, unrelated code. Combined with the three st-onboard-a5 passes
earlier today (10:27 / 11:03 / 11:18), the card clearly failed between 11:18 and 11:56 and
is now blocking every a5 onboard job repo-wide.

I am not re-running it: task-submit refuses at submission, so a retry burns a runner
slot and fails the same way. It needs a maintainer to drop card 7 from the a5 runner's
DEVICE_RANGE, switch that step to --device auto, or replace the card.

Everything else on this PR is green, including all four checks that actually exercise the
diff:

check	result	what it covers here
ut-a2a3	pass	builds + runs test_thread_completion_gate, test_hbg_core_tracker
ut-a5	pass	same tests against a5 headers (MAX_CLUSTERS=36, bit 105)
st-onboard-a2a3	pass	full examples + tests/st sweep on real silicon
st-sim-a2a3sim	pass	host_build_graph_validation + host_build_graph_wide_dispatch

pre-commit also passes, which confirms the clang-tidy failure I saw locally was a stale
PYTHONPATH in my environment, not a defect in the change.

[ChaoWao]

✅ All checks green — st-onboard-a5 passed on re-run

The a5 card recovered on its own (st-onboard-a5 succeeded at 12:24 and 12:36 UTC on other
branches), so I re-ran the failed job. It passed in 7m35s — a real sweep, versus the
1m8s submission refusal earlier. That confirms the original failure was the broken card and
nothing latent in this diff.

Final: 17 pass, 0 fail (deploy: skipping is expected — docs deploy does not run on PRs).

One caveat worth recording for whoever sees this next: the card fault was transient, not
fixed. It passed at 10:27/11:03/11:18, failed at 11:56 (here) and 12:03
(worker-async-b6a-enqueue-drain, unrelated), then passed again from 12:24 onward. Nobody
intervened. Expect it to recur and redden arbitrary PRs; the signature is a task-submit
refusal (NPU 卡 7 已损坏) with a ~60-70s job runtime, versus ~7-8 min for a genuine run —
the duration alone tells the two apart.