Refactor: centralize chip native run ownership

[Crane-Liu]

Summary

• add ChipRunLane / ChipRun as the C++ owner of native FIFO admission, generation, prepare/launch/poll/finalize, terminal state, poison, drain, and close
• route both the direct adapter and local endpoint child loop through that lane while keeping public direct Worker.submit() blocking
• split mailbox phase from preparation disposition (VALIDATED_ONLY / NATIVE_PREPARED) and bump the task protocol to v3
• remove native run policy from Python; Python retains frame validation, activation publication, registry control deferral, and mailbox state publication

Scope

This is W1a + E2 only. It does not add public asynchronous submit, W1d cleanup/folding, B5 compatibility, or W1b.

Testing

• all pre-commit hooks passed, including clang-tidy, cpplint, Ruff, and Pyright
• Python no-hardware UT: 1175 passed, 13 skipped, 14 deselected
• C++ no-hardware UT: 91/91 passed
• targeted ChipRunLane: 12/12 passed; scheduler: 63/63 passed
• same-lease repeated dispatch regression: current generation accepted, older generation rejected
• a2a3sim and a5sim consecutive group reservation passed; all CI cases that previously reported stale lease generation passed on both relevant sim paths
• a2a3sim native lifecycle and A2/A3 + A5 wide-dispatch/vector simulation passed
• A2/A3 queue architecture probe: task_20260807_044537_40856118009
• A2/A3 endpoint, whole-run FIFO, and native lifecycle: task_20260807_050221_7162399608 passed
• A2/A3 launch-seam and stale-generation onboard regressions: task_20260807_061626_126859527092 passed
• A2/A3 SDMA/AICore fault terminal error propagation: task_20260807_064449_306323420340 passed

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 00b4d9cb-d278-4ea8-bd39-a706297f22e7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds bounded concurrent native-run preparation for eligible backends, asynchronous L2 submission with RunHandle, run identity propagation, per-slot resource management, FIFO launch gating, non-throwing dispatch failures, and expanded lifecycle tests and documentation.

Changes

Concurrent native-run pipeline

Layer / File(s)	Summary
Native-run contracts and bindings src/common/worker/*, python/bindings/task_interface.cpp, python/simpler/task_interface.py	Native runs now carry identity metadata and expose backend concurrent-preparation capability.
Per-run resource reservation and provisioning src/common/platform/onboard/host/*, src/a2a3/platform/onboard/host/*	Pipeline and arena selections use thread-local state. Slot reservations and stream resources support one prepared successor.
Runtime preparation, launch, and finalization src/common/platform/onboard/host/c_api_shared.cpp, src/common/log/include/common/strace.h	Preparation, launch, polling, and finalization manage identities, resources, trace attributes, and outstanding-run checks.
ChipWorker native-run state machine src/common/worker/chip_worker.cpp	Native-run phases validate tokens and handle preparation, launch failures, polling, waiting, finalization, and cleanup.
Hierarchical staged-frame and dispatch behavior python/simpler/worker.py, src/common/hierarchical/*	Two-frame execution supports prepared successors and FIFO activation. Dispatch rejection now completes through callbacks without throwing.
Asynchronous L2 submission and handles python/simpler/worker.py	L2 submission returns live handles after launch, applies two-run admission limits, and performs ordered completion and finalization.
Lifecycle validation and documentation tests/*, docs/*	Tests cover overlap, ordering, failures, cleanup, resource reuse, and handle completion. Documentation describes the updated lifecycle.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Worker
  participant ChipWorker
  participant Runtime
  participant RunHandle

  Client->>Worker: submit L2 task
  Worker->>ChipWorker: prepare native run
  ChipWorker->>Runtime: reserve and prepare run identity
  Worker->>ChipWorker: launch FIFO front run
  ChipWorker->>Runtime: launch native run
  Worker-->>Client: return RunHandle
  Client->>RunHandle: wait
  RunHandle->>Worker: poll and finalize
  Worker->>ChipWorker: finalize native run

Loading

Possibly related PRs

• hw-native-sys/simpler#1583: Adds generation-safe two-frame preparation and FIFO admission infrastructure extended by this PR.
• hw-native-sys/simpler#1466: Introduces the run-handle execution model extended with concurrent preparation here.
• hw-native-sys/simpler#1464: Adds pipeline-slot stream isolation used by the new resource lifecycle.

Poem

A rabbit queues runs in a neat little line,
One hops while the next waits in pipeline time.
Handles hold dreams till completion is near,
FIFO keeps every launch crystal clear.
Streams bloom, then retire when the work is done—
Thump-thump, concurrent execution has begun!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 7.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: centralizing ownership and lifecycle management for chip native runs.
Description check	✅ Passed	The description directly explains the native lane, ownership refactor, protocol changes, scope, and validation for this changeset.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 8

🧹 Nitpick comments (8)

src/common/platform/onboard/host/device_runner_base.cpp (1)

1356-1359: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Guard the division by cores_per_blockdim_.

block_dim_ = worker_count_ / cores_per_blockdim_ introduces a division on a field that the previous code only multiplied by. If any arch leaves cores_per_blockdim_ at 0, this is undefined behaviour instead of a benign zero.

🛡️ Proposed guard

 void DeviceRunnerBase::activate_launch_shape(const Runtime &runtime) {
     worker_count_ = runtime.get_worker_count();
-    block_dim_ = worker_count_ / cores_per_blockdim_;
+    block_dim_ = cores_per_blockdim_ > 0 ? worker_count_ / cores_per_blockdim_ : 0;
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/device_runner_base.cpp` around lines 1356 -
1359, Update DeviceRunnerBase::activate_launch_shape to guard
cores_per_blockdim_ before dividing worker_count_. Preserve the benign zero
behavior by setting block_dim_ to zero when cores_per_blockdim_ is zero;
otherwise retain the existing worker-count division.

src/common/platform/onboard/host/c_api_shared.cpp (2)

711-713: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Explain why runner_resources_owned is set before provisioning.

Line 711 sets the flag to true before provision_native_run_resources runs at Line 712. On a provisioning failure, cleanup_failed_prepare therefore calls abandon_native_run_resources for a slot that provisioning did not complete.

That is the right choice for a partial provision, but the pre-set reads as a sequencing mistake. Add a one-line comment stating that a failed provision may still hold partial resources, so ownership is claimed before the attempt.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/c_api_shared.cpp` around lines 711 - 713, In
the preparation flow around runner_resources_owned and
provision_native_run_resources, add a concise one-line comment explaining that
provisioning failure may leave partial native resources held, so ownership must
be claimed before the provisioning attempt for cleanup_failed_prepare to release
them.

---

697-701: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

A rejected reservation drops its allocated trace invocation.

Line 676 allocates trace_inv and Line 677 stamps trace_start_ns. When try_reserve_native_run fails, Line 700 returns without calling emit_native_run_host_wall. Every other failure path routes through cleanup_failed_prepare, which emits the span. Admission rejections therefore leave a gap in the host trace exactly where contention analysis needs a record.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/c_api_shared.cpp` around lines 697 - 701,
Update the admission-rejection branch in the native-run preparation flow around
try_reserve_native_run so it emits the allocated trace invocation via the
existing cleanup_failed_prepare path or equivalent before destroying state and
returning. Preserve the current error logging and -1 return while ensuring
rejected reservations produce the same host-wall trace record as other failure
paths.

src/common/platform/onboard/host/device_runner_base.h (1)

132-135: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

pipeline_slot(), selected_arena_bank(), and arena_bank() can now throw.

These accessors were plain field reads. They now route through the pthread TLS helper, which throws on key-creation or allocation failure. arena_bank() at Line 1021 is on the setup_static_arena and acquire_pooled_* paths, and pipeline_slot() is read at the top of DeviceRunner::run. Callers that previously treated these as infallible now have a new exception edge.

Document the new throwing contract on the declarations so callers on noexcept paths do not adopt them by accident.

Also applies to: 148-148, 1021-1021

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/device_runner_base.h` around lines 132 -
135, Document on the declarations of pipeline_slot(), selected_arena_bank(), and
arena_bank() that each accessor may throw due to pthread TLS key creation or
allocation failure. Keep the existing signatures unchanged and make the contract
visible to callers before they use these accessors in run, setup_static_arena,
or acquire_pooled_* paths.

src/common/worker/chip_worker.cpp (1)

684-688: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Simplify the occupancy check.

occupied != 0 && occupied != 1 is occupied > 1. The direct form states the intent: at most one predecessor may exist.

♻️ Proposed simplification

-        if (occupied != 0 && occupied != 1) {
+        if (occupied > 1) {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/worker/chip_worker.cpp` around lines 684 - 688, In the occupancy
validation near the prepare_native_run ownership check, replace the `occupied !=
0 && occupied != 1` condition with the equivalent `occupied > 1` check,
preserving the existing runtime error and identity formatting.

src/common/hierarchical/worker_manager.cpp (1)

318-325: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Make the non-queued mapping explicit in dispatch.

dispatch passes no staged_run_id, so enqueue_dispatch cannot return STAGED_IDENTITY_CHANGED today. The else branch nevertheless reports any future non-STOPPING result as "endpoint capacity exceeded". dispatch_prepared already switches on each enumerator. Match that shape so a new enumerator cannot be reported under the wrong message.

♻️ Proposed change

     if (result == EnqueueDispatchResult::STOPPING) {
         complete_unpublished(d, "WorkerThread::dispatch: worker is stopping");
-    } else {
+    } else if (result == EnqueueDispatchResult::CAPACITY_EXCEEDED) {
         complete_unpublished(d, "WorkerThread::dispatch: endpoint capacity exceeded");
+    } else {
+        complete_unpublished(d, "WorkerThread::dispatch: enqueue rejected the dispatch");
     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/worker_manager.cpp` around lines 318 - 325, Update
dispatch to explicitly handle every non-queued EnqueueDispatchResult, matching
dispatch_prepared’s switch structure. Preserve the STOPPING message, map the
currently impossible STAGED_IDENTITY_CHANGED result explicitly, and ensure any
newly added enumerator cannot fall through to “endpoint capacity exceeded.”

python/simpler/worker.py (1)

2405-2414: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Share one diagnostics predicate.

config_has_diagnostics here and Worker._l2_config_has_diagnostics at lines 8194-8202 contain the identical field list. Both must track CallConfig::diagnostics_any(). Extract one module-level helper and call it from both sites, so a new diagnostic flag cannot be added to only one copy.

♻️ Proposed shared helper

def _call_config_has_diagnostics(config: CallConfig) -> bool:
    # Mirrors CallConfig::diagnostics_any().
    return bool(
        config.enable_l2_swimlane
        or config.enable_dump_args
        or config.enable_pmu
        or config.enable_dep_gen
        or config.enable_scope_stats
    )

-        def config_has_diagnostics(config: CallConfig) -> bool:
-            # Mirrors CallConfig::diagnostics_any(); these modes share native
-            # diagnostic state and therefore use the serial prepare fallback.
-            return bool(
-                config.enable_l2_swimlane
-                or config.enable_dump_args
-                or config.enable_pmu
-                or config.enable_dep_gen
-                or config.enable_scope_stats
-            )
+        config_has_diagnostics = _call_config_has_diagnostics

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/simpler/worker.py` around lines 2405 - 2414, Extract the shared
field-list logic from the local config_has_diagnostics function into a
module-level _call_config_has_diagnostics(config: CallConfig) helper, preserving
the fields that mirror CallConfig::diagnostics_any(). Update both
config_has_diagnostics and Worker._l2_config_has_diagnostics to delegate to this
helper so future diagnostic flags are maintained in one place.

tests/ut/py/test_worker/test_host_worker.py (1)

405-432: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Let the harness vary the frame run id.

publish writes a constant run_id of 5 into every frame. The new concurrency tests therefore stage an active frame and a successor frame that share one run id and differ only by dispatch_id. Production stages a successor that belongs to a different run. The tests still prove dispatch-id ordering, so nothing is wrong today. If run_two_frame_loop later gates preparation on run identity, these tests would pass without exercising that gate. Add a run_id parameter with a default of 5 and give the successor a distinct value in the concurrent-prepare tests.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/py/test_worker/test_host_worker.py` around lines 405 - 432, Update
the test harness publish method to accept a run_id parameter defaulting to 5,
and write that value into _OFF_FRAME_RUN_ID instead of the hardcoded constant.
In the concurrent-prepare tests, pass a distinct run_id for the successor frame
while preserving the existing default for other callers.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/worker-manager.md`:
- Around line 241-247: Update the registry-control deferral sentence in the
worker-manager documentation to wait for both the active native run and any
prepared successor holding a native token. Keep the rule aligned with
run_two_frame_loop, where controls remain deferred while any staged frame owns a
native token.

In `@python/simpler/worker.py`:
- Around line 8570-8583: Update _wait_run_handle_accepted so it advances the L2
FIFO in bounded steps, rechecking the target run state after each
_l2_progress_locked call rather than allowing one call to run through terminal
completion. Return as soon as the target phase is no longer "prepared", while
preserving unknown-run and propagated-error handling; leave the orchestrator
path unchanged.

In `@src/common/platform/onboard/host/c_api_shared.cpp`:
- Around line 873-881: Update simpler_finalize_run to wrap
capture_native_run_thread_selection, select_pipeline_slot, and select_arena_bank
in exception handling that releases runner_claimed and runner_reserved, calls
destroy_native_run_state, and returns -1 on selection failure or thrown
exceptions. Apply the equivalent exception guard to simpler_launch_run around
its native thread selection calls, preserving its existing claim-release
behavior.

In `@src/common/platform/onboard/host/device_runner_base.cpp`:
- Around line 1659-1665: Protect the native_launch_signal_ read in
DeviceRunnerBase::publish_task_accepted with native_run_mu_, matching the
synchronization used by try_acquire_native_run and release_native_run. Hold the
mutex while copying or checking the pointer and notifying it, so the pointer
cannot race with updates or destruction; preserve the existing accepted_state
release store.
- Around line 72-107: Update the native-run selection flow around
run_selection(), select_pipeline_slot(), and select_arena_bank() so selection
state is isolated per DeviceRunnerBase instance rather than shared by thread
alone. Attach or copy NativeRunThreadSelection to the owning runner, or make the
existing TLS state resolve through a runner-specific key, ensuring contexts used
on the same host thread cannot reuse each other’s arena bank, retained buffers,
or run identity.

In `@src/common/platform/onboard/host/device_runner_base.h`:
- Around line 136-140: Remove noexcept from restore_native_run_thread_selection
in both its declaration and definition, allowing exceptions from run_selection
during create_thread’s initial TLS allocation to propagate safely instead of
terminating the process.

In `@src/common/worker/chip_worker.cpp`:
- Around line 814-826: Re-validate both lease_generation and run_epoch under
native_run_mu_ before writing completion state in poll_native_run and
wait_native_run. If the slot identity no longer matches, do not update the
successor state; otherwise preserve the existing writes to phase REAPED and, in
wait_native_run, wait_rc.

In
`@tests/st/a2a3/host_build_graph/native_run_lifecycle/test_native_run_lifecycle.py`:
- Around line 196-199: Remove the timing-dependent torch.count_nonzero assertion
from the direct L2 submit test, while retaining the run_handle._terminal
assertion to verify submit returns a non-completed compatibility handle. Keep
the subsequent run_handle.wait(30.0) lifecycle check unchanged.

---

Nitpick comments:
In `@python/simpler/worker.py`:
- Around line 2405-2414: Extract the shared field-list logic from the local
config_has_diagnostics function into a module-level
_call_config_has_diagnostics(config: CallConfig) helper, preserving the fields
that mirror CallConfig::diagnostics_any(). Update both config_has_diagnostics
and Worker._l2_config_has_diagnostics to delegate to this helper so future
diagnostic flags are maintained in one place.

In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 318-325: Update dispatch to explicitly handle every non-queued
EnqueueDispatchResult, matching dispatch_prepared’s switch structure. Preserve
the STOPPING message, map the currently impossible STAGED_IDENTITY_CHANGED
result explicitly, and ensure any newly added enumerator cannot fall through to
“endpoint capacity exceeded.”

In `@src/common/platform/onboard/host/c_api_shared.cpp`:
- Around line 711-713: In the preparation flow around runner_resources_owned and
provision_native_run_resources, add a concise one-line comment explaining that
provisioning failure may leave partial native resources held, so ownership must
be claimed before the provisioning attempt for cleanup_failed_prepare to release
them.
- Around line 697-701: Update the admission-rejection branch in the native-run
preparation flow around try_reserve_native_run so it emits the allocated trace
invocation via the existing cleanup_failed_prepare path or equivalent before
destroying state and returning. Preserve the current error logging and -1 return
while ensuring rejected reservations produce the same host-wall trace record as
other failure paths.

In `@src/common/platform/onboard/host/device_runner_base.cpp`:
- Around line 1356-1359: Update DeviceRunnerBase::activate_launch_shape to guard
cores_per_blockdim_ before dividing worker_count_. Preserve the benign zero
behavior by setting block_dim_ to zero when cores_per_blockdim_ is zero;
otherwise retain the existing worker-count division.

In `@src/common/platform/onboard/host/device_runner_base.h`:
- Around line 132-135: Document on the declarations of pipeline_slot(),
selected_arena_bank(), and arena_bank() that each accessor may throw due to
pthread TLS key creation or allocation failure. Keep the existing signatures
unchanged and make the contract visible to callers before they use these
accessors in run, setup_static_arena, or acquire_pooled_* paths.

In `@src/common/worker/chip_worker.cpp`:
- Around line 684-688: In the occupancy validation near the prepare_native_run
ownership check, replace the `occupied != 0 && occupied != 1` condition with the
equivalent `occupied > 1` check, preserving the existing runtime error and
identity formatting.

In `@tests/ut/py/test_worker/test_host_worker.py`:
- Around line 405-432: Update the test harness publish method to accept a run_id
parameter defaulting to 5, and write that value into _OFF_FRAME_RUN_ID instead
of the hardcoded constant. In the concurrent-prepare tests, pass a distinct
run_id for the successor frame while preserving the existing default for other
callers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6baf5a84-ef87-4133-9053-7b0416dc8098

📥 Commits

Reviewing files that changed from the base of the PR and between 810fbcd and b160bcf.

📒 Files selected for processing (24)

• docs/task-flow.md
• docs/worker-manager.md
• python/bindings/task_interface.cpp
• python/simpler/task_interface.py
• python/simpler/worker.py
• src/a2a3/platform/onboard/host/device_runner.cpp
• src/a2a3/platform/onboard/host/device_runner.h
• src/common/hierarchical/scheduler.cpp
• src/common/hierarchical/worker_manager.cpp
• src/common/hierarchical/worker_manager.h
• src/common/log/include/common/strace.h
• src/common/platform/onboard/host/c_api_shared.cpp
• src/common/platform/onboard/host/device_runner_base.cpp
• src/common/platform/onboard/host/device_runner_base.h
• src/common/worker/chip_worker.cpp
• src/common/worker/chip_worker.h
• src/common/worker/native_run_state.h
• src/common/worker/pto_runtime_c_api.h
• tests/st/a2a3/host_build_graph/native_run_lifecycle/kernels/orchestration/long_vector_orch.cpp
• tests/st/a2a3/host_build_graph/native_run_lifecycle/test_native_run_lifecycle.py
• tests/ut/cpp/hierarchical/test_run_stream_slots.cpp
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/py/test_worker/test_host_worker.py
• tests/ut/py/test_worker/test_startup_readiness.py

[ChaoWao]

Reviewed against main @ 673aecea. Three things to flag before this can go further, plus a rebase.

1. This reverts D1, which merged 4 hours ago in #1587

The title says "align native prepared lane with v2 ownership", but the diff also undoes the uniform host-runtime pipeline ABI:

• load_optional_symbol is reintroduced, and all six pipeline symbols go back to optional loading
• get_pipeline_contract is deleted from both a5 runtime_maker.cpp files
• supports_concurrent_native_prepare_ctx, set_task_accepted_state_ctx, and set_native_run_identity_ctx are deleted from the sim c_api_shared.cpp
• tests/ut/py/test_host_runtime_abi.py is deleted entirely — the check that all eight built DSOs export the required set

The commit message justifies this as "keep pipeline metadata optional for older runtimes". There are no older runtimes: host_runtime.so is built from this tree by the same pip install that installs its consumer, which is the reasoning #1587's own body gave for making the loads strict. A stale build/lib/ is the only way to hit the "older runtime" case, and failing loudly at ChipWorker::init is the intended handling — #1653 added a rebuild hint to that exact error for this reason.

If the v2 ownership rework genuinely needs a symbol to be optional, please name which one and why, rather than reverting the set.

2. Deleting the sim set_task_accepted_state_ctx re-creates the defect #1649 was written to catch

#1649's commit message describes it precisely: with no sim export, ChipWorker's optional load yields nullptr, both bind sites are skipped, and SimDeviceRunnerBase::publish_task_accepted stores through a null target — so a sim child never publishes acceptance and the launch fence silently degrades into a completion fence.

That is observable on the stacked #1588 right now: st-sim-a2a3 fails tests/st/a2a3/tensormap_and_ringbuffer/test_l3_launch_acceptance.py with the chip worker never published launch acceptance: [0, 0, 0].

3. The new worker_async_endpoint assertion fails on real hardware

st-onboard-a2a3 fails at test_worker_async_endpoint.py:214:

AssertionError: the successor staged only after the predecessor native run had already terminalized
assert 0 == 9

To be clear about attribution: that assertion is added by this PR, so this is not a pre-existing invariant being broken — it is a new claim that does not hold on device. 0 is IDLE, so when the successor published FRAME_STAGED the predecessor's frame had already been reset. Worth deciding which is true before fixing: the assertion's timing assumption is too strong (the predecessor is pinned by a SubTask fence, but nothing pins its frame state at _TASK_LAUNCHED), or the Worker-side rework genuinely delays staging past the predecessor's terminal transition. ut-a5 and st-onboard-a5 are also red.

4. Rebase

Merge-base is 2a650f2d; main is 673aecea. #1653 landed in between and touched chip_worker.cpp, device_runner_base.{h,cpp}, and run_stream_slots.h — all files this PR also edits, so the conflict only gets worse with time.

Happy to help dig into (3) if useful — that one blocks #1588 as well, since #1588 stacks on this branch and inherits the same failure.

[ChaoWao]

@ChaoWao Addressed the review summary:

• Rebased the single commit onto the latest main and kept every pipeline ABI
  symbol required across all eight host-runtime DSOs; optional symbol loading
  is not used.
• Kept simulation launch-acceptance and run-identity exports in parity with
  onboard, with the host-runtime ABI tests retained.
• Removed the timing-dependent output/frame-state assertions while preserving
  deterministic handle and lifecycle checks.
• Added stable cross-runner TLS ownership, serialized native phase access, and
  interruption-safe direct-L2 registry/control cleanup found during the final
  audit.

Final validation includes all pre-commit hooks, 1093 Python unit tests, 78 C++
unit tests, clean A2A3sim/A5sim sweeps, and final-HEAD A2A3 hardware coverage for
worker_async_endpoint plus native_run_lifecycle.

[ChaoWao]

Splitting out PR-A of this change as an independent, narrower PR: #1685 ("Refactor: bind platform host callbacks per run instead of per thread").

#1685 lands only the platform-layer prerequisite this PR bundles first:

• HostApi → immutable HostApiFunctions table + per-run HostApi value object (runner + slot + bank).
• NativeRunDescriptor carried across the C ABI; simpler_prepare_run / simpler_run take it (required).
• Launch acceptance moves from the set_task_accepted_state_ctx TLS setter into the per-run launch signal (configure_acceptance / publish_acceptance).
• Deletes the pthread-TLS resource-selection mechanism (NativeRunThreadSelection, both pthread keys, capture/restore) and all four setter exports.
• PTO_PIPELINE_CONTRACT_ABI_VERSION bumped 1 → 2 so a stale .so is rejected at load.

It deliberately excludes the other two concerns still on this branch:

• PR-B — the direct-L2 two-slot async lane (_L2NativeRun / _l2_fifo / live RunHandle in worker.py + the binding). ChipWorker::run_on_slot keeps main's synchronous _run_slot path.
• PR-C — the HostTensorAccessScope RAII conversion (a minimal host_tensor_access_reset(const HostApi*) bridge lands in Refactor: carry host callbacks and launch identity per native run #1685 so HBG still compiles; the full RAII scope follows).

Implemented independently from this branch (not pushed to it). Once #1685 merges, the remaining two can rebase onto it — HostApi already exists, so neither depends on the new lane. PR body has the per-thread-state safety argument for doing B6c while the per-run executor thread still exists, and the ABI-version answer.

[ChaoWao]

Reviewed at b277c25a against the .docs ruling for this PR. The record's requirements are met, and I've pushed one fix on top (5e4e186b, fast-forward — your commit is untouched).

The .docs ruling is satisfied

.docs/others-ref/worker-async-pipeline/implementation-record.md §#1650 marked the original "superseded as a landing shape" with four named rejection reasons. All four are genuinely addressed here:

Rejection reason	Status
Python _L2NativeRun/FIFO/phase/poison as a third lifecycle authority	✅ grep _L2NativeRun = 0 hits; Python now only reads chip_run.lane_poisoned / .done() / .launched
Dispatch-path sleep(0)	✅ Gone. The four remaining time.sleep(_STARTUP_POLL_INTERVAL_S) are all in _await_children_ready / _abort_hierarchical / _reap_child_groups — the init/teardown paths rule 5 explicitly exempts
C++ ChipRunLane must own the lifecycle	✅ chip_run_lane.{h,cpp}, single-mutex FIFO state machine
"Public direct-chip capacity two only after capacity-one is proven"	✅ ChipWorker::run() is submit + wait_until(max()), so public capacity stays one; the lane's 2 slots serve the child loop's existing two-frame staging

The HBG-globals removal is correctly deferred — your Scope section says W1a + E2 only, and the record agrees that's a separate stage.

What I fixed

ChipWorker::run() used to make one blocking call into the platform:

int rc = run_fn_(device_ctx_, rt, callable_id, args, &config, &descriptor);   // pre-PR

simpler_run blocks inside the device layer on aclrtSynchronizeStreamWithTimeout — no host CPU. It became:

ChipRun run = run_lane_->submit(...);
(void)run.wait_until(ChipRun::Deadline::max());

and wait_until re-entered progress() in a tight loop with no pause and no wakeup primitive. progress() calls poll_native_run, so each iteration is a real device poll — and with Deadline::max() the Clock::now() >= deadline exit can never fire.

I measured it rather than eyeballing it, by temporarily instrumenting your own test harness (completion delayed 200 ms on a second thread, reverted afterwards):

[MEASURE] 200 ms wait -> 636336 poll_run calls

~3.2 M/s against the stubbed poll; in production every one is an rtSetDevice plus a device read. So every blocking chip run saturated a core for its duration, and the chip child did it for its whole lifetime.

This is legal under codestyle.md rule 5 by the letter — a dispatch-path wait may spin. But the rule names this exact scenario and prescribes the remedy:

When an idle spin is genuinely too expensive to leave running (a forked child that would hold a core for its whole lifetime), the answer is a blocking wakeup primitive, not a sleep — blocking costs no CPU and wakes in single-digit microseconds.

The primitive was already in the lane: drain_front() calls wait_native_run at line 202. An unbounded waiter now blocks on the launched front through that same call. Only the front can be LAUNCHED, so its completion is what lets any waiter in the FIFO advance; block_on_front() returns whether it actually blocked, so a waiter with nothing blockable falls back to the existing poll loop rather than spinning on a no-op. Finite deadlines are unchanged — they need polling to honor the deadline, and their loop terminates on its own.

Same measurement after the change: 1 poll_run call.

Worth flagging the framing: rejection reason #4 was the sleep(0). Removing it was right, but the replacement went to a raw spin rather than to the wakeup primitive the rule points at — so that concern was arguably worse, not resolved.

Two smaller things, not fixed

• The spin was structurally untestable. The stubbed poll_run returns COMPLETE as soon as g_complete is set, so every wait_until in all 12 cases returns on its first pass — ExpiredWaitLeavesTheRunLive passes an already-elapsed deadline, so it too does one iteration. That's why CI was 18/18 green with this in it. A g_poll_count bound on a delayed completion (the harness already has the counter) would pin the invariant. I left this to you since it's your harness.
• ChipRun::wait_until_launched has zero callers — dead API on arrival. Worth deleting or wiring up.

Validation of the pushed commit

• 91/91 cpp UT under CI's own -LE requires_hardware filter (your 12 lane tests included, unchanged)
• 1190 passed / 13 skipped — full tests/ut/py
• a2a3 onboard sweep: 54 passed, plus 24 passed / 2 skipped in the resource phase
• a2a3sim sweep: 45 passed, plus 21 passed / 4 skipped

One note before merge

This branch forked at 9a03d92c (#1728), but upstream/main is now d1ec68a4 (#1739), which deleted WorkerEndpoint::run / run_with_accept / dispatch_process and touched worker_manager.{h,cpp} — files this PR also edits. Worth a rebase to confirm they don't collide.

[ChaoWao]

Rebased onto c9e5f3cc (current main) — the branch had conflicted. Force-pushed to 7560854c with --force-with-lease pinned to the SHA I had last pushed, so it would have aborted rather than overwrite anything you pushed in the meantime. Your commit keeps its authorship; the rebase resolution is folded into it, and my lane fix stays a separate commit on top.

Three commits had landed since you forked at 9a03d92c: #1739, #1729, #1696.

The one conflict that needed a judgement call

#1729 (aa1d7c7d, self-describing Tensor wire ABI) rewrote the same submit_frame / prepare path this PR rewrites, in the opposite direction:

• Update: cut task args over to the self-describing Tensor wire ABI #1729 moved tensor materialization into Python — import_registry.materialize_blob + materialize_tensor_blob produce a ChipStorageTaskArgs, and it renamed the binding _prepare_native_run_from_blob → _prepare_native_run_materialized, dropping the blob-based entry point entirely.
• This PR pushed the raw blob into C++, with _submit_chip_run_from_blob calling read_blob + view_to_chip_storage inside the binding.

Taking either side wholesale would have been wrong: keeping the blob binding would have resurrected an API main deleted and bypassed the new import-registry resolution (so remote/imported tensor descriptors would never be materialized to local bases); taking main's alone would have dropped the lane.

Resolution: the lane keeps its ownership, but adopts main's materialized contract.

• _submit_chip_run_from_blob → _submit_chip_run_materialized, taking const ChipStorageTaskArgs & like its _prepare_* sibling. The _from_blob suffix would have been a lie after the change.
• submit_frame now materializes first, mirroring what prepare_frame_native_run did on main:

  args_ptr = frame.frame_addr + _OFF_TASK_ARGS_BLOB
  resolved = import_registry.materialize_blob(args_ptr, _MAILBOX_ARGS_CAPACITY)
  chip_args = materialize_tensor_blob(args_ptr, _MAILBOX_ARGS_CAPACITY, resolved)
  frame.chip_run = cw._impl._submit_chip_run_materialized(frame.cid, chip_args, ...)

• The _FakeChipWorker in test_host_worker.py follows, since it explicitly "mirrors the production binding".

Two smaller hunks were mechanical: your deletion of prepare_frame_native_run / finalize_frame_native_run (the lane subsumes them) applied over main's edits to their bodies, and in stage_frame main dropped _rewrite_blob_host_addrs while you dropped the separate config / activation_required locals — both deletions compose, and config= is inlined into your _StagedFrame(...) call.

Please sanity-check the materialization placement. Doing it in submit_frame means it happens at submission rather than at prepare; that matches where main does it relative to the native call, but you know the intended lane sequencing better than I do.

Validation of the rebased branch

• 91/91 cpp UT under CI's own -LE requires_hardware filter (your 12 lane tests included)
• 1252 passed / 13 skipped — full tests/ut/py
• a2a3 onboard sweep: 54 passed, plus 24 passed / 2 skipped in the resource phase

⚠️ One flake worth your eyes, in this PR's own area. The first onboard sweep after the rebase failed with TestWorkerAsyncEndpoint::test_run (tests/st/a2a3/host_build_graph/worker_async_endpoint/) rc=1 under 4-device contention. It passed standalone and the full sweep passed clean on re-run (exit=0), so I could not reproduce it — but it is an async-endpoint test on the path this PR restructures, so I would rather flag it than call it noise. If st-onboard-a2a3 shows it, it is not new with the rebase.