CI: move additional simulator scenes to daily sweep

[doraemonmj]

Summary

• Extend the Per-PR scene-test downselection introduced by CI: move redundant scene tests to daily sweep #1790 by moving additional simulator-only lifecycle, error-matrix, stress, scale, and redundant multi-device cases behind the existing manual selection.
• Keep --manual exclude for Per-PR CI and rely on the existing Daily workflow's --manual include to retain every migrated case.
• Scope every shared case to Sim only, so supported A3/A5 Onboard coverage remains unchanged.
• Keep the dedicated DFX smoke steps and both architecture/runtime-specific BGEMM cases unchanged.
• Keep kernels, case parameters, runtime configuration, golden checks, and timeout thresholds unchanged.

Cases moved from Per-PR Sim

Test group	A2/A3 Sim moved to Daily	A5 Sim moved to Daily	Coverage retained in Per-PR	Reason
Dynamic registration	Parallel two-chip broadcast; capacity overflow; duplicate-handle unregister; unregister/reprepare	Capacity overflow; duplicate-handle unregister; unregister/reprepare	prepare_new_identity_after_start_then_run on each architecture	The retained happy path proves post-start registration and execution; the migrated cases repeatedly create Workers to exercise lifecycle boundaries and error handling.
Runtime fatal codes	scope_deadlock, heap_ring_deadlock, flow_control_deadlock, dep_pool_overflow, invalid_args, require_sync_start_invalid, async_wait_overflow	Same	explicit_fatal and async_completion_invalid	Retains one orchestration-originated and one scheduler-originated failure path. Error-name mapping also has C++ UT coverage. The two onboard-only hang cases already skip on Sim and follow the migrated parameter selection.
SPMD sync-start stress	TestSpmdSyncStartStress::Case1	Same	Basic and mixed SPMD scene tests	This is a wide sync-start concurrency stress case rather than the minimum SPMD correctness path.
HBG wide dispatch	TestHostBuildGraphWideDispatch::TwoThreads	Same	HBG 2D graph execution and mixed-SPMD record/replay	The migrated case intentionally widens dispatch across two AICPU threads; smaller graph execution remains in Per-PR.
P2 collectives	AllToAll, Allgather, Broadcast, ReduceScatter, and Allreduce Ring	Same supported set	Allreduce One-phase P2 and ep_dispatch_combine on both Sim lanes	These cases repeat multi-device setup and the same communication substrate. P4 and the other P2 allreduce algorithms were already moved to Daily by #1790.
Multi-device resource/examples	P3 domain_rank_map, P3 overlapping group_reservation, and the A2/A3 L3 allreduce demo	P3 domain_rank_map and P3 overlapping group_reservation	ep_dispatch_combine and Allreduce One-phase P2	Keeps one representative two-chip collective and an end-to-end expert path while moving rank-scale and reservation-specific expansion.
Dependency overflow chain	n_64_no_chain, n_200_single_overflow, n_391_two_overflow	Same	n_65_single_overflow	The 64→65 boundary is the minimum regression detector; the other sizes expand below-boundary and larger-chain scale.
Dummy dependency graph	DenseFanoutFanin	Same	SingleDummyAutoDep and DummyExplicitDepBarrier	Keeps automatic and explicit dependency semantics while moving the 18-way density stress case.
HBG graph execution	AIC+AIV record/replay	AIC+AIV record/replay and 1D record/replay	2D record/replay and mixed-SPMD record/replay	The retained paths cover ordinary shape replay and the richer mixed-SPMD graph; A2/A3 1D was already Daily-only.
HBG invalid-input matrix	mixed_subtask_overflow, unbound_owner_read, and unbound_owner_write	Same	zero_block_num	All four cases build the same callable, create a fresh Worker, and assert the same code-5/INVALID_ARGS propagation. Per-PR keeps one end-to-end invalid-input smoke; Daily keeps the branch-specific matrix.
HBG prepared-callable lifecycle	Entire HBG prepared-callable suite	Same	Full TMR prepared-callable suite on both architectures; ordinary HBG graph execution remains	The dlopen/slot/unregister lifecycle matrix is host-loader behavior already exercised by the retained TMR suite. Moving the HBG duplicate also removes a grouped block of Sim executions.
Paged-attention expansions	Manual-scope small case and multi-round case	Manual-scope, unroll-manual-scope, multi-round, and 4D-unroll cases	Basic TMR paged attention, batch paged attention, and HBG paged attention	These expand scope mode, rounds, unrolling, or dimensionality over the same kernel pipeline. Representative ordinary, batched, and HBG paths remain in Per-PR.
Redundant SPMD variants	Pure-AIV multiblock, pure-AIV sync-start, and starvation stress	Same	spmd_basic, mixed AIC+AIV multiblock, ordinary sync-start, sync-start mix/spill, edge, and early-dispatch paths	Richer mixed cases retain AIC+AIV scheduling coverage; the pure-AIV duplicates and starvation stress are better suited to Daily.
Simple multi-chip dispatch	multi_chip_dispatch	Same	Allreduce One-phase P2 and ep_dispatch_combine	The simple two-chip dispatch repeats Worker/domain setup while the retained collective and expert-dispatch paths exercise the same substrate with stronger assertions.
Available AICore counts	HBG Default	None	A2/A3 TMR copy and A5 TMR case	A2/A3 has equivalent TMR and HBG geometry-query coverage; A5 has only the TMR case and keeps it in Per-PR.

Coverage policy

• All new manual annotations are platform-scoped where the case also supports Onboard, so this PR only changes a2a3sim and a5sim selection.
• Daily continues to execute migrated cases through manual_mode: include.
• Per-PR still covers dynamic registration success, orchestration and scheduler fatal propagation, ordinary and mixed SPMD execution, HBG record/replay, automatic and explicit dependency handling, and representative multi-chip communication.
• Dedicated dep-gen, chip-swimlane, PMU, and args-dump smokes are deliberately unchanged because they are a separate serial cost and not the main target of this downselection.

Observed timing impact

The complete fresh-CI run at current head 0cf94764 passed all four Sim lanes. To reduce corpus drift, the comparison below uses run 31687370652 from the same day as the nearest pre-change reference and compares only the Run pytest scene tests step; toolchain and package setup are excluded.

Sim lane	Pre-change pytest	This PR pytest	Reduction
A2/A3 Ubuntu	5m33s	3m43s	1m50s
A2/A3 macOS	5m27s	4m35s	52s
A5 Ubuntu	5m21s	3m01s	2m20s
A5 macOS	5m48s	3m30s	2m18s

The four-lane pytest reduction is 7m20s of runner time (about 33%). Both A5 lanes improved by more than two minutes and A2/A3 Ubuntu improved by 1m50s. The slowest pytest lane fell from 5m48s to 4m35s, so the observed critical-path improvement is 1m13s; A2/A3 macOS remains the variable long tail. The spread confirms that simulator contention and limited parallel scheduling make the wall-time benefit non-linear; migrated-case count should not be interpreted as additive duration.

Measured run 31695897393

Validation

• python -m pytest tests/ut/py/test_manual_selection.py tests/ut/py/test_scene_level_selection.py -q: 36 passed.
• Full A2/A3 Sim collection with --manual exclude: 78/266 items retained; --manual only: 79/266 items selected, including pre-existing Daily coverage.
• Full A5 Sim collection with --manual exclude: 52/266 items retained; --manual only: 72/266 items selected, including pre-existing Daily coverage.
• Full A3 and A5 Onboard collection retained every newly platform-scoped case (148/266 and 83/266 items respectively).
• A5 Sim local low-parallel (--max-parallel 4) smoke passed all 52 selected items; its warm-cache 1m09s wall time is validation only, not used as the CI estimate.
• Pre-commit checks for all 38 changed Python files passed.
• git diff --check upstream/main...HEAD: passed.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9d27801b-2ae8-48b5-b515-6faa6be92035

📥 Commits

Reviewing files that changed from the base of the PR and between 300fb6d and 0967e54.

📒 Files selected for processing (22)

• examples/workers/l3/allreduce/test_allreduce.py
• examples/workers/l3/domain_rank_map/test_domain_rank_map.py
• tests/st/a2a3/host_build_graph/available_aicore_counts/test_available_aicore_counts.py
• tests/st/a2a3/host_build_graph/graph_execution/test_graph_execution_aic_aiv.py
• tests/st/a2a3/tensormap_and_ringbuffer/dfx/dep_gen/test_dep_gen_chain.py
• tests/st/a2a3/tensormap_and_ringbuffer/dummy_task/test_dummy_task.py
• tests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.py
• tests/st/a2a3/tensormap_and_ringbuffer/spmd_sync_start_stress/test_spmd_sync_start_stress.py
• tests/st/a5/host_build_graph/graph_execution/test_graph_execution.py
• tests/st/a5/host_build_graph/graph_execution/test_graph_execution_aic_aiv.py
• tests/st/a5/tensormap_and_ringbuffer/dfx/dep_gen/test_dep_gen_chain.py
• tests/st/a5/tensormap_and_ringbuffer/dummy_task/test_dummy_task.py
• tests/st/a5/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.py
• tests/st/a5/tensormap_and_ringbuffer/spmd_sync_start_stress/test_spmd_sync_start_stress.py
• tests/st/host_build_graph_wide_dispatch/test_host_build_graph_wide_dispatch.py
• tests/st/runtime_fatal_codes/test_runtime_fatal_codes.py
• tests/st/worker/collectives/all_to_all/test_all_to_all.py
• tests/st/worker/collectives/allgather/test_allgather.py
• tests/st/worker/collectives/allreduce/test_allreduce.py
• tests/st/worker/collectives/broadcast/test_broadcast.py
• tests/st/worker/collectives/group_reservation/test_group_reservation.py
• tests/st/worker/collectives/reduce_scatter/test_reduce_scatter.py

---

📝 Walkthrough

Walkthrough

Selected simulator tests now use manual-execution markers for a2a3sim and a5sim. Runtime-fatal tests retain only specified per-PR simulator cases for automatic execution.

Changes

Simulator execution policy

Layer / File(s)	Summary
Runtime simulator case selection tests/st/runtime_fatal_codes/test_runtime_fatal_codes.py	The simulator test now parameterizes automatic execution over selected per-PR cases. Other cases remain available for manual simulator runs.
A2A3 and A5 specialized test markers tests/st/a2a3/..., tests/st/a5/..., tests/st/host_build_graph_wide_dispatch/test_host_build_graph_wide_dispatch.py	Selected host-build-graph, tensor-map, ring-buffer, dynamic-register, stress, dummy-task, and wide-dispatch cases now include manual execution markers.
Collective and example test markers tests/st/worker/collectives/*, examples/workers/l3/*	Selected collective and worker example tests now mark a2a3sim and/or a5sim for manual execution.

Estimated code review effort: 2 (Simple) | ~10 minutes

Mergeability Score: ⚪ Minimal · up to 0967e

This change only adjusts simulator test selection while retaining representative Per-PR coverage and daily execution for migrated cases; no actionable merge-blocking risk remains after normal checks and review.

Possibly related issues

• Reduce CI per-job wall time #1772: This PR applies manual execution markers to simulator-heavy tests.

Possibly related PRs

• hw-native-sys/simpler#1206: Both changes adjust platform-specific execution in the runtime-fatal test.
• hw-native-sys/simpler#1790: Both changes add manual-test filtering for simulator platforms.
• hw-native-sys/simpler#1591: Both changes modify simulator test execution for a2a3sim and a5sim.

Poem

A rabbit marks the tests with care,
“Simulators wait for manual air.”
Per-PR cases run in line,
The rest await a daily sign.
Hop, hop—cleaner queues now shine!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the primary change: moving additional simulator scenes to the Daily CI sweep.
Description check	✅ Passed	The description directly explains the simulator test migration, coverage policy, timing impact, and validation results.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}