[tohtana]

Problem

Recent two-L40S Modal runs have hit an environment-level NCCL/CUDA failure or
the existing no-progress watchdog after capacity was already acquired. The
current AWS fallback is entered only when Modal cannot acquire capacity, so
those runs never exercise the alternate backend. This does not imply that
every downstream test failure is caused by Modal.

Approach

Temporarily route eligible modal-torch-latest executions directly through
the existing two-GPU AWS controller. This keeps the required job names, trusted
checkout and exact candidate SHA handling, test selection, runtime inputs,
timeouts, and primary/reconciliation cleanup unchanged. The explicit
infrastructure-smoke mode remains available for provider diagnostics.

Testing

I ran the focused controller/workflow suite because it covers trusted checkout,
direct AWS execution, timeout classification, and cleanup behavior.

• python3 ci/test_torch_latest.py (49/49 passed)
• python3 ci/tests_fetcher.py --base 3b809329fb760de6832d02020e156a9870f5cbdd --explain (mode=all)
• pre-commit run --files .github/workflows/modal-torch-latest.yml ci/test_torch_latest.py (passed)
• pre-commit run --all-files (passed)

A manual exact-head AWS run
selected 2,492 tests/unit/v1 items and exercised two GPUs. It did not pass:
after 414 passed and 92 skipped, the unchanged no-progress watchdog terminated
pytest at 20% while
TestPerHeadMuonUnderAutoTP::test_the_tags_describe_the_shard_and_not_the_model
was active. No assertion failure was emitted, and workflow cleanup completed.

Compatibility and disclosure

This CI-only routing change does not alter DeepSpeed runtime behavior, so ZeRO,
offload, precision, and parallelism matrix cells are not applicable. Searches
of open issues and PRs found no existing Torch-latest change that temporarily
bypasses Modal for direct AWS execution. AI assistance was used to prepare this
change; the upstream PR remains a draft for human line-by-line review.