Skip to content

[CI] Restore Modal-first torch-latest routing #8849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
delock merged 1 commit into from
Oct 17, 2026
+161 −83
Merged

[CI] Restore Modal-first torch-latest routing #8849

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 60 additions & 55 deletions .github/workflows/TEST_SELECTION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,9 @@ This document explains the diff-driven test-selection system that lets some GPU
workflows run only the tests a PR could affect, instead of the whole suite, while
still satisfying Required status checks.

It currently drives **`modal-torch-latest`**, which runs `tests/unit/v1/` on a
bounded AWS GPU backend, but is built to drive more workflows from one config —
see [Adding a workflow](#add-a-new-workflow).

> **Temporary AWS-only mode:** Modal execution is disabled while its GPU driver
> issue is unresolved. The Modal implementation and its tests remain in
> `ci/torch_latest.py` so the original Modal-first route can be restored, but the
> active workflow does not install Modal, start a Modal Sandbox, or run a Modal
> infrastructure smoke check.
It currently drives **`modal-torch-latest`** (which runs `tests/unit/v1/` on a
Modal GPU or its bounded AWS capacity fallback), but is built to drive more
workflows from one config — see [Adding a workflow](#add-a-new-workflow).

- [TL;DR](#tldr)
- [Why](#why)
Expand All @@ -31,19 +25,19 @@ see [Adding a workflow](#add-a-new-workflow).
the import graph from your changed files to the impacted tests, and writes the
list to `ci/.test_selection/test_list.txt`. A nightly scheduled run always
executes the full suite.
- The trusted controller validates that list, authenticates to AWS through the
deploy job's GitHub OIDC permission, then tries a fixed, ordered pool of AWS
`g7.12xlarge` Spot locations. The selected instance fetches, installs, and
tests the exact candidate SHA without repository secrets. The backend runs
only for merge queue entries, the nightly scheduled full run, and manual runs
— a plain PR event never spends GPU quota.
- The trusted controller validates that list, then fetches, installs, and tests
the exact candidate SHA inside a no-secret Modal Sandbox. If the fixed
`L40S:2` Sandbox cannot start within its acquisition window, the controller
tries a fixed, ordered pool of AWS `g7.12xlarge` Spot locations. The backend
runs only for merge queue entries, the nightly scheduled full run, and manual
runs — a plain PR event never spends GPU quota.
- A manual `infrastructure_smoke` dispatch skips selection and pytest entirely.
It starts three independent regional AWS jobs in parallel. Each job directly
selects one region, launches a G7.12 Spot instance, reaches it through SSM,
verifies two host and container GPUs, stores a checksum-verified proof in the
private regional log bucket, and terminates its own run-tagged instance. The
matrix uses `fail-fast: false`, so one regional result cannot cancel the other
regions.
It starts one short OCI `L40S:2` Sandbox check and three independent regional
AWS jobs in parallel. Each AWS job directly selects one region, launches a
G7.12 Spot instance, reaches it through SSM, verifies two host and container
GPUs, stores a checksum-verified proof in the private regional log bucket,
and terminates its own run-tagged instance. The matrix uses
`fail-fast: false`, so one regional result cannot cancel the other regions.
- It is **fail-safe**: anything it can't reason about safely → run the *full* suite.
It never silently runs *fewer* tests than reality.
- Preview locally:
Expand Down Expand Up @@ -107,10 +101,9 @@ only documentation-only diffs. Preview the decision with

Scheduled, manual, and push runs retain their existing behavior. Required
workflows still start and report their statuses; only their heavy jobs or steps
skip. The torch-latest selector imports the same `is_docs_path()` and drops
documentation paths before applying its run-all path globs and import graph, so
documentation never widens a selection; explicit `[test all]` / `[no filter]`
overrides still win.
skip. The Modal selector imports the same `is_docs_path()` and drops documentation
paths before applying its run-all path globs and import graph, so documentation
never widens a selection; explicit `[test all]` / `[no filter]` overrides still win.

Root `README.md` is package metadata, so the formatting workflow always runs
`python scripts/check-readme.py` to check that it exists and is valid UTF-8
Expand All @@ -125,7 +118,7 @@ without installing DeepSpeed or PyTorch.
| `ci/tests_fetcher.py` | The selector. AST-parses the repo, builds an import graph, decides `all` / `subset` / `none`, writes the test-list file, emits a job summary. |
| `ci/check_paths.py` | Defines documentation paths (`is_docs_path()`) for the selector and the `check-paths.yml` CPU/GPU workflow gate. |
| `ci/test_tests_fetcher.py` | Self-tests for the selector and `ci/check_paths.py` (pure stdlib; run in `collect-tests` and, on the PR's own code, in the formatting workflow). |
| `ci/torch_latest.py` | Pure-stdlib metadata/selection validation plus the active AWS controller and retained Modal-first controller. |
| `ci/torch_latest.py` | Pure-stdlib metadata/selection validation plus the trusted Modal-first, AWS-capacity-fallback controller. |
| `ci/.test_selection/test_list.txt` | The hand-off artifact (one pytest target per line). Git-ignored. |

### Job flow
Expand All @@ -149,9 +142,11 @@ without installing DeepSpeed or PyTorch.
│ if mode != none │
│ checkout exact base SHA │
│ validate mode/list + SHA │
│ assume AWS role via OIDC │
│ try G7.12 Spot by region │
│ via SSM, no inbound port │
│ try one L40S:2 Sandbox │
│ no secrets or mounts │
│ startup capacity only: │
│ try G7.12 Spot by region │
│ via GitHub OIDC + SSM │
│ fetch exact head SHA │
│ install + run pytest │
│ terminate chosen backend │
Expand All @@ -163,12 +158,14 @@ A manual dispatch with `infrastructure_smoke=true` takes a separate path:
```text
workflow_dispatch (infrastructure_smoke=true)
│
AWS regional matrix
fail-fast: false
east-1 / east-2 / west-2
launch + SSM + two GPUs
container + private proof
per-region termination
┌─────────────────┴─────────────────┐
│ │
short Modal OCI smoke AWS regional matrix
startup + two GPUs fail-fast: false
always terminates east-1 / east-2 / west-2
launch + SSM + two GPUs
container + private proof
per-region termination
```

That path never creates a test-selection artifact, installs DeepSpeed, or runs
Expand Down Expand Up @@ -197,8 +194,10 @@ the nightly full suite exists to catch.

For a manual infrastructure-only check, select `infrastructure_smoke` when
dispatching the workflow. That skips both `collect-tests` and `deploy`; the
per-region AWS smoke jobs run directly and concurrently instead. Leaving the
input at its default `false` preserves the ordinary manual full scope.
Modal and per-region AWS smoke jobs run directly and concurrently instead.
Leaving the input at its default `false` preserves the ordinary manual full
scope. The AWS smoke path does not depend on Modal's result, so a successful
Modal allocation cannot bypass the regional checks.

### Nightly full-suite runs and regression triage

Expand All @@ -209,7 +208,7 @@ suite. Its outcome feeds `nightly-triage.yml`:
anchors the last verified-good master revision (and is exactly the good
endpoint a later manual bisect would need), so days that fail for operational
reasons (no GPU instance, timeout) simply leave the tag in place.
- **Infra** (no configured AWS location acquired a GPU) →
- **Infra** (neither Modal nor any configured AWS location acquired a GPU) →
nothing is concluded; no report, no tag move.
- **Timeout** (the backend/job time budget ran out) → an issue is opened; a
timeout points at an operational problem, not a candidate regression.
Expand All @@ -226,11 +225,13 @@ suite. Its outcome feeds `nightly-triage.yml`:
is the good endpoint it would start from.

The controller (`ci/torch_latest.py`) classifies its own failures for this
routing and prints exactly one `DS_CI_FAILURE_CLASS=infra|timeout|test` sentinel
on failure. If every configured AWS location reports insufficient capacity, no
backend runs and the outcome is infra (75); 300 seconds without pytest output is
a timeout (124); setup and test errors are test failures (1). A failed run with
no sentinel routes to the unknown class instead of guessing.
routing. It waits up to 10 minutes for Modal Sandbox startup; only that
startup-capacity timeout returns 75 to the workflow without a final sentinel
so that AWS can run. The selected backend prints exactly one final
`DS_CI_FAILURE_CLASS=infra|timeout|test` sentinel on failure. If no backend runs
the outcome is infra (75); 300 seconds without pytest output is a timeout
(124); setup and test errors are test failures (1). A failed run with no
sentinel routes to the unknown class instead of guessing.

Triage reports (timeout, unknown, test failures) go through `ci/nightly_report.sh`:
every report carries the `nightly-triage` label, and a recurring outcome
Expand Down Expand Up @@ -413,7 +414,8 @@ tests.
## Security model

The workflow triggers on **`pull_request_target`**, so it runs in the base repo's
context and the `deploy` controller can assume its AWS role. The trust boundary is:
context and the `deploy` controller can authenticate to Modal or, only for the
capacity fallback, assume its AWS role. The trust boundary is:

- Both jobs check out only the exact trusted base SHA (or the exact
push/manual SHA), with persisted credentials, LFS, and submodules disabled.
Expand All @@ -432,22 +434,24 @@ context and the `deploy` controller can assume its AWS role. The trust boundary
- `deploy` runs only the trusted controller. Candidate metadata is restricted
to a public `owner/repository` and full 40-hex SHA; event values enter Python
through environment variables, not workflow shell interpolation. GitHub's
OIDC permission is scoped to this job. AWS credentials are requested only
after the trusted checkout, Python setup, and validated selection artifact
are available; cleanup runs only if that authentication step succeeded.
OIDC permission is scoped to this job and AWS credentials are requested only
after the Modal controller returns its dedicated startup-capacity result.
- The controller role ARN and opaque regional fallback JSON are stored as
masked repository secrets. The controller logs region and ordinal location
transitions, but never the launch-template, subnet, bucket, instance, or SSM
command identifiers, and AWS CLI failures expose only a bounded error code.
The trusted job exchanges its GitHub OIDC token directly for only the three
credential fields, masks each value before exporting it, and never requests
or prints the STS assumed-role identity.
- The active job receives no Modal service token and does not install or invoke
Modal. Its retained implementation preserves the no-secret Sandbox boundary
for a future re-enable.
- The Modal service token remains in the controller process. The Sandbox gets
no Modal, GitHub, Hugging Face, OIDC, or other secret; no local checkout,
mount, volume, network filesystem, port, proxy, or workload identity is
attached. The controller constructs a positive allowlist of non-sensitive
git/pip environment flags instead of forwarding its environment.
- Candidate acquisition, dependency installation, DeepSpeed installation, and
pytest all happen on one selected `g7.12xlarge` Spot instance. The controller
tries configured subnets in fixed `us-east-1`,
pytest all happen inside one selected backend. Modal uses an isolated
`l40s:2` Sandbox with a bounded server-side lifetime. Its capacity fallback
uses only `g7.12xlarge` Spot, tries configured subnets in fixed `us-east-1`,
`us-east-2`, `us-west-2` order, exposes exactly GPUs 0 and 1 to the container,
and controls it through SSM. The instance has no inbound rules and a
60-minute termination guard. SSM sends prepare and test output to a scoped,
Expand Down Expand Up @@ -499,9 +503,10 @@ The selector is built to **fail safe — to `all`, never to `none`**:
- Missing or invalid repository/SHA metadata, an inconsistent mode/list pair,
and any failed clone, install, version probe, or pytest command fail the
controller. There is no moving-branch fallback.
- AWS authentication failures are terminal. After an AWS instance is allocated,
boot, SSM, setup, test, and cleanup failures are also terminal and never move
to another location.
- Only the bounded 10-minute Modal Sandbox startup timeout enables AWS. Modal API,
authentication, image-build, setup, test, cleanup, and lifetime failures are
terminal. After an AWS instance is allocated, boot, SSM, setup, test, and
cleanup failures are also terminal and never move to another location.
- Before allocation, AWS advances through the fixed regional subnet list only
for explicit EC2 insufficient-capacity responses. If every location reports
capacity unavailable, no backend ran and the Required check fails as infra.
Expand Down
91 changes: 82 additions & 9 deletions .github/workflows/modal-torch-latest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
name: modal-torch-latest

# This Required workflow selects tests on a no-secret GitHub runner, then uses
# the trusted controller to make one bounded G7.12 Spot acquisition pass across
# the configured AWS regions and test the exact candidate SHA. Under
# This Required workflow selects tests on a no-secret GitHub runner, then tests
# the exact candidate SHA inside a no-secret Modal Sandbox. If that Sandbox
# cannot start within its bounded acquisition window, the same trusted
# controller makes one bounded G7.12 Spot acquisition pass across the
# configured AWS regions. Under
# pull_request_target, every GitHub-side checkout and Python entrypoint comes
# from the trusted base commit; merge queue runs resolve that trusted base from
# merge_group.base_sha. A GPU backend runs only for merge queue entries, the
Expand All @@ -17,12 +19,12 @@ on:
workflow_dispatch:
inputs:
infrastructure_smoke:
description: Run only the AWS infrastructure smoke checks
description: Run only the parallel backend infrastructure smoke checks
required: false
default: false
type: boolean
torch_preset:
description: AWS PyTorch/CUDA image preset for manual runs
description: Modal PyTorch/CUDA image preset for manual runs
required: false
default: '2.10.0-cuda12.8'
type: choice
Expand Down Expand Up @@ -71,6 +73,41 @@ concurrency:
cancel-in-progress: true

jobs:
modal-infrastructure-smoke:
name: modal-torch-latest / Modal infrastructure smoke
if: ${{ github.event_name == 'workflow_dispatch' && inputs.infrastructure_smoke == true }}
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
steps:
- name: Checkout trusted controller code
uses: actions/checkout@v7
with:
ref: ${{ github.sha }}
fetch-depth: 1
persist-credentials: false
lfs: false
submodules: false

- name: Install Python
uses: actions/setup-python@v7
with:
python-version: "3.10"
cache: 'pip'

- name: Install trusted controller dependency
run: |
pip install uv
uv pip install --system modal==1.2.6
- name: Check Modal OCI startup, two GPUs, and cleanup
env:
MODAL_TORCH_PRESET: ${{ inputs.torch_preset }}
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
run: python3 ci/torch_latest.py modal-infrastructure-smoke

aws-infrastructure-smoke:
name: modal-torch-latest / AWS infrastructure smoke / ${{ matrix.region }}
if: ${{ github.event_name == 'workflow_dispatch' && inputs.infrastructure_smoke == true }}
Expand Down Expand Up @@ -265,8 +302,43 @@ jobs:
name: modal-torch-latest-test-selection
path: ci/.test_selection

- name: Configure AWS credentials for test backend
- name: Install trusted controller dependency
run: |
pip install uv
uv pip install --system modal==1.2.6
- name: Try tests in isolated Modal Sandbox
id: modal
env:
DS_CI_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
DS_CI_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
DS_CI_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || '' }}
DS_TEST_SELECTION_MODE: ${{ needs.collect-tests.outputs.mode }}
DS_TEST_LIST_FILE: ci/.test_selection/test_list.txt
MODAL_TORCH_PRESET: ${{ github.event.inputs.torch_preset || '2.10.0-cuda12.8' }}
MODAL_TRANSFORMERS_SOURCE: ${{ github.event.inputs.transformers_source || 'git' }}
MODAL_TRANSFORMERS_REF: ${{ github.event.inputs.transformers_ref || 'main' }}
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
run: |
set +e
python3 ci/torch_latest.py controller
status=$?
set -e
echo "status=$status" >> "$GITHUB_OUTPUT"
if [ "$status" -eq 0 ]; then
echo "fallback=false" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$status" -eq 75 ]; then
echo "fallback=true" >> "$GITHUB_OUTPUT"
exit 0
fi
exit "$status"
- name: Configure AWS credentials for capacity fallback
id: aws-auth
if: steps.modal.outputs.fallback == 'true'
env:
AWS_ROLE_ARN: ${{ secrets.AWS_MODAL_FALLBACK_ROLE_ARN }}
run: |
Expand Down Expand Up @@ -301,7 +373,8 @@ jobs:
printf 'AWS_DEFAULT_REGION=us-east-1\n'
} >> "$GITHUB_ENV"
- name: Run tests on AWS backend
- name: Run tests on AWS capacity fallback
if: steps.modal.outputs.fallback == 'true'
env:
DS_CI_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
DS_CI_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
Expand All @@ -314,8 +387,8 @@ jobs:
AWS_MODAL_FALLBACK_CONFIG: ${{ secrets.AWS_MODAL_FALLBACK_CONFIG }}
run: python3 ci/torch_latest.py aws-controller

- name: Reconcile AWS test cleanup
if: ${{ always() && steps.aws-auth.outcome == 'success' }}
- name: Reconcile AWS capacity fallback cleanup
if: ${{ always() && steps.modal.outputs.fallback == 'true' && steps.aws-auth.outcome == 'success' }}
env:
AWS_MODAL_FALLBACK_CONFIG: ${{ secrets.AWS_MODAL_FALLBACK_CONFIG }}
GITHUB_RUN_ID: ${{ github.run_id }}
Expand Down
5 changes: 2 additions & 3 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,8 @@ make test

### Diff-based CI test selection
Some GPU CI workflows (currently `modal-torch-latest`, which runs `tests/unit/v1/`)
run their GPU tests on AWS for the merge queue entry instead of on every PR push,
to conserve GPU quota. Modal execution is temporarily disabled while its GPU
driver issue is unresolved. `ci/tests_fetcher.py` looks at the files your PR
run their Modal tests on the merge queue entry instead of on every PR push, to
conserve GPU quota. `ci/tests_fetcher.py` looks at the files your PR
changes, builds an import graph over `deepspeed/` and the `unit` test helpers,
and selects only the tests that could be affected. The selection is previewed on
the PR by a cheap no-secret job and executed once the PR enters the merge queue.
Expand Down
Loading
Loading