diff --git a/.github/workflows/TEST_SELECTION.md b/.github/workflows/TEST_SELECTION.md index 9f9c3f9b61e8..c80aa0d6e1a8 100644 --- a/.github/workflows/TEST_SELECTION.md +++ b/.github/workflows/TEST_SELECTION.md @@ -4,15 +4,9 @@ This document explains the diff-driven test-selection system that lets some GPU workflows run only the tests a PR could affect, instead of the whole suite, while still satisfying Required status checks. -It currently drives **`modal-torch-latest`**, which runs `tests/unit/v1/` on a -bounded AWS GPU backend, but is built to drive more workflows from one config — -see [Adding a workflow](#add-a-new-workflow). - -> **Temporary AWS-only mode:** Modal execution is disabled while its GPU driver -> issue is unresolved. The Modal implementation and its tests remain in -> `ci/torch_latest.py` so the original Modal-first route can be restored, but the -> active workflow does not install Modal, start a Modal Sandbox, or run a Modal -> infrastructure smoke check. +It currently drives **`modal-torch-latest`** (which runs `tests/unit/v1/` on a +Modal GPU or its bounded AWS capacity fallback), but is built to drive more +workflows from one config — see [Adding a workflow](#add-a-new-workflow). - [TL;DR](#tldr) - [Why](#why) @@ -31,19 +25,19 @@ see [Adding a workflow](#add-a-new-workflow). the import graph from your changed files to the impacted tests, and writes the list to `ci/.test_selection/test_list.txt`. A nightly scheduled run always executes the full suite. -- The trusted controller validates that list, authenticates to AWS through the - deploy job's GitHub OIDC permission, then tries a fixed, ordered pool of AWS - `g7.12xlarge` Spot locations. The selected instance fetches, installs, and - tests the exact candidate SHA without repository secrets. The backend runs - only for merge queue entries, the nightly scheduled full run, and manual runs - — a plain PR event never spends GPU quota. +- The trusted controller validates that list, then fetches, installs, and tests + the exact candidate SHA inside a no-secret Modal Sandbox. If the fixed + `L40S:2` Sandbox cannot start within its acquisition window, the controller + tries a fixed, ordered pool of AWS `g7.12xlarge` Spot locations. The backend + runs only for merge queue entries, the nightly scheduled full run, and manual + runs — a plain PR event never spends GPU quota. - A manual `infrastructure_smoke` dispatch skips selection and pytest entirely. - It starts three independent regional AWS jobs in parallel. Each job directly - selects one region, launches a G7.12 Spot instance, reaches it through SSM, - verifies two host and container GPUs, stores a checksum-verified proof in the - private regional log bucket, and terminates its own run-tagged instance. The - matrix uses `fail-fast: false`, so one regional result cannot cancel the other - regions. + It starts one short OCI `L40S:2` Sandbox check and three independent regional + AWS jobs in parallel. Each AWS job directly selects one region, launches a + G7.12 Spot instance, reaches it through SSM, verifies two host and container + GPUs, stores a checksum-verified proof in the private regional log bucket, + and terminates its own run-tagged instance. The matrix uses + `fail-fast: false`, so one regional result cannot cancel the other regions. - It is **fail-safe**: anything it can't reason about safely → run the *full* suite. It never silently runs *fewer* tests than reality. - Preview locally: @@ -107,10 +101,9 @@ only documentation-only diffs. Preview the decision with Scheduled, manual, and push runs retain their existing behavior. Required workflows still start and report their statuses; only their heavy jobs or steps -skip. The torch-latest selector imports the same `is_docs_path()` and drops -documentation paths before applying its run-all path globs and import graph, so -documentation never widens a selection; explicit `[test all]` / `[no filter]` -overrides still win. +skip. The Modal selector imports the same `is_docs_path()` and drops documentation +paths before applying its run-all path globs and import graph, so documentation +never widens a selection; explicit `[test all]` / `[no filter]` overrides still win. Root `README.md` is package metadata, so the formatting workflow always runs `python scripts/check-readme.py` to check that it exists and is valid UTF-8 @@ -125,7 +118,7 @@ without installing DeepSpeed or PyTorch. | `ci/tests_fetcher.py` | The selector. AST-parses the repo, builds an import graph, decides `all` / `subset` / `none`, writes the test-list file, emits a job summary. | | `ci/check_paths.py` | Defines documentation paths (`is_docs_path()`) for the selector and the `check-paths.yml` CPU/GPU workflow gate. | | `ci/test_tests_fetcher.py` | Self-tests for the selector and `ci/check_paths.py` (pure stdlib; run in `collect-tests` and, on the PR's own code, in the formatting workflow). | -| `ci/torch_latest.py` | Pure-stdlib metadata/selection validation plus the active AWS controller and retained Modal-first controller. | +| `ci/torch_latest.py` | Pure-stdlib metadata/selection validation plus the trusted Modal-first, AWS-capacity-fallback controller. | | `ci/.test_selection/test_list.txt` | The hand-off artifact (one pytest target per line). Git-ignored. | ### Job flow @@ -149,9 +142,11 @@ without installing DeepSpeed or PyTorch. │ if mode != none │ │ checkout exact base SHA │ │ validate mode/list + SHA │ - │ assume AWS role via OIDC │ - │ try G7.12 Spot by region │ - │ via SSM, no inbound port │ + │ try one L40S:2 Sandbox │ + │ no secrets or mounts │ + │ startup capacity only: │ + │ try G7.12 Spot by region │ + │ via GitHub OIDC + SSM │ │ fetch exact head SHA │ │ install + run pytest │ │ terminate chosen backend │ @@ -163,12 +158,14 @@ A manual dispatch with `infrastructure_smoke=true` takes a separate path: ```text workflow_dispatch (infrastructure_smoke=true) │ - AWS regional matrix - fail-fast: false - east-1 / east-2 / west-2 - launch + SSM + two GPUs - container + private proof - per-region termination + ┌─────────────────┴─────────────────┐ + │ │ + short Modal OCI smoke AWS regional matrix + startup + two GPUs fail-fast: false + always terminates east-1 / east-2 / west-2 + launch + SSM + two GPUs + container + private proof + per-region termination ``` That path never creates a test-selection artifact, installs DeepSpeed, or runs @@ -197,8 +194,10 @@ the nightly full suite exists to catch. For a manual infrastructure-only check, select `infrastructure_smoke` when dispatching the workflow. That skips both `collect-tests` and `deploy`; the -per-region AWS smoke jobs run directly and concurrently instead. Leaving the -input at its default `false` preserves the ordinary manual full scope. +Modal and per-region AWS smoke jobs run directly and concurrently instead. +Leaving the input at its default `false` preserves the ordinary manual full +scope. The AWS smoke path does not depend on Modal's result, so a successful +Modal allocation cannot bypass the regional checks. ### Nightly full-suite runs and regression triage @@ -209,7 +208,7 @@ suite. Its outcome feeds `nightly-triage.yml`: anchors the last verified-good master revision (and is exactly the good endpoint a later manual bisect would need), so days that fail for operational reasons (no GPU instance, timeout) simply leave the tag in place. -- **Infra** (no configured AWS location acquired a GPU) → +- **Infra** (neither Modal nor any configured AWS location acquired a GPU) → nothing is concluded; no report, no tag move. - **Timeout** (the backend/job time budget ran out) → an issue is opened; a timeout points at an operational problem, not a candidate regression. @@ -226,11 +225,13 @@ suite. Its outcome feeds `nightly-triage.yml`: is the good endpoint it would start from. The controller (`ci/torch_latest.py`) classifies its own failures for this -routing and prints exactly one `DS_CI_FAILURE_CLASS=infra|timeout|test` sentinel -on failure. If every configured AWS location reports insufficient capacity, no -backend runs and the outcome is infra (75); 300 seconds without pytest output is -a timeout (124); setup and test errors are test failures (1). A failed run with -no sentinel routes to the unknown class instead of guessing. +routing. It waits up to 10 minutes for Modal Sandbox startup; only that +startup-capacity timeout returns 75 to the workflow without a final sentinel +so that AWS can run. The selected backend prints exactly one final +`DS_CI_FAILURE_CLASS=infra|timeout|test` sentinel on failure. If no backend runs +the outcome is infra (75); 300 seconds without pytest output is a timeout +(124); setup and test errors are test failures (1). A failed run with no +sentinel routes to the unknown class instead of guessing. Triage reports (timeout, unknown, test failures) go through `ci/nightly_report.sh`: every report carries the `nightly-triage` label, and a recurring outcome @@ -413,7 +414,8 @@ tests. ## Security model The workflow triggers on **`pull_request_target`**, so it runs in the base repo's -context and the `deploy` controller can assume its AWS role. The trust boundary is: +context and the `deploy` controller can authenticate to Modal or, only for the +capacity fallback, assume its AWS role. The trust boundary is: - Both jobs check out only the exact trusted base SHA (or the exact push/manual SHA), with persisted credentials, LFS, and submodules disabled. @@ -432,9 +434,8 @@ context and the `deploy` controller can assume its AWS role. The trust boundary - `deploy` runs only the trusted controller. Candidate metadata is restricted to a public `owner/repository` and full 40-hex SHA; event values enter Python through environment variables, not workflow shell interpolation. GitHub's - OIDC permission is scoped to this job. AWS credentials are requested only - after the trusted checkout, Python setup, and validated selection artifact - are available; cleanup runs only if that authentication step succeeded. + OIDC permission is scoped to this job and AWS credentials are requested only + after the Modal controller returns its dedicated startup-capacity result. - The controller role ARN and opaque regional fallback JSON are stored as masked repository secrets. The controller logs region and ordinal location transitions, but never the launch-template, subnet, bucket, instance, or SSM @@ -442,12 +443,15 @@ context and the `deploy` controller can assume its AWS role. The trust boundary The trusted job exchanges its GitHub OIDC token directly for only the three credential fields, masks each value before exporting it, and never requests or prints the STS assumed-role identity. -- The active job receives no Modal service token and does not install or invoke - Modal. Its retained implementation preserves the no-secret Sandbox boundary - for a future re-enable. +- The Modal service token remains in the controller process. The Sandbox gets + no Modal, GitHub, Hugging Face, OIDC, or other secret; no local checkout, + mount, volume, network filesystem, port, proxy, or workload identity is + attached. The controller constructs a positive allowlist of non-sensitive + git/pip environment flags instead of forwarding its environment. - Candidate acquisition, dependency installation, DeepSpeed installation, and - pytest all happen on one selected `g7.12xlarge` Spot instance. The controller - tries configured subnets in fixed `us-east-1`, + pytest all happen inside one selected backend. Modal uses an isolated + `l40s:2` Sandbox with a bounded server-side lifetime. Its capacity fallback + uses only `g7.12xlarge` Spot, tries configured subnets in fixed `us-east-1`, `us-east-2`, `us-west-2` order, exposes exactly GPUs 0 and 1 to the container, and controls it through SSM. The instance has no inbound rules and a 60-minute termination guard. SSM sends prepare and test output to a scoped, @@ -499,9 +503,10 @@ The selector is built to **fail safe — to `all`, never to `none`**: - Missing or invalid repository/SHA metadata, an inconsistent mode/list pair, and any failed clone, install, version probe, or pytest command fail the controller. There is no moving-branch fallback. -- AWS authentication failures are terminal. After an AWS instance is allocated, - boot, SSM, setup, test, and cleanup failures are also terminal and never move - to another location. +- Only the bounded 10-minute Modal Sandbox startup timeout enables AWS. Modal API, + authentication, image-build, setup, test, cleanup, and lifetime failures are + terminal. After an AWS instance is allocated, boot, SSM, setup, test, and + cleanup failures are also terminal and never move to another location. - Before allocation, AWS advances through the fixed regional subnet list only for explicit EC2 insufficient-capacity responses. If every location reports capacity unavailable, no backend ran and the Required check fails as infra. diff --git a/.github/workflows/modal-torch-latest.yml b/.github/workflows/modal-torch-latest.yml index 8ea6bb40d77a..9cfcc10a8dac 100644 --- a/.github/workflows/modal-torch-latest.yml +++ b/.github/workflows/modal-torch-latest.yml @@ -1,8 +1,10 @@ name: modal-torch-latest -# This Required workflow selects tests on a no-secret GitHub runner, then uses -# the trusted controller to make one bounded G7.12 Spot acquisition pass across -# the configured AWS regions and test the exact candidate SHA. Under +# This Required workflow selects tests on a no-secret GitHub runner, then tests +# the exact candidate SHA inside a no-secret Modal Sandbox. If that Sandbox +# cannot start within its bounded acquisition window, the same trusted +# controller makes one bounded G7.12 Spot acquisition pass across the +# configured AWS regions. Under # pull_request_target, every GitHub-side checkout and Python entrypoint comes # from the trusted base commit; merge queue runs resolve that trusted base from # merge_group.base_sha. A GPU backend runs only for merge queue entries, the @@ -17,12 +19,12 @@ on: workflow_dispatch: inputs: infrastructure_smoke: - description: Run only the AWS infrastructure smoke checks + description: Run only the parallel backend infrastructure smoke checks required: false default: false type: boolean torch_preset: - description: AWS PyTorch/CUDA image preset for manual runs + description: Modal PyTorch/CUDA image preset for manual runs required: false default: '2.10.0-cuda12.8' type: choice @@ -71,6 +73,41 @@ concurrency: cancel-in-progress: true jobs: + modal-infrastructure-smoke: + name: modal-torch-latest / Modal infrastructure smoke + if: ${{ github.event_name == 'workflow_dispatch' && inputs.infrastructure_smoke == true }} + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + steps: + - name: Checkout trusted controller code + uses: actions/checkout@v7 + with: + ref: ${{ github.sha }} + fetch-depth: 1 + persist-credentials: false + lfs: false + submodules: false + + - name: Install Python + uses: actions/setup-python@v7 + with: + python-version: "3.10" + cache: 'pip' + + - name: Install trusted controller dependency + run: | + pip install uv + uv pip install --system modal==1.2.6 + + - name: Check Modal OCI startup, two GPUs, and cleanup + env: + MODAL_TORCH_PRESET: ${{ inputs.torch_preset }} + MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }} + MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }} + run: python3 ci/torch_latest.py modal-infrastructure-smoke + aws-infrastructure-smoke: name: modal-torch-latest / AWS infrastructure smoke / ${{ matrix.region }} if: ${{ github.event_name == 'workflow_dispatch' && inputs.infrastructure_smoke == true }} @@ -265,8 +302,43 @@ jobs: name: modal-torch-latest-test-selection path: ci/.test_selection - - name: Configure AWS credentials for test backend + - name: Install trusted controller dependency + run: | + pip install uv + uv pip install --system modal==1.2.6 + + - name: Try tests in isolated Modal Sandbox + id: modal + env: + DS_CI_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name || github.repository }} + DS_CI_SHA: ${{ github.event.pull_request.head.sha || github.sha }} + DS_CI_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || '' }} + DS_TEST_SELECTION_MODE: ${{ needs.collect-tests.outputs.mode }} + DS_TEST_LIST_FILE: ci/.test_selection/test_list.txt + MODAL_TORCH_PRESET: ${{ github.event.inputs.torch_preset || '2.10.0-cuda12.8' }} + MODAL_TRANSFORMERS_SOURCE: ${{ github.event.inputs.transformers_source || 'git' }} + MODAL_TRANSFORMERS_REF: ${{ github.event.inputs.transformers_ref || 'main' }} + MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }} + MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }} + run: | + set +e + python3 ci/torch_latest.py controller + status=$? + set -e + echo "status=$status" >> "$GITHUB_OUTPUT" + if [ "$status" -eq 0 ]; then + echo "fallback=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + if [ "$status" -eq 75 ]; then + echo "fallback=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + exit "$status" + + - name: Configure AWS credentials for capacity fallback id: aws-auth + if: steps.modal.outputs.fallback == 'true' env: AWS_ROLE_ARN: ${{ secrets.AWS_MODAL_FALLBACK_ROLE_ARN }} run: | @@ -301,7 +373,8 @@ jobs: printf 'AWS_DEFAULT_REGION=us-east-1\n' } >> "$GITHUB_ENV" - - name: Run tests on AWS backend + - name: Run tests on AWS capacity fallback + if: steps.modal.outputs.fallback == 'true' env: DS_CI_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name || github.repository }} DS_CI_SHA: ${{ github.event.pull_request.head.sha || github.sha }} @@ -314,8 +387,8 @@ jobs: AWS_MODAL_FALLBACK_CONFIG: ${{ secrets.AWS_MODAL_FALLBACK_CONFIG }} run: python3 ci/torch_latest.py aws-controller - - name: Reconcile AWS test cleanup - if: ${{ always() && steps.aws-auth.outcome == 'success' }} + - name: Reconcile AWS capacity fallback cleanup + if: ${{ always() && steps.modal.outputs.fallback == 'true' && steps.aws-auth.outcome == 'success' }} env: AWS_MODAL_FALLBACK_CONFIG: ${{ secrets.AWS_MODAL_FALLBACK_CONFIG }} GITHUB_RUN_ID: ${{ github.run_id }} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 976f42aa234f..5dbbd426d6fd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -53,9 +53,8 @@ make test ### Diff-based CI test selection Some GPU CI workflows (currently `modal-torch-latest`, which runs `tests/unit/v1/`) -run their GPU tests on AWS for the merge queue entry instead of on every PR push, -to conserve GPU quota. Modal execution is temporarily disabled while its GPU -driver issue is unresolved. `ci/tests_fetcher.py` looks at the files your PR +run their Modal tests on the merge queue entry instead of on every PR push, to +conserve GPU quota. `ci/tests_fetcher.py` looks at the files your PR changes, builds an import graph over `deepspeed/` and the `unit` test helpers, and selects only the tests that could be affected. The selection is previewed on the PR by a cheap no-secret job and executed once the PR enters the merge queue. diff --git a/ci/test_torch_latest.py b/ci/test_torch_latest.py index dff1b23ba5ab..96e89a462ee5 100644 --- a/ci/test_torch_latest.py +++ b/ci/test_torch_latest.py @@ -1657,12 +1657,12 @@ def test_workflow_keeps_github_execution_trusted_and_preserves_modes(): assert "allow-unsafe-pr-checkout" not in text assert "Use base-branch CI scripts" not in text assert "HF_TOKEN" not in text - assert "modal==1.2.6" not in text + assert "modal==1.2.6" in text assert "timeout-minutes: 20" in text assert "timeout-minutes: 180" in text - assert text.count("persist-credentials: false") == 3 - assert text.count("lfs: false") == 3 - assert text.count("submodules: false") == 3 + assert text.count("persist-credentials: false") == 4 + assert text.count("lfs: false") == 4 + assert text.count("submodules: false") == 4 assert "github.event.pull_request.head.repo.full_name" in text assert "github.event.pull_request.head.sha" in text assert "github.event.pull_request.base.repo.full_name" in text @@ -1672,11 +1672,11 @@ def test_workflow_keeps_github_execution_trusted_and_preserves_modes(): assert text.count("modal-torch-latest-test-selection") == 2 assert "needs.collect-tests.outputs.mode != 'none'" in text assert "needs.collect-tests.result != 'success'" in text - assert 'python3 ci/torch_latest.py controller' not in text + assert 'python3 ci/torch_latest.py controller' in text assert 'python3 ci/torch_latest.py aws-controller' in text assert 'python3 ci/torch_latest.py cleanup-aws' in text - assert "steps.modal.outputs.fallback == 'true'" not in text - assert 'if [ "$status" -eq 75 ]' not in text + assert "steps.modal.outputs.fallback == 'true'" in text + assert 'if [ "$status" -eq 75 ]' in text assert "aws-actions/configure-aws-credentials" not in text assert "sts assume-role-with-web-identity" in text assert "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" in text @@ -1688,11 +1688,11 @@ def test_workflow_keeps_github_execution_trusted_and_preserves_modes(): assert "vars.AWS_MODAL_FALLBACK" not in text assert "infrastructure_smoke:" in text assert "default: false" in text - assert text.count("inputs.infrastructure_smoke == true") == 1 + assert text.count("inputs.infrastructure_smoke == true") == 2 assert text.count("inputs.infrastructure_smoke != true") == 2 assert "region: [us-east-1, us-east-2, us-west-2]" in text assert "fail-fast: false" in text - assert 'python3 ci/torch_latest.py modal-infrastructure-smoke' not in text + assert 'python3 ci/torch_latest.py modal-infrastructure-smoke' in text assert 'python3 ci/torch_latest.py aws-infrastructure-smoke --region "${{ matrix.region }}"' in text assert "needs: modal-infrastructure-smoke" not in text assert "DS_CI_AWS_RUN_SUFFIX: ${{ matrix.region }}" in text @@ -1705,14 +1705,15 @@ def test_workflow_keeps_github_execution_trusted_and_preserves_modes(): assert "checkout-candidate" not in deploy assert "pull_request.head.sha || github.sha" in deploy assert "pull_request.head.repo.full_name || github.repository" in deploy - assert "MODAL_TOKEN_ID" not in deploy - assert "MODAL_TOKEN_SECRET" not in deploy - deploy_auth = deploy.split("- name: Configure AWS credentials for test backend", - 1)[1].split("- name: Run tests on AWS backend", 1)[0] - deploy_cleanup = deploy.split("- name: Reconcile AWS test cleanup", 1)[1] + assert "MODAL_TOKEN_ID" in deploy + assert "MODAL_TOKEN_SECRET" in deploy + deploy_auth = deploy.split("- name: Configure AWS credentials for capacity fallback", + 1)[1].split("- name: Run tests on AWS capacity fallback", 1)[0] + deploy_cleanup = deploy.split("- name: Reconcile AWS capacity fallback cleanup", 1)[1] assert "id: aws-auth" in deploy_auth - assert "if: ${{ always() && steps.aws-auth.outcome == 'success' }}" in deploy_cleanup - assert "if: ${{ always() }}" not in deploy_cleanup + assert "if: steps.modal.outputs.fallback == 'true'" in deploy_auth + assert ("if: ${{ always() && steps.modal.outputs.fallback == 'true' && steps.aws-auth.outcome == 'success' }}" + in deploy_cleanup) def test_launcher_source_has_no_local_packaging_or_shell_execution():