[DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE by delock · Pull Request #8381 · deepspeedai/DeepSpeed

delock · 2026-09-01T01:21:36Z

This is an on going work. Note I'll rebase when revealed bugs are fixed. The goal is fix all issues exposed by this CI, then turn on multi-rank test in CPU workflow.

Problem

On the cpu-torch-latest runner (single-socket ubuntu-24.04), every unit test that needs more than one rank is skipped:

SKIPPED tests/unit/common.py:275: Skipping test because not enough GPUs are available: 2 required, 1 available

Root cause: CPU_Accelerator.device_count() reports the number of NUMA nodes (accelerator/cpu_accelerator.py), which is 1 on the single-socket runner, and DistributedExec._launch_procs() in tests/unit/common.py gates process count on device_count(). Multi-rank CPU tests do not actually need one device per rank — ranks are ordinary processes communicating over gloo.

Change

Set LOCAL_SIZE=4 for the unit-tests job. device_count() reads LOCAL_SIZE first, so the launch gate now admits world_size<=4 tests. This is the same signal the DeepSpeed launcher sets for spawned processes; the unit-test harness re-sets LOCAL_SIZE per worker in _dist_run(), so the CI-level value only affects the gate and cannot leak into test bodies.

Evidence this is safe

TestDistIsendIrecv (tests/unit/comm/test_dist.py, world_size=2) and the autotp universal-checkpoint test (tests/unit/checkpoint/test_autotp_uc_checkpoint.py, world_size=4) already bypass the per-device gate for CPU and run green in this very CI job.
LOCAL_SIZE has a single reader in the codebase (CPU_Accelerator.device_count()); the launcher only writes it.

Expected impact

~207 world_size=2 and ~85 world_size=4 tests that are currently skipped will now execute on CPU CI (world_size>=8 stays skipped). Some of them may have latent failures — this PR intentionally surfaces them so they can be triaged.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af3841378c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-01T01:24:04Z

+      # multi-rank test. CPU ranks are plain processes over gloo, so advertise 4
+      # local devices to let world_size<=4 tests run. The test harness re-sets
+      # LOCAL_SIZE per worker, so this value only affects the launch gate.
+      LOCAL_SIZE: '4'


Keep CUDA-only distributed tests out of the CPU run

In the cpu-torch-latest job, advertising four devices admits every distributed test with world_size <= 4, not only CPU-safe tests. For example, tests/unit/ulysses_alst/test_ulysses_sp_hf.py:240-263 defines an unguarded two-rank test that creates tensors on cuda:<rank>; because CPU_Accelerator.is_available() returns true, the harness does not skip it, and the all-unit pytest invocation at workflow line 283 will fail on the CPU-only PyTorch installation. Scope this override to an explicitly CPU-compatible subset or add CPU capability checks before enabling the previously skipped tests.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-01T01:24:04Z

+      # multi-rank test. CPU ranks are plain processes over gloo, so advertise 4
+      # local devices to let world_size<=4 tests run. The test harness re-sets
+      # LOCAL_SIZE per worker, so this value only affects the launch gate.
+      LOCAL_SIZE: '4'


Add the required sign-off trailer

This is a non-merge commit, but its commit message has no Signed-off-by trailer. Add the author sign-off so the commit satisfies the repository's commit and CI requirements.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

delock · 2026-09-01T05:59:13Z

This is an on going work. Note I'll rebase when revealed bugs are fixed. The goal is fix all issues exposed by this CI, then turn on multi-rank test in CPU workflow.

delock · 2026-09-02T15:01:44Z

Experiment results: enabling multi-rank CPU tests via `LOCAL_SIZE=4`

Baseline comparison: same-commit master run (all multi-rank tests skipped) vs this branch. Zero regressions: every failure is a test that was previously skipped.

What now runs

~1800 previously-skipped multi-rank tests execute on the CPU runner (gloo). Final complete numbers from the last run (split invocations):

unit/v1 half (complete): 916 passed / 131 failed / 206 skipped
sequential (complete): 54 passed / 9 failed
non-v1 half: killed by the 150m guard (still too slow on 4 vCPU), 59 failures listed inline

Failure inventory & root causes (triaged)

Count	Test	Root cause
92	`v1/zero/test_offload_states.py`	Asserts `memory_allocated()` drops after offload — a VRAM concept. `CPU_Accelerator.memory_allocated()` returns RSS, which never shrinks on free. Test/accelerator semantic gap; needs discussion (gate on device semantics or a better CPU metric).
24	`v1/zero/test_zero_autocast.py`	Hardcoded `dist_backend='nccl'` + a CUDA/NCCL-oriented bf16 gate. Needs accelerator-derived backend + capability-based skip.
25	ulysses SP tests	Numerical mismatches on CPU (real correctness questions, need deep dive).
16	onebit optimizer tests	`NoneType.size` at `onebit/adam.py:108` — runtime bug on CPU path.
~20	pipe / zeropp / moe-checkpoint / coalesce	Smaller follow-ups, same GPU-assumption patterns.

Test-side fixes already in this branch (validated by CI)

All 5 DDP reference sites route through wrap_ddp_reference() (CPU modules must not pin device_ids) — dropped this class from 114 failures to 7.
reduce_boolean_flags / autotp tests use current_device_name() instead of current_device() (a LOCAL_RANK string on CPU, not a device).
fp16-config tests get a capability skipif (not get_accelerator().is_fp16_supported()) — GH ubuntu-24.04 runners are hardware-heterogeneous w.r.t. AVX512-FP16, so hardcoded fp16 was a runner lottery.
fork_rng probes the device module for a per-device RNG instead of matching accelerator names.

CI infrastructure findings (need maintainer decisions)

--maxfail=100 in PYTEST_OPTS + no timeout in DistributedExec._close_pool = permanent wedge: when the 100th failure trips the interrupt, teardown (pool.starmap(_dist_destroy), close/join) can block forever on wedged gloo peers; the run then dies at the 6h job limit without printing any summary. This is what cancelled the first three attempts. (This branch works around it by splitting the suite and raising maxfail.)
Suite size vs 4 vCPU: the full multi-rank suite does not fit one 6h job. Options: split invocations (done here), lower -n, or a dedicated larger runner.
CPU_Accelerator.device_count()'s NUMA-node semantics remain the root cause of the original gap; the long-term fix could be a test-harness-level exemption for CPU (ranks are processes, not devices).

Happy to split the commits into separate PRs (workflow change / mechanical test fixes / triage follow-ups) per maintainer preference — the failure inventory above is intended as the working list.

delock · 2026-09-03T00:05:41Z

Given the CPU UT would run ~hrs after enabling LOCAL_SIZE=4, I wouldn't suggest to turn on this in master. However it is beneficial to use this branch to reveal problems in CPU accelerator and fix them. I'm mark this PR as do not merge.

…dai#8398) ## Problem `TestMultipleModels::test_zero_optimizer`, `TestSimpleMoE`, `TestMoE`, `TestPRMoE`, and `TestMOETensorParallel` hardcode `"fp16": {"enabled": True}` in their DeepSpeed configs. The engine's sanity check then raises: ``` ValueError: Type fp16 is not supported on your device. ``` on any accelerator whose `is_fp16_supported()` is false. On CPU that maps to the AVX512-FP16 capability of the host, and GitHub's `ubuntu-24.04` runners are hardware-heterogeneous: **the same test passes on one runner and fails on the next** (observed directly in deepspeedai#8381 — 146 failures appeared on one runner generation and none on another, with identical code). ## Change Skip these tests via a capability query: ```python @pytest.mark.skipif(not get_accelerator().is_fp16_supported(), reason="fp16 is not supported on this accelerator") ``` - capability only, no accelerator-name matching; - mirrors the existing bf16 skip precedent in `tests/unit/v1/zero/test_zero_user_backward.py`; - deliberately a **skip** rather than silently running bf16 — these tests exist to cover the fp16 paths. ## Validation Validated as part of the multi-rank CPU CI experiment in deepspeedai#8381: the 146 hardware-lottery failures became deterministic skips, zero regressions on previously-passing tests. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

## Problem `tests/unit/v1/zero/test_zero_user_backward.py` builds torch DDP **reference** models (the known-good baseline that ZeRO results are compared against) at five sites: ```python model_ddp = DDP(model_ddp, device_ids=[rank], output_device=rank) ``` `device_ids=[rank]` assumes rank ↔ GPU index. torch's DDP contract only allows `device_ids`/`output_device` for single-device GPU modules; **CPU modules live on one shared device and must omit them**, so multi-rank CPU runs died inside the DDP constructor with: ``` ValueError: DistributedDataParallel device_ids and output_device arguments only work with single-device/multiple-device GPU modules or CPU modules, ... ``` ## Change Route all five sites through one helper: ```python def wrap_ddp_reference(model, device, rank): # Only indexed devices take device_ids/output_device; CPU modules live on one shared device. if torch.device(device).type == 'cpu': return DDP(model) return DDP(model, device_ids=[rank], output_device=rank) ``` Design notes: - the condition restates the exact precondition torch's own DDP constructor enforces, in torch's device vocabulary — it follows the model's actual device rather than the global accelerator configuration; - a single helper means new reference-model sites cannot forget the branch (the first fix round in deepspeedai#8381 missed 4 of the 5 sites for exactly this reason); - GPU behavior is unchanged. ## Validation Validated as part of the multi-rank CPU CI experiment in deepspeedai#8381: all DDP-constructor failures were eliminated (the file's few remaining failures there are unrelated — see the triage table in that PR), zero regressions vs the same-commit baseline. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…eepspeedai#8397) ## Problem `get_accelerator().current_device()` returns a **device index** on GPU backends (`torch.cuda.current_device()` → int), but on CPU it returns the `LOCAL_RANK` environment value — a plain **string** like `'1'`. Two test-side consumers fed that value straight into tensor/device placement: - `reduce_boolean_flags` in `tests/unit/common.py` (backbone of `allclose_on_all_ranks`, the "all ranks succeed or fail together" check) - 15 call sites in `tests/unit/v1/autotp/test_autotp_training.py` On CPU this fails immediately with `RuntimeError: Invalid device string: '1'` — before the first collective even runs. ## Change - Use `current_device_name()`, which returns a full device string on every backend (`'cpu'`, `'cuda:N'`, `'mps:0'`, …) and is equivalent to the index on GPU backends. - In `reduce_boolean_flags`, carry the flag in a 1-dim tensor: gloo rejects 0-dim inputs to `all_gather_into_tensor` (NCCL tolerates them), so the previous form would have failed on the very next line. ## Validation Validated as part of the multi-rank CPU CI experiment in deepspeedai#8381 (same-commit baseline comparison): this failure class disappeared, zero regressions on previously-passing tests. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Signed-off-by: Ma, Guokai <guokai.ma@intel.com>

cpu-torch-latest runs on a single-socket runner where CPU_Accelerator.device_count() reports 1 NUMA node, so the per-device gate in tests/unit/common.py skips every test that needs more than one rank. CPU ranks are plain processes over gloo and need no per-rank hardware, so advertise 4 local devices via LOCAL_SIZE, the env var device_count() reads first. The test harness re-sets LOCAL_SIZE per worker, so this value only affects the launch gate. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

With LOCAL_SIZE=4 the suite now runs to ~63% and then all xdist workers go silent for hours until the 6h job limit cancels the run - the pool worker cleanup hang that DS_DISABLE_REUSE_DIST_ENV was added for. Fresh pools per test cost some wall time but let the run finish and print the failure summary. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

Both full-suite attempts wedge before printing a summary: once the 100th failure trips PYTEST_OPTS' --maxfail, pytest-xdist's interrupt path stalls forever in mp pool teardown (no timeout guards _close_pool), and the 6h job limit cancels the run. Run the suite as two fresh-worker halves, override maxfail so all failures are listed, and cap each half with timeout so the sequential tail always runs. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…epspeedai#8407) ## Description `train_cifar` unconditionally passed `devices=[get_accelerator().current_device_name()]` to `fork_rng`. On CPU that becomes `devices=['cpu']`, and `torch.cpu` has no `get_rng_state`, so the context manager raises `AttributeError` before the test body starts — the CPU RNG lives in the global generator that `fork_rng` already forks. Probe the device module for a per-device RNG instead of matching accelerator names: - backends whose device module has `get_rng_state` (e.g. cuda) behave exactly as before; - backends without one (cpu) pass `devices=[]`, which changes nothing beyond the global-RNG save/restore `fork_rng` always performs. The crash is only reachable from the multi-rank tests that call `train_cifar` (`test_onebit.py`, `test_pipe.py`), which the CPU runner currently skips at the device gate; it was exposed by the `LOCAL_SIZE=4` experiment in deepspeedai#8381. Sibling fixes from the same series landed as deepspeedai#8397, deepspeedai#8398 and deepspeedai#8399. ## Test plan - pre-commit (yapf / flake8 / check-torchdist / codespell) passes on the changed file; - exercised under the `LOCAL_SIZE=4` multi-rank CPU run tracked in deepspeedai#8381: the `test_onebit` / `test_pipe` callers got past `fork_rng` and into their test bodies with this exact change. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…peedai#8409) ## Description The dynamic offload-state tests assert strict allocated-memory deltas around `offload_states()` / `reload_states()`: - `alloc_after_offload < alloc_before_offload` - `alloc_after_reload > alloc_after_offload` That contract assumes `memory_allocated()` is allocator bookkeeping, which holds on cuda (`torch.cuda.memory_allocated()`). On cpu, `CPU_Accelerator.memory_allocated()` reports process RSS (psutil), and RSS does not shrink when tensors are freed — so all 92 parameterized cases fail even when the offload itself is correct (the device-placement and data-integrity checks in the same tests pass). Gate only the memory-delta asserts on whether the accelerator's torch device module exposes `memory_allocated` (cuda does; `torch.cpu` does not), mirroring the capability probe used for `fork_rng` in `train_cifar` (deepspeedai#8407): - cuda and other allocator-backed backends: behavior unchanged - cpu: the unobservable deltas are skipped; all device-placement validations still run Exposed by the `LOCAL_SIZE=4` multi-rank CPU run in deepspeedai#8381 (92 of the 131 v1-half failures there). ## Validation (executed on real hardware) - 20-core x86_64 CPU, torch 2.13.0+cpu, gloo backend, 2 ranks (`LOCAL_SIZE=2`) - Before: `TestDynamicOffloadStatesZero12[False-1-False-False-optim_states]` fails on the persistent-state delta assert - After: 5 representative cases pass (persistent and grad paths, ZeRO stage 1/2/3, `static_offload_optimizer=True` branch) — 5 passed in 55.6s - pre-commit (yapf / flake8 / check-torchdist / codespell) passes on the changed file Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

delock · 2026-09-16T09:41:27Z

Status update

Fixes extracted so far (spin-off PRs)

Investigating the failures exposed by this CI change surfaced three real bugs, each with a standalone fix:

split_half_float_double() silently drops all gradient buckets on CPU — legacy type strings built from the accelerator device name never match CPU tensors (torch.FloatTensor has no device prefix), so with contiguous_gradients=False + reduce_scatter=False ZeRO-1/2 skips the gradient all-reduce entirely; mean configs mask it, sum is off by world_size. Fix in Fix silent gradient-bucket drop on CPU in ZeRO-1/2 non-contiguous reduction #8382 (also fixes the TestCoalesceCollectiveCount "baseline issued no collectives" failure).
Test-harness comparison helper crashes on CPU — reduce_boolean_flags() passed current_device() (a bare rank string) to torch and used a 0-dim all-gather that gloo rejects. The current_device_name() part already landed on master via Use device names, not rank ids, for device placement in test helpers #8397; the rest is in Fix silent gradient-bucket drop on CPU in ZeRO-1/2 non-contiguous reduction #8382.
ZeRO-3 coalesced parameter gather only waits for the last async handle — gloo handles are independent and CPU synchronize() is a no-op, so earlier gathers can still be in flight when param.data is read, exposing uninitialized memory as NaN. Reproduced flaky (9/20 → 0/20 with the fix); a separate PR for partition_parameters.py is being prepared.

Current failures on this branch (84 total: 75 in the `unit/v1` half + 9 sequential)

~50: CPU gating missing for GPU-only tests (bf16-requires-NCCL checks, fp16-unsupported, remaining DDP device_ids usages in test_zero_autocast.py)
15: HF download PermissionError for bert-base-uncased — runner environment issue, not code
~5: covered by Fix silent gradient-bucket drop on CPU in ZeRO-1/2 non-contiguous reduction #8382 + the ZeRO-3 wait fix above
16: test_offload_states remainder after Gate the offload-state memory deltas on allocator-backed stats #8409's gating; needs triage

Current focus: phase-1 timeout / worker wedge

The job takes 4h05m and the first half (unit/ non-v1, timeout 150m) never finishes — it reaches 94% and a worker wedges for ~40min until the timeout kills it, losing that half's results. From the job log:

gw1 claims TestDistInferenceAllReduce::test[128-dtype2] (fp16) at 09:54:17 and produces no output for the remaining 111 minutes;
gw0/gw2 wedge later, but the tests they were running were later passed by other workers — secondary casualties of state left by the first wedge;
gw3 runs alone until the timeout.

Prime suspect is dist.inference_all_reduce() on an fp16 tensor over CPU/gloo. Investigating next; fixing this should also bring the job back to ~2h45m and recover phase-1 results.

The shm-based inference_all_reduce keys its /dev/shm segments by MASTER_ADDR/MASTER_PORT. The bind-and-release probe in get_master_port() hands every pool the same first-free port when tests run one process each (--forked), so concurrent pools share segments: ranks of different pools cross-write the collective state and produce wrong results or spin forever (the CI worker wedge). Also the op never unlinks its ~66MB-per- rank segments, which exhausts /dev/shm over a full run. Offset the port probe per process and per pool so each pool gets a distinct key, and remove the segments once the pool (or spawned procs) is done. Verified with two concurrent gloo pools calling inference_all_reduce: shared key gives wrong results (rank sees 2.0 instead of 3.0), distinct keys are correct; three rounds of -n 2 --forked TestDistInferenceAllReduce pass with zero /dev/shm leftovers. Signed-off-by: Ma, Guokai <guokai.ma@intel.com>

Concurrent xdist workers each downloading bert-base-uncased race on the transformers cache lock and fail with PermissionError. Populate a shared HF_HOME once before the suite and point both unit-test halves at it (the sequential phase already used it), so tests only read the cache. Signed-off-by: Ma, Guokai <guokai.ma@intel.com>

huggingface-cli is now a deprecation stub in the runner's huggingface_hub: it prints a warning and exits 1, killing the job in the pre-download step before any test runs. Use its replacement, hf download, which is already installed on the runner. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

bert-base-uncased is stored on Xet, and the anonymous xet-read-token request 404s, so hf download dies before populating the cache. Fall back to the classic resolve endpoint, which works without a token. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

ZeRO-3 also rounds the partitioned parameters through the autocast dtype on every all-gather, so the engine's forward runs on fp16-rounded weights while the fp32 baseline rounds per op; the loss carries percent-level dtype noise (measured 0.4%-2.7% across runs) instead of the sub-ulp noise covered below ZeRO-3. Extend the parity tolerance accordingly (rtol=5e-2, atol=5e-2 with 2x margin over the measured band); bf16 and fp32 paths are unchanged. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The default preset (2.10.0) predates four stable releases and carries a known CPU SDPA integer-divide-by-zero on zero-head tensors, which kills a rank with SIGFPE in the uneven-TP tests (trailing ranks hold size-0 attention shards; fixed in newer kernels, verified on 2.13). Add the 2.14.0-cpu preset case and make it the default for this branch so the experiment runs against a current torch; the older presets stay selectable for manual dispatch. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The plain cpu-torch-latest run already guards every single-rank test, so this branch's budget should go only to the multi-rank tests its LOCAL_SIZE gate admits. Gate a collection filter on DS_MULTIRANK_ONLY: it deselects non-DistributedTest items and world_size<=1 classes (honoring the per-test world_size mark with the launcher's precedence) and the workflow sets it. This also gives the non-v1 half enough of its time budget to expose the full multi-rank backlog instead of a slice. Groundwork for a dedicated cpu-torch-multi-latest workflow that will own exactly this selection. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

Two latent bugs surfaced by the multi-rank CPU run (gloo): - register_with_transformers validated core_attn_implementation against a bare AutoConfig's _attn_implementation when given a string model path. A config that has not gone through model loading still holds the unresolved 'eager' default, so every string-path caller asking for sdpa/flex/FA2 was rejected with a spurious mismatch. Only compare when the caller passed an actual model, whose config carries the implementation resolved at load time. - UlyssesSPDataLoaderAdapter exchanged the local sequence length as a 0-dim tensor but allocated 1-element receive buffers. Backends that move raw bytes (nccl) tolerate that; gloo validates gather shapes strictly and rejects it. Send a 1-element tensor to match. With both fixed the sdpa Ulysses SP tests pass end to end on cpu/gloo (verified 2/2 locally, incl. the numerical-parity assertions). Signed-off-by: Guokai Ma <guokai.ma@intel.com>

With the engine's string-path registration fixed, the sdpa classes run on cpu/gloo. Guard the CUDA-only remainder (the disable-in-eval test hardcodes cuda:<rank> tensors, flex_attention needs CUDA) and make the mock PEFT model carry the load-resolved attention implementation a real PEFT wrapper would have, instead of the bare-config 'eager' default that trips the consistency check. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The multi-rank-only selection leaves enough of the 6h job budget to let the non-v1 half finish its last 5% instead of losing its failure report to the 150m kill. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The test pinned its tensors and models to cuda:<rank> although nothing in it is CUDA-specific (it runs sdpa); use the accelerator device like the rest of the file, so it runs on any backend instead of being skipped. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

TestMoECheckpoint hardcodes fp16 configs, so deepspeed.initialize's sanity check fails on accelerators without fp16 support — the same hardware lottery deepspeedai#8398 handled elsewhere. Add the class-level skipif. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

TestZeroPartialOffloadConfigSweep hardcodes fp16 and test_checkpoint_pipe_engine enables it for zero_stage > 0; both trip initialize's sanity check on accelerators without fp16 support (deepspeedai#8398's hardware lottery). Skip those parametrizations so the gap is a skip. With this, every fp16-config test in the suite guards on is_fp16_supported(). Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The 180m guard still cut the final 6% of the multi-rank non-v1 half on a slow runner (zero test failures up to the kill; the job only failed on the timeout's exit code). 210m keeps the whole job, worst case, inside the 6h limit while letting the half finish and report green. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…dai#8684) ## Description Every fp16-config test that reaches `deepspeed.initialize` crashes its sanity check (`Type fp16 is not supported on your device.`) on accelerators whose `is_fp16_supported()` is false. On CPU that maps to the AVX512-FP16 capability of the host, and GitHub's `ubuntu-24.04` runners are hardware-heterogeneous, so these tests flip between failure and skip depending on which runner they land on. deepspeedai#8398 added the first skipifs; the multi-rank CPU run in deepspeedai#8381 flushed out six more files: | File | Shape of the gap | |---|---| | `checkpoint/test_universal_checkpoint.py` | fp16 parametrizations fail **inside the baseline DistributedFixture's distributed run** — pytest reports a setup **ERROR** for every dependent test (48 on the multi-rank run) instead of a skip | | `v1/zero/test_zero_coalesce_grad_reduction.py` | `TestCoalesceFP16` forces an fp16 config | | `runtime/test_no_sync_ctxt.py` | dtype=float16 parametrizations of three methods; stages 2/3 additionally never reach their expected no_sync AssertionError on such hosts | | `checkpoint/test_moe_checkpoint.py` | whole class hardcodes fp16 | | `runtime/zero/test_zero_offloadpp.py` | `TestZeroPartialOffloadConfigSweep` hardcodes fp16 | | `checkpoint/test_pipeline.py` | fp16 enabled for zero_stage > 0; only that parametrization skips, zero_stage=0 keeps running | With this PR, **every fp16-config test in the suite guards on `is_fp16_supported()`** — the capability gap is a skip, not a failure, on any accelerator. ## Validation (executed on real hardware) - 20-core x86_64 CPU without AVX512-FP16, gloo, 2-4 ranks (`LOCAL_SIZE=2/4`) - Before: 18 failures + 48 setup ERRORs across these files on the multi-rank CPU run - After: every affected parametrization skips; adjacent non-fp16 parametrizations keep passing (e.g. pipeline zero_stage=0 runs to completion) - Full-suite evidence in deepspeedai#8381: the multi-rank CPU run went 8 failures -> 0 with these guards Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

A rank that hangs inside a test blocks its pool worker indefinitely; the remaining xdist workers drain the queue, the phase sits idle until the outer timeout kills it, and the summary is lost. Arm a faulthandler timer per rank at exec_timeout so a wedged rank dumps its stack and exits, turning a silent wedge into a visible failure with a traceback. Signed-off-by: Ma, Guokai <guokai.ma@intel.com>

@delock

Follow up deepspeedai#8381 Fix 3 FAILED `unit/v1/ops/muon/test_per_head_muon_accelerator.py` in https://github.com/deepspeedai/DeepSpeed/actions/runs/35675316100/job/106580600974?pr=8381 @delock --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: pengdurice <pengduhit@gmail.com>

delock · 2026-09-29T01:53:35Z

Note the inference guard on host ISA needs to be validated on a machine with AVX512 support to make sure no degredation on AVX512 environment.

The shm comm op's bf16/fp16 reduce kernels are compiled with target("avx512bw") and use _mm512_* conversions (the fp32 kernels only use 256-bit AVX). Executing them on a host without the ISA faults, and because the reduce synchronizes through shm flags, the fault surfaces downstream as a permanent hang of the peers instead of an error - on the multi-rank CPU run it froze an xdist worker for two hours until the phase timeout. Export a vector_kernel_level() probe from the op (2 = avx512bw, 1 = avx2, 0 = neither; non-x86 hosts report 2 since their kernels are JIT-built for the host ISA) and gate the dispatch in inference_all_reduce: fall back to the process-group all_reduce when the tensor dtype needs a level the host lacks. Verified on an avx2-only host: fp32 keeps the shm op, bf16/fp16 now fall back and all three dtype parametrizations of TestDistInferenceAllReduce pass (they previously segfaulted/hung); a standalone 4-rank repro that crashed before returns correct sums. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The test drives DeepSpeedCPUAdam with a parameter on device_name(0) and expects step() to raise; on the cpu accelerator the premise is void (step returns normally), and the test additionally deadlocks inside the rank pool. Skip until the pool-side hang is understood. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…ocal-size

…es (deepspeedai#8702) ## Description Two latent bugs in Ulysses SP, both backend-independent (neither is CPU-specific — the CPU/gloo multi-rank run in deepspeedai#8381 is simply what surfaced them), plus test adaptations so the file can also run on non-CUDA accelerators. ### 1. `register_with_transformers` rejected every string model path `deepspeed/runtime/sequence_parallel/ulysses_sp.py` compared `core_attn_implementation` against the config's `_attn_implementation` even when the caller passed a **model path string**: ```python hf_model_config = AutoConfig.from_pretrained(model_name_or_path) # string branch model_attn_implementation = getattr(hf_model_config, "_attn_implementation", None) if model_attn_implementation is not None and model_attn_implementation != core_attn_implementation: raise ValueError(...) ``` In transformers, `_attn_implementation` is a **property resolved at model load time** — `from_pretrained` resolves which implementation the runtime can actually use and writes it back to the config. A bare `AutoConfig` never goes through that resolution, so the property returns its fallback value for "unresolved": ``` AutoConfig.from_pretrained(path)._attn_implementation -> eager (fallback, not a choice) AutoModelForCausalLM.from_pretrained(path).config._attn... -> sdpa (resolved) ``` So every string-path caller asking for `sdpa`/`flex_attention`/FA2 was rejected with a spurious mismatch against `'eager'`. The check is only meaningful for a config that was resolved, i.e. when the caller handed us a constructed model (or a PEFT wrapper exposing `.config`); scope it to that case. The comparison expression itself is unchanged, and `TestUlyssesSPHFAttnImplMismatch` (which passes a model object) still raises as intended. ### 2. `UlyssesSPDataLoaderAdapter` used a 0-dim tensor in `all_gather` ```python seqlen = torch.tensor(n, ...) # 0-dim seqlens = [torch.zeros(1, ...) for _ in range(sp_world_size)] # shape (1,) dist.all_gather(seqlens, seqlen, group=self.sp_group) ``` `c10d.all_gather` requires each receive tensor to have the same size as the send tensor. NCCL moves flattened bytes and tolerates the mismatch; gloo validates shapes and rejects it (`ProcessGroupGloo::allgather: invalid tensor size at index 0 (expected (), got (1))`). Send a 1-element tensor to match. ### 3. Tests - `TestUlyssesSPHFFlexAttention` needs the CUDA flex kernel -> skip on non-CUDA accelerators (same guard idiom as `tests/unit/util.py`, the nvme/triton CUDA-only tests). - `TestUlyssesSPHFPEFT`: the mock attached a **bare** `AutoConfig`, whose `_attn_implementation` is the unresolved `'eager'` — so this test failed on every backend, not just CPU. Give the mock the load-resolved value a real PEFT wrapper carries (the sibling `TestUlyssesSPHFAttnImplMismatch` in the same file already sets it the same way). - `TestUlyssesSPHFDisableInEval` pinned its tensors/models to `cuda:<rank>` although it runs sdpa and nothing in it is CUDA-specific. Use `get_accelerator().current_device_name()` like the rest of the file (on CUDA that is exactly `cuda:<rank>`, since the test harness sets the device per rank in `tests/unit/common.py`), so the test runs on any backend instead of erroring. ## Validation Hardware: 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks (`LOCAL_SIZE=2`), transformers 4.51.3, torch 2.13. - `tests/unit/ulysses_alst/test_ulysses_sp_hf.py`: **5 passed, 2 skipped** (flex skipped by the new guard). The sdpa classes pass end to end including their numerical-parity assertions; before the fix they died at registration. - `TestUlyssesSPHFDisableInEval` now actually **executes** on CPU rather than being skipped. - `TestUlyssesSPHFAttnImplMismatch` still raises the expected `ValueError` (the mismatch check is preserved where a resolved config exists). - Full multi-rank CPU CI in deepspeedai#8381: this file went from 5 failures to 0, with the overall job green. (`TestUlyssesSPHFHubKernel` is not exercised by the version pin above — it needs the `lazy_import_flash_attention` API from a newer transformers; it passes in CI, which installs transformers from git main.) Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648, deepspeedai#8684. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: pengdurice <pengduhit@gmail.com>

The dynamic offload-state tests already relax their memory-delta assertions for non-allocator backends (deepspeedai#8409), but validate_device's contiguous_grad_buffer check still assumed the state was moved out of the accelerator: when it is reported on cpu, the buffer is required to be gone. On the cpu accelerator the offload target is the accelerator itself, so the buffer is never moved and that assertion fails. Gate the check on the offload actually crossing memory tiers. On CUDA the new flag is always true, so the assertion behaves exactly as before; on cpu the state falls through to the device comparison, which holds. Measured on cpu/gloo with 2 ranks (master's version, no skip): 112 of the 128 parametrizations already pass, the 16 failures are all this one assertion. With the guard the whole file passes, so the cpu run keeps the coverage a module-level skip would have dropped. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…deepspeedai#8738) ## Description `validate_device` in `tests/unit/v1/zero/test_offload_states.py` requires the contiguous grad buffer to be **gone** once it is reported on cpu, because a cpu device means `offload_states()` moved it to the host tier: ```python if state == OffloadStateTypeEnum.contiguous_grad_buffer and state_device[state] == torch.device("cpu"): assert len(get_state_devices(model, state)) == 0, f"State {state} must be removed after offload_states()" ``` On the cpu accelerator the offload target **is** the accelerator, so `state_device[state] == torch.device("cpu")` holds for everything, yet the buffer was never moved anywhere — the assertion fails for every parametrization that includes it. This is the device-state counterpart of deepspeedai#8409, which gated the **memory-delta** assertions on `allocator_backed_memory_stats` for the same reason. Add a sibling `offload_leaves_accelerator` flag (next to that one) and require it for the removal check. ## Behavior | Accelerator | `offload_leaves_accelerator` | Effect | |---|---|---| | CUDA / XPU | `True` | assertion unchanged, bit for bit | | cpu | `False` | state falls through to the device comparison instead, which holds | ## Validation (executed on real hardware) 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks (`LOCAL_SIZE=2`), `-n 4`: - **Without** this change on master: `16 failed, 112 passed` — and all 16 failures are this single assertion (`State OffloadStateTypeEnum.contiguous_grad_buffer must be removed after offload_states()`). - **With** this change: `128 passed, 0 failed`. So the cpu accelerator keeps the coverage of the other 112 parametrizations instead of losing them to a blanket skip. The enclosing cpu CI run is deepspeedai#8381, which exercises this file at multi-rank and reports the same result (`128 PASSED`). Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…ocal-size

## What Adds the `world_size` axis to CI, which the current suite does not cover: | File | Purpose | |---|---| | `.github/workflows/cpu-torch-multi-latest.yml` | Nightly + manual dispatch: runs the multi-rank CPU tests on a 4-vCPU runner | | `.github/workflows/cpu-multi-triage.yml` | Routes its outcome through the existing `ci/nightly_classify.sh` / `ci/nightly_report.sh` protocol | | `tests/conftest.py` | The `DS_MULTIRANK_ONLY` collection hook the workflow selects with (required) | | `.github/workflows/TEST_SELECTION.md` | Registration of the new nightly | ## Why `cpu-torch-latest` advertises a single device, so the per-device gate in `tests/unit/common.py` skips **every `world_size > 1` test** on that runner. The multi-rank path is therefore only exercised on GPU machines, and issues that show up there only in a multi-rank configuration go unnoticed. A recent example: PR deepspeedai#8638 changed the overflow semantics so that non-finite group norms force an optimizer step to be skipped, bypassing `bf16.check_grad_overflow: False`. `TestZeROBFloat16::test_ignore_grad_overflow` is `world_size = 2`, so it is collected and *skipped* by the CPU CI and by any run with one device; it fails as soon as two ranks execute. Reported as issue deepspeedai#8805. ## Why a nightly rather than a pre-merge check The selection is cheap to state and expensive to run (measured in deepspeedai#8381, 4-vCPU runner): | Run | Wall time | |---|---| | This suite: non-v1 multi-rank half | 68-89 min | | This suite: `sequential`-marked phase | 13-16 min | | Longest current pre-merge job (`cpu-torch-latest`) | 39-43 min | | Longest merge-queue job (`modal-torch-latest`) | 64-71 min | Putting it in the pre-merge path would roughly triple the critical path of every PR, and a single wedged rank would block the merge queue (the suite's known hang — the shm op segfaulting on hosts without AVX512 — is fixed by deepspeedai#8701, but the suite is the most expensive one to re-run). ## What it runs - `LOCAL_SIZE=4` so the gate admits `world_size <= 4`; `DS_MULTIRANK_ONLY=1` deselects the single-rank tests that `cpu-torch-latest` already guards. - Non-v1 half, `timeout 120m`, plus a `sequential`-marked phase, `timeout 25m` (the parallel phase deselects those by default, so they would otherwise never run here). - Same preset/install/HF-predownload steps as `cpu-torch-latest`, so both runs agree on the environment. - On failure the last step prints `DS_CI_FAILURE_CLASS=timeout|test` for triage; a run that dies before that step is reported as unclassified rather than guessed. Triage is a separate workflow with its own issue titles and its own `nightly-cpu-multi-last-green` tag, so `nightly-last-green` keeps meaning "the last revision the full GPU suite verified" and the modal issue dedupe is untouched. ## Validation - Workflow YAML parses; `pre-commit` clean. - Timings above come from the real runs of this configuration in deepspeedai#8381. - Not yet run as `cpu-torch-multi-latest`: GitHub only exposes `workflow_dispatch` for workflows present on the default branch, so the first manual dispatch has to happen after this merges. Happy to run it then and follow up with the numbers. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Signed-off-by: Ma, Guokai <guokai.ma@intel.com>

…ocal-size # Conflicts: # csrc/cpu/comm/shm_interface.cpp

delock requested a review from loadams as a code owner September 1, 2026 01:21

chatgpt-codex-connector Bot reviewed Sep 1, 2026

View reviewed changes

delock mentioned this pull request Sep 1, 2026

Fix silent gradient-bucket drop on CPU in ZeRO-1/2 non-contiguous reduction #8382

Merged

delock requested review from tjruwase and tohtana as code owners September 1, 2026 14:28

delock changed the title ~~Run multi-rank CPU unit tests in CI via LOCAL_SIZE~~ [DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE Sep 3, 2026

This was referenced Sep 3, 2026

Use device names, not rank ids, for device placement in test helpers #8397

Merged

Skip fp16-config tests on accelerators without fp16 support #8398

Merged

Do not pin DDP device_ids for CPU reference models #8399

Merged

delock added 3 commits September 3, 2026 21:33

delock mentioned this pull request Sep 3, 2026

Probe the device module for train_cifar's fork_rng device entries #8407

Merged

delock force-pushed the ci/cpu-multi-rank-local-size branch from f46af32 to f2793e7 Compare September 3, 2026 14:01

delock mentioned this pull request Sep 4, 2026

Gate the offload-state memory deltas on allocator-backed stats #8409

Merged

Merge branch 'master' into ci/cpu-multi-rank-local-size

4d4019d

delock mentioned this pull request Sep 16, 2026

Wait for all async all-gathers in coalesced ZeRO-3 parameter gather #8539

Merged

delock added 3 commits September 16, 2026 18:50

delock added 9 commits September 26, 2026 10:32

Raise the non-v1 half timeout to 180m

c81a577

The multi-rank-only selection leaves enough of the 6h job budget to let the non-v1 half finish its last 5% instead of losing its failure report to the 150m kill. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

delock mentioned this pull request Sep 27, 2026

Skip fp16-config tests on accelerators without fp16 support #8684

Merged

delock added 3 commits September 29, 2026 17:43

Merge remote-tracking branch 'origin/master' into ci/cpu-multi-rank-l…

e3d6f5c

…ocal-size

delock force-pushed the ci/cpu-multi-rank-local-size branch from 9fb4ae9 to e3d6f5c Compare September 29, 2026 09:44

delock mentioned this pull request Sep 29, 2026

Fix Ulysses SP registration for string model paths and gloo collectives #8702

Merged

delock added 2 commits October 3, 2026 22:40

Merge remote-tracking branch 'origin/master' into pr-8381-new

ead6cce

This was referenced Oct 4, 2026

Assert offload-state removal only when offload leaves the accelerator #8738

Merged

fix: attention_unfused alpha missing norm_factor squaring #8666

Closed

Merge remote-tracking branch 'origin/master' into ci/cpu-multi-rank-l…

23c4e93

…ocal-size

delock mentioned this pull request Oct 12, 2026

Add a nightly multi-rank CPU suite #8806

Merged

Merge remote-tracking branch 'origin/master' into ci/cpu-multi-rank-l…

271b774

…ocal-size # Conflicts: # csrc/cpu/comm/shm_interface.cpp

Navigation Menu

Repository navigation

[DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE - #8381

[DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE#8381
delock wants to merge 45 commits into
deepspeedai:masterfrom
delock:ci/cpu-multi-rank-local-size

delock commented Sep 1, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 1, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 1, 2026

Uh oh!

delock commented Sep 1, 2026

Uh oh!

delock commented Sep 2, 2026

Uh oh!

delock commented Sep 3, 2026

Uh oh!

delock commented Sep 16, 2026

Uh oh!

delock commented Sep 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

delock commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Evidence this is safe

Expected impact

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

delock commented Sep 1, 2026

Uh oh!

delock commented Sep 2, 2026

Experiment results: enabling multi-rank CPU tests via LOCAL_SIZE=4

What now runs

Failure inventory & root causes (triaged)

Test-side fixes already in this branch (validated by CI)

CI infrastructure findings (need maintainer decisions)

Uh oh!

delock commented Sep 3, 2026

Uh oh!

delock commented Sep 16, 2026

Status update

Fixes extracted so far (spin-off PRs)

Current failures on this branch (84 total: 75 in the unit/v1 half + 9 sequential)

Current focus: phase-1 timeout / worker wedge

Uh oh!

delock commented Sep 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

delock commented Sep 1, 2026 •

edited

Loading

Experiment results: enabling multi-rank CPU tests via `LOCAL_SIZE=4`

Current failures on this branch (84 total: 75 in the `unit/v1` half + 9 sequential)