Skip to content

Derive the autocast test backend and scope the bf16 gate to NCCL - #8559

Merged
delock merged 1 commit into
deepspeedai:masterfrom
delock:pr-f-autocast-cpu
Sep 17, 2026
Merged

Derive the autocast test backend and scope the bf16 gate to NCCL#8559
delock merged 1 commit into
deepspeedai:masterfrom
delock:pr-f-autocast-cpu

Conversation

@delock

@delock delock commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

bf16_required_version_check() (tests/unit/util.py) requires torch >= 1.10, CUDA >= 11.0 and NCCL >= 2.10.3. On the cpu accelerator, bf16 collectives run over gloo/ccl and none of those transport dependencies exist, so the check always returns False and every bf16 test is skipped — about 40 call sites across 15 files. test_zero_autocast.py is worse off: it raises instead of skipping, so each of its cases counts as a failure (24 on the multi-rank CPU run in #8381).

This PR scopes the version floors inside the check itself:

if (cpu_accelerator and accelerator_pass) or (torch_version_available and cuda_version_available
                                              and nccl_version_available and accelerator_pass):
    return True
  • cpu: only the accelerator's own bf16 support (is_bf16_supported()) is required — the torch/CUDA/NCCL floors are transport dependencies that gloo/ccl does not have
  • every other accelerator (cuda, npu, hpu, xpu, mlu, ...): evaluates the exact original floors; cpu_accelerator is False so the expression is bit-identical to before, and the npu/hpu/xpu exemption branches are untouched
  • test_zero_autocast.py: the bf16 gate goes back to the bare call every other caller uses, and skips instead of raising

The hardcoded init_distributed(dist_backend='nccl') in the same test is deliberately left alone: it is a no-op (the harness already initialized the process group, comm.py:838-839), and deriving the backend per accelerator would change behavior on non-cuda accelerators (npu/hpu resolve to hccl etc.). The baseline DDP(device_ids=[i]) pinning is left for a follow-up (#8399 fixed the same pattern elsewhere).

Validation (executed on real hardware)

  • 20-core x86_64 CPU, torch 2.13.0+cpu, gloo backend
  • Direct call: bf16_required_version_check() on cpu returns False before, True after. Note: CPU_Accelerator.is_bf16_supported() is currently a stub that always returns True, so the cpu path does not gate on the hardware's bf16 instructions — giving it a real capability probe is left as a follow-up
  • Non-cpu equivalence: with cpu_accelerator == False the new expression reduces exactly to the original A and C and N and P
  • pre-commit (yapf / flake8 / check-torchdist / codespell) passes on both changed files

Sibling PRs from the same series: #8397, #8398, #8399, #8407, #8409. Exposed by the LOCAL_SIZE=4 multi-rank CPU run in #8381.

@delock
Scope the bf16 version floors to NCCL transports in the test check
baf0caa 
bf16_required_version_check() requires torch >= 1.10, CUDA >= 11 and
NCCL >= 2.10.3, so on the cpu accelerator, where bf16 collectives run
over gloo/ccl and none of those dependencies exist, the check always
fails and every bf16 test is skipped, about 40 call sites across 15
files. test_zero_autocast is worse off: it raises instead of skipping,
so each case counts as a failure.

Exempt the cpu accelerator inside the check itself so all call sites
benefit: on cpu only the accelerator's own bf16 support is required.
Every other accelerator evaluates the exact original floors, and
test_zero_autocast now skips like every other caller instead of
raising.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
@sfc-gh-truwase
sfc-gh-truwase added this pull request to the merge queue Sep 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 17, 2026
@delock
delock added this pull request to the merge queue Sep 17, 2026
Merged via the queue into deepspeedai:master with commit b7b197e Sep 17, 2026
13 checks passed
@delock
delock deleted the pr-f-autocast-cpu branch September 17, 2026 16:21
@delock delock mentioned this pull request Sep 27, 2026
Merged
yermakoffivan pushed a commit to yermakoffivan/deepspeed that referenced this pull request Sep 28, 2026
@delock
Skip fp16-config tests on accelerators without fp16 support (deepspee…
df00e0b 
…dai#8684)

## Description

Every fp16-config test that reaches `deepspeed.initialize` crashes its
sanity check (`Type fp16 is not supported on your device.`) on
accelerators whose `is_fp16_supported()` is false. On CPU that maps to
the AVX512-FP16 capability of the host, and GitHub's `ubuntu-24.04`
runners are hardware-heterogeneous, so these tests flip between failure
and skip depending on which runner they land on. deepspeedai#8398 added the first
skipifs; the multi-rank CPU run in deepspeedai#8381 flushed out six more files:

| File | Shape of the gap |
|---|---|
| `checkpoint/test_universal_checkpoint.py` | fp16 parametrizations fail
**inside the baseline DistributedFixture's distributed run** — pytest
reports a setup **ERROR** for every dependent test (48 on the multi-rank
run) instead of a skip |
| `v1/zero/test_zero_coalesce_grad_reduction.py` | `TestCoalesceFP16`
forces an fp16 config |
| `runtime/test_no_sync_ctxt.py` | dtype=float16 parametrizations of
three methods; stages 2/3 additionally never reach their expected
no_sync AssertionError on such hosts |
| `checkpoint/test_moe_checkpoint.py` | whole class hardcodes fp16 |
| `runtime/zero/test_zero_offloadpp.py` |
`TestZeroPartialOffloadConfigSweep` hardcodes fp16 |
| `checkpoint/test_pipeline.py` | fp16 enabled for zero_stage > 0; only
that parametrization skips, zero_stage=0 keeps running |

With this PR, **every fp16-config test in the suite guards on
`is_fp16_supported()`** — the capability gap is a skip, not a failure,
on any accelerator.

## Validation (executed on real hardware)

- 20-core x86_64 CPU without AVX512-FP16, gloo, 2-4 ranks
(`LOCAL_SIZE=2/4`)
- Before: 18 failures + 48 setup ERRORs across these files on the
multi-rank CPU run
- After: every affected parametrization skips; adjacent non-fp16
parametrizations keep passing (e.g. pipeline zero_stage=0 runs to
completion)
- Full-suite evidence in deepspeedai#8381: the multi-rank CPU run went 8 failures
-> 0 with these guards

Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409,
deepspeedai#8559, deepspeedai#8648.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
@delock delock mentioned this pull request Sep 29, 2026
Merged
meefs pushed a commit to meefs/DeepSpeed that referenced this pull request Oct 2, 2026
@delock @pengdurice
Fix Ulysses SP registration for string model paths and gloo collectiv…
04e0be2 
…es (deepspeedai#8702)

## Description

Two latent bugs in Ulysses SP, both backend-independent (neither is
CPU-specific — the CPU/gloo multi-rank run in deepspeedai#8381 is simply what
surfaced them), plus test adaptations so the file can also run on
non-CUDA accelerators.

### 1. `register_with_transformers` rejected every string model path

`deepspeed/runtime/sequence_parallel/ulysses_sp.py` compared
`core_attn_implementation` against the config's `_attn_implementation`
even when the caller passed a **model path string**:

```python
hf_model_config = AutoConfig.from_pretrained(model_name_or_path)   # string branch
model_attn_implementation = getattr(hf_model_config, "_attn_implementation", None)
if model_attn_implementation is not None and model_attn_implementation != core_attn_implementation:
    raise ValueError(...)
```

In transformers, `_attn_implementation` is a **property resolved at
model load time** — `from_pretrained` resolves which implementation the
runtime can actually use and writes it back to the config. A bare
`AutoConfig` never goes through that resolution, so the property returns
its fallback value for "unresolved":

```
AutoConfig.from_pretrained(path)._attn_implementation      -> eager   (fallback, not a choice)
AutoModelForCausalLM.from_pretrained(path).config._attn... -> sdpa    (resolved)
```

So every string-path caller asking for `sdpa`/`flex_attention`/FA2 was
rejected with a spurious mismatch against `'eager'`. The check is only
meaningful for a config that was resolved, i.e. when the caller handed
us a constructed model (or a PEFT wrapper exposing `.config`); scope it
to that case. The comparison expression itself is unchanged, and
`TestUlyssesSPHFAttnImplMismatch` (which passes a model object) still
raises as intended.

### 2. `UlyssesSPDataLoaderAdapter` used a 0-dim tensor in `all_gather`

```python
seqlen = torch.tensor(n, ...)                    # 0-dim
seqlens = [torch.zeros(1, ...) for _ in range(sp_world_size)]   # shape (1,)
dist.all_gather(seqlens, seqlen, group=self.sp_group)
```

`c10d.all_gather` requires each receive tensor to have the same size as
the send tensor. NCCL moves flattened bytes and tolerates the mismatch;
gloo validates shapes and rejects it (`ProcessGroupGloo::allgather:
invalid tensor size at index 0 (expected (), got (1))`). Send a
1-element tensor to match.

### 3. Tests

- `TestUlyssesSPHFFlexAttention` needs the CUDA flex kernel -> skip on
non-CUDA accelerators (same guard idiom as `tests/unit/util.py`, the
nvme/triton CUDA-only tests).
- `TestUlyssesSPHFPEFT`: the mock attached a **bare** `AutoConfig`,
whose `_attn_implementation` is the unresolved `'eager'` — so this test
failed on every backend, not just CPU. Give the mock the load-resolved
value a real PEFT wrapper carries (the sibling
`TestUlyssesSPHFAttnImplMismatch` in the same file already sets it the
same way).
- `TestUlyssesSPHFDisableInEval` pinned its tensors/models to
`cuda:<rank>` although it runs sdpa and nothing in it is CUDA-specific.
Use `get_accelerator().current_device_name()` like the rest of the file
(on CUDA that is exactly `cuda:<rank>`, since the test harness sets the
device per rank in `tests/unit/common.py`), so the test runs on any
backend instead of erroring.

## Validation

Hardware: 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks
(`LOCAL_SIZE=2`), transformers 4.51.3, torch 2.13.

- `tests/unit/ulysses_alst/test_ulysses_sp_hf.py`: **5 passed, 2
skipped** (flex skipped by the new guard). The sdpa classes pass end to
end including their numerical-parity assertions; before the fix they
died at registration.
- `TestUlyssesSPHFDisableInEval` now actually **executes** on CPU rather
than being skipped.
- `TestUlyssesSPHFAttnImplMismatch` still raises the expected
`ValueError` (the mismatch check is preserved where a resolved config
exists).
- Full multi-rank CPU CI in deepspeedai#8381: this file went from 5 failures to 0,
with the overall job green.

(`TestUlyssesSPHFHubKernel` is not exercised by the version pin above —
it needs the `lazy_import_flash_attention` API from a newer
transformers; it passes in CI, which installs transformers from git
main.)

Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409,
deepspeedai#8559, deepspeedai#8648, deepspeedai#8684.

---------

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Co-authored-by: pengdurice <pengduhit@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-truwase sfc-gh-truwase sfc-gh-truwase approved these changes
@tjruwase tjruwase Awaiting requested review from tjruwase tjruwase is a code owner
@loadams loadams Awaiting requested review from loadams loadams is a code owner
@tohtana tohtana Awaiting requested review from tohtana tohtana is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@delock @sfc-gh-truwase