Skip fp16-config tests on accelerators without fp16 support by delock · Pull Request #8684 · deepspeedai/DeepSpeed

delock · 2026-09-27T15:21:33Z

Description

Every fp16-config test that reaches deepspeed.initialize crashes its sanity check (Type fp16 is not supported on your device.) on accelerators whose is_fp16_supported() is false. On CPU that maps to the AVX512-FP16 capability of the host, and GitHub's ubuntu-24.04 runners are hardware-heterogeneous, so these tests flip between failure and skip depending on which runner they land on. #8398 added the first skipifs; the multi-rank CPU run in #8381 flushed out six more files:

File	Shape of the gap
`checkpoint/test_universal_checkpoint.py`	fp16 parametrizations fail inside the baseline DistributedFixture's distributed run — pytest reports a setup ERROR for every dependent test (48 on the multi-rank run) instead of a skip
`v1/zero/test_zero_coalesce_grad_reduction.py`	`TestCoalesceFP16` forces an fp16 config
`runtime/test_no_sync_ctxt.py`	dtype=float16 parametrizations of three methods; stages 2/3 additionally never reach their expected no_sync AssertionError on such hosts
`checkpoint/test_moe_checkpoint.py`	whole class hardcodes fp16
`runtime/zero/test_zero_offloadpp.py`	`TestZeroPartialOffloadConfigSweep` hardcodes fp16
`checkpoint/test_pipeline.py`	fp16 enabled for zero_stage > 0; only that parametrization skips, zero_stage=0 keeps running

With this PR, every fp16-config test in the suite guards on is_fp16_supported() — the capability gap is a skip, not a failure, on any accelerator.

Validation (executed on real hardware)

20-core x86_64 CPU without AVX512-FP16, gloo, 2-4 ranks (LOCAL_SIZE=2/4)
Before: 18 failures + 48 setup ERRORs across these files on the multi-rank CPU run
After: every affected parametrization skips; adjacent non-fp16 parametrizations keep passing (e.g. pipeline zero_stage=0 runs to completion)
Full-suite evidence in [DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE #8381: the multi-rank CPU run went 8 failures -> 0 with these guards

Sibling PRs from the same series: #8397, #8398, #8399, #8407, #8409, #8559, #8648.

Every fp16-config test that reaches deepspeed.initialize crashes its sanity check ("Type fp16 is not supported on your device.") on accelerators whose is_fp16_supported() is false — on CPU that maps to the AVX512-FP16 capability of the host, and GitHub's ubuntu-24.04 runners are hardware-heterogeneous. deepspeedai#8398 added skipifs for the first three files; the multi-rank CPU run in deepspeedai#8381 flushed out six more: - checkpoint/test_universal_checkpoint.py: the fp16 parametrizations fail inside the baseline DistributedFixture's distributed run, which reports a setup ERROR for every dependent test instead of a skip. - v1/zero/test_zero_coalesce_grad_reduction.py: TestCoalesceFP16 forces an fp16 config. - runtime/test_no_sync_ctxt.py: the dtype=float16 parametrizations of the three dtype-parametrized methods; stages 2/3 additionally never reach their expected no_sync AssertionError on such hosts. - checkpoint/test_moe_checkpoint.py: the whole class hardcodes fp16. - runtime/zero/test_zero_offloadpp.py: TestZeroPartialOffloadConfigSweep hardcodes fp16. - checkpoint/test_pipeline.py: fp16 is enabled for zero_stage > 0; skip only that parametrization so zero_stage=0 keeps running. With this, every fp16-config test in the suite guards on is_fp16_supported(): the gap is a skip, not a failure, on any accelerator. Verified on an fp16-incapable CPU (the affected parametrizations skip; adjacent non-fp16 ones keep passing) and by the multi-rank CPU run in deepspeedai#8381. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

hwchen2017 · 2026-09-27T18:04:49Z

    def test_zero_stage(self, zero_stage, dtype):
+        # The fp16 parametrization crashes initialize's sanity check on accelerators
+        # without fp16 support (#8398's hardware lottery); skip instead of failing.
+        if dtype == torch.float16 and not get_accelerator().is_fp16_supported():


Should we have a check for bf16 here as well?

Thanks for the comments. Currently CPU with accelerated instruction set (AVX or AMX) has BF16 support. We can add bf16 skip check when we need to run validation on accelerators without bf16 support.

…es (deepspeedai#8702) ## Description Two latent bugs in Ulysses SP, both backend-independent (neither is CPU-specific — the CPU/gloo multi-rank run in deepspeedai#8381 is simply what surfaced them), plus test adaptations so the file can also run on non-CUDA accelerators. ### 1. `register_with_transformers` rejected every string model path `deepspeed/runtime/sequence_parallel/ulysses_sp.py` compared `core_attn_implementation` against the config's `_attn_implementation` even when the caller passed a **model path string**: ```python hf_model_config = AutoConfig.from_pretrained(model_name_or_path) # string branch model_attn_implementation = getattr(hf_model_config, "_attn_implementation", None) if model_attn_implementation is not None and model_attn_implementation != core_attn_implementation: raise ValueError(...) ``` In transformers, `_attn_implementation` is a **property resolved at model load time** — `from_pretrained` resolves which implementation the runtime can actually use and writes it back to the config. A bare `AutoConfig` never goes through that resolution, so the property returns its fallback value for "unresolved": ``` AutoConfig.from_pretrained(path)._attn_implementation -> eager (fallback, not a choice) AutoModelForCausalLM.from_pretrained(path).config._attn... -> sdpa (resolved) ``` So every string-path caller asking for `sdpa`/`flex_attention`/FA2 was rejected with a spurious mismatch against `'eager'`. The check is only meaningful for a config that was resolved, i.e. when the caller handed us a constructed model (or a PEFT wrapper exposing `.config`); scope it to that case. The comparison expression itself is unchanged, and `TestUlyssesSPHFAttnImplMismatch` (which passes a model object) still raises as intended. ### 2. `UlyssesSPDataLoaderAdapter` used a 0-dim tensor in `all_gather` ```python seqlen = torch.tensor(n, ...) # 0-dim seqlens = [torch.zeros(1, ...) for _ in range(sp_world_size)] # shape (1,) dist.all_gather(seqlens, seqlen, group=self.sp_group) ``` `c10d.all_gather` requires each receive tensor to have the same size as the send tensor. NCCL moves flattened bytes and tolerates the mismatch; gloo validates shapes and rejects it (`ProcessGroupGloo::allgather: invalid tensor size at index 0 (expected (), got (1))`). Send a 1-element tensor to match. ### 3. Tests - `TestUlyssesSPHFFlexAttention` needs the CUDA flex kernel -> skip on non-CUDA accelerators (same guard idiom as `tests/unit/util.py`, the nvme/triton CUDA-only tests). - `TestUlyssesSPHFPEFT`: the mock attached a **bare** `AutoConfig`, whose `_attn_implementation` is the unresolved `'eager'` — so this test failed on every backend, not just CPU. Give the mock the load-resolved value a real PEFT wrapper carries (the sibling `TestUlyssesSPHFAttnImplMismatch` in the same file already sets it the same way). - `TestUlyssesSPHFDisableInEval` pinned its tensors/models to `cuda:<rank>` although it runs sdpa and nothing in it is CUDA-specific. Use `get_accelerator().current_device_name()` like the rest of the file (on CUDA that is exactly `cuda:<rank>`, since the test harness sets the device per rank in `tests/unit/common.py`), so the test runs on any backend instead of erroring. ## Validation Hardware: 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks (`LOCAL_SIZE=2`), transformers 4.51.3, torch 2.13. - `tests/unit/ulysses_alst/test_ulysses_sp_hf.py`: **5 passed, 2 skipped** (flex skipped by the new guard). The sdpa classes pass end to end including their numerical-parity assertions; before the fix they died at registration. - `TestUlyssesSPHFDisableInEval` now actually **executes** on CPU rather than being skipped. - `TestUlyssesSPHFAttnImplMismatch` still raises the expected `ValueError` (the mismatch check is preserved where a resolved config exists). - Full multi-rank CPU CI in deepspeedai#8381: this file went from 5 failures to 0, with the overall job green. (`TestUlyssesSPHFHubKernel` is not exercised by the version pin above — it needs the `lazy_import_flash_attention` API from a newer transformers; it passes in CI, which installs transformers from git main.) Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648, deepspeedai#8684. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: pengdurice <pengduhit@gmail.com>

delock requested review from loadams, tjruwase and tohtana as code owners September 27, 2026 15:21

hwchen2017 reviewed Sep 27, 2026

View reviewed changes

hwchen2017 approved these changes Sep 28, 2026

View reviewed changes

delock added this pull request to the merge queue Sep 28, 2026

Merged via the queue into deepspeedai:master with commit df00e0b Sep 28, 2026
13 of 15 checks passed

delock deleted the pr-o-fp16-lottery-skips branch September 28, 2026 07:03

delock mentioned this pull request Sep 29, 2026

Fix Ulysses SP registration for string model paths and gloo collectives #8702

Merged

Navigation Menu

Repository navigation

Skip fp16-config tests on accelerators without fp16 support - #8684

Skip fp16-config tests on accelerators without fp16 support#8684
delock merged 1 commit into
deepspeedai:masterfrom
delock:pr-o-fp16-lottery-skips

delock commented Sep 27, 2026

Uh oh!

hwchen2017 Sep 27, 2026

Uh oh!

delock Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

delock commented Sep 27, 2026

Description

Validation (executed on real hardware)

Uh oh!

hwchen2017 Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

delock Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants