Skip fp16-config tests on accelerators without fp16 support by delock · Pull Request #8398 · deepspeedai/DeepSpeed

delock · 2026-09-03T00:27:25Z

Problem

TestMultipleModels::test_zero_optimizer, TestSimpleMoE, TestMoE, TestPRMoE, and TestMOETensorParallel hardcode "fp16": {"enabled": True} in their DeepSpeed configs. The engine's sanity check then raises:

ValueError: Type fp16 is not supported on your device.

on any accelerator whose is_fp16_supported() is false. On CPU that maps to the AVX512-FP16 capability of the host, and GitHub's ubuntu-24.04 runners are hardware-heterogeneous: the same test passes on one runner and fails on the next (observed directly in #8381 — 146 failures appeared on one runner generation and none on another, with identical code).

Change

Skip these tests via a capability query:

@pytest.mark.skipif(not get_accelerator().is_fp16_supported(), reason="fp16 is not supported on this accelerator")

capability only, no accelerator-name matching;
mirrors the existing bf16 skip precedent in tests/unit/v1/zero/test_zero_user_backward.py;
deliberately a skip rather than silently running bf16 — these tests exist to cover the fp16 paths.

Validation

Validated as part of the multi-rank CPU CI experiment in #8381: the 146 hardware-lottery failures became deterministic skips, zero regressions on previously-passing tests.

test_multiple_models/test_zero_optimizer and the MoE tests hardcode "fp16": {"enabled": True} in their DeepSpeed configs. The engine's sanity check then raises "Type fp16 is not supported on your device" on any accelerator whose is_fp16_supported() is false - on CPU that depends on the host's AVX512-FP16 capability, and GitHub's ubuntu-24.04 runners are heterogeneous enough that the same test passes on one runner and fails on the next. Skip these tests via a capability query instead of depending on runner hardware, mirroring the existing bf16 skip precedent in test_zero_user_backward. Validated as part of the multi-rank CPU experiment in deepspeedai#8381: 146 failures of this class became deterministic skips, zero regressions. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31e794e4ed

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-03T00:29:06Z

 import deepspeed
 import deepspeed.comm as dist
 import torch
+from deepspeed import get_accelerator


Add the required Signed-off-by trailer

This is a non-merge commit, but its message has no Signed-off-by trailer. Add the author name and email from the Git configuration using git commit --signoff so the commit satisfies the repository's mandatory DCO/CI requirement.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

…epspeedai#8407) ## Description `train_cifar` unconditionally passed `devices=[get_accelerator().current_device_name()]` to `fork_rng`. On CPU that becomes `devices=['cpu']`, and `torch.cpu` has no `get_rng_state`, so the context manager raises `AttributeError` before the test body starts — the CPU RNG lives in the global generator that `fork_rng` already forks. Probe the device module for a per-device RNG instead of matching accelerator names: - backends whose device module has `get_rng_state` (e.g. cuda) behave exactly as before; - backends without one (cpu) pass `devices=[]`, which changes nothing beyond the global-RNG save/restore `fork_rng` always performs. The crash is only reachable from the multi-rank tests that call `train_cifar` (`test_onebit.py`, `test_pipe.py`), which the CPU runner currently skips at the device gate; it was exposed by the `LOCAL_SIZE=4` experiment in deepspeedai#8381. Sibling fixes from the same series landed as deepspeedai#8397, deepspeedai#8398 and deepspeedai#8399. ## Test plan - pre-commit (yapf / flake8 / check-torchdist / codespell) passes on the changed file; - exercised under the `LOCAL_SIZE=4` multi-rank CPU run tracked in deepspeedai#8381: the `test_onebit` / `test_pipe` callers got past `fork_rng` and into their test bodies with this exact change. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…peedai#8409) ## Description The dynamic offload-state tests assert strict allocated-memory deltas around `offload_states()` / `reload_states()`: - `alloc_after_offload < alloc_before_offload` - `alloc_after_reload > alloc_after_offload` That contract assumes `memory_allocated()` is allocator bookkeeping, which holds on cuda (`torch.cuda.memory_allocated()`). On cpu, `CPU_Accelerator.memory_allocated()` reports process RSS (psutil), and RSS does not shrink when tensors are freed — so all 92 parameterized cases fail even when the offload itself is correct (the device-placement and data-integrity checks in the same tests pass). Gate only the memory-delta asserts on whether the accelerator's torch device module exposes `memory_allocated` (cuda does; `torch.cpu` does not), mirroring the capability probe used for `fork_rng` in `train_cifar` (deepspeedai#8407): - cuda and other allocator-backed backends: behavior unchanged - cpu: the unobservable deltas are skipped; all device-placement validations still run Exposed by the `LOCAL_SIZE=4` multi-rank CPU run in deepspeedai#8381 (92 of the 131 v1-half failures there). ## Validation (executed on real hardware) - 20-core x86_64 CPU, torch 2.13.0+cpu, gloo backend, 2 ranks (`LOCAL_SIZE=2`) - Before: `TestDynamicOffloadStatesZero12[False-1-False-False-optim_states]` fails on the persistent-state delta assert - After: 5 representative cases pass (persistent and grad paths, ZeRO stage 1/2/3, `static_offload_optimizer=True` branch) — 5 passed in 55.6s - pre-commit (yapf / flake8 / check-torchdist / codespell) passes on the changed file Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…pspeedai#8559) ## Description `bf16_required_version_check()` (`tests/unit/util.py`) requires torch >= 1.10, **CUDA >= 11.0 and NCCL >= 2.10.3**. On the cpu accelerator, bf16 collectives run over gloo/ccl and none of those transport dependencies exist, so the check always returns False and **every bf16 test is skipped — about 40 call sites across 15 files**. `test_zero_autocast.py` is worse off: it **raises** instead of skipping, so each of its cases counts as a failure (24 on the multi-rank CPU run in deepspeedai#8381). This PR scopes the version floors inside the check itself: ```python if (cpu_accelerator and accelerator_pass) or (torch_version_available and cuda_version_available and nccl_version_available and accelerator_pass): return True ``` - **cpu**: only the accelerator's own bf16 support (`is_bf16_supported()`) is required — the torch/CUDA/NCCL floors are transport dependencies that gloo/ccl does not have - **every other accelerator** (cuda, npu, hpu, xpu, mlu, ...): evaluates the exact original floors; `cpu_accelerator` is False so the expression is bit-identical to before, and the npu/hpu/xpu exemption branches are untouched - `test_zero_autocast.py`: the bf16 gate goes back to the bare call every other caller uses, and skips instead of raising The hardcoded `init_distributed(dist_backend='nccl')` in the same test is deliberately left alone: it is a no-op (the harness already initialized the process group, `comm.py:838-839`), and deriving the backend per accelerator would change behavior on non-cuda accelerators (npu/hpu resolve to hccl etc.). The baseline `DDP(device_ids=[i])` pinning is left for a follow-up (deepspeedai#8399 fixed the same pattern elsewhere). ## Validation (executed on real hardware) - 20-core x86_64 CPU, torch 2.13.0+cpu, gloo backend - Direct call: `bf16_required_version_check()` on cpu returns `False` before, `True` after. Note: `CPU_Accelerator.is_bf16_supported()` is currently a stub that always returns True, so the cpu path does not gate on the hardware's bf16 instructions — giving it a real capability probe is left as a follow-up - Non-cpu equivalence: with `cpu_accelerator == False` the new expression reduces exactly to the original `A and C and N and P` - pre-commit (yapf / flake8 / check-torchdist / codespell) passes on both changed files Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409. Exposed by the `LOCAL_SIZE=4` multi-rank CPU run in deepspeedai#8381. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The baseline DistributedFixture builds an fp16 engine config for every dtype=float16 parametrization, and deepspeed.initialize's sanity check raises "Type fp16 is not supported on your device." on accelerators without fp16 support. Because the failure happens inside the fixture's distributed run, pytest reports a setup ERROR for every dependent test (48 on the multi-rank CPU run) instead of a skip. Skip inside the fixture like deepspeedai#8398 did for the fp16-config tests; DistributedFixture propagates the skip to all dependent tests. Devices with fp16 support are unchanged. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

TestCoalesceFP16 forces an fp16 config, and deepspeed.initialize's sanity check raises "Type fp16 is not supported on your device." on accelerators without fp16 support — the same hardware lottery deepspeedai#8398 handled for the other fp16-config tests. Add the same class-level skipif so the gap is a skip, not three failures. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

The dtype=float16 parametrizations of TestNoSyncCtxt crash deepspeed.initialize's sanity check ("Type fp16 is not supported on your device.") on accelerators without fp16 support — the same hardware lottery deepspeedai#8398 handled elsewhere. Guard the three dtype-parametrized methods so the gap is a skip, not eight failures; stages 2/3 additionally never reach their expected no_sync AssertionError on such hosts. Verified locally on an fp16-incapable CPU: 8 fail -> 8 skip, all 17 remaining parametrizations pass. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

TestMoECheckpoint hardcodes fp16 configs, so deepspeed.initialize's sanity check fails on accelerators without fp16 support — the same hardware lottery deepspeedai#8398 handled elsewhere. Add the class-level skipif. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

TestZeroPartialOffloadConfigSweep hardcodes fp16 and test_checkpoint_pipe_engine enables it for zero_stage > 0; both trip initialize's sanity check on accelerators without fp16 support (deepspeedai#8398's hardware lottery). Skip those parametrizations so the gap is a skip. With this, every fp16-config test in the suite guards on is_fp16_supported(). Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…dai#8684) ## Description Every fp16-config test that reaches `deepspeed.initialize` crashes its sanity check (`Type fp16 is not supported on your device.`) on accelerators whose `is_fp16_supported()` is false. On CPU that maps to the AVX512-FP16 capability of the host, and GitHub's `ubuntu-24.04` runners are hardware-heterogeneous, so these tests flip between failure and skip depending on which runner they land on. deepspeedai#8398 added the first skipifs; the multi-rank CPU run in deepspeedai#8381 flushed out six more files: | File | Shape of the gap | |---|---| | `checkpoint/test_universal_checkpoint.py` | fp16 parametrizations fail **inside the baseline DistributedFixture's distributed run** — pytest reports a setup **ERROR** for every dependent test (48 on the multi-rank run) instead of a skip | | `v1/zero/test_zero_coalesce_grad_reduction.py` | `TestCoalesceFP16` forces an fp16 config | | `runtime/test_no_sync_ctxt.py` | dtype=float16 parametrizations of three methods; stages 2/3 additionally never reach their expected no_sync AssertionError on such hosts | | `checkpoint/test_moe_checkpoint.py` | whole class hardcodes fp16 | | `runtime/zero/test_zero_offloadpp.py` | `TestZeroPartialOffloadConfigSweep` hardcodes fp16 | | `checkpoint/test_pipeline.py` | fp16 enabled for zero_stage > 0; only that parametrization skips, zero_stage=0 keeps running | With this PR, **every fp16-config test in the suite guards on `is_fp16_supported()`** — the capability gap is a skip, not a failure, on any accelerator. ## Validation (executed on real hardware) - 20-core x86_64 CPU without AVX512-FP16, gloo, 2-4 ranks (`LOCAL_SIZE=2/4`) - Before: 18 failures + 48 setup ERRORs across these files on the multi-rank CPU run - After: every affected parametrization skips; adjacent non-fp16 parametrizations keep passing (e.g. pipeline zero_stage=0 runs to completion) - Full-suite evidence in deepspeedai#8381: the multi-rank CPU run went 8 failures -> 0 with these guards Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648. Signed-off-by: Guokai Ma <guokai.ma@intel.com>

…es (deepspeedai#8702) ## Description Two latent bugs in Ulysses SP, both backend-independent (neither is CPU-specific — the CPU/gloo multi-rank run in deepspeedai#8381 is simply what surfaced them), plus test adaptations so the file can also run on non-CUDA accelerators. ### 1. `register_with_transformers` rejected every string model path `deepspeed/runtime/sequence_parallel/ulysses_sp.py` compared `core_attn_implementation` against the config's `_attn_implementation` even when the caller passed a **model path string**: ```python hf_model_config = AutoConfig.from_pretrained(model_name_or_path) # string branch model_attn_implementation = getattr(hf_model_config, "_attn_implementation", None) if model_attn_implementation is not None and model_attn_implementation != core_attn_implementation: raise ValueError(...) ``` In transformers, `_attn_implementation` is a **property resolved at model load time** — `from_pretrained` resolves which implementation the runtime can actually use and writes it back to the config. A bare `AutoConfig` never goes through that resolution, so the property returns its fallback value for "unresolved": ``` AutoConfig.from_pretrained(path)._attn_implementation -> eager (fallback, not a choice) AutoModelForCausalLM.from_pretrained(path).config._attn... -> sdpa (resolved) ``` So every string-path caller asking for `sdpa`/`flex_attention`/FA2 was rejected with a spurious mismatch against `'eager'`. The check is only meaningful for a config that was resolved, i.e. when the caller handed us a constructed model (or a PEFT wrapper exposing `.config`); scope it to that case. The comparison expression itself is unchanged, and `TestUlyssesSPHFAttnImplMismatch` (which passes a model object) still raises as intended. ### 2. `UlyssesSPDataLoaderAdapter` used a 0-dim tensor in `all_gather` ```python seqlen = torch.tensor(n, ...) # 0-dim seqlens = [torch.zeros(1, ...) for _ in range(sp_world_size)] # shape (1,) dist.all_gather(seqlens, seqlen, group=self.sp_group) ``` `c10d.all_gather` requires each receive tensor to have the same size as the send tensor. NCCL moves flattened bytes and tolerates the mismatch; gloo validates shapes and rejects it (`ProcessGroupGloo::allgather: invalid tensor size at index 0 (expected (), got (1))`). Send a 1-element tensor to match. ### 3. Tests - `TestUlyssesSPHFFlexAttention` needs the CUDA flex kernel -> skip on non-CUDA accelerators (same guard idiom as `tests/unit/util.py`, the nvme/triton CUDA-only tests). - `TestUlyssesSPHFPEFT`: the mock attached a **bare** `AutoConfig`, whose `_attn_implementation` is the unresolved `'eager'` — so this test failed on every backend, not just CPU. Give the mock the load-resolved value a real PEFT wrapper carries (the sibling `TestUlyssesSPHFAttnImplMismatch` in the same file already sets it the same way). - `TestUlyssesSPHFDisableInEval` pinned its tensors/models to `cuda:<rank>` although it runs sdpa and nothing in it is CUDA-specific. Use `get_accelerator().current_device_name()` like the rest of the file (on CUDA that is exactly `cuda:<rank>`, since the test harness sets the device per rank in `tests/unit/common.py`), so the test runs on any backend instead of erroring. ## Validation Hardware: 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks (`LOCAL_SIZE=2`), transformers 4.51.3, torch 2.13. - `tests/unit/ulysses_alst/test_ulysses_sp_hf.py`: **5 passed, 2 skipped** (flex skipped by the new guard). The sdpa classes pass end to end including their numerical-parity assertions; before the fix they died at registration. - `TestUlyssesSPHFDisableInEval` now actually **executes** on CPU rather than being skipped. - `TestUlyssesSPHFAttnImplMismatch` still raises the expected `ValueError` (the mismatch check is preserved where a resolved config exists). - Full multi-rank CPU CI in deepspeedai#8381: this file went from 5 failures to 0, with the overall job green. (`TestUlyssesSPHFHubKernel` is not exercised by the version pin above — it needs the `lazy_import_flash_attention` API from a newer transformers; it passes in CI, which installs transformers from git main.) Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648, deepspeedai#8684. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: pengdurice <pengduhit@gmail.com>

delock requested review from loadams, tjruwase and tohtana as code owners September 3, 2026 00:27

chatgpt-codex-connector Bot reviewed Sep 3, 2026

View reviewed changes

hwchen2017 approved these changes Sep 3, 2026

View reviewed changes

delock added this pull request to the merge queue Sep 3, 2026

Merged via the queue into deepspeedai:master with commit 493dafa Sep 3, 2026
12 of 15 checks passed

delock deleted the pr-b-fp16-skip branch September 3, 2026 10:26

This was referenced Sep 3, 2026

Probe the device module for train_cifar's fork_rng device entries #8407

Merged

Gate the offload-state memory deltas on allocator-backed stats #8409

Merged

delock mentioned this pull request Sep 17, 2026

Derive the autocast test backend and scope the bf16 gate to NCCL #8559

Merged

delock mentioned this pull request Sep 27, 2026

Skip fp16-config tests on accelerators without fp16 support #8684

Merged

delock mentioned this pull request Sep 29, 2026

Fix Ulysses SP registration for string model paths and gloo collectives #8702

Merged

Navigation Menu

Repository navigation

Skip fp16-config tests on accelerators without fp16 support - #8398

Skip fp16-config tests on accelerators without fp16 support#8398
delock merged 1 commit into
deepspeedai:masterfrom
delock:pr-b-fp16-skip

delock commented Sep 3, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

delock commented Sep 3, 2026

Problem

Change

Validation

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants