Skip to content

Skip the accelerator-only CPU optimizer tests on the cpu accelerator - #8836

Open
delock wants to merge 1 commit into
deepspeedai:masterfrom
delock:ci/skip-accelerator-only-tests
Open

delock wants to merge 1 commit into
deepspeedai:masterfrom
delock:ci/skip-accelerator-only-tests

Conversation

@delock

@delock delock commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

TestCPUAdamGPUError and TestCPUAdagradGPUError assert that the CPU optimizer rejects a param living on a discrete accelerator:

device = get_accelerator().device_name(0)  # 'cuda:0' or 'xpu:0'
param = torch.nn.Parameter(torch.randn(model_size, device=device))
...
with pytest.raises(AssertionError):
    optimizer.step()

On the cpu accelerator the param is already on cpu, so the assert cannot fire — and the test never returns. In the first cpu-torch-multi-latest nightly (issue #8834, run https://github.com/deepspeedai/DeepSpeed/actions/runs/38849719748) two of the four xdist workers were stuck on exactly these two tests: they were the only items of the 851-test phase that started and never reported a result, and the run sat at 93-94% until the 120m phase budget killed it.

This marks both tests with an item-level skipif so they are skipped before tests/unit/common.py launches the rank pool, matching the guard TestCPULionGPUError already has:

@pytest.mark.skipif(get_accelerator().device_name() == 'cpu', reason="requires a discrete accelerator device")

Verified on a CPU-only machine (8 vCPU), with DS_MULTIRANK_ONLY=1 LOCAL_SIZE=4: both tests skip in ~2s without launching a rank pool, where the unmarked versions leave pytest wedged.

The shared test harness has related robustness gaps (an unbounded pool.starmap/join in DistributedTest._close_pool, one master port handed to every pool, and /dev/shm allreduce segments the op never unlinks). Those are deliberately left out of this PR; it only stops the nightly from selecting tests that cannot pass on cpu.

TestCPUAdamGPUError and TestCPUAdagradGPUError assert that a param on a
discrete accelerator is rejected by the CPU optimizer. On the cpu
accelerator the param is already on cpu, so the assert cannot fire, and
the rank pool never returns: the first cpu-torch-multi-latest nightly
(issue deepspeedai#8834) lost two of its four xdist workers to these two tests until
the 120m phase budget killed the run.

Use an item-level marker so the test is skipped before the rank pool is
launched, the way TestCPULionGPUError already guards itself.

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T09:05:52.591760Z dfb8393 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant