Skip to content

AutoEP: preserve native Qwen3.5 wrapper text routing - #8821

Closed
hazelduan wants to merge 2 commits into
deepspeedai:masterfrom
hazelduan:codex/qwen35-autoep-compat-20261014
Closed

hazelduan wants to merge 2 commits into
deepspeedai:masterfrom
hazelduan:codex/qwen35-autoep-compat-20261014

Conversation

@hazelduan

@hazelduan hazelduan commented Oct 14, 2026 •

Copy link
Copy Markdown

AutoEP rejects the Qwen3.5 MoE wrapper and diverges from its native router semantics. This change accepts text input through both the text backbone and the wrapper's language-model path, reads nested text configuration, and preserves native sorted top-k routing, gate-projection score dtype (including BF16 autocast with FP32 inputs), and raw router logits.

It uses the existing preset/spec/router extension points, with defaults preserving other presets. The regression compares Transformers' native models for both classes, with and without CPU BF16 autocast. Vision input remains untested.

Split from #8784; no HiFloat8 or NPU permutation dependency. Related RFC: #8777. Base: 3b80932; initial validated source: c0eb9e1. An open-PR search for Qwen3.5 returned no match on 2026-10-14.

Validation:

  • macOS CPU, Python 3.12.2: initial AutoEP suite 194 passed, 6 CUDA-only skips; final four native-router/text-output cases 4 passed. Both autocast cases fail before the gate-dtype correction and pass after it. Changed-file pre-commit and git diff --check passed.
  • A5 software environment (Ascend 950PR, driver 25.7.rc1, CANN 9.1, Python 3.11.15, PyTorch/TorchNPU 2.11.0, Transformers 5.16.1): final standalone AutoEP suite 196 passed, 6 CUDA-only skips. The four Qwen parity cases use CPU tensors, not NPU tensors. Command: PYTHONPATH=<isolated-qwen-repo> python -m pytest tests/unit/v1/moe/test_autoep_unit.py -q -rs.

The private combined-stack validation also completed Qwen3.5-35B-A3B text-only EP4/ZeRO2 for 30 matched steps. All 40 layers execute; only routed/shared experts in layers 0/39 train (12 tensors/rank), batch 1/GAS1, sequence length 256, eager attention/experts, no activation checkpointing or checkpoint save, seed 42, AdamW 1e-5/cosine, same model weights and 400-row data. Every rank has exact input hashes, loss, gradient norms, native operator counts and all final trainable tensors. Maximum-rank steady-step median (excluding 5 warm-up steps) is 4.1886s baseline vs 4.1906s candidate (+0.046%); training NPU peak is 33,487,014,400 bytes and initialization peak is 86,776,612,352 bytes in both. This combined-stack result does not attribute an independent speedup or numerical improvement to this split.

Native commands and logs are retained at /data/dxq/hif8_a5_20260914/experiments/quantize_convergence_20261014_01: bash native_gates.sh, bash paired_ep4.sh, qwen_native_stack.log, permutation_native.log, comparison_ep4_baseline_r1_ep4_candidate_r1.json. The runner bind-mounts the original Conda environment (dxq_hif8_torchao_py311) and CANN/driver; it changes no global packages or device permissions. Standalone runtime file hashes match each immutable source revision below. The model run uses a private composition of main PR and split runtime hunks, with the overlapping dispatch hunks reconciled; this composition is not the main PR contents.

Standalone model parity covers BF16 text inference at EP1 with eager attention and sequential experts. Vision, FP16, CUDA-native parity, ZeRO/offload combinations, activation checkpointing, AutoTP, LoRA and multi-node EP remain untested by this split.

AI assistance was used to implement and review this change; the human submitter owns review and validation.

Final published source: d8045b1. Post-push cleanup only removes three duplicate local imports; all four parity cases pass again on macOS and in the A5 software environment. Changed-file pre-commit passes. Runtime source is identical to the initial validated revision.

Resolve wrapper language-model paths and nested text configuration through existing presets. Preserve sorted top-k scores in the gate projection dtype under autocast and return raw router logits.

Native Transformers model oracle: text backbone and wrapper, with and without BF16 autocast. Standalone suite: 196 passed, 6 CUDA-only skips. Changed-file pre-commit passed.

Signed-off-by: hazelduan <duanxueqi@outlook.com>
Remove redundant local imports after moving the native model oracle into the AutoEP suite. All four parity cases and changed-file pre-commit pass.

Signed-off-by: hazelduan <duanxueqi@outlook.com>
@hazelduan hazelduan closed this Oct 14, 2026
@hazelduan
hazelduan deleted the codex/qwen35-autoep-compat-20261014 branch October 14, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant