Skip to content

[Review] MoE fixed-route export and independent numerical reference - #80

Draft
Happymic wants to merge 4 commits into
mainfrom
review/moe-normal-export-20260905
Draft

Happymic wants to merge 4 commits into
mainfrom
review/moe-normal-export-20260905

Conversation

@Happymic

@Happymic Happymic commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add fixed-route MoE workload export and an independent numerical reference for the paired Simulator experiment. Given ready activations, expert weights and routes, the exporter produces workload.json, encoded weights.bin and golden.json.

Draft for review; not ready to merge. Companion: Simulator #117.

Implementation

  • Group work by expert while preserving token IDs, routing slots, routing weights and shared-expert semantics.
  • Encode weights with the local PLENA E4M3/E8M0 codec: eight elements share one scale. Record addresses, row strides and tail padding explicitly.
  • Independently decode the exported bytes and evaluate gate/up → SwiGLU → down → combine, with ordered FP32 accumulation and BF16 rounding at defined boundaries.
  • Support full-dimension fixtures by vectorizing independent reference outputs while preserving each dot product's accumulation order.

This is a workload-manifest interface, not complete dual-core instruction lowering. Earlier grouping and layout experiments are excluded from this diff.

Validation

All 21 exporter tests passed on the review branch. Coverage includes codec bytes, tails, scales, repeated expert selections, shared experts and numerical-reference agreement. The paired Simulator smoke test produces bit-exact BF16 outputs on both the single-core and large/small-core configurations.

The completed performance campaign used D=2048, F=512 or 1408, and 8/32-token windows, with archived routes and synthetic nonzero weights and activations. Core dimensions, SRAM budgets and performance results are documented in the companion Simulator PR.

See the review guide and run commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant