From b0aea59a4497f86dc8c47b952c116041a51f1baf Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 05:41:52 +0000 Subject: [PATCH 1/4] simd: a codegen oracle, and the u64 rotate it immediately found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `crates/simd-codegen-oracle` (+ script, baseline, CI job) and removes blake3's C/asm FFI. The oracle answers "does this need intrinsics?" with an assembly measurement instead of intuition — and its first run falsified the design assumptions it was built to check. ## Why TD-T22 measured that under the pinned `-Ctarget-cpu=x86-64-v3`, the "scalar polyfill" SIMD types already compile to packed AVX2 at the instruction floor. A PR had hand-written 700 lines of intrinsics against that nonexistent gap. Nothing in the repo could have caught it: there is no codegen check, only correctness parity harnesses. ## What the oracle measured 13 probes, three groups. Group A expected to vectorize, Group B expected NOT to, Group C (u64 rotate) deliberately unclassified. Group A — all vectorized, as expected. `arx_rounds_u32x16` (10-round ChaCha double-round): 52 packed, 0 scalar arithmetic on lane data. Group B — **3 of 5 predictions were WRONG.** LLVM vectorized: * saturating_abs_i8x32 → vpxor/vpsubsb/vpblendvb, i.e. it synthesized the VPABSB abs+clamp trick on its own * widening_u16_to_f32 → vpmovzxwd + vcvtdq2ps * cross_lane_reverse_u8x64 → vbroadcasti128 + vpshufb + vpermq — a cross-lane permute, from a scalar index loop Only serial_dependent_chain (loop-carried dependency) and gather_lookup_u8 stayed scalar. The "cross-lane/widening/saturating obviously needs intrinsics" intuition is measurably false. Group C — **the u64 rotate does NOT vectorize.** rot_u64x8 / rot_u64x4: 0 packed, one scalar `rorq %cl` per lane. blake2b_g_u64x8: the leading `a+b` goes packed (vpaddq), then LLVM extracts every lane to a GPR and stays scalar through all four rotate stages — including the byte-granular amounts 32/24/16, which at u32 width fold to vpshufb. AVX2 has vpsllq/vpsrlq and LLVM applies exactly that shift-or to u32 rotate-by-12/7; it declines to at 64-bit width. That matters because BLAKE2b is a 64-bit ARX cipher and argon2 uses BLAKE2b. The crate has 8 `rotate_left` methods (all u32) and zero `rotate_right` at any width, so argon2's kernel is unexpressible today. This is the first intrinsic override meeting the entry criterion: a probe proving the generic form fails. Same crate, same week, opposite answers for u32 and u64. That contrast is the argument for the oracle. ## blake3: no more C Operator directive — C is a contamination of ndarray, and vendored builds fail without a C toolchain. blake3's default build ran cc over c/blake3_{sse2,sse41,avx2,avx512}_x86-64_unix.S: measured 33 .o files plus libblake3_avx512_assembly.a, with blake3_*_ffi cfgs set. Now pinned to `default-features = false, features = ["pure"]`, which routes build.rs to build_sse2_sse41_avx2_rust_intrinsics() ("No C code to compile here"). Verified after `cargo clean -p blake3`: the only cfgs actually SET are blake3_{sse2,sse41,avx2}_rust; 0 object files, 0 archives, no cc. (The _ffi names still appear in the build output as rustc-check-cfg DECLARATIONS — cargo listing valid cfg names — not assignments. A grep that cannot tell the two apart reports a false positive here.) Remaining cc build-deps in the lockfile (cmake, openblas-build, openssl-sys) are reachable only through the optional `blas` feature, not `default = ["std", "hpc-extras"]`. blake3 was the only C on the default path — which is why vendoring broke for everyone. ## Honesty rule baked into the tool The analyzer separates "scalar arithmetic on lane data" from loop control, so a trip-counter `decl` is never reported as scalar lane work. That exact overclaim was made and corrected earlier this week; the tool cannot reproduce it. ## Docs * .claude/knowledge/simd-one-spec-design.md — design for collapsing 5 backends (31 macro-generated + 57 hand-written types, 13,253 LoC, three authoring strategies) into one spec, with the oracle as the entry criterion for intrinsic overrides. Includes the table of my three wrong predictions. * .claude/knowledge/crypto-lane-status.md — u32 ARX proven optimal (ChaCha20/BLAKE3 need no SIMD work); u64 ARX absent and now measured as genuinely needing intrinsics. --- .claude/knowledge/crypto-lane-status.md | 118 +++++ .claude/knowledge/simd-one-spec-design.md | 174 ++++++++ .github/workflows/ci.yaml | 17 + Cargo.toml | 35 +- crates/simd-codegen-oracle/Cargo.lock | 170 +++++++ crates/simd-codegen-oracle/Cargo.toml | 31 ++ .../baselines/x86_64-unknown-linux-gnu.toml | 122 +++++ crates/simd-codegen-oracle/src/main.rs | 379 ++++++++++++++++ scripts/codegen-oracle.sh | 73 +++ scripts/codegen_oracle_analyze.py | 416 ++++++++++++++++++ 10 files changed, 1533 insertions(+), 2 deletions(-) create mode 100644 .claude/knowledge/crypto-lane-status.md create mode 100644 .claude/knowledge/simd-one-spec-design.md create mode 100644 crates/simd-codegen-oracle/Cargo.lock create mode 100644 crates/simd-codegen-oracle/Cargo.toml create mode 100644 crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml create mode 100644 crates/simd-codegen-oracle/src/main.rs create mode 100755 scripts/codegen-oracle.sh create mode 100755 scripts/codegen_oracle_analyze.py diff --git a/.claude/knowledge/crypto-lane-status.md b/.claude/knowledge/crypto-lane-status.md new file mode 100644 index 00000000..7658cc40 --- /dev/null +++ b/.claude/knowledge/crypto-lane-status.md @@ -0,0 +1,118 @@ +# The crypto lane — what is proven, what is missing + +> **Status: MEASURED, 2026-07-28.** Every claim here is backed by an +> assembly probe or a grep across all six backends. No estimates. + +## READ BY: +- Anyone building on `crates/encryption` (argon2 / BLAKE3 / ChaCha20 / AEAD) +- Anyone asked "do we need new SIMD work for cipher X?" + +--- + +## The u32 ARX lane: PROVEN, at the instruction floor + +`crate::simd::U32x16` Add / BitXor / `rotate_left` is the ChaCha20 and BLAKE3 +mixing triple. TD-T22 measured it (`td-t22-asm-investigation.md`): the +scalar-storage polyfill compiles to **8 `vpaddd` for 64 u32 lanes — the AVX2 +instruction floor** — with no scalar op touching lane data, and +`rotate_left(16)` strength-reduced to `vpshufb`, cheaper than the +`shl|shr|or` triple an intrinsic emits. + +**Consequence: ChaCha20 and BLAKE3 need no new SIMD work.** The lane they +ride is already optimal on the default tier. + +The float side is likewise done: `add_mul_f32` emits real `vfmadd213ps` — +one rounding, mantissa preserved — and `array_chunks` / `array_windows` +(+ `_checked`) already exist as the slice-level primitives in `simd_ops.rs`. + +## The u64 ARX lane: DOES NOT EXIST + +Measured across `simd_avx512`, `simd_avx2`, `simd_scalar`, `simd_neon`, +`simd_wasm`, and `simd_nightly`: **zero `rotate_left` or `rotate_right` +methods on `U64x8` or `U64x4`. On any backend.** + +Whole-crate census of rotate methods — `grep -rhoE "fn rotate_(left|right)" src/`: + +| method | count | +|---|---| +| `rotate_left` | 8 (all u32 lanes) | +| `rotate_right` | **0, at any width** | + +*(Search validated by control: the same pattern finds `U32x16::rotate_left` +in all four backends that define it, so the empty u64 result is a true +negative and not a broken query.)* + +Note the second row. BLAKE2b specifies **right** rotations. `rotr(n)` is +expressible as `rotl(64 - n)`, so this is a naming/API gap rather than a +mathematical one — but a caller writing BLAKE2b today has neither. + +This matters because **BLAKE2b is a 64-bit ARX cipher**, and BLAKE2b is what +**argon2** uses. Its G-function is +`a+=b; d=(d^a).rotr(32); c+=d; b=(b^c).rotr(24); a+=b; d=(d^a).rotr(16); c+=d; b=(b^c).rotr(63)` +— four u64 rotates per mixing step, none of which the crate can express +today. + +`crates/encryption` currently references exactly one SIMD type: +`simd::U32x16`. + +**This is a real gap, unlike the u32 one.** The distinction is the whole +lesson of TD-T22: a missing *source-level* lowering is not a gap when LLVM +already emits the instruction; a missing *method* is a gap regardless of +what LLVM would do with it. + +### ANSWERED by the oracle: no, it does not vectorize + +Measured on x86_64 v3 via `crates/simd-codegen-oracle`: + +| probe | packed | scalar lane-arith | verdict | +|---|---|---|---| +| `rot_u64x8` | **0** | 8 | one scalar `rorq %cl` per lane | +| `rot_u64x4` | **0** | 4 | one scalar `rorq %cl` per lane | +| `blake2b_g_u64x8` | 22 | ~88 | leading add only | + +`blake2b_g_u64x8` in detail: the opening `a = a + b` vectorizes (`vpaddq` +across both ymm halves). The moment a rotate is needed LLVM extracts every +lane to a GPR (`vmovq` / `vpextrq`) and stays scalar (`rorxq`/`addq`/`xorq`) +through all four rotate stages, going packed again only for the return +struct's reassembly. + +Two things make this a genuine finding rather than a shrug: + +1. **The byte-granular amounts stay scalar too.** Rotates by 32/24/16 are + byte-aligned — the class LLVM folds to `vpshufb` for u32 — yet all four + BLAKE2b amounts (32/24/16/63) lowered identically to scalar `rorxq`. +2. **The mechanism exists and is unused.** AVX2 has `vpsllq`/`vpsrlq`, and + LLVM *does* use exactly that shift-or composition for u32's rotate-by-12 + and rotate-by-7. It has the tools and declines to apply them at 64-bit + width. + +**So the u64 ARX lane is the crate's first intrinsic override that meets the +entry criterion** (a probe proving the generic form fails). AVX-512: +`_mm512_rorv_epi64` / `VPROLVQ`, one instruction. AVX2 / NEON / wasm: write +the `vpsllq`/`vpsrlq`-shaped shift-or explicitly, since LLVM will not. + +Contrast with the u32 lane, where hand-writing intrinsics *lost* to the +optimizer. Same crate, same week, opposite answers — which is the argument +for the oracle existing at all. + +## Decision gates (operator, not engineering) + +Neither of these is blocked on SIMD work: + +1. **FIPS.** If any deployment needs FIPS-adjacent claims, BLAKE3 is out and + SHA-384 stays the KDF hash. Settle before investing in a BLAKE3 lane. +2. **`x448` audit provenance.** Gates whether an X25519 port is worth it. + The tripwire test already on master + (`channel::tests::low_order_peer_keys_are_refused_and_honest_ones_are_not`) + asserts both halves, so a mechanical port cannot silently drop RFC 7748's + contributory check — `x448::x448()` returns `Option` where + `x25519_dalek::x25519()` returns a bare `[u8; 32]`. + +## Summary + +| lane | status | blocks | +|---|---|---| +| u32 ARX (ChaCha20, BLAKE3) | **proven optimal** | nothing | +| f32 FMA (`add_mul`) | **proven fused** | nothing | +| slice chunking | **exists** | nothing | +| **u64 ARX (BLAKE2b → argon2)** | **absent on all 6 backends** | argon2 SIMD | diff --git a/.claude/knowledge/simd-one-spec-design.md b/.claude/knowledge/simd-one-spec-design.md new file mode 100644 index 00000000..9a3a352c --- /dev/null +++ b/.claude/knowledge/simd-one-spec-design.md @@ -0,0 +1,174 @@ +# One spec, N backends — collapsing the SIMD type surface + +> **Status: DESIGN.** Not implemented. The enabling measurement is done +> (TD-T22, merged 2026-07-28); the migration is a staged epic, not a PR. + +## READ BY: +- Anyone about to hand-write a lane type in `src/simd_.rs` +- Anyone proposing to "add the missing types" to a backend +- `simd-savant`, `truth-architect` + +--- + +## The measurement that makes this possible + +TD-T22 (`.claude/knowledge/td-t22-asm-investigation.md`) established, with +assembly evidence: **under a pinned `target-cpu` baseline, LLVM compiles a +scalar-shaped lane loop to optimal packed SIMD.** On the ChaCha20 ARX triple +over the scalar-storage `U32x16`, the emitted code hits the AVX2 instruction +floor — 8 `vpaddd` for 64 u32 lanes — with no scalar op touching lane data, +and `rotate_left(16)` strength-reduced to `vpshufb`, which is *cheaper* than +the `shl|shr|or` triple a hand-written intrinsic emits. + +The consequence is not "the polyfill is fine." It is: **for lane-wise +operations, the scalar spec IS the implementation, on every backend.** + +## What exists today + +| backend | types from a macro | hand-written structs | +|---|---|---| +| `simd_avx2` | 12 | 8 | +| `simd_avx512` | 0 | 21 | +| `simd_scalar` | 19 | 6 | +| `simd_neon` | 0 | 15 | +| `simd_wasm` | 0 | 7 | +| **total** | **31** | **57** | + +Plus 72 `impl_bin_op!`-family invocations in `simd_avx512` layered on top of +its hand-written structs. **13,253 LoC across five files, three different +authoring strategies, and the same logical type written five times.** + +Three strategies for one problem: +1. `avx2` / `scalar` — type-generating macros (`avx2_int_type!`, `impl_int_type!`) +2. `avx512` — hand-written struct + operator-generating macros +3. `neon` / `wasm` — fully hand-written + +This is why adding one lane type is a five-file change, why ten AVX2 int +types are still "unlowered," and why the same boilerplate was hand-typed +twice for `U16x16` and then again for `U32x8`. + +## The design + +One declaration per logical type. Backends are *generated*, not authored. + +```rust +simd_type! { + name: U32x16, elem: u32, lanes: 16, repr: align(64), + + // Lane-wise ops. Emitted as the scalar loop form for EVERY backend. + // LLVM vectorizes them under the pinned target-cpu baseline; the + // codegen oracle proves it, per target, in CI. + lanewise: [ + add(wrapping), sub(wrapping), mul(wrapping), + and, or, xor, not, + rotate_left, shl(zero_on_overshift), shr(zero_on_overshift), + reduce_sum(wrapping), + ], + + // ESCAPE HATCH. A per-backend intrinsic override may be added ONLY + // with an oracle probe showing the generic form does not vectorize, + // or a measured win the generic form cannot reach. + intrinsic: { + avx512: { rotate_left: "_mm512_rolv_epi32" }, // VPROLVD, 1 instr + }, +} +``` + +**The entry criterion is the whole point.** Today "should this be an +intrinsic?" is answered by intuition, and intuition said yes to a case where +LLVM was already at the instruction floor. Under this design the question is +answered by `scripts/codegen-oracle.sh`: if the generic form vectorizes, the +override is rejected; if it doesn't, the override is justified and the probe +that justified it is committed alongside. + +## What must NOT be generated — MEASURED, and my predictions were wrong + +I predicted five classes LLVM could not synthesize from scalar source. The +oracle ran them. **Three of the five vectorized anyway.** Recorded here +because the wrong list is more instructive than the right one: the intuition +that "cross-lane / widening / saturating obviously needs intrinsics" is +exactly the intuition that produced a 700-line PR against a nonexistent gap. + +| predicted scalar | actual | what LLVM emitted | +|---|---|---| +| `saturating_abs_i8x32` | **VECTORIZED** (4 packed / 0 scalar) | `vpxor` → `vpsubsb` (saturating 0−x) → `vpblendvb` — the exact abs+clamp trick the VPABSB correction documents, synthesized on its own | +| `widening_u16_to_f32` | **VECTORIZED** (6 packed / 0 scalar) | `vpmovzxwd` + `vcvtdq2ps` | +| `cross_lane_reverse_u8x64` | **VECTORIZED** (9 packed / 0 scalar) | `vbroadcasti128` + `vpshufb` + `vpermq` — it invented a cross-lane permute from a scalar index loop | +| `serial_dependent_chain` | scalar, as predicted | GPR `rorxl`/`addl`/`xorl` chain — a loop-carried dependency cannot vectorize | +| `gather_lookup_u8` | scalar, as predicted | pure `movzbl`/`movb`; no arithmetic at all | + +So the genuine "cannot be generated" list is much shorter than assumed: + +- **Loop-carried dependencies.** Structural; no compiler escapes them. +- **Gather / table lookup.** No contiguous load to widen. +- **u64 lane rotates** — see below. The one case where LLVM has the + mechanism and declines to use it. + +**Everything else measured so far is free.** `U16x16`'s hand-written +`permute2x128`/`blend_epi32` may still be justified — they are *explicit API +surface* consumers call directly, not something to be synthesized — but the +claim that cross-lane work inherently requires intrinsics is false. + +## The u64 rotate — the first earned intrinsic override + +Measured (`rot_u64x8`, `rot_u64x4`, `blake2b_g_u64x8`): + +- `rot_u64x8` / `rot_u64x4`: **0 packed.** Each lane's `u64::rotate_right(n)` + becomes a scalar GPR `rorq %cl, reg`, one per lane. +- `blake2b_g_u64x8`: the leading `a = a + b` vectorizes (`vpaddq` over both + ymm halves); the moment a rotate appears LLVM extracts every lane + (`vmovq`/`vpextrq`) and stays scalar (`rorxq`/`addq`/`xorq`) through all + four rotate stages, reassembling only at the return. + +The striking part: **the byte-granular amounts 32/24/16 stay scalar here**, +while the same class of amounts (16, 8) fold to `vpshufb` for u32. And AVX2 +has `vpsllq`/`vpsrlq` — the exact shift-or mechanism LLVM *does* apply to +u32's rotate-by-12 and rotate-by-7. It has the tools and does not reach for +them on u64. + +This is the crate's first intrinsic override that meets the entry criterion: +a probe showing the generic form does not vectorize. `_mm512_rorv_epi64` +(`VPROLVQ`) is a single instruction on AVX-512; AVX2/NEON/wasm get the +shift-or composition written explicitly. + +## Staged migration (not one PR) + +1. **Oracle first.** `crates/simd-codegen-oracle` + CI job. Without it the + entry criterion is unenforceable and this design is just a refactor. +2. **Characterize.** Run the oracle across x86-64-v3 / v4 / aarch64 / wasm32. + Produce the per-target table of what vectorizes and what doesn't. That + table *is* the specification of which intrinsic overrides are legitimate. +3. **Pilot on one type family.** The u32 lanes (`U32x8`, `U32x16`) — smallest + blast radius, best-understood semantics, already measured. +4. **Migrate the 31 macro-generated types.** Mechanical; the macros already + prove the shape is regular. +5. **Migrate the 57 hand-written types**, keeping every intrinsic the oracle + justifies and deleting the rest. Expect the survivors to be concentrated + in the cross-lane / widening / saturating families above. + +## Invariants the design must preserve + +- **`repr(align(64))` on every lane type.** Nine sites across + `scalar`/`neon`/`wasm` carry it; it is a cacheline guarantee, not an + accident. A `repr(transparent)` wrapper over `__m256i` LOSES it (measured: + `U32x8` size 64→32, `U32x16` align 64→32). The spec must emit `align(64)` + by default. +- **One API on every backend.** The generated surface is identical by + construction — which structurally eliminates the class of bug where a + method exists on x86_64 and nowhere else. +- **`U32x8` must not be `U32x16`'s building block** (operator ruling, + 2026-07-28). Composition, where needed, is an implementation detail of the + generated backend, never a public half-width type standing in for the lane + the substrate actually uses. +- **No `core::simd` or `hpc::` in a public signature; consumers only ever + name `crate::simd::*`.** Backend-internal construction uses concrete + backend types — a backend file is compiled even when its dispatch arm is + not selected. + +## What this buys + +- Adding a lane type: one declaration instead of a five-file change. +- Ten currently-unlowered AVX2 int types: free. +- The "is this fast enough?" argument: replaced by a CI check. +- ~13k LoC of hand-maintained backend code: substantially reduced, with the + remainder being exactly the intrinsics that earn their place. diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index f2fbcab2..75f92942 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -163,6 +163,22 @@ jobs: - name: NEON SIMD parity (cross-build + qemu run) run: ./scripts/neon-parity.sh + simd-codegen-oracle: + # Proves (in both directions) what `.cargo/config.toml`'s + # `-Ctarget-cpu=x86-64-v3` baseline actually does to this crate's + # "scalar polyfill" SIMD storage types: Group A probes must show packed + # AVX2 codegen from scalar *source*, Group B probes must show none, and + # Group C probes (the u64 rotate gap) are reported without a pass/fail + # verdict. See crates/simd-codegen-oracle/src/main.rs and + # scripts/codegen_oracle_analyze.py for the full picture. + runs-on: ubuntu-latest + name: simd-codegen-oracle/instruction-histogram + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: SIMD codegen oracle (build --emit asm + classify + baseline compare) + run: ./scripts/codegen-oracle.sh + tests: runs-on: ubuntu-latest needs: pass-msrv @@ -388,6 +404,7 @@ jobs: - nostd - wasm_simd - neon_simd + - simd-codegen-oracle - tests - native-backend - hpc-stream-parallel diff --git a/Cargo.toml b/Cargo.toml index 472bc1f7..e8689c66 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -185,7 +185,32 @@ matrixmultiply = { version = "0.3.2", default-features = false, features=["cgemm # If not, leave the `optional = true` + `std`-feature pinning in place. # # ===================================================================== -blake3 = { version = "1", optional = true } +# `default-features = false` + `pure`: NO C/ASM FFI. Operator directive +# 2026-07-28 — "I don't want any ffi with c". +# +# blake3's default build runs `cc::Build` over `c/blake3_{sse2,sse41,avx2, +# avx512}_x86-64_unix.S` and links them (measured before this change: 33 `.o` +# files plus `libblake3_avx512_assembly.a` in target/, with `blake3_*_ffi` +# cfgs set). The `pure` feature routes build.rs to +# `build_sse2_sse41_avx2_rust_intrinsics()` — its own comment: "No C code to +# compile here" — which only sets `blake3_{sse2,sse41,avx2}_rust` cfgs and +# lets the normal cargo build compile the Rust intrinsics modules. +# +# What `pure` costs (build.rs:344-371): the hand-tuned x86-64 assembly is +# replaced by Rust intrinsics, the AVX-512 path is dropped entirely, and the +# aarch64 NEON C intrinsics are dropped. What it removes: every byte of C. +# +# `default-features = false` also drops blake3's own `std` (which only gates +# `constant_time_eq/std`); this crate's `std` feature is what pulls blake3 in +# at all, so nothing here needs blake3's. +# +# NOTE: `pure` still leaves blake3's Rust SSE2/SSE4.1/AVX2 intrinsics — a +# second SIMD surface beside `ndarray::simd`, which the matryoshka pattern +# exists to prevent. Closing that means implementing BLAKE3's compression on +# `ndarray::simd::U32x16` (it is a ChaCha-derived u32 ARX kernel, and that +# lane is proven at the AVX2 instruction floor — see +# `.claude/knowledge/td-t22-asm-investigation.md`). Tracked, not done here. +blake3 = { version = "1", optional = true, default-features = false, features = ["pure"] } # p64 + fractal — specialized convergence / manifold math. Gated behind # `hpc-extras` since they pull in a dep tree burn-ndarray doesn't need. @@ -402,7 +427,13 @@ members = [ "ndarray-rand", "crates/*", ] -exclude = ["crates/burn", "crates/wasm-simd-parity", "crates/neon-simd-parity", "vendor/chacha20"] +exclude = [ + "crates/burn", + "crates/wasm-simd-parity", + "crates/neon-simd-parity", + "crates/simd-codegen-oracle", + "vendor/chacha20", +] default-members = [ ".", "ndarray-rand", diff --git a/crates/simd-codegen-oracle/Cargo.lock b/crates/simd-codegen-oracle/Cargo.lock new file mode 100644 index 00000000..42a296a7 --- /dev/null +++ b/crates/simd-codegen-oracle/Cargo.lock @@ -0,0 +1,170 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.7.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "blake3" +version = "1.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +dependencies = [ + "arrayref", + "arrayvec", + "cc", + "cfg-if", + "constant_time_eq", + "cpufeatures", +] + +[[package]] +name = "cc" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "matrixmultiply" +version = "0.3.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7" +dependencies = [ + "autocfg", + "rawpointer", +] + +[[package]] +name = "ndarray" +version = "0.17.2" +dependencies = [ + "blake3", + "matrixmultiply", + "num-complex", + "num-integer", + "num-traits", + "paste", + "portable-atomic", + "portable-atomic-util", + "rawpointer", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "rawpointer" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-codegen-oracle" +version = "0.0.0" +dependencies = [ + "ndarray", +] diff --git a/crates/simd-codegen-oracle/Cargo.toml b/crates/simd-codegen-oracle/Cargo.toml new file mode 100644 index 00000000..e8115274 --- /dev/null +++ b/crates/simd-codegen-oracle/Cargo.toml @@ -0,0 +1,31 @@ +# simd-codegen-oracle — proves (or disproves) what actually vectorizes. +# +# Context: `.cargo/config.toml` pins `-Ctarget-cpu=x86-64-v3`, so this crate's +# "scalar polyfill" SIMD storage types (macro-generated `[T; N]`, e.g. +# `avx2_int_type!(U32x16, u32, 16, 0u32)` in `src/simd_avx2.rs`) already +# compile to packed AVX2 via LLVM's loop/SLP vectorizer — they are not scalar +# codegen despite being scalar *source*. This crate is the tested proof of +# that claim, in both directions: probes that DO vectorize (Group A) and +# probes that provably DO NOT (Group B), plus a third bucket (Group C) whose +# classification is the open question being measured (the u64 rotate gap). +# +# EXCLUDED from the workspace (see root Cargo.toml `exclude`) so it has zero +# effect on the default build/test jobs; CI's `simd-codegen-oracle` job +# builds + runs it via --manifest-path, mirroring `neon-simd-parity` / +# `wasm-simd-parity`. +[package] +name = "simd-codegen-oracle" +version = "0.0.0" +edition = "2021" +publish = false + +[[bin]] +name = "simd-codegen-oracle" +path = "src/main.rs" + +[dependencies] +ndarray = { path = "../..", default-features = false, features = ["std"] } + +[profile.release] +debug = false +panic = "abort" diff --git a/crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml b/crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml new file mode 100644 index 00000000..22f0b8c0 --- /dev/null +++ b/crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml @@ -0,0 +1,122 @@ +# Baseline instruction-class expectations for simd-codegen-oracle on +# x86_64-unknown-linux-gnu, generated from the first run on this host +# (AVX2 baseline, `-Ctarget-cpu=x86-64-v3`, rustc 1.95.0). +# +# Bounds are class ASSERTIONS, not exact instruction counts -- LLVM version +# drift changes exact counts across toolchain updates. `min_packed` is set at +# roughly 60% of the first observed count (see each probe's `note` for the +# exact observed numbers) so a regression trips the gate without noise- +# triggering on harmless codegen wobble. `max_scalar_lane_arith` bounds the +# "scalar arithmetic on lane data" bucket -- see +# scripts/codegen_oracle_analyze.py's module docstring for exactly what is +# and isn't counted there. +# +# expect = "vectorized" | "scalar" | "unknown" +# vectorized -> require packed-vector >= min_packed (and, if set, +# scalar-lane-arith <= max_scalar_lane_arith) +# scalar -> require packed-vector <= max_packed (default 0) +# unknown -> report only, no pass/fail (Group C: the open question) + +# ============================================================================ +# Group A -- expected to fully vectorize. All 5 confirmed. +# ============================================================================ + +[probe.arx_u32x16] +expect = "vectorized" +min_packed = 6 +max_scalar_lane_arith = 0 +note = "ChaCha/BLAKE ARX triple; rotate_left(16) folds to vpshufb (byte-granular rotate). Observed 11 packed / 0 scalar-lane-arith / 0 loop-control on rustc 1.95.0." + +[probe.arx_rounds_u32x16] +expect = "vectorized" +min_packed = 30 +max_scalar_lane_arith = 0 +note = "10-round ChaCha double-round over 4x U32x16 lanes; NOT fully unrolled (real loop, trip count 10) -- retains exactly 2 GPR instructions (movl $10,%eax / decl %eax, the loop counter) which correctly land in loop-control, not lane-arith. rotate(16)/(8) fold to vpshufb; rotate(12)/(7) fold to vpslld+vpsrld+vpor. Observed 52 packed / 0 scalar-lane-arith / 2 loop-control." + +[probe.fma_f32x16] +expect = "vectorized" +min_packed = 16 +max_scalar_lane_arith = 3 +note = "ndarray::simd::add_mul_f32 is a general slice-length FMA: unrolled x4 vfmadd213ps main loop (8-wide ymm) + a second x4-unrolled 8-wide ymm loop + xmm remainder loop + a genuine single-element vfmadd213ss tail for n%4 leftovers. The tail's vfmadd213ss IS real scalar-lane-arith (by design, not a codegen failure) -- a nonzero count here is EXPECTED, not a regression signal by itself. Observed 28 packed (vmovups+vfmadd213ps) / 1 scalar-lane-arith (the vfmadd213ss tail) / 62 loop-control (three-way min(acc.len(),a.len(),b.len()) bound tracking -- rax index, r10 combined bound, rbx/r14/r15 parallel countdown copies, rcx aligned trip count -- all correctly identified as index/bound bookkeeping, not lane data) / 18 memory, for n=64 (exactly divisible by 16, so the tail paths are present in the static code but do not execute at runtime for this input)." + +[probe.reduce_u32x16] +expect = "vectorized" +min_packed = 4 +max_scalar_lane_arith = 0 +note = "avx2_int_type! reduce_sum: horizontal add via vpaddd + vextracti128 + vpshufd shuffle-reduce tree. Observed 8 packed / 0 scalar-lane-arith." + +[probe.bitwise_u8x64] +expect = "vectorized" +min_packed = 7 +max_scalar_lane_arith = 0 +note = "(a^b)&a|b over U8x64, both 32-byte ymm halves. Observed 12 packed / 0 scalar-lane-arith -- LLVM chose the *ps float-domain packed-bitwise forms (vxorps/vandps/vorps instead of vpxor/vpand/vpor), still packed-vector under our classification (domain choice, not width)." + +# ============================================================================ +# Group B -- expected to NOT fully vectorize. THREE surprises measured here: +# saturating_abs_i8x32, widening_u16_to_f32, and cross_lane_reverse_u8x64 ALL +# vectorized despite being hand-written as scalar loops. Only +# serial_dependent_chain and gather_lookup_u8 came out scalar as hypothesized. +# `expect` below reflects the MEASURED reality, not the original hypothesis; +# that 3-of-5 mismatch IS the finding -- kept visible in each probe's `note` +# rather than silently "fixed" by relabeling without comment. +# ============================================================================ + +[probe.serial_dependent_chain] +expect = "scalar" +max_packed = 0 +note = "Genuinely serial (nonlinear rotate+xor chain, runtime trip count). Confirmed scalar: rorxl/addl/xorl on GPRs (BMI2 rorxl chosen over the plain scalar ror, since -Ctarget-cpu=x86-64-v3 includes BMI2), real backward branches. Observed 0 packed / 27 scalar-lane-arith (rorxl+addl+xorl x9, the unrolled-by-8 main loop plus the up-to-7-iteration remainder loop) / 12 loop-control (trip-count bookkeeping for both loops, correctly separated from the arithmetic)." + +[probe.saturating_abs_i8x32] +expect = "vectorized" +min_packed = 2 +max_scalar_lane_arith = 0 +note = "SURPRISE: hand-written as a 32-iteration scalar loop over to_array()/i8::saturating_abs(), but LLVM recognized the idiom and emitted vpxor+vpsubsb+vpblendvb (subtract-saturate-from-zero, then blend by original sign) -- the exact abs+clamp trick the VPABSB correction doc (.claude/knowledge/vertical-simd-consumer-contract.md) describes for i8::MIN, discovered automatically by the vectorizer with zero scalar loop surviving. Observed 4 packed / 0 scalar-lane-arith. Originally hypothesized Group B; measured result is fully vectorized -- do not 'fix' this, it is real." + +[probe.gather_lookup_u8] +expect = "scalar" +max_packed = 0 +note = "Gather-shaped table lookup (table[idx[i]]), fully unrolled x32, ZERO arithmetic instructions at all -- pure movzbl/movb chains, addressing folded entirely into immediate-offset operands. Confirmed scalar as hypothesized. Observed 0 packed / 0 scalar-lane-arith / 0 loop-control (no loop at all, fully unrolled -- there is no trip count to track) / 96 memory (movzbl+movzbl+movb x32)." + +[probe.widening_u16_to_f32] +expect = "vectorized" +min_packed = 3 +max_scalar_lane_arith = 0 +note = "SURPRISE: hand-written as a 16-iteration scalar `as f32` cast loop, but LLVM recognized the zero-extend-then-convert idiom and emitted vpmovzxwd+vcvtdq2ps (two 8-wide ymm chunks). Observed 6 packed / 0 scalar-lane-arith. Originally hypothesized Group B; measured result is fully vectorized -- do not 'fix' this, it is real." + +[probe.cross_lane_reverse_u8x64] +expect = "vectorized" +min_packed = 5 +max_scalar_lane_arith = 0 +note = "SURPRISE: hand-written as a 64-iteration scalar index-reversal loop (out[i] = in[63-i]), but LLVM recognized it and emitted vbroadcasti128 + vpshufb (byte-reverse within each 16-byte lane) + vpermq (swap the two 128-bit halves across the ymm) -- a real cross-lane permute, not just a within-lane shuffle. Observed 9 packed / 0 scalar-lane-arith. Originally hypothesized Group B; measured result is fully vectorized -- do not 'fix' this, it is real." + +# ============================================================================ +# Group C -- UNKNOWN. Report only, no pass/fail. This is the open question +# that motivated extending this oracle: does a scalar-shaped u64 rotate loop, +# mirroring U32x16::rotate_left's own shape line for line, get the same free +# ride from LLVM that the u32 lane gets? No backend in this crate defines +# rotate_left/rotate_right on any u64 lane type today. +# +# MEASURED ANSWER: NO. All three probes below show packed-vector == 0 for +# every rotate-dependent instruction; the rotate itself always lowers to a +# scalar GPR rorq/rorxq, regardless of whether the rotate amount is a runtime +# variable (rot_u64x8/rot_u64x4) or a compile-time constant +# (blake2b_g_u64x8's 32/24/16/63) -- unlike the u32 lane, where BOTH the +# byte-granular constant case (rotate_left(16)/(8) -> vpshufb) and the +# bit-granular constant case (rotate_left(12)/(7) -> vpslld+vpsrld+vpor) +# vectorize (see arx_rounds_u32x16 above). This is the one probe family in +# this oracle where the "scalar source still gets packed codegen" claim does +# NOT hold -- the first probe result in this crate that is a legitimate case +# for a hand-written intrinsic, per the task brief's own framing. +# ============================================================================ + +[probe.rot_u64x8] +expect = "unknown" +note = "MEASURED SCALAR. 8x (movq-load + rorq %cl,reg [variable count] + movq-store), zero packed instructions anywhere in the function. AVX2 has vpsllq/vpsrlq (uniform-count packed 64-bit shift, which is exactly the mechanism LLVM used to vectorize u32's rotate(12)/(7) via shift+or), so the ingredients for a packed shift-or lowering exist on this target -- LLVM simply did not apply them here for a runtime-variable-count u64 rotate. Observed 0 packed / 8 scalar-lane-arith (the 8 rorq) / 0 loop-control (straight-line, no loop) / 19 memory (movq loads+stores + the count setup)." + +[probe.rot_u64x4] +expect = "unknown" +note = "Same pattern as rot_u64x8, 4 lanes instead of 8, same verdict. Observed 0 packed / 4 scalar-lane-arith (4x rorq) / 0 loop-control / 9 memory." + +[probe.blake2b_g_u64x8] +expect = "unknown" +note = "MEASURED MIXED, and the most interesting single result in this oracle. The G-function's straight-line body has NO loop at all (fully unrolled by construction -- 8 lanes, no trip count, confirmed by 0 jumps in the disassembly). The LEADING a=a+b step DOES vectorize (vpaddq over both 32-byte ymm halves -- 2 of the function's 22 packed instructions; the rest of the 22 are the final vmovaps/vmovups reassembly into the return-struct). The moment a rotate is needed, LLVM extracts every lane out to a GPR (vmovq/vpextrq, in the 'other'/memory buckets) and the REST of the function -- all four rotate stages (32/24/16/63) and essentially every subsequent add/xor -- stays scalar (rorxq/addq/xorq on GPRs) all the way through, only reassembling into ymm registers at the very end for the tuple-return store. Byte-granular rotate amounts (32/24/16 -- whole bytes, the exact shape that folds to vpshufb for u32's rotate(16)/(8)) did NOT fold to a shuffle here either -- all four rotate amounts, byte-granular or not, lowered to scalar rorxq identically. This directly answers the motivating question: no, u64 rotate does not get the u32 lane's free ride, not even for the byte-granular special case, not even with compile-time-constant rotate amounts (unlike rot_u64x8/x4's runtime-variable case, ruling out 'maybe it only needed a constant amount' as an explanation). Observed 22 packed (the leading vpaddq pair + the trailing reassembly moves) / 74 scalar-lane-arith (rorxq/addq/xorq on GPRs, automated count) / 14 loop-control (automated count) / 98 memory (lane extract/insert + loads/stores) / 2 other. KNOWN UNDER-COUNT: the 14 instructions the automated classifier placed in loop-control are, on manual inspection, ALSO genuine scalar lane arithmetic (xorq/addq on %r8/%rsi) -- they were misclassified because those two registers were used as memory-operand address bases earlier in the function (dereferencing the incoming d-pointer) before being overwritten and reused to carry rotated lane data, and the whole-block heuristic in scripts/codegen_oracle_analyze.py cannot distinguish a register's role before vs. after such a reuse (documented in that script's module docstring, 'KNOWN LIMITATION' paragraph). The true scalar-lane-arith count is therefore ~88, not 74; the qualitative verdict (packed only for the leading add and the trailing reassembly; scalar for the entire rotate-dependent chain in between) is unaffected either way and was cross-checked by hand against the raw --verbose disassembly." diff --git a/crates/simd-codegen-oracle/src/main.rs b/crates/simd-codegen-oracle/src/main.rs new file mode 100644 index 00000000..d8e9b63b --- /dev/null +++ b/crates/simd-codegen-oracle/src/main.rs @@ -0,0 +1,379 @@ +//! SIMD codegen oracle — probe kernels for `scripts/codegen-oracle.sh`. +//! +//! # Why this exists +//! +//! `.cargo/config.toml` pins `-Ctarget-cpu=x86-64-v3`. Because of that, this +//! crate's "scalar polyfill" SIMD storage types — macro-generated `[T; N]` +//! arrays, e.g. `avx2_int_type!(U32x16, u32, 16, 0u32)` at +//! `src/simd_avx2.rs:1542` — already compile to packed AVX2 instructions: +//! LLVM's loop/SLP vectorizer sees a fixed-trip-count loop over an aligned +//! array with no cross-lane data dependency and lowers it to `vpaddd` / +//! `vpxor` / `vpsrld` etc. A recent PR hand-wrote ~700 lines of intrinsics to +//! "fix" a gap that did not exist. This binary makes that a TESTED invariant +//! (Group A below) instead of a fact someone has to rediscover by reading +//! disassembly. +//! +//! The oracle must discriminate in BOTH directions: it must also confirm the +//! *absence* of vectorization where the shape of the code makes it +//! impossible (Group B) — a tool that reports "everything vectorizes" is as +//! useless as one that reports nothing does. +//! +//! Group C is neither: it is the open empirical question that motivated +//! extending this oracle — whether a hand-written scalar `u64` rotate loop +//! (mirroring the *actual* library pattern used for `U32x16::rotate_left`) +//! gets the same free ride from LLVM that the `u32` lane does. No backend in +//! this crate (avx512 / avx2 / scalar / neon / wasm / nightly) currently +//! defines a `rotate_left`/`rotate_right` on any `u64` lane type, so there is +//! no existing library function to call here — the probes below are written +//! by hand, deliberately mirroring `U32x16::rotate_left`'s shape line for +//! line, so the comparison is apples-to-apples. +//! +//! # Rules every probe kernel follows +//! +//! - `#[inline(never)]` so it survives as its own labeled symbol in the +//! emitted assembly (`scripts/codegen-oracle.sh` locates probes by name via +//! the `.type ,@function` label). +//! - Inputs arrive as parameters, never as `const`/literal-folded values — +//! `main` builds every input from a runtime seed (wall-clock time XORed +//! with argc, see `runtime_seed()` below) and pipes it through +//! `std::hint::black_box` before the call, so LLVM cannot constant-fold +//! into the callee even under an optimizer aggressive enough to see past +//! `#[inline(never)]` at the call site. +//! - The return value is consumed (folded into the printed report), so nothing +//! is dead-code-eliminated. + +use ndarray::simd::{add_mul_f32, I8x32, U32x16, U64x4, U64x8, U8x64}; +use std::hint::black_box; +use std::time::{SystemTime, UNIX_EPOCH}; + +// ============================================================================ +// Group A — expected to FULLY VECTORIZE +// ============================================================================ +// The scalar-source-is-not-scalar-codegen claim. Every kernel here operates +// on a full ndarray::simd lane type (or a fixed-width slice loop through the +// library's own `add_mul_f32`), with no loop-carried dependency *between* +// lanes and a compile-time-fixed lane count. LLVM has everything it needs to +// lower these to packed AVX2 without any hand-written intrinsic. + +/// The ChaCha/BLAKE ARX triple over `U32x16`: `(a+b)^b`, then `rotate_left(16)`. +/// `U32x16` is the `[u32; 16]` scalar-polyfill storage (`simd_avx2.rs`); its +/// `Add` / `BitXor` / `rotate_left` are themselves per-lane scalar loops in +/// the library source — this probe measures whether that scalar *source* +/// still lowers to packed AVX2 *codegen* at `-Ctarget-cpu=x86-64-v3`. +#[inline(never)] +pub fn arx_u32x16(a: U32x16, b: U32x16) -> U32x16 { + ((a + b) ^ b).rotate_left(16) +} + +/// Ten iterations of a ChaCha-style ARX double-round over four `U32x16` +/// lanes (16 independent ChaCha block-instances processed in parallel, one +/// per SIMD lane) — the actual production shape: a small, compile-time-fixed +/// trip count wrapped around several `U32x16` ARX ops per iteration. +#[inline(never)] +pub fn arx_rounds_u32x16(state: [U32x16; 4]) -> [U32x16; 4] { + let [mut a, mut b, mut c, mut d] = state; + for _ in 0..10 { + a += b; + d = (d ^ a).rotate_left(16); + c += d; + b = (b ^ c).rotate_left(12); + a += b; + d = (d ^ a).rotate_left(8); + c += d; + b = (b ^ c).rotate_left(7); + } + [a, b, c, d] +} + +/// Fused multiply-add into an accumulator slice via the library's own +/// `ndarray::simd::add_mul_f32` — built on `F32x16::mul_add` (native +/// `vfmadd*ps` on AVX2+FMA hosts, which `x86-64-v3` guarantees). +#[inline(never)] +pub fn fma_f32x16(acc: &mut [f32], a: &[f32], b: &[f32]) { + add_mul_f32(acc, a, b); +} + +/// Horizontal reduce over `U32x16` — `avx2_int_type!`'s `reduce_sum`, itself +/// a per-lane `wrapping_add` fold loop in the library source. +#[inline(never)] +pub fn reduce_u32x16(v: U32x16) -> u32 { + v.reduce_sum() +} + +/// Byte-lane bitwise chain over `U8x64`: `(a ^ b) & a | b`. +#[inline(never)] +pub fn bitwise_u8x64(a: U8x64, b: U8x64) -> U8x64 { + (a ^ b) & a | b +} + +// ============================================================================ +// Group B — expected to NOT fully vectorize +// ============================================================================ +// This half is the point. Do NOT "fix" these if they come out scalar — a +// scalar verdict here is the expected, correct finding. Each kernel's shape +// defeats auto-vectorization for a distinct, structural reason (loop-carried +// serial dependency, gather-shaped indexing, or an inherently cross-lane +// permute), independent of whatever the library's storage type happens to be. + +/// A loop where iteration `i` depends on iteration `i-1` through a +/// rotate+xor — an inherently serial ARX-shaped chain (as opposed to +/// `arx_rounds_u32x16` above, where the 10 "rounds" only chain full 16-lane +/// vector registers together, never individual scalar values). `n` is a +/// runtime parameter, not a compile-time constant, so LLVM cannot unroll and +/// then discover parallelism across iterations even if it wanted to — the +/// data dependency itself is nonlinear (rotate is not linear over XOR/add), +/// so there is no parallel prefix-scan trick available either. +#[inline(never)] +pub fn serial_dependent_chain(seed: u32, n: u32) -> u32 { + let mut x = seed; + for _ in 0..n { + x = x.rotate_left(7) ^ x.wrapping_add(0x9E37_79B9); + } + x +} + +/// Lane-wise `i8::saturating_abs` written as a scalar loop over +/// `I8x32::to_array()`. `abs(i8::MIN)` does not fit in `i8` (`+128` +/// overflows), which is exactly the correction recorded in +/// `.claude/knowledge/vertical-simd-consumer-contract.md` § "VPABSB +/// correction": `_mm512_abs_epi8` does NOT saturate `i8::MIN` by itself: the +/// real hardware-correct implementation needs `abs` + `min_epu8(_, 0x7f)`. +/// Whether LLVM's auto-vectorizer discovers that two-instruction idiom from +/// a scalar `saturating_abs()` call in a 32-iteration loop is exactly the +/// kind of thing this oracle exists to measure rather than assume. +#[inline(never)] +pub fn saturating_abs_i8x32(v: I8x32) -> I8x32 { + let arr = v.to_array(); + let mut out = [0i8; 32]; + for i in 0..32 { + out[i] = arr[i].saturating_abs(); + } + I8x32::from_array(out) +} + +/// Per-lane table lookup (gather-shaped: `table[idx[i]]`, no contiguous +/// load). LLVM's auto-vectorizer does not synthesize gather instructions +/// from a scalar indexed-load loop by default. +#[inline(never)] +pub fn gather_lookup_u8(table: &[u8; 256], idx: &[u8; 32]) -> [u8; 32] { + let mut out = [0u8; 32]; + for i in 0..32 { + out[i] = table[idx[i] as usize]; + } + out +} + +/// Zero-extend 16 × `u16` to `f32` via a scalar loop (`as` cast per lane). +#[inline(never)] +pub fn widening_u16_to_f32(v: [u16; 16]) -> [f32; 16] { + let mut out = [0.0f32; 16]; + for i in 0..16 { + out[i] = v[i] as f32; + } + out +} + +/// Reverse all 64 byte lanes of a `U8x64` via a scalar index loop — an +/// arbitrary cross-lane permute with no per-128-bit-lane locality (unlike +/// e.g. `unpack_lo_epi8` in `simd_avx2.rs`, which stays within 16-byte +/// sub-lanes and is realizable with `vpshufb`). +#[inline(never)] +pub fn cross_lane_reverse_u8x64(v: U8x64) -> U8x64 { + let arr = v.to_array(); + let mut out = [0u8; 64]; + for i in 0..64 { + out[i] = arr[63 - i]; + } + U8x64::from_array(out) +} + +// ============================================================================ +// Group C — UNKNOWN. Do not pre-classify. This is the open question. +// ============================================================================ +// No backend in this crate defines rotate_left/rotate_right on any u64 lane +// type today (measured: zero hits across avx512/avx2/scalar/neon/wasm/ +// nightly). BLAKE2b — which argon2 uses — is a 64-bit ARX cipher, so this is +// a genuine gap, not a stylistic one. The question: does a scalar-shaped u64 +// rotate loop, mirroring U32x16::rotate_left's own shape line for line, get +// the same free ride from LLVM that the u32 lane gets? If yes, the u64 ARX +// lane is free (no intrinsic needed, same as U32x16::rotate_left today). If +// no, this is the first legitimately justified case in this crate for a +// hand-written intrinsic override (AVX-512 has `_mm512_rorv_epi64` / +// VPROLVQ as a single instruction; AVX2 baseline has no direct equivalent). + +/// Mirrors `U32x16::rotate_left`'s exact shape (`simd_avx2.rs:1553`) at u64 +/// width, 8 lanes: `to_array()` → per-lane `u64::rotate_right` → `from_array()`. +/// `#[inline(always)]` so its expansion appears directly in the probe +/// symbols below (`rot_u64x8` / `blake2b_g_u64x8`), not behind a `call`. +#[inline(always)] +fn scalar_rotr_u64x8(v: U64x8, n: u32) -> U64x8 { + let arr = v.to_array(); + let mut out = [0u64; 8]; + for i in 0..8 { + out[i] = arr[i].rotate_right(n); + } + U64x8::from_array(out) +} + +/// 256-bit sibling of [`scalar_rotr_u64x8`] — 4 lanes. +#[inline(always)] +fn scalar_rotr_u64x4(v: U64x4, n: u32) -> U64x4 { + let arr = v.to_array(); + let mut out = [0u64; 4]; + for i in 0..4 { + out[i] = arr[i].rotate_right(n); + } + U64x4::from_array(out) +} + +/// Lane-wise `u64::rotate_right(n)` over `U64x8`, 8-wide (512-bit-equivalent +/// storage). The probe symbol itself, `#[inline(never)]`. +#[inline(never)] +pub fn rot_u64x8(v: U64x8, n: u32) -> U64x8 { + scalar_rotr_u64x8(v, n) +} + +/// Lane-wise `u64::rotate_right(n)` over `U64x4`, 4-wide (256-bit storage). +#[inline(never)] +pub fn rot_u64x4(v: U64x4, n: u32) -> U64x4 { + scalar_rotr_u64x4(v, n) +} + +/// One BLAKE2b G-function mixing step over `U64x8` — the u64 analogue of +/// `arx_rounds_u32x16` above, and the actual kernel an argon2/BLAKE2b lane +/// would need. Rotate amounts 32/24/16 are byte-granular (like ChaCha's 16 +/// and 8, which fold to `vpshufb` on the u32 lane); 63 is not — the +/// instruction histogram may show a split between shuffle-lowered and +/// shift-or-lowered rotates, which is itself a finding worth reporting +/// separately, not averaging away. +#[inline(never)] +pub fn blake2b_g_u64x8(a: U64x8, b: U64x8, c: U64x8, d: U64x8) -> (U64x8, U64x8, U64x8, U64x8) { + let mut a = a; + let mut b = b; + let mut c = c; + let mut d = d; + a += b; + d = scalar_rotr_u64x8(d ^ a, 32); + c += d; + b = scalar_rotr_u64x8(b ^ c, 24); + a += b; + d = scalar_rotr_u64x8(d ^ a, 16); + c += d; + b = scalar_rotr_u64x8(b ^ c, 63); + (a, b, c, d) +} + +// ============================================================================ +// Driver — runtime-derived inputs, every result consumed. +// ============================================================================ + +/// Wall-clock nanoseconds XORed with argc — a runtime value no build-time +/// constant folder can predict, used to seed a small PRNG for building probe +/// inputs. Piped through `black_box` at every call site below as well, so +/// nothing about a probe's actual argument values is visible to the +/// optimizer at compile time. +fn runtime_seed() -> u64 { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_nanos() as u64; + let argc = std::env::args().count() as u64; + black_box(nanos ^ argc.wrapping_mul(0x9E37_79B9_7F4A_7C15)) +} + +/// splitmix64 — deterministic given a seed, but the seed itself is +/// runtime-derived (see `runtime_seed`), so the sequence is not a +/// compile-time constant anywhere it is consumed. +struct SplitMix64(u64); + +impl SplitMix64 { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } +} + +fn main() { + let mut rng = SplitMix64(runtime_seed()); + let mut acc: u64 = 0; + + // ---- Group A ---- + let a16: [u32; 16] = std::array::from_fn(|_| rng.next() as u32); + let b16: [u32; 16] = std::array::from_fn(|_| rng.next() as u32); + let r = arx_u32x16(black_box(U32x16::from_array(a16)), black_box(U32x16::from_array(b16))); + acc ^= r.reduce_sum() as u64; + + let state: [U32x16; 4] = std::array::from_fn(|_| U32x16::from_array(std::array::from_fn(|_| rng.next() as u32))); + let rounds = arx_rounds_u32x16(black_box(state)); + for lane in &rounds { + acc ^= lane.reduce_sum() as u64; + } + + let n = 64usize; + let mut acc_v: Vec = (0..n).map(|_| (rng.next() as u32 as f32) * 1e-9).collect(); + let a_v: Vec = (0..n).map(|_| (rng.next() as u32 as f32) * 1e-9).collect(); + let b_v: Vec = (0..n).map(|_| (rng.next() as u32 as f32) * 1e-9).collect(); + fma_f32x16(black_box(&mut acc_v), black_box(&a_v), black_box(&b_v)); + acc ^= acc_v.iter().fold(0u64, |s, &x| s ^ x.to_bits() as u64); + + let rv = reduce_u32x16(black_box(U32x16::from_array(std::array::from_fn(|_| rng.next() as u32)))); + acc ^= rv as u64; + + let bw = bitwise_u8x64( + black_box(U8x64::from_array(std::array::from_fn(|_| rng.next() as u8))), + black_box(U8x64::from_array(std::array::from_fn(|_| rng.next() as u8))), + ); + acc ^= bw.reduce_sum() as u64; + + // ---- Group B ---- + let sdc = serial_dependent_chain(black_box(rng.next() as u32), black_box(50 + (rng.next() % 8) as u32)); + acc ^= sdc as u64; + + let sat = saturating_abs_i8x32(black_box(I8x32::from_array(std::array::from_fn(|_| rng.next() as i8)))); + acc ^= sat + .to_array() + .iter() + .map(|&x| x as i64 as u64) + .fold(0u64, |s, x| s ^ x); + + let mut table = [0u8; 256]; + for (i, t) in table.iter_mut().enumerate() { + *t = (i as u8).wrapping_mul(0x9B).wrapping_add(rng.next() as u8); + } + let idx: [u8; 32] = std::array::from_fn(|_| rng.next() as u8); + let gathered = gather_lookup_u8(black_box(&table), black_box(&idx)); + acc ^= gathered.iter().fold(0u64, |s, &x| s ^ x as u64); + + let u16in: [u16; 16] = std::array::from_fn(|_| rng.next() as u16); + let widened = widening_u16_to_f32(black_box(u16in)); + acc ^= widened.iter().fold(0u64, |s, &x| s ^ x.to_bits() as u64); + + let rev = cross_lane_reverse_u8x64(black_box(U8x64::from_array(std::array::from_fn(|_| rng.next() as u8)))); + acc ^= rev.reduce_sum() as u64; + + // ---- Group C ---- + let rot8 = rot_u64x8( + black_box(U64x8::from_array(std::array::from_fn(|_| rng.next()))), + black_box(1 + (rng.next() % 63) as u32), + ); + acc ^= rot8.reduce_sum(); + + let rot4 = rot_u64x4( + black_box(U64x4::from_array(std::array::from_fn(|_| rng.next()))), + black_box(1 + (rng.next() % 63) as u32), + ); + acc ^= rot4.reduce_sum(); + + let (ga, gb, gc, gd) = blake2b_g_u64x8( + black_box(U64x8::from_array(std::array::from_fn(|_| rng.next()))), + black_box(U64x8::from_array(std::array::from_fn(|_| rng.next()))), + black_box(U64x8::from_array(std::array::from_fn(|_| rng.next()))), + black_box(U64x8::from_array(std::array::from_fn(|_| rng.next()))), + ); + acc ^= ga.reduce_sum() ^ gb.reduce_sum() ^ gc.reduce_sum() ^ gd.reduce_sum(); + + println!("simd-codegen-oracle: probes executed, combined checksum = {acc:#018x}"); +} diff --git a/scripts/codegen-oracle.sh b/scripts/codegen-oracle.sh new file mode 100755 index 00000000..d65e8801 --- /dev/null +++ b/scripts/codegen-oracle.sh @@ -0,0 +1,73 @@ +#!/bin/sh +# SIMD codegen oracle -- proves what actually vectorizes (crates/simd-codegen-oracle), +# in both directions: Group A probes must show packed AVX2 instructions, Group B +# probes must show none, Group C probes are reported without a pass/fail verdict +# (the open question this oracle was extended to answer). Mirrors the build/locate +# shape of scripts/neon-parity.sh / scripts/wasm-parity.sh, but the analysis itself +# (instruction classification, baseline comparison) lives in the Python helper +# scripts/codegen_oracle_analyze.py -- see its module docstring for the exact +# packed-vector / scalar-lane-arith / loop-control / memory / other classification +# rules and the documented honesty rule (a loop-counter decl is not lane +# arithmetic). +# +# Usage: scripts/codegen-oracle.sh [target-triple] [-- --verbose] +# target-triple defaults to the host triple (`rustc -vV | grep ^host`). +# Everything after the target is forwarded to the analyzer (e.g. --verbose +# to print the raw instruction list per bucket). +set -eu + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +MANIFEST="$ROOT/crates/simd-codegen-oracle/Cargo.toml" + +TARGET="${1:-}" +case "$TARGET" in + "" | -*) + TARGET="$(rustc -vV | sed -n 's/^host: //p')" + ;; + *) + shift + ;; +esac +# Remaining args (an optional `--` and/or analyzer flags like --verbose) forward as-is. +if [ "${1:-}" = "--" ]; then + shift +fi + +BASELINE="$ROOT/crates/simd-codegen-oracle/baselines/$TARGET.toml" +if [ ! -f "$BASELINE" ]; then + echo "==> no baseline for target $TARGET at $BASELINE" >&2 + exit 90 +fi + +echo "==> building simd-codegen-oracle (--emit asm) for $TARGET" +if [ "$TARGET" = "$(rustc -vV | sed -n 's/^host: //p')" ]; then + cargo rustc --release --manifest-path "$MANIFEST" -- --emit asm -C debuginfo=0 +else + cargo rustc --release --manifest-path "$MANIFEST" --target "$TARGET" -- --emit asm -C debuginfo=0 +fi + +# Excluded crate -> cargo places `deps/` either under the crate's own target +# dir or falls back to the workspace target dir, exactly like neon/wasm-parity. +ASM="" +for CAND_ROOT in "$ROOT/crates/simd-codegen-oracle/target" "$ROOT/target"; do + if [ "$TARGET" = "$(rustc -vV | sed -n 's/^host: //p')" ]; then + CAND_DIR="$CAND_ROOT/release/deps" + else + CAND_DIR="$CAND_ROOT/$TARGET/release/deps" + fi + if [ -d "$CAND_DIR" ]; then + FOUND="$(ls -t "$CAND_DIR"/simd_codegen_oracle-*.s 2>/dev/null | head -n1 || true)" + if [ -n "$FOUND" ]; then + ASM="$FOUND" + break + fi + fi +done + +if [ -z "$ASM" ]; then + echo "==> could not locate emitted simd_codegen_oracle-*.s under $ROOT" >&2 + exit 91 +fi + +echo "==> analyzing $ASM against $BASELINE" +python3 "$ROOT/scripts/codegen_oracle_analyze.py" "$ASM" "$BASELINE" "$@" diff --git a/scripts/codegen_oracle_analyze.py b/scripts/codegen_oracle_analyze.py new file mode 100755 index 00000000..6e1d10a1 --- /dev/null +++ b/scripts/codegen_oracle_analyze.py @@ -0,0 +1,416 @@ +#!/usr/bin/env python3 +"""Instruction-histogram analyzer for `scripts/codegen-oracle.sh`. + +Parses the `--emit asm` output of `simd-codegen-oracle`, extracts each probe +kernel's assembly block by its `.type ,@function` label, classifies each +instruction, and compares the result against the committed baseline TOML. + +# Classification buckets + + packed-vector AVX/AVX2 instructions operating on a full ymm/zmm width + lane group: `vp*` (packed integer), `v*ps`/`v*pd` + (packed single/double), `vfmadd*ps`/`vfmadd*pd` etc., + `vmovaps`/`vmovups`/`vmovdqa`/`vmovdqu`, broadcast/ + permute/blend/extract-128/insert-128 forms. Explicitly + EXCLUDES the AVX *scalar* forms (`v*ss`/`v*sd`, single + lane despite the `v` prefix) and single-lane extract/ + insert forms (`vmovq`/`vmovd`/`vpextr*`/`vpinsr*`) -- + those move or compute exactly one lane, not a packed + width, and are bucketed as scalar-lane-arith / memory + respectively. + + scalar-lane-arith Scalar arithmetic that is NOT part of the loop-control + idiom -- i.e. GPR add/sub/and/or/xor/shl/shr/sar/rol/ + ror/rorx/imul/neg/not/inc/dec (or the AVX *scalar* + ss/sd arithmetic forms) operating on data, not on an + index/pointer/trip-count register. See "Loop-control + vs lane-arith" below for exactly how the two are told + apart -- this is the CRITICAL HONESTY distinction the + oracle exists to get right: a `decl` that decrements a + loop counter is scalar-ALU but does not touch lane + data, and must not be counted here. + + loop-control Conditional/unconditional jumps, `cmp`/`test`, `lea` + (address computation), and any GPR arithmetic + instruction identified as operating on an index / + pointer / trip-count register (see below). + + memory Plain `mov`-family (register<->register, register<-> + memory, zero/sign-extending loads), `push`/`pop`, and + single-lane vector<->GPR data movement (`vmovq`, + `vmovd`, `vpextr*`, `vpinsr*`) that is not itself an + arithmetic op. + + other Everything else (`vzeroupper`, `.cfi_*` already + stripped at extraction, prefetch/nop, etc). + +# Loop-control vs lane-arith (the honesty-critical rule) + +`cmp`/`test`/`j*`/`lea` are ALWAYS loop-control -- across every probe in this +oracle, no kernel performs a per-element scalar *comparison* or *branch* on +lane data (they are pure straight-line ARX/gather/widen/reverse kernels), so +this is a safe rule for this specific probe set, not a general-purpose +disassembler heuristic. + +For the remaining GPR arithmetic mnemonics, a register is classified as an +"index/pointer/bookkeeping register" for the whole block if, ANYWHERE in the +block, it is (a) used as a base or index register inside a memory operand +`offset(%base[,%index[,scale]])`, or (b) an operand of any `cmp`/`test` +instruction. An arithmetic instruction whose destination register is in that +set is loop-control; otherwise it is scalar-lane-arith. + +Two exceptions, both load-bearing: + + - `inc`/`dec` are ALWAYS loop-control (the spec's own canonical example: + a loop-counter decrement is scalar-ALU but never touches lane data). + - `rol`/`ror`/`rorx`/`rolx` (rotate, in any width) are ALWAYS + scalar-lane-arith. No probe in this oracle uses rotate for index + bookkeeping -- every rotate instruction that appears is a real ARX/ + BLAKE2b/ChaCha rotate on lane data, so treating rotate as + "index-adjacent" would hide exactly the finding this oracle exists to + surface. + +KNOWN LIMITATION (stated here, not hidden): this is a whole-block, name-based +heuristic, not a real dataflow/liveness analysis. A register that is +reused across two unrelated logical roles within one block (e.g. briefly +holding a copied pointer, then later reused -- after being overwritten -- +to hold a scalar data value) can be misclassified, because the heuristic +does not track *when* a register held which role, only *whether* it ever +played an address/compare role anywhere in the block. Measured impact: this +under-counts scalar-lane-arith in `blake2b_g_u64x8` specifically (a straight- +line, register-heavy kernel with real register reuse) by a handful of +instructions out of several dozen; the AGGREGATE verdict (substantial +scalar-lane-arith present, zero packed-vector coverage for the rotate- +dependent chain) is unaffected and was cross-checked by hand against the raw +disassembly. Every other probe in this oracle has no register-reuse-across- +roles and is classified exactly. +""" +import argparse +import re +import sys + +try: + import tomllib # Python 3.11+ +except ModuleNotFoundError: # pragma: no cover - CI runners pin 3.11+, this is a courtesy fallback + tomllib = None + + +PACKED_RE = re.compile( + r"^v(" + r"p(?!extr|insr)[a-z0-9]+" # vp* packed-integer, but not vpextr*/vpinsr* (single-lane) + r"|mova?p[sd]" + r"|movu?p[sd]" + r"|movdqa|movdqu" + r"|broadcast[a-z0-9]*" + r"|perm[a-z0-9]*" + r"|extracti[0-9]+" + r"|inserti[0-9]+" + r"|blendvb|blendps|blendpd" + r"|add[ps][sd]?p[sd]|addp[sd]|subp[sd]|mulp[sd]|divp[sd]" + r"|xorp[sd]|andnp[sd]|andp[sd]|orp[sd]" + r"|cmpp[sd]" + r"|f(n?m)?add[0-9]+p[sd]|f(n?m)?sub[0-9]+p[sd]" + r"|cvt[a-z0-9]*p[sd]|cvttp[sd]2[a-z0-9]+" + r"|roundp[sd]|maxp[sd]|minp[sd]|sqrtp[sd]" + r"|unpcklp[sd]|unpckhp[sd]|shufp[sd]|movmskp[sd]" + r")$" +) +# Single-lane AVX forms: NOT packed, despite the `v` prefix. +SCALAR_VECTOR_ARITH_RE = re.compile(r"^v(f(n?m)?add[0-9]+s[sd]|adds[sd]|subs[sd]|muls[sd]|divs[sd])$") +SCALAR_VECTOR_MOVE_RE = re.compile(r"^v(movs[sd]|movq|movd|pextr[bwdq]|pinsr[bwdq])$") + +ROTATE_RE = re.compile(r"^(rol|ror|rorx|rolx)[lqwb]?$") +INCDEC_RE = re.compile(r"^(inc|dec)[lqwb]?$") +GPR_ARITH_RE = re.compile(r"^(add|adc|sub|sbb|and|or|xor|shl|sal|shr|sar|imul|mul|neg|not)[lqwb]?$") +CMP_TEST_RE = re.compile(r"^(cmp|test)[lqwb]?$") +JUMP_RE = re.compile(r"^j[a-z]*$") +LEA_RE = re.compile(r"^lea[qwl]?$") +MOV_RE = re.compile(r"^(mov(z|s)?[bwlq]{0,2}|movabs[qlwb]?|push[qwl]?|pop[qwl]?)$") +NOP_RE = re.compile(r"^(nop[lwq]?|endbr(32|64)|ud2)$") + +REG_TOKEN_RE = re.compile(r"%([a-z][a-z0-9]*)") +IMM_RE = re.compile(r"\$-?(\d+)") + + +def normalize_reg(tok: str) -> str: + """Collapse register width aliases to one canonical family name.""" + t = tok.lower() + if re.fullmatch(r"r(\d+)[bwd]?", t): + return re.match(r"r\d+", t).group(0) # r8b/r8w/r8d -> r8 + table = { + "al": "ax", "ah": "ax", "ax": "ax", "eax": "ax", "rax": "ax", + "bl": "bx", "bh": "bx", "bx": "bx", "ebx": "bx", "rbx": "bx", + "cl": "cx", "ch": "cx", "cx": "cx", "ecx": "cx", "rcx": "cx", + "dl": "dx", "dh": "dx", "dx": "dx", "edx": "dx", "rdx": "dx", + "sil": "si", "si": "si", "esi": "si", "rsi": "si", + "dil": "di", "di": "di", "edi": "di", "rdi": "di", + "bpl": "bp", "bp": "bp", "ebp": "bp", "rbp": "bp", + "spl": "sp", "sp": "sp", "esp": "sp", "rsp": "sp", + } + return table.get(t, t) + + +def split_operands(rest: str): + """Split an AT&T operand list on top-level commas (parens protect memory operands).""" + depth = 0 + field = [] + out = [] + for ch in rest: + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + if ch == "," and depth == 0: + out.append("".join(field).strip()) + field = [] + else: + field.append(ch) + if field: + out.append("".join(field).strip()) + return out + + +def mem_operand_regs(operand: str): + """Registers used as base/index inside a memory operand, e.g. `16(%rdx,%rax,4)`.""" + m = re.search(r"\(([^)]*)\)", operand) + if not m: + return set() + return {normalize_reg(r) for r in REG_TOKEN_RE.findall(m.group(1))} + + +class Instr: + __slots__ = ("raw", "mnemonic", "operands") + + def __init__(self, raw: str): + self.raw = raw.strip() + parts = self.raw.split(None, 1) + self.mnemonic = parts[0].lower() if parts else "" + self.operands = split_operands(parts[1]) if len(parts) > 1 else [] + + +def is_instruction_line(line: str) -> bool: + s = line.strip() + if not s or s.startswith("#") or s.startswith("//"): + return False + if s.startswith("."): + return False + if s.endswith(":"): + return False + first = s.split(None, 1)[0] + return bool(re.match(r"^[a-z]", first, re.IGNORECASE)) + + +def extract_probe_block(lines, sym_needle: str): + """Return the raw instruction lines between `.type ,@function` and the + matching `ret`/`retq`, falling back to the next `.type ...,@function` label + (a different function starting) as a safety bound if no ret is found first.""" + start = None + for i, line in enumerate(lines): + if ".type" in line and "@function" in line and sym_needle in line: + start = i + break + if start is None: + return None + body = [] + i = start + 1 + # Skip the symbol's own label line (`SYM:`) if present. + if i < len(lines) and lines[i].strip().endswith(":"): + i += 1 + while i < len(lines): + line = lines[i] + stripped = line.strip() + if re.match(r"^ret[q]?$", stripped, re.IGNORECASE): + body.append(line) + break + if ".type" in line and "@function" in line: + # Next function started; no ret found (shouldn't happen for our + # probes, but bound the extraction defensively). + break + body.append(line) + i += 1 + return body + + +def classify_block(lines): + instrs = [Instr(l) for l in lines if is_instruction_line(l)] + + # Pass 1: collect the whole-block "bookkeeping register" set (addressing + # base/index registers, and any register compared/tested). + bookkeeping = set() + for ins in instrs: + for op in ins.operands: + bookkeeping |= mem_operand_regs(op) + if CMP_TEST_RE.match(ins.mnemonic): + for op in ins.operands: + bookkeeping |= {normalize_reg(r) for r in REG_TOKEN_RE.findall(op)} + + counts = {"packed-vector": 0, "scalar-lane-arith": 0, "loop-control": 0, "memory": 0, "other": 0} + detail = {k: [] for k in counts} + + for ins in instrs: + m = ins.mnemonic + bucket = None + + if PACKED_RE.match(m): + bucket = "packed-vector" + elif SCALAR_VECTOR_ARITH_RE.match(m): + bucket = "scalar-lane-arith" + elif SCALAR_VECTOR_MOVE_RE.match(m): + bucket = "memory" + elif JUMP_RE.match(m) or CMP_TEST_RE.match(m) or LEA_RE.match(m): + bucket = "loop-control" + elif INCDEC_RE.match(m): + bucket = "loop-control" # canonical loop-counter idiom, never lane data + elif ROTATE_RE.match(m): + bucket = "scalar-lane-arith" # never index bookkeeping in this probe set + elif GPR_ARITH_RE.match(m): + dest = normalize_reg(REG_TOKEN_RE.findall(ins.operands[-1])[0]) if ins.operands and REG_TOKEN_RE.findall(ins.operands[-1]) else None + bucket = "loop-control" if dest in bookkeeping else "scalar-lane-arith" + elif MOV_RE.match(m): + bucket = "memory" + elif NOP_RE.match(m) or m in ("vzeroupper", "vzeroall"): + bucket = "other" + else: + bucket = "other" + + counts[bucket] += 1 + detail[bucket].append(ins.raw) + + return counts, detail + + +def load_baseline(path): + with open(path, "rb") as f: + if tomllib is not None: + return tomllib.load(f) + return _parse_toml_fallback(f.read().decode("utf-8")) + + +def _parse_toml_fallback(text: str): + """Minimal TOML subset parser (only what baselines/*.toml actually uses): + `[section.name]` headers and `key = value` where value is a bare word, + quoted string, or integer. Used only if `tomllib` is unavailable.""" + data = {} + section = data + for raw_line in text.splitlines(): + line = raw_line.split("#", 1)[0].strip() + if not line: + continue + if line.startswith("[") and line.endswith("]"): + keys = line[1:-1].split(".") + node = data + for k in keys: + node = node.setdefault(k, {}) + section = node + continue + if "=" in line: + k, v = line.split("=", 1) + k = k.strip() + v = v.strip() + if v.startswith('"') and v.endswith('"'): + v = v[1:-1] + elif re.fullmatch(r"-?\d+", v): + v = int(v) + section[k] = v + return data + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("asm_path") + ap.add_argument("baseline_path") + ap.add_argument("--verbose", action="store_true") + args = ap.parse_args() + + with open(args.asm_path, "r", errors="replace") as f: + lines = f.readlines() + + baseline = load_baseline(args.baseline_path) + probes = baseline.get("probe", {}) + # Stable order: declaration order in the baseline file isn't preserved by + # a naive dict in older parsers; sort by name for determinism instead. + names = sorted(probes.keys()) + + print("=" * 100) + print("simd-codegen-oracle -- instruction histogram") + print( + "NOTE: 'scalar arithmetic on lane data' EXCLUDES loop-control-idiom ops\n" + " (cmp/test/jumps/lea, and any GPR arithmetic on an index/pointer/\n" + " trip-count register, including inc/dec). A loop-counter decl does\n" + " NOT count as lane arithmetic. See this script's module docstring\n" + " for the exact rule and its documented limitation." + ) + print("=" * 100) + header = f"{'probe':<26}{'packed':>8}{'lane-arith':>12}{'loop-ctl':>10}{'memory':>8}{'other':>8} verdict" + print(header) + print("-" * len(header)) + + exit_mask = 0 + any_missing = False + for idx, name in enumerate(names): + spec = probes[name] + needle = f"{len(name)}{name}" + block = extract_probe_block(lines, needle) + if block is None: + print(f"{name:<26} MISSING FROM ASM (symbol not found)") + any_missing = True + exit_mask |= 1 << idx + continue + counts, detail = classify_block(block) + + expect = spec.get("expect", "unknown") + min_packed = int(spec.get("min_packed", 0)) + max_scalar = spec.get("max_scalar_lane_arith") + max_packed = spec.get("max_packed") + + ok = True + reasons = [] + if expect == "vectorized": + if counts["packed-vector"] < min_packed: + ok = False + reasons.append(f"packed={counts['packed-vector']} < min_packed={min_packed}") + if max_scalar is not None and counts["scalar-lane-arith"] > int(max_scalar): + ok = False + reasons.append(f"lane-arith={counts['scalar-lane-arith']} > max_scalar_lane_arith={max_scalar}") + elif expect == "scalar": + cap = int(max_packed) if max_packed is not None else 0 + if counts["packed-vector"] > cap: + ok = False + reasons.append(f"packed={counts['packed-vector']} > max_packed={cap}") + elif expect == "unknown": + pass # Group C: report only, no pass/fail assertion. + else: + ok = False + reasons.append(f"unrecognized expect={expect!r} in baseline") + + if expect == "unknown": + verdict = "OBSERVE" + elif ok: + verdict = "PASS" + else: + verdict = "FAIL: " + "; ".join(reasons) + exit_mask |= 1 << idx + + print( + f"{name:<26}{counts['packed-vector']:>8}{counts['scalar-lane-arith']:>12}" + f"{counts['loop-control']:>10}{counts['memory']:>8}{counts['other']:>8} {verdict}" + ) + if args.verbose: + for bucket in ("packed-vector", "scalar-lane-arith", "loop-control"): + if detail[bucket]: + print(f" [{bucket}]") + for raw in detail[bucket]: + print(f" {raw}") + + print("-" * len(header)) + if any_missing: + print("ERROR: one or more probes were not found in the emitted assembly.") + if exit_mask == 0: + print("simd-codegen-oracle: ALL PROBES MATCH BASELINE EXPECTATIONS") + else: + print(f"simd-codegen-oracle: FAILURES (exit bitmask 0x{exit_mask:x}) -- see FAIL rows above") + return exit_mask + + +if __name__ == "__main__": + sys.exit(main()) From 4ce42d6ecd31d1d3cce2d801e4fb03327a565042 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 05:46:47 +0000 Subject: [PATCH 2/4] oracle: declare the measured baseline instead of inheriting it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The oracle's first CI run failed with packed=0 on every Group-A probe — arx_rounds_u32x16 came back 0 packed / 146 scalar / 491 memory against 52 packed / 0 scalar locally. Not codegen drift: no AVX2 at all. Cause, verified empirically: $ cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*' target-cpu=x86-64-v3 $ RUSTFLAGS="-D warnings" cargo build ... | grep -o 'target-cpu=...' (empty) Cargo's RUSTFLAGS env var REPLACES `[target.'cfg(...)'].rustflags` from .cargo/config.toml; they do not merge. ci.yaml:23 sets `RUSTFLAGS: "-D warnings"` at workflow level, so the v3 baseline is silently dropped for every job that inherits it. Fix: the script now DECLARES its baseline, passing `-C target-cpu=x86-64-v3` after the `--` so it lands on the final rustc invocation and wins over ambient RUSTFLAGS. An oracle that inherits its baseline measures a different machine depending on where it runs — precisely the class of error it exists to catch. Verified with RUSTFLAGS="-D warnings" exported: all 13 probes match. Also recorded in the TD-T22 artifact, because it narrows that finding: "the scalar polyfill compiles to packed AVX2" holds UNDER the v3 baseline, and CI never had it. The baseline does the work, not the source form. NOT changed here: whether CI itself should build at v3 so tests exercise the shipped instruction set. That flips every job's codegen and carries the SIGILL risk TD-SIMD-1 documented — an operator decision, not a drive-by fix in a tooling PR. tier4-avx512-check is unaffected; it uses the more specific CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS, which does not collide. --- .claude/knowledge/td-t22-asm-investigation.md | 50 +++++++++++++++++++ scripts/codegen-oracle.sh | 36 ++++++++++++- 2 files changed, 84 insertions(+), 2 deletions(-) diff --git a/.claude/knowledge/td-t22-asm-investigation.md b/.claude/knowledge/td-t22-asm-investigation.md index 1c960dae..31b0a0e0 100644 --- a/.claude/knowledge/td-t22-asm-investigation.md +++ b/.claude/knowledge/td-t22-asm-investigation.md @@ -172,3 +172,53 @@ shipped ~700 lines of hand-written intrinsics on the premise that these polyfills executed scalar code. They do not. The PR was closed on the operator's `U32x8`-composition ruling; this investigation is the deliverable the ticket originally asked for. + +--- + +## ⚠ CI DOES NOT APPLY THE v3 BASELINE (found 2026-07-29 by the oracle) + +Everything above was measured on a developer host, where +`.cargo/config.toml` supplies `-Ctarget-cpu=x86-64-v3`. **On CI it does +not.** The oracle's first CI run reported `packed = 0` for every Group-A +probe — `arx_rounds_u32x16` came back 0 packed / 146 scalar / 491 memory, +against 52 packed / 0 scalar locally. + +**Cause:** `.github/workflows/ci.yaml:23` sets `RUSTFLAGS: "-D warnings"` at +workflow level, and cargo's `RUSTFLAGS` env var **replaces** +`[target.'cfg(...)'].rustflags` from `.cargo/config.toml` — the two do not +merge. Verified locally: + +```sh +$ cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*' +target-cpu=x86-64-v3 +$ RUSTFLAGS="-D warnings" cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*' + # (empty — flag silently dropped) +``` + +### Consequences + +1. **Every CI job that inherits the workflow `env:` compiles at baseline + x86-64 (SSE2)** — `tests/{stable,beta,1.95.0}`, `clippy`, + `native-backend`, `hpc-stream-parallel`. Not the AVX2 tier developers + build and run locally. CI has been testing different machine code than + anyone reviews. +2. **The comment in `.cargo/config.toml` — "This is what GitHub CI runs + against" — is false**, and has been since `RUSTFLAGS` was added. +3. **TD-T22's finding is unaffected but narrower than stated.** "The scalar + polyfill compiles to packed AVX2" is true *under the v3 baseline*. It is + the baseline, not the source form, that does the work — and CI never had + it. `tier4-avx512-check` is unaffected because it uses the more specific + `CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS`, which does not collide. + +### What was done here + +`scripts/codegen-oracle.sh` now **declares** its baseline instead of +inheriting it, passing `-C target-cpu=x86-64-v3` after the `--` so it lands +on the final rustc invocation and wins regardless of ambient `RUSTFLAGS`. +Confirmed: with `RUSTFLAGS="-D warnings"` exported, all 13 probes match. + +**The wider question — whether CI should build at v3 so tests exercise the +code that ships — is deliberately NOT changed here.** Making every CI job +switch instruction sets is an operator decision with real consequences +(SIGILL risk on runners without AVX2 is what TD-SIMD-1 was about), not a +drive-by fix in a tooling PR. diff --git a/scripts/codegen-oracle.sh b/scripts/codegen-oracle.sh index d65e8801..79c3caa8 100755 --- a/scripts/codegen-oracle.sh +++ b/scripts/codegen-oracle.sh @@ -39,11 +39,43 @@ if [ ! -f "$BASELINE" ]; then exit 90 fi +# The measured baseline is DECLARED here, never inherited from the ambient +# environment. This is load-bearing: +# +# `.cargo/config.toml` sets `-Ctarget-cpu=x86-64-v3` via +# `[target.'cfg(target_arch = "x86_64")'].rustflags`, but cargo's RUSTFLAGS +# env var REPLACES that config wholesale — the two do not merge. CI sets +# `RUSTFLAGS: "-D warnings"` at workflow level (.github/workflows/ci.yaml), +# so on CI the target-cpu flag is silently DROPPED and everything compiles +# at baseline x86-64 (SSE2). Verified: +# $ cargo build -v | grep target-cpu -> target-cpu=x86-64-v3 +# $ RUSTFLAGS="-D warnings" cargo build -v | grep target-cpu -> (empty) +# +# An oracle that inherits the ambient baseline therefore measures a +# DIFFERENT machine's codegen depending on where it runs, which is exactly +# the class of error it exists to catch. Passing `-C target-cpu` after the +# `--` puts it on the final rustc invocation, where it wins regardless of +# RUSTFLAGS. +case "$TARGET" in + x86_64-*) BASELINE_CPU="x86-64-v3" ;; + *) BASELINE_CPU="" ;; +esac + +if [ -n "$BASELINE_CPU" ]; then + CPU_FLAG="-C target-cpu=$BASELINE_CPU" + echo "==> baseline: $TARGET @ target-cpu=$BASELINE_CPU (declared, not inherited)" +else + CPU_FLAG="" + echo "==> baseline: $TARGET @ target default (no target-cpu override)" +fi + echo "==> building simd-codegen-oracle (--emit asm) for $TARGET" if [ "$TARGET" = "$(rustc -vV | sed -n 's/^host: //p')" ]; then - cargo rustc --release --manifest-path "$MANIFEST" -- --emit asm -C debuginfo=0 + # shellcheck disable=SC2086 + cargo rustc --release --manifest-path "$MANIFEST" -- --emit asm -C debuginfo=0 $CPU_FLAG else - cargo rustc --release --manifest-path "$MANIFEST" --target "$TARGET" -- --emit asm -C debuginfo=0 + # shellcheck disable=SC2086 + cargo rustc --release --manifest-path "$MANIFEST" --target "$TARGET" -- --emit asm -C debuginfo=0 $CPU_FLAG fi # Excluded crate -> cargo places `deps/` either under the crate's own target From 78ac854b8629fb92370928b64d06ba8546aa38a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 05:49:31 +0000 Subject: [PATCH 3/4] oracle: remove the CI job; fix a silent-success exit status Two changes. 1. The oracle is no longer a CI job (operator ruling). A standing check that parses assembly to catch a designer skipping a measurement is machinery guarding a lapse the design step should prevent. The question "does this vectorize?" is answered once, during design, and recorded; it is not re-answered on every commit. The crate and script remain as an on-demand instrument. `.github/workflows/ci.yaml` job and its `conclusion.needs` entry removed; YAML re-validated. 2. `codegen_oracle_analyze.py` returned the raw failure bitmask as the process exit status. Unix truncates exit status to the low 8 bits, so a failure confined to probe index >= 8 exits 0x100/0x200/... and the shell observes 0 -- the script printed "FAILURES" and reported success. Demonstrated: a probe-9-only failure exited 512, seen as 0. With probes sorted alphabetically this covered rot_u64x4, rot_u64x8, saturating_abs_i8x32, serial_dependent_chain and widening_u16_to_f32 -- both u64-rotate probes among them, the finding the tool exists to protect. Now exits 1 on any failure; the bitmask stays in the printed line for diagnosis. Found by Codex review. Docs updated to stop claiming CI enforcement and to state why the tool is deliberately on-demand. --- .claude/knowledge/simd-one-spec-design.md | 32 ++++++++++++++++++++--- .github/workflows/ci.yaml | 17 ------------ scripts/codegen_oracle_analyze.py | 16 +++++++++--- 3 files changed, 41 insertions(+), 24 deletions(-) diff --git a/.claude/knowledge/simd-one-spec-design.md b/.claude/knowledge/simd-one-spec-design.md index 9a3a352c..9457b36b 100644 --- a/.claude/knowledge/simd-one-spec-design.md +++ b/.claude/knowledge/simd-one-spec-design.md @@ -57,7 +57,8 @@ simd_type! { // Lane-wise ops. Emitted as the scalar loop form for EVERY backend. // LLVM vectorizes them under the pinned target-cpu baseline; the - // codegen oracle proves it, per target, in CI. + // an on-demand codegen probe proves it, per target, when the question + // is actually open. NOT a standing CI job — see "On tooling" below. lanewise: [ add(wrapping), sub(wrapping), mul(wrapping), and, or, xor, not, @@ -133,8 +134,9 @@ shift-or composition written explicitly. ## Staged migration (not one PR) -1. **Oracle first.** `crates/simd-codegen-oracle` + CI job. Without it the - entry criterion is unenforceable and this design is just a refactor. +1. **Answer the codegen question once, in the design.** Run + `scripts/codegen-oracle.sh` for the type family under consideration and + record the result. This is a design activity, not a standing check. 2. **Characterize.** Run the oracle across x86-64-v3 / v4 / aarch64 / wasm32. Produce the per-target table of what vectorizes and what doesn't. That table *is* the specification of which intrinsic overrides are legitimate. @@ -169,6 +171,28 @@ shift-or composition written explicitly. - Adding a lane type: one declaration instead of a five-file change. - Ten currently-unlowered AVX2 int types: free. -- The "is this fast enough?" argument: replaced by a CI check. +- The "is this fast enough?" argument: answered by a twenty-minute + measurement during design, instead of debated. + +## On tooling — why this is NOT a CI job + +`crates/simd-codegen-oracle` is an **on-demand instrument**, deliberately +not wired into CI (operator ruling, 2026-07-29). A standing job that parses +assembly to catch a designer skipping the measurement is machinery guarding +against a lapse the design step should prevent — it institutionalizes the +lapse instead of fixing it, and it carries permanent cost: an asm parser, a +per-target baseline, and brittleness that already bit once (the job's first +run failed because it inherited its baseline from the ambient environment +rather than declaring it). + +The question "does this vectorize?" is answered **once, during design**, and +the answer goes in a doc. It is not re-answered on every commit. + +Run it when a codegen question is genuinely open: + +```sh +sh scripts/codegen-oracle.sh # host, x86-64-v3 baseline +sh scripts/codegen-oracle.sh aarch64-unknown-linux-gnu +``` - ~13k LoC of hand-maintained backend code: substantially reduced, with the remainder being exactly the intrinsics that earn their place. diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 75f92942..f2fbcab2 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -163,22 +163,6 @@ jobs: - name: NEON SIMD parity (cross-build + qemu run) run: ./scripts/neon-parity.sh - simd-codegen-oracle: - # Proves (in both directions) what `.cargo/config.toml`'s - # `-Ctarget-cpu=x86-64-v3` baseline actually does to this crate's - # "scalar polyfill" SIMD storage types: Group A probes must show packed - # AVX2 codegen from scalar *source*, Group B probes must show none, and - # Group C probes (the u64 rotate gap) are reported without a pass/fail - # verdict. See crates/simd-codegen-oracle/src/main.rs and - # scripts/codegen_oracle_analyze.py for the full picture. - runs-on: ubuntu-latest - name: simd-codegen-oracle/instruction-histogram - steps: - - uses: actions/checkout@v4 - - uses: dtolnay/rust-toolchain@stable - - name: SIMD codegen oracle (build --emit asm + classify + baseline compare) - run: ./scripts/codegen-oracle.sh - tests: runs-on: ubuntu-latest needs: pass-msrv @@ -404,7 +388,6 @@ jobs: - nostd - wasm_simd - neon_simd - - simd-codegen-oracle - tests - native-backend - hpc-stream-parallel diff --git a/scripts/codegen_oracle_analyze.py b/scripts/codegen_oracle_analyze.py index 6e1d10a1..b8ff3778 100755 --- a/scripts/codegen_oracle_analyze.py +++ b/scripts/codegen_oracle_analyze.py @@ -407,9 +407,19 @@ def main(): print("ERROR: one or more probes were not found in the emitted assembly.") if exit_mask == 0: print("simd-codegen-oracle: ALL PROBES MATCH BASELINE EXPECTATIONS") - else: - print(f"simd-codegen-oracle: FAILURES (exit bitmask 0x{exit_mask:x}) -- see FAIL rows above") - return exit_mask + return 0 + + # The bitmask is DIAGNOSTIC OUTPUT ONLY, never the exit status. + # + # Unix truncates an exit status to its low 8 bits. Returning the raw + # mask means a failure confined to probe index >= 8 exits 0x100 / 0x200 + # / ..., which the shell observes as 0: the script prints "FAILURES" + # and reports success. With probes sorted alphabetically that silently + # swallowed regressions in rot_u64x4, rot_u64x8, saturating_abs_i8x32, + # serial_dependent_chain and widening_u16_to_f32 -- including both + # u64-rotate probes, the finding this tool exists to protect. + print(f"simd-codegen-oracle: FAILURES (probe bitmask 0x{exit_mask:x}) -- see FAIL rows above") + return 1 if __name__ == "__main__": From 605e33ad278b5e0acc701b9397661b5b5facd93b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 05:54:31 +0000 Subject: [PATCH 4/4] oracle: relocate to .claude/knowledge as an investigation instrument The tool is knowledge, not a product artifact. It was adding a crate, two scripts, and a workspace-exclude entry to a published library in order to answer a question that gets asked during design and then recorded. Moved crates/simd-codegen-oracle + scripts/codegen-oracle.sh + scripts/codegen_oracle_analyze.py into .claude/knowledge/simd-codegen-oracle/ (probes.rs, analyze.py, run.sh, baseline-x86_64-v3.toml, README.md). Reverted the Cargo.toml exclude entry. Net effect on the shipped crate: this PR now changes exactly one dependency line (blake3 -> default-features = false, features = ["pure"]) plus its comment. Nothing else outside .claude/. The README carries the runnable recipe, the recorded per-probe results, the declared-baseline rationale, and the exit-status bug, so the instrument is reproducible from where it lives. --- .claude/knowledge/crypto-lane-status.md | 2 +- .../knowledge/simd-codegen-oracle/README.md | 117 ++++++++++++ .../knowledge/simd-codegen-oracle/analyze.py | 0 .../baseline-x86_64-v3.toml | 0 .../knowledge/simd-codegen-oracle/probes.rs | 0 .../knowledge/simd-codegen-oracle/run.sh | 0 .claude/knowledge/simd-one-spec-design.md | 10 +- .claude/knowledge/td-t22-asm-investigation.md | 2 +- Cargo.toml | 1 - crates/simd-codegen-oracle/Cargo.lock | 170 ------------------ crates/simd-codegen-oracle/Cargo.toml | 31 ---- 11 files changed, 124 insertions(+), 209 deletions(-) create mode 100644 .claude/knowledge/simd-codegen-oracle/README.md rename scripts/codegen_oracle_analyze.py => .claude/knowledge/simd-codegen-oracle/analyze.py (100%) rename crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml => .claude/knowledge/simd-codegen-oracle/baseline-x86_64-v3.toml (100%) rename crates/simd-codegen-oracle/src/main.rs => .claude/knowledge/simd-codegen-oracle/probes.rs (100%) rename scripts/codegen-oracle.sh => .claude/knowledge/simd-codegen-oracle/run.sh (100%) delete mode 100644 crates/simd-codegen-oracle/Cargo.lock delete mode 100644 crates/simd-codegen-oracle/Cargo.toml diff --git a/.claude/knowledge/crypto-lane-status.md b/.claude/knowledge/crypto-lane-status.md index 7658cc40..b789e737 100644 --- a/.claude/knowledge/crypto-lane-status.md +++ b/.claude/knowledge/crypto-lane-status.md @@ -62,7 +62,7 @@ what LLVM would do with it. ### ANSWERED by the oracle: no, it does not vectorize -Measured on x86_64 v3 via `crates/simd-codegen-oracle`: +Measured on x86_64 v3 via `.claude/knowledge/simd-codegen-oracle/`: | probe | packed | scalar lane-arith | verdict | |---|---|---|---| diff --git a/.claude/knowledge/simd-codegen-oracle/README.md b/.claude/knowledge/simd-codegen-oracle/README.md new file mode 100644 index 00000000..206e2a4e --- /dev/null +++ b/.claude/knowledge/simd-codegen-oracle/README.md @@ -0,0 +1,117 @@ +# SIMD codegen oracle — an on-demand instrument + +Answers one question with evidence: **does this lane operation actually +vectorize, or does the source only look like it should?** + +Lives here, not in `crates/` or `scripts/`, because it is an investigation +instrument rather than a product artifact. It ships no code, adds no +workspace member, and runs no CI job. + +## When to use it + +When a codegen question is genuinely open — before hand-writing intrinsics, +before "lowering" a scalar-looking polyfill, before claiming a lane is slow. +Answer it once, record the answer in a doc, move on. + +**Not a standing check.** A permanent job that re-answers a settled question +on every commit is machinery guarding a lapse the design step should +prevent. Measure during design instead. + +## Files + +| file | role | +|---|---| +| `probes.rs` | 13 `#[inline(never)]` kernels over `ndarray::simd` types | +| `analyze.py` | extracts per-symbol instruction histograms from emitted asm, classifies, compares to baseline | +| `run.sh` | builds with `--emit asm`, locates the `.s`, invokes the analyzer | +| `baseline-x86_64-v3.toml` | expected class assertions per probe | + +## Running it + +The tool needs a throwaway crate to compile the probes against `ndarray`: + +```sh +cd /path/to/ndarray +mkdir -p /tmp/oracle/src +cp .claude/knowledge/simd-codegen-oracle/probes.rs /tmp/oracle/src/main.rs +cat > /tmp/oracle/Cargo.toml <<'EOF' +[package] +name = "simd-codegen-oracle" +version = "0.0.0" +edition = "2021" +[dependencies] +ndarray = { path = "/path/to/ndarray", features = ["std"] } +[profile.release] +debug = false +EOF + +cargo rustc --release --manifest-path /tmp/oracle/Cargo.toml -- \ + --emit asm -C debuginfo=0 -C target-cpu=x86-64-v3 +python3 .claude/knowledge/simd-codegen-oracle/analyze.py \ + "$(ls -t /tmp/oracle/target/release/deps/simd_codegen_oracle-*.s | head -1)" \ + .claude/knowledge/simd-codegen-oracle/baseline-x86_64-v3.toml +``` + +`run.sh` automates this against a crate laid out as above; adjust +`MANIFEST`/`BASELINE` at the top for your scratch location. + +## Two properties that make the result trustworthy + +**The baseline is declared, never inherited.** `-C target-cpu=x86-64-v3` is +passed on the final rustc invocation. This is load-bearing: cargo's +`RUSTFLAGS` env var *replaces* `[target.'cfg(…)'].rustflags` from +`.cargo/config.toml` — they do not merge — so a tool that inherits its +baseline measures a different machine depending on where it runs. Verified: + +```sh +$ cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*' +target-cpu=x86-64-v3 +$ RUSTFLAGS="-D warnings" cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*' + # (empty — silently dropped) +``` + +**Scalar arithmetic on lane data is separated from loop control**, so a +trip-counter `decl` is never miscounted as scalar lane work. The distinction +matters: "zero scalar instructions" is almost always false (a loop counter +exists); "no scalar op touches lane data" is the claim that means something. + +## Recorded results + +Measured on x86_64 + `x86-64-v3`, rustc 1.95.0. Full narrative in +`../td-t22-asm-investigation.md` and `../crypto-lane-status.md`. + +| probe | packed | scalar (lane data) | lowering | +|---|---|---|---| +| `arx_rounds_u32x16` | 52 | 0 | `vpaddd`/`vpxor`/`vpshufb`; 8 `vpaddd` per round = the AVX2 floor for 64 lanes | +| `arx_u32x16` | 11 | 0 | `rotate_left(16)` → `vpshufb` | +| `reduce_u32x16` | 8 | 0 | logarithmic reduction tree | +| `fma_f32x16` | 28 | 0 | `vfmadd213ps` | +| `bitwise_u8x64` | 12 | 0 | packed | +| `saturating_abs_i8x32` | 4 | 0 | `vpxor`/`vpsubsb`/`vpblendvb` — synthesized the abs+clamp trick unprompted | +| `widening_u16_to_f32` | 6 | 0 | `vpmovzxwd` + `vcvtdq2ps` | +| `cross_lane_reverse_u8x64` | 9 | 0 | `vbroadcasti128`/`vpshufb`/`vpermq` — invented a cross-lane permute from a scalar index loop | +| **`rot_u64x8`** | **0** | 8 | scalar `rorq %cl`, one per lane | +| **`rot_u64x4`** | **0** | 4 | scalar `rorq %cl`, one per lane | +| **`blake2b_g_u64x8`** | 22 | ~88 | packed leading add; scalar through all four rotates | +| `gather_lookup_u8` | 0 | 0 | `movzbl` chain, no arithmetic | +| `serial_dependent_chain` | 0 | 27 | loop-carried dependency | + +**The headline:** LLVM vectorizes far more than intuition suggests — +including cross-lane permutes, widening converts, and saturating +arithmetic, all from plain scalar loops. It does **not** vectorize u64 +rotates, even for byte-granular amounts that fold to `vpshufb` at u32 +width, and even though `vpsllq`/`vpsrlq` are available and it uses exactly +that shift-or for u32 `rotate_left(12)`/`(7)`. + +That single row is why the tool was worth building: it is the one place a +hand-written intrinsic is currently justified, and it is what argon2 needs +(BLAKE2b is a 64-bit ARX cipher). + +## A bug worth remembering + +`analyze.py` originally returned its failure bitmask as the process exit +status. Unix truncates exit status to the low 8 bits, so a failure confined +to probe index ≥ 8 exits `0x100`/`0x200`/… and the shell observes **0** — +printing `FAILURES` while reporting success. Alphabetically that covered +both u64-rotate probes. Now it exits 1 on any failure, with the bitmask +kept as diagnostic text. diff --git a/scripts/codegen_oracle_analyze.py b/.claude/knowledge/simd-codegen-oracle/analyze.py similarity index 100% rename from scripts/codegen_oracle_analyze.py rename to .claude/knowledge/simd-codegen-oracle/analyze.py diff --git a/crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml b/.claude/knowledge/simd-codegen-oracle/baseline-x86_64-v3.toml similarity index 100% rename from crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml rename to .claude/knowledge/simd-codegen-oracle/baseline-x86_64-v3.toml diff --git a/crates/simd-codegen-oracle/src/main.rs b/.claude/knowledge/simd-codegen-oracle/probes.rs similarity index 100% rename from crates/simd-codegen-oracle/src/main.rs rename to .claude/knowledge/simd-codegen-oracle/probes.rs diff --git a/scripts/codegen-oracle.sh b/.claude/knowledge/simd-codegen-oracle/run.sh similarity index 100% rename from scripts/codegen-oracle.sh rename to .claude/knowledge/simd-codegen-oracle/run.sh diff --git a/.claude/knowledge/simd-one-spec-design.md b/.claude/knowledge/simd-one-spec-design.md index 9457b36b..35a41283 100644 --- a/.claude/knowledge/simd-one-spec-design.md +++ b/.claude/knowledge/simd-one-spec-design.md @@ -78,7 +78,7 @@ simd_type! { **The entry criterion is the whole point.** Today "should this be an intrinsic?" is answered by intuition, and intuition said yes to a case where LLVM was already at the instruction floor. Under this design the question is -answered by `scripts/codegen-oracle.sh`: if the generic form vectorizes, the +answered by the oracle (`.claude/knowledge/simd-codegen-oracle/`): if the generic form vectorizes, the override is rejected; if it doesn't, the override is justified and the probe that justified it is committed alongside. @@ -135,7 +135,7 @@ shift-or composition written explicitly. ## Staged migration (not one PR) 1. **Answer the codegen question once, in the design.** Run - `scripts/codegen-oracle.sh` for the type family under consideration and + the oracle (`.claude/knowledge/simd-codegen-oracle/`) for the type family under consideration and record the result. This is a design activity, not a standing check. 2. **Characterize.** Run the oracle across x86-64-v3 / v4 / aarch64 / wasm32. Produce the per-target table of what vectorizes and what doesn't. That @@ -176,7 +176,7 @@ shift-or composition written explicitly. ## On tooling — why this is NOT a CI job -`crates/simd-codegen-oracle` is an **on-demand instrument**, deliberately +`.claude/knowledge/simd-codegen-oracle/` is an **on-demand instrument**, deliberately not wired into CI (operator ruling, 2026-07-29). A standing job that parses assembly to catch a designer skipping the measurement is machinery guarding against a lapse the design step should prevent — it institutionalizes the @@ -191,8 +191,8 @@ the answer goes in a doc. It is not re-answered on every commit. Run it when a codegen question is genuinely open: ```sh -sh scripts/codegen-oracle.sh # host, x86-64-v3 baseline -sh scripts/codegen-oracle.sh aarch64-unknown-linux-gnu +see .claude/knowledge/simd-codegen-oracle/README.md # host, x86-64-v3 baseline +see .claude/knowledge/simd-codegen-oracle/README.md aarch64-unknown-linux-gnu ``` - ~13k LoC of hand-maintained backend code: substantially reduced, with the remainder being exactly the intrinsics that earn their place. diff --git a/.claude/knowledge/td-t22-asm-investigation.md b/.claude/knowledge/td-t22-asm-investigation.md index 31b0a0e0..610e548b 100644 --- a/.claude/knowledge/td-t22-asm-investigation.md +++ b/.claude/knowledge/td-t22-asm-investigation.md @@ -212,7 +212,7 @@ $ RUSTFLAGS="-D warnings" cargo build -p ndarray --lib -v | grep -o 'target-cpu= ### What was done here -`scripts/codegen-oracle.sh` now **declares** its baseline instead of +the oracle (`.claude/knowledge/simd-codegen-oracle/`) now **declares** its baseline instead of inheriting it, passing `-C target-cpu=x86-64-v3` after the `--` so it lands on the final rustc invocation and wins regardless of ambient `RUSTFLAGS`. Confirmed: with `RUSTFLAGS="-D warnings"` exported, all 13 probes match. diff --git a/Cargo.toml b/Cargo.toml index e8689c66..a93181ae 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -431,7 +431,6 @@ exclude = [ "crates/burn", "crates/wasm-simd-parity", "crates/neon-simd-parity", - "crates/simd-codegen-oracle", "vendor/chacha20", ] default-members = [ diff --git a/crates/simd-codegen-oracle/Cargo.lock b/crates/simd-codegen-oracle/Cargo.lock deleted file mode 100644 index 42a296a7..00000000 --- a/crates/simd-codegen-oracle/Cargo.lock +++ /dev/null @@ -1,170 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "arrayref" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" - -[[package]] -name = "arrayvec" -version = "0.7.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "blake3" -version = "1.8.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" -dependencies = [ - "arrayref", - "arrayvec", - "cc", - "cfg-if", - "constant_time_eq", - "cpufeatures", -] - -[[package]] -name = "cc" -version = "1.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9" -dependencies = [ - "find-msvc-tools", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "constant_time_eq" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" - -[[package]] -name = "cpufeatures" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" -dependencies = [ - "libc", -] - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "libc" -version = "0.2.189" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" - -[[package]] -name = "matrixmultiply" -version = "0.3.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7" -dependencies = [ - "autocfg", - "rawpointer", -] - -[[package]] -name = "ndarray" -version = "0.17.2" -dependencies = [ - "blake3", - "matrixmultiply", - "num-complex", - "num-integer", - "num-traits", - "paste", - "portable-atomic", - "portable-atomic-util", - "rawpointer", -] - -[[package]] -name = "num-complex" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-integer" -version = "0.1.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - -[[package]] -name = "portable-atomic" -version = "1.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "rawpointer" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "simd-codegen-oracle" -version = "0.0.0" -dependencies = [ - "ndarray", -] diff --git a/crates/simd-codegen-oracle/Cargo.toml b/crates/simd-codegen-oracle/Cargo.toml deleted file mode 100644 index e8115274..00000000 --- a/crates/simd-codegen-oracle/Cargo.toml +++ /dev/null @@ -1,31 +0,0 @@ -# simd-codegen-oracle — proves (or disproves) what actually vectorizes. -# -# Context: `.cargo/config.toml` pins `-Ctarget-cpu=x86-64-v3`, so this crate's -# "scalar polyfill" SIMD storage types (macro-generated `[T; N]`, e.g. -# `avx2_int_type!(U32x16, u32, 16, 0u32)` in `src/simd_avx2.rs`) already -# compile to packed AVX2 via LLVM's loop/SLP vectorizer — they are not scalar -# codegen despite being scalar *source*. This crate is the tested proof of -# that claim, in both directions: probes that DO vectorize (Group A) and -# probes that provably DO NOT (Group B), plus a third bucket (Group C) whose -# classification is the open question being measured (the u64 rotate gap). -# -# EXCLUDED from the workspace (see root Cargo.toml `exclude`) so it has zero -# effect on the default build/test jobs; CI's `simd-codegen-oracle` job -# builds + runs it via --manifest-path, mirroring `neon-simd-parity` / -# `wasm-simd-parity`. -[package] -name = "simd-codegen-oracle" -version = "0.0.0" -edition = "2021" -publish = false - -[[bin]] -name = "simd-codegen-oracle" -path = "src/main.rs" - -[dependencies] -ndarray = { path = "../..", default-features = false, features = ["std"] } - -[profile.release] -debug = false -panic = "abort"