oracle: repair the run script, then use it — chacha20 reach and the BLAKE3 shuffle gap

.claude/knowledge/blake3-on-ndarray-simd.md

@@ -0,0 +1,213 @@
+# BLAKE3 without C and without intrinsics — what is actually missing
+
+> **Status: MEASURED, 2026-07-29.** Every number below comes from
+> `.claude/knowledge/simd-codegen-oracle/` on x86_64 @ `x86-64-v3`,
+> rustc 1.95.0. The intrinsic census comes from `AdaWorldAPI/BLAKE3`
+> at `8aa5145`.
+
+## READ BY:
+- Anyone porting BLAKE3 onto `ndarray::simd`
+- Anyone about to hand-write shuffle intrinsics for a lane type
+- `simd-savant`, `truth-architect`
+
+## P0 TRIGGER
+About to write `_mm256_unpacklo_epi32` wrappers because `U32x16` has no
+shuffle surface? **Don't. Measured below: the interleave primitive compiles
+to a real packed permute from a plain scalar index loop. What is missing is
+a method, not an intrinsic.**
+
+---
+
+## The starting position
+
+Two constraints, both operator-set:
+
+1. **No FFI with C.** `blake3` is now pinned to `default-features = false,
+   features = ["pure"]`, which removed 33 `.o` objects and
+   `libblake3_avx512_assembly.a` from the build. Measured, and already
+   shipped.
+2. **No second SIMD surface.** `pure` gets rid of the C, but its Rust
+   backends (`rust_avx2.rs`, `rust_sse41.rs`, `rust_sse2.rs`,
+   `wasm32_simd.rs` — 2,910 lines) are raw `core::arch` intrinsics. That is
+   exactly what the matryoshka pattern exists to prevent: all SIMD lives
+   once, audited, inside `ndarray::simd`.
+
+So the question is what a `ndarray::simd`-native BLAKE3 backend would need.
+
+## The census: 15 intrinsics, and 12 already exist
+
+`src/rust_avx2.rs` (496 lines) uses fifteen distinct intrinsics. Grouped by
+whether `crate::simd::U32x16` can already express them:
+
+| intrinsic | uses | `U32x16` today |
+|---|---|---|
+| `_mm256_add_epi32` | 1 | `Add` |
+| `_mm256_xor_si256` | 1 | `BitXor` |
+| `_mm256_or_si256` | 4 | `BitOr` |
+| `_mm256_slli_epi32` / `_mm256_srli_epi32` | 4 / 4 | folded into `rotate_left` |
+| `_mm256_set1_epi32` | 1 | `splat` |
+| `_mm256_setr_epi32` / `_mm256_set_epi32` | 1 / 1 | `from_array` |
+| `_mm256_loadu_si256` / `_mm256_storeu_si256` | 1 / 1 | `from_slice` / `copy_to_slice` |
+| **`_mm256_unpacklo_epi32` / `_mm256_unpackhi_epi32`** | **4 / 4** | **absent** |
+| **`_mm256_unpacklo_epi64` / `_mm256_unpackhi_epi64`** | **4 / 4** | **absent** |
+| **`_mm256_permute2x128_si256`** | **2** | **absent** |
+
+The twelve available ones cover the whole compression core. The three
+missing families — 18 call sites — exist for one thing: the transpose in
+`hash_many`, which turns N chunk states into word-major vectors so the
+rounds run N ways in parallel.
+
+`U32x16` is generated by `avx2_int_type!` and exposes `splat`, `from_slice`,
+`from_array`, `to_array`, `copy_to_slice`, `reduce_sum`, the operators, and
+`rotate_left`. There is no shuffle surface at all.
+
+## Four measurements
+
+The tempting move is to hand-write the missing shuffles as intrinsics. TD-T22
+is the standing reason not to assume that: a 700-line intrinsic PR was closed
+after measurement showed LLVM was already at the instruction floor. So the
+shuffles got measured first.
+
+| probe | packed | scalar (lane data) | what LLVM emitted |
+|---|---|---|---|
+| `blake3_g_u32x16` | **72** | 0 | `vpshufb` ×2 (rotr 16, rotr 8), `vpsrld`+`vpslld`+`vpor` ×2 (rotr 12, rotr 7), `vpaddd`/`vpxor` throughout |
+| `interleave_lo_u32x16` | **11** | 0 | `vbroadcasti128` + `vpmovzxdq` + `vpermd` (constant mask) + `vpblendd $170` |
+| `transpose_16x16_u32` | **0** | 1 | 1088 B of stack, a 256-iteration scalar element copy, stride-64 scatter |
+| `transpose_stage_u32x16` | **27** | 1 | `vunpcklps` / `vunpckhps` — one 32-bit stage only |
+| `transpose_16x16_composed` | **79** | 0 | 60 vector moves + **19 shuffles**: `vinserti128` ×7, `vpshufd` ×3, `vpblendd` ×3, `vpunpcklqdq` ×2, `vpermq` ×2, `vunpcklpd`, `vpermpd` |
+
+Rows 3 and 5 are the same transpose, correctness-checked to be the same
+function. The difference is only how it is written.
+
+### The G function is free
+
+72 packed instructions, zero scalar on lane data. BLAKE3 specifies *right*
+rotations by 16/12/8/7 and this crate has no `rotate_right` at any width, so
+each is written `rotate_left(32 - n)` — exact, since rotation is modular. The
+resulting amounts split as the u32 lane always does: byte-granular 16 and 24
+fold to `vpshufb`, non-byte-granular 20 and 25 take the shift-or triple.
+
+**`compress_in_place` and `compress_xof` need nothing new.** That half of the
+port is a transcription.
+
+### The interleave primitive is free
+
+11 packed, zero scalar, from a plain `for i in 0..8 { out[2i]=a[i];
+out[2i+1]=b[i] }`. LLVM synthesized a two-source cross-lane permutation —
+`vpermd` against a constant mask, then `vpblendd`.
+
+This extends `cross_lane_reverse_u8x64`'s earlier result (a *single*-source
+permute synthesized from a scalar reverse loop) to the two-source case, which
+was the open half. Both were written as index loops; both came out as real
+shuffles.
+
+### The monolithic transpose is not free
+
+Zero packed. LLVM allocated 1088 bytes of stack (`subq $1088,%rsp`, 64-byte
+aligned) and emitted a genuine 256-iteration element-at-a-time copy —
+`cmpq $256,%rax` / `jne` — striding the output by 64 bytes each step.
+
+The stride-64 scatter is what defeats it: each output vector gathers one
+element from each of sixteen inputs. Nothing in the loop looks like a
+shuffle to the vectorizer.
+
+### …but the composed transpose is
+
+A complete, correctness-checked 16×16 transpose, written as four butterfly
+stages at granularities 1 / 2 / 4 / 8 (32-bit unpack, 64-bit unpack, 128-bit
+lane exchange, 256-bit half exchange) over a `const G` parameter so every
+shuffle pattern is compile-time constant: **79 packed, zero scalar on lane
+data**, of which 19 are real shuffles — `vinserti128`, `vpshufd`,
+`vpblendd`, `vpunpcklqdq`, `vpermq`, `vunpcklpd`, `vpermpd`. All four
+granularities appear in the emitted code, including the 64-bit unpack and the
+256-bit cross-lane permute.
+
+The single-stage probe (`transpose_stage_u32x16`, 27 packed, `vunpcklps` /
+`vunpckhps`) remains in the oracle as a narrower data point, but it is **not**
+what supports the conclusion. It is one 32-bit stage, exercises neither wider
+exchange, and its whole-vector helper semantics do not compose into a real
+transpose — so an earlier version of this document that rested the conclusion
+on it was inferring, not measuring. Raised by codex on PR #265; the finding
+was correct, and `transpose_16x16_composed` is the answer to it.
+
+**Correctness is checked, not asserted.** The driver compares the composed
+result against a naive nested-loop transpose and aborts on mismatch, and
+`run.sh` now executes the probe binary — `--emit asm` links nothing, so the
+assertion would otherwise never run. The check was control-tested by deleting
+`stage::<8>`, which makes it fire (exit 101). A packed-but-wrong network
+cannot be reported here as a success.
+
+**Two honest caveats.** Unlike the single stage, this is not clean
+straight-line code: LLVM kept loop structure (38 loop-control instructions)
+and there is real spill traffic (41 memory). That is expected — sixteen
+512-bit vectors of live state are 32 ymm registers against 16 architectural
+ones, and a hand-written intrinsic backend faces exactly the same pressure.
+So the measurement establishes that the shuffle network is *synthesized*
+rather than degraded to scalar copies. It does not establish throughput
+parity with `rust_avx2.rs`.
+
+## What this means
+
+**No hand-written intrinsic is earned.** Under the entry criterion in
+`simd-one-spec-design.md`, an override needs a probe showing the generic form
+fails. The generic form does not fail here — the primitive vectorizes, and so
+does a composition of primitives. Only the monolithic index-loop spelling
+fails, and that is a spelling, not a capability.
+
+So what is missing is a **method surface on `U32x16`**, implemented as the
+scalar index loops LLVM already handles. The measured probe suggests it wants
+to be *one* const-generic operation rather than five named ones:
+
+```rust
+fn exchange<const G: usize>(lo: Self, hi: Self) -> (Self, Self)
+```
+
+`G = 1` is the 32-bit unpack, `G = 2` the 64-bit unpack, `G = 4` the 128-bit
+lane exchange, `G = 8` the 256-bit half exchange:
+
+| `G` | intrinsic role | replaces in `rust_avx2.rs` |
+|---|---|---|
+| 1 | 32-bit unpack | `_mm256_unpack{lo,hi}_epi32` (8 sites) |
+| 2 | 64-bit unpack | `_mm256_unpack{lo,hi}_epi64` (8 sites) |
+| 4 | 128-bit lane exchange | `_mm256_permute2x128_si256` (2 sites) |
+| 8 | 256-bit half exchange | (no AVX2 analogue — a 512-bit lane needs it) |
+
+About fifteen lines, once. **Zero `unsafe`, zero `core::arch`, zero C.** The
+transpose is a composition of four stages over it — which is what the
+intrinsic backends already are, spelled in the lane vocabulary instead of in
+x86. The measured `transpose_16x16_composed` probe is that composition,
+verified to be a correct transpose.
+
+That is the whole gap. It is a method surface, not a porting effort.
+
+## Deliberate scope limits
+
+- **Degree 16, not 8.** `U32x16` is the lane the substrate uses, so the
+  natural backend matches BLAKE3's AVX-512 degree rather than AVX2's degree 8.
+  Degree 8 would want a `U32x8`, ruled out as a building block (operator
+  ruling, 2026-07-28).
+- **`transpose_stage_u32x16` measures codegen shape, not correctness.** Its
+  helpers use whole-vector interleave semantics rather than x86's
+  per-128-bit-lane `unpack`, so four stages of *those* helpers do not compose
+  a correct transpose. Getting the lane semantics right is a detail of the
+  real implementation; the question the probe answers is whether composing
+  interleave calls stays packed, and it does.
+- **Nothing here is measured on aarch64 or wasm32.** The oracle needs a
+  baseline per target and only `baseline-x86_64-v3.toml` exists.
+- **No benchmark.** Every claim above is about *instruction class*, not
+  throughput. "Emits packed shuffles" is not "is faster than the intrinsic
+  backend" — that needs a bench against `rust_avx2.rs`, which has not been
+  run.
+
+## Still open
+
+1. **Does the composed transpose beat `rust_avx2.rs`?** Unmeasured on both
+   sides. The instruction-class result says a port is *possible* without
+   intrinsics; it does not say it wins.
+2. **The FIPS gate.** If any deployment needs FIPS-adjacent claims, BLAKE3 is
+   out and SHA-384 stays the KDF hash (`crypto-lane-status.md`). Settle
+   before investing in the port.
+3. **`hash_many`'s degree contract.** BLAKE3's `Platform::simd_degree()`
+   feeds chunk scheduling. A degree-16 backend on an AVX2 host is legal but
+   changes the parallelism/transpose-cost balance — worth measuring, not
+   assuming.

.claude/knowledge/chacha20-vendoring-blast-radius.md

@@ -0,0 +1,181 @@
+# `vendor/chacha20` — what it actually reaches
+
+> **Status: MEASURED, 2026-07-29.** Every row below is read from `Cargo.lock`,
+> a manifest on disk, `rustc --print cfg`, or a git object hash. No estimates.
+
+## READ BY:
+- Anyone about to re-sync, extend, or delete `vendor/chacha20`
+- Anyone citing "the AdaWorldAPI chacha20 fork" as an accelerated dependency
+- Anyone applying the P0 fork rule to a crate that also has a vendored copy
+
+## P0 TRIGGER
+About to state that a consumer's ChaCha20 keystream is accelerated because
+ndarray patches `chacha20`? **Read the two reach tables first. The patch and
+the acceleration have different, and both narrower-than-assumed, reach.**
+
+---
+
+## Two different questions, two different answers
+
+"Does the fork reach consumer X?" is really two questions, and conflating
+them is what produced the overstated claims in the manifest comments.
+
+1. **Does the `[patch.crates-io]` apply** — i.e. does the build link the
+   vendored source at all?
+2. **Does `ndarray_simd` compile** — i.e. does the added backend get
+   selected, or does the vendored crate fall through to RustCrypto's own?
+
+A build can answer yes to (1) and no to (2). Most do.
+
+## Reach of the patch
+
+`[patch.crates-io]` takes effect **only from the top-level workspace
+manifest**. It is ignored in a dependency's manifest, so ndarray's patch
+does not travel with ndarray — each consumer must declare its own.
+
+| repo | declares a chacha20 patch | links the vendored source |
+|---|---|---|
+| ndarray | yes — `Cargo.toml:488` | **yes** |
+| MedCare-rs | yes — `vendor/ndarray/vendor/chacha20` | **yes** |
+| lance-graph | no | no — registry crate |
+| OGAR | no | no — registry crate |
+| tesseract-rs | no | no — registry crate |
+| stockfish-rs | no | no — registry crate |
+| a2ui-rs | no | no — registry crate |
+
+Evidence for the two "yes" rows: ndarray's `Cargo.lock` carries
+`chacha20 0.9.1` with **no `source` line**, which is how a path override
+appears; MedCare-rs's root manifest declares
+`chacha20 = { path = "vendor/ndarray/vendor/chacha20" }`, and its
+`vendor/ndarray` symlink canonicalizes to the same checkout, so one
+`ndarray` node resolves.
+
+Evidence for the five "no" rows: `grep 'chacha20'` and `grep '\[patch'`
+across each root manifest. lance-graph's manifest contains only a comment
+*about* patching, explicitly recording that no `[patch]` block exists there.
+
+## Reach of the `ndarray_simd` backend
+
+`vendor/chacha20/src/backends.rs` selects the added backend under
+
+```rust
+all(target_arch = "x86_64", target_feature = "avx512f", …)   // or
+all(target_arch = "wasm32",  target_feature = "simd128")
+```
+
+`avx512f` is a **compile-time** feature cfg, so it is present only if the
+build pins a target-cpu that includes it. Measured:
+
+```console
+$ rustc --print cfg -Ctarget-cpu=x86-64-v3 | grep avx
+target_feature="avx2"            # <- no avx512f
+$ rustc --print cfg -Ctarget-cpu=x86-64-v4 | grep avx512f
+target_feature="avx512f"
+```
+
+| build | target-cpu | `avx512f` | backend compiled |
+|---|---|---|---|
+| ndarray, default | `x86-64-v3` (`.cargo/config.toml`) | absent | RustCrypto soft/avx2/sse2 |
+| ndarray, `--config .cargo/config-avx512.toml` | `x86-64-v4` | present | **`ndarray_simd`** |
+| MedCare-rs, default | none — no `.cargo/config.toml` at all | absent | RustCrypto soft/avx2/sse2 |
+| wasm32 + `simd128` | — | n/a | **`ndarray_simd`** |
+| the five unpatched repos | — | — | not present in the source they link |
+
+**No default build of any repo in the workspace runs `ndarray_simd`.** Two
+opt-in configurations do: an explicitly-selected AVX-512 x86_64 build, and a
+wasm32 build with `simd128`.
+
+This is not a defect — the fallthrough is correct and the comments in
+MedCare-rs's manifest describe it accurately ("non-avx512 builds fall through
+to RustCrypto's own backends"). It is a scope fact that keeps being stated
+one size too large.
+
+## One manifest comment overstates it
+
+`Cargo.toml:481-482` reads:
+
+> This transitively accelerates the `encryption` crate's XChaCha20-Poly1305
+> keystream … **on any x86_64+avx512f build (the workspace's
+> `target-cpu=x86-64-v4`)**.
+
+The workspace's pinned target-cpu is **v3**, in `.cargo/config.toml`. `v4`
+is the opt-in `.cargo/config-avx512.toml`. The conditional half of the
+sentence is right; the parenthetical asserts the default build satisfies it,
+and it does not.
+
+**Not edited here.** Manifest string changes in this repo need an operator
+ruling; the finding is recorded instead of applied.
+
+## The fork the P0 rule names is not this tree
+
+`AdaWorldAPI/stream-ciphers` exists and is now cloned in-session.
+
+| | `AdaWorldAPI/stream-ciphers` | `ndarray/vendor/chacha20` |
+|---|---|---|
+| what it is | mirror of RustCrypto upstream | hand-copied source tree |
+| head | `5f3430b` "Release chacha20 v0.10.1 (#574)" | — |
+| branches | `master` only | — |
+| chacha20 version | **0.10.1** | **0.9.1** |
+| `cipher` | 0.5 | 0.4.4 |
+| edition / rust-version | 2024 / 1.85 | 2021 / 1.95 |
+| `backends/ndarray_simd.rs` | **absent** | present |
+| `backends/avx512.rs` | **present (upstream's own)** | absent |
+| `repository` field | RustCrypto | rewritten to `AdaWorldAPI/ndarray` |
+
+Three consequences worth stating plainly:
+
+1. **The fork carries no AdaWorldAPI delta**, and this is checkable rather
+   than inferred from a branch count. Against `RustCrypto/stream-ciphers`
+   fetched as `upstream`:
+
+   ```console
+   $ git rev-parse HEAD
+   5f3430b7531a33aa14957b6bd407b46687635124
+   $ git rev-parse upstream/master
+   5f3430b7531a33aa14957b6bd407b46687635124
+   $ git rev-list --count upstream/master..HEAD
+   0
+   $ git rev-parse HEAD:chacha20 upstream/master:chacha20
+   3f58304be34b79a3a38430a5752fb8db223d1a20
+   3f58304be34b79a3a38430a5752fb8db223d1a20
+   ```
+
+   The fork's head *is* upstream's head — same commit hash, zero commits
+   ahead, and the `chacha20/` subtree hashes to the same object. Git tree
+   hashes cover content recursively, so identical tree ids mean byte equality
+   for that directory. "Depend on the fork" and "depend on upstream" are
+   therefore the same bytes today — as a verified fact, not an inference.
+   (Repeat the check after any upstream sync; it is one command.)
+2. **The vendored tree is not a checkout of the fork**, and is one minor
+   release behind it. It cannot be re-synced by `git pull`; the re-sync
+   procedure its own header describes (bump the source, re-apply the backend
+   and the four cfg branches, re-run the vectors) is a manual port, and
+   0.9.1 → 0.10.1 crosses a `cipher` major (0.4 → 0.5).
+3. **Upstream now ships its own AVX-512 backend.** `backends/avx512.rs`
+   at 0.10.1 occupies the exact niche `ndarray_simd.rs` was written to fill.
+   Whether the ndarray-lane version still earns its place is a measurement
+   question — and the instrument for it already exists
+   (`.claude/knowledge/simd-codegen-oracle/`), with the relevant lane already
+   measured: the u32 ARX triple on `U32x16` hits the AVX2 instruction floor
+   (`td-t22-asm-investigation.md`).
+
+## What is NOT claimed here
+
+- Not that the vendored backend is wrong. It is untested by default builds,
+  which is a different statement.
+- Not that the patch should be removed. That is a decision, and it depends on
+  (3) above plus the AVX-512 deployment question, neither of which this audit
+  settles.
+- Not that MedCare-rs's wiring is incorrect. It is the one consumer that
+  declared its patch deliberately and documented the fallthrough honestly.
+
+## Open, for an operator ruling
+
+1. **Point `vendor/chacha20` at the fork, or keep the copy?** The P0 rule
+   says depend on the AdaWorldAPI fork. Today the tree is neither — a copy of
+   a version the fork does not carry, with `repository` rewritten to point at
+   ndarray itself.
+2. **Does `ndarray_simd` still beat upstream's own `avx512.rs`?** Answerable
+   with the oracle. Unmeasured today, on both sides.
+3. **The v3/v4 manifest comment** — correct in place, or leave the record and
+   annotate?