Merge pull request #268 from AdaWorldAPI/claude/simd-ladder-plan

The SIMD ladder: the cycle map, portable BLAKE3, and the u64 ARX rotate

Author: AdaWorldAPI

.claude/board/EPIPHANIES.md

@@ -1,5 +1,89 @@
 # ndarray — Epiphanies (append-only)
 
+## 2026-07-29 — Which PACKAGE pulls a dep decides whether it can consume you
+**Status:** FINDING
+**Scope:** @simd-savant @truth-architect domain:build-graph
+**Cross-ref:** PR #267, `.claude/knowledge/the-simd-ladder.md`
+
+The ladder's goal is that every dependency carrying its own SIMD instead
+consumes `ndarray::simd`. Whether a given dependency *can* is not a property
+of the dependency, and not of "the workspace" — it is decided by **which
+package in the workspace pulls it**:
+
+| dependency | entry point | can consume `ndarray::simd`? |
+|---|---|---|
+| `chacha20` | `crates/encryption` → … → `chacha20` | **yes** |
+| `curve25519-dalek` | `crates/encryption` → `ed25519-dalek` → … | **yes** |
+| `blake3` | **root `ndarray`** (`std` feature) | **no — cycle** |
+
+Measured — and measured the right way round. The evidence is the **positive**
+reverse tree (which terminates at `encryption`, with root `ndarray` absent),
+plus a control proving the method can produce a hit:
+
+```console
+$ cargo tree -p encryption -i curve25519-dalek     # positive: full path shown
+curve25519-dalek v4.1.3
+└── ed25519-dalek v2.2.0
+    └── encryption v0.1.0 (crates/encryption)
+
+$ cargo tree -p ndarray -i blake3                  # control: a real edge DOES hit
+blake3 v1.8.4
+└── ndarray v0.17.2 (/workspace/ndarray)
+```
+
+**Not** an error message. A first draft rested on
+`cargo tree -p ndarray -i curve25519-dalek` failing with `package ID
+specification ... did not match any packages`; a typo produces that message
+byte-for-byte, so it cannot distinguish "no such edge" from "no such
+package". Corrected by CodeRabbit on #268 — and it is the same defect this
+repo keeps hitting one level down: a check that cannot fail for the reason
+you think it is failing.
+
+Only blake3 is pulled by the ROOT package, so only blake3 closes a loop when
+it depends back on ndarray. The other two ride a shape that already works in
+this repo today — `crates/encryption` → `chacha20` → `ndarray(root)` is
+exactly it.
+
+**Why this is worth recording rather than re-derived.** "ndarray depends on
+X, so X can't depend on ndarray" is the intuitive rule and it is wrong at
+workspace granularity. A sub-crate is a different package; the cycle is
+per-package, not per-workspace. Reasoning at workspace granularity says all
+three are blocked, which would have parked two rungs that have no blocker at
+all.
+
+**On the diagnostic — be precise, because two different things were
+conflated here.** Cargo *does* report the cycle when the patched package
+would be selected; `cargo update -p blake3` printed the chain
+(`blake3 ... satisfies dependency of ndarray ... satisfies path dependency
+ndarray of blake3`). What is NOT a cycle diagnostic is `[[patch.unused]]`.
+That only says the patch was not selected, and the usual causes are a version
+that does not satisfy the requirement or a stale lockfile.
+
+An earlier version of this entry read the unused patch as the cycle's
+signature and called the failure "silent." Both halves were wrong, and codex
+caught it on #268. Treating `[[patch.unused]]` as a cycle report teaches the
+next reader to misdiagnose an ordinary stale patch — and undermines the very
+check this entry prescribes.
+
+The check stands, but ONLY in its positive form — and the wording here was
+itself an instance of the bug it warns about, corrected on #268:
+
+Consequence: **before planning any "make X consume our crate" work, run
+`cargo tree -p <our-root> -i <X>`.**
+
+1. First confirm `<X>` is a real package in the selected graph (a typo, or a
+   package absent from that graph, produces the byte-identical error).
+2. A tree that resolves and shows the root package = the edge must be cut
+   first.
+3. A tree that resolves and does NOT show it = unblocked.
+4. **A package-ID error is inconclusive — never "no cycle".**
+
+An earlier draft of this very entry said "an error there means no cycle and
+the work is unblocked", one paragraph after explaining that the error is
+ambiguous. Read literally it would mark blocked work as unblocked on the
+strength of a typo.
+
+
 ## 2026-07-29 — BLAKE3 needs a method surface, not intrinsics (measured)
 **Status:** FINDING
 **Scope:** @simd-savant domain:codec

.claude/knowledge/blake3-ab-bench/blake3_ab.rs

@@ -0,0 +1,33 @@
+//! A/B: ndarray's in-tree BLAKE3 vs the external `blake3` crate, on the input
+//! sizes ndarray actually hashes. Throwaway; deleted after the measurement.
+use std::time::Instant;
+
+fn bench<F: FnMut() -> [u8; 32]>(name: &str, iters: u32, mut f: F) -> f64 {
+    let mut sink = 0u8;
+    for _ in 0..iters / 10 { sink ^= f()[0]; }          // warm
+    let t = Instant::now();
+    for _ in 0..iters { sink ^= f()[0]; }
+    let ns = t.elapsed().as_nanos() as f64 / iters as f64;
+    std::hint::black_box(sink);
+    println!("    {name:<12} {ns:>10.1} ns/op");
+    ns
+}
+
+fn main() {
+    // Sizes chosen to match what this substrate ACTUALLY hashes:
+    //   16 B   a word (crystal_encoder)
+    //   256 B  short text
+    //   480 B  the SoA node's value region (512 - key(16) - edges(16))
+    //   512 B  THE canonical SoA node -- 4096 bit, the default unit
+    //   1024 B one BLAKE3 chunk exactly (the hash_many threshold)
+    //   2 KB   VSA_BYTES
+    //   64 KB  bulk
+    for &n in &[16usize, 256, 480, 512, 1024, 2048, 65536] {
+        let data: Vec<u8> = (0..n).map(|i| (i % 251) as u8).collect();
+        let iters = if n >= 65536 { 2_000 } else { 50_000 };
+        println!("  input = {n} B");
+        let ours = bench("in-tree", iters, || *ndarray::hpc::blake3::hash(&data).as_bytes());
+        let theirs = bench("blake3 crate", iters, || *blake3::hash(&data).as_bytes());
+        println!("    ratio        {:>10.2}x  (in-tree / crate)\n", ours / theirs);
+    }
+}

.claude/knowledge/blake3-ab-bench/run.sh

@@ -0,0 +1,32 @@
+#!/bin/sh
+# A/B: ndarray's in-tree BLAKE3 vs the external `blake3` crate.
+#
+# On-demand instrument, same posture as ../simd-codegen-oracle: run it when the
+# question is open, record the answer, stop. NOT a CI job.
+#
+# Requires the external `blake3` crate to still be a dependency -- once it is
+# dropped, this bench can no longer build, which is by design: at that point
+# there is nothing left to compare against.
+set -eu
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO="$(cd "$HERE/../../.." && pwd)"
+# The bench needs its source under $REPO/examples/ for cargo to see it, and
+# removes it afterwards. Both halves must refuse to touch anything they did not
+# create: a fixed destination plus an unconditional `rm` in the EXIT trap would
+# clobber a pre-existing examples/blake3_ab.rs and then delete it.
+EXAMPLE_DIR="$REPO/examples"
+EXAMPLE="$EXAMPLE_DIR/blake3_ab.rs"
+created_example_dir=0
+if [ -e "$EXAMPLE" ] || [ -L "$EXAMPLE" ]; then
+    echo "refusing to overwrite $EXAMPLE" >&2
+    exit 1
+fi
+if [ ! -d "$EXAMPLE_DIR" ]; then
+    mkdir "$EXAMPLE_DIR"
+    created_example_dir=1
+fi
+# Only remove the directory if this script created it.
+trap 'rm -f "$EXAMPLE"; [ "$created_example_dir" -eq 0 ] || rmdir "$EXAMPLE_DIR" 2>/dev/null || true' EXIT
+cp "$HERE/blake3_ab.rs" "$EXAMPLE"
+cd "$REPO"
+cargo run --release --quiet --example blake3_ab

.claude/knowledge/blake3-in-tree-measured.md

@@ -0,0 +1,175 @@
+# In-tree BLAKE3 — correct, and what it costs
+
+> **Status: MEASURED, 2026-07-29.** Correctness against the official vectors;
+> throughput against the external crate. Both numbers below are reproducible
+> with the instruments in this directory.
+
+## READ BY:
+- Anyone about to drop the external `blake3` dependency
+- Anyone continuing rung 3 of `the-simd-ladder.md`
+
+## P0 TRIGGER
+About to swap `blake3::` call sites onto `crate::hpc::blake3`? **The swap is
+correct but costs 1.3× on typical inputs and ~5× at 64 KB. Read the table.**
+
+---
+
+## Why it exists
+
+Root `ndarray` depends on `blake3`, so `blake3 → ndarray::simd` is a cargo
+cycle — the only rung of the ladder that has one (`the-simd-ladder.md`).
+Cutting it means ndarray owning BLAKE3 rather than consuming the crate.
+
+Scoping finding that made this small: **ndarray's usage is entirely
+single-input.** 14 call sites across 8 files use only `hash`,
+`Hasher::{new, new_keyed, update, finalize, finalize_xof().fill()}`,
+`Hash::as_bytes`, and `Hash` as a signature type. **No `hash_many`.** So the
+serial core suffices, and it needs no SIMD at all.
+
+## Correctness — proven
+
+`src/hpc/blake3.rs`, 771 lines, transcribed from upstream's own
+`reference_impl/reference_impl.rs` (the spec-referenced serial
+implementation, BLAKE3 spec §5.1). No `unsafe`, no `core::arch`, no new
+dependencies.
+
+Against the official `test_vectors.json`, vendored to
+`src/hpc/blake3_test_vectors.json`:
+
+- **35/35** cases, unkeyed `hash` **and** keyed `keyed_hash`,
+- each checked at **both** 32-byte length and the full extended length via
+  `finalize_xof().fill()` — 140 assertions,
+- input lengths 0 … 102 400.
+
+Plus: streaming (one `update` vs many 37-byte `update`s over 102 400 bytes),
+empty input, and incremental `fill` (7 bytes at a time vs one shot).
+
+`derive_key` was included too — it is another `Hasher` invocation with
+different flags, so it came free.
+
+## Throughput — the cost, measured
+
+`sh .claude/knowledge/blake3-ab-bench/run.sh`, release build, two runs:
+
+| input | in-tree | `blake3` crate | ratio |
+|---|---|---|---|
+| 16 B (a word) | 134 ns | 100 ns | **1.34–1.39×** |
+| 256 B (text) | 437 ns | 350 ns | **1.25–1.29×** |
+| 2 KB (`VSA_BYTES`) | 3.4 µs | 2.6 µs | **1.30×** |
+| 64 KB (bulk) | 109 µs | 23 µs | **4.7–4.9×** |
+
+### The `array_chunks` fast path — operator's lead, measured
+
+Hypothesis (operator, citing the blasgraph JIT-gap precedent): the gap might
+be closed by proper use of the existing slice primitives rather than by new
+SIMD. The staging path copies every byte **twice** — input → `self.block` →
+`block_words` — and for a full block the first copy is pure overhead.
+
+Implemented as a `crate::simd_ops::array_chunks::<u8, 64>` fast path in
+`ChunkState::update`, guarded `input.len() > BLOCK_LEN` so a chunk's final
+block is never compressed early (it carries `CHUNK_END`). **Measured, three
+runs:**
+
+| input | before | after | change |
+|---|---|---|---|
+| 16 B | 137 ns | 134 ns | — (never reaches the fast path) |
+| 256 B | 445 ns | 437 ns | — |
+| **2 KB** | **4322 ns** | **3421 ns** | **−21 %** |
+| 64 KB | 114.8 µs | 109.0 µs | −5 % |
+
+**Verdict: real, and bounded.** The double copy was costing ~21 % at the mid
+sizes — not nothing, and free to remove. But it does **not** replace the two
+structural gaps: inputs ≤ 1 block never reach the fast path at all, and at
+64 KB the copy is noise beside the absent `hash_many`. The ratio at 2 KB
+moved 1.34–1.60× → a stable 1.30×; the small-input 1.3× and the bulk 4.8×
+both stand.
+
+So the answer to "does it just need proper `array_chunks` use?" is **partly,
+and the part it fixes is now fixed.** Rungs 3b (`hash_many`) and 3c (SIMD
+single-compress) remain the load-bearing ones.
+
+Correctness is gated, not assumed: the official vectors cover every boundary
+the fast path turns on — 63/64/65 (the `> BLOCK_LEN` guard itself),
+127/128/129, and 1023/1024/1025 (the chunk boundary).
+
+**The two gaps have different causes, and only one is about `hash_many`.**
+
+- **The 64 KB gap is `hash_many`.** Above one chunk (1024 B) the crate
+  switches to its degree-8/16 parallel path with the transpose. We have none.
+  This is exactly rung 3b, and the `U32x16` shuffle surface merged in #267 is
+  what it would be built on.
+- **The 1.3× small-input gap is NOT.** At 16 B there is a single compression
+  and no parallelism to be had — the crate is still faster because it
+  SIMD-accelerates *the single compress itself* (its sse41 backend). Closing
+  that needs a `U32x4`-shaped compress, which is a rung the ladder plan does
+  not currently have. Call it 3c.
+
+So the honest shape is:
+
+```text
+3a  in-tree core, correct          DONE, costs 1.3x typical / 5x bulk
+3b  hash_many on U32x16            closes the bulk gap
+3c  SIMD single-compress (U32x4)   closes the small-input gap
+```
+
+## What this means for the swap
+
+**The cycle-cut is available now and is correct.** It removes a cargo cycle
+and 2,910 lines of second-surface `core::arch`.
+
+It does **not** remove a C build — `Cargo.toml:213` already sets
+`default-features = false, features = ["pure"]`, which removed all C/ASM
+compilation back in #264. An earlier revision of this line credited the swap
+with that too; overstating the benefit matters here specifically, because
+what it is being weighed against is transcribing a cryptographic
+implementation.
+
+**It is not free**, and the previous framing ("removes things, needs no
+benchmark to justify") was true about what it *removes* and silent about what
+it *costs*. With the numbers in hand that framing is incomplete: this is a
+trade, and which side wins depends on how hot ndarray's hashing actually is.
+
+Where the call sites sit on the curve: `crystal_encoder` hashes a word
+(16 B band), `vsa` XOF-expands to 2 KB, `merkle_tree`/`seal`/`spo_bundle`
+hash small nodes, `deepnsm`/`compression_curves` small. So ndarray's real
+exposure is the **1.3–1.6× band**, not the 5× one — but 1.3× on a hot encoder
+path is a real cost, not a rounding error.
+
+**Not swapped here.** The module lands and is tested; the call sites still
+use the external crate. Flipping them is a decision with a measured price
+tag, and it is the operator's.
+
+## One deliberate deviation from upstream, and one restored
+
+- **Deviated:** transcribed from `reference_impl.rs` rather than
+  `portable.rs` + `lib.rs`. Upstream ships the reference implementation as
+  the readable, algorithmically-identical serial version, which is what
+  "take the serial branch everywhere" reduces to. Correctness is proven by
+  the vectors.
+
+  An earlier revision of this document guessed that "part of the 1.3× is
+  likely this choice rather than the SIMD gap — `portable.rs` avoids a
+  per-block staging copy." **That guess is now separated out and was wrong
+  for the small-input case.** Removing the staging copy (the `array_chunks`
+  fast path above) bought 21 % at 2 KB and **nothing at ≤ 1 block**, because
+  inputs that small never reach the fast path. So the small-input 1.3× is not
+  the staging copy — it is the crate's SIMD single-compress, as originally
+  suspected.
+- **Restored:** `Hash::eq` is **constant-time**. Upstream uses the
+  `constant_time_eq` crate; the transcription initially used a plain `==`,
+  which leaks match-prefix length through timing when a BLAKE3 output is used
+  as a MAC. Rewritten as an XOR-fold with a `black_box` on the accumulator,
+  no dependency added. No call site in this crate compares two `Hash` values
+  today — `seal.rs` compares the truncated `MerkleRoot` — so this is a guard
+  for future consumers, not a live-leak fix.
+
+## Not claimed
+
+- Not that the in-tree version should replace the crate. That is the open
+  decision this document exists to inform.
+- Not that the remaining 1.3× has been fully attributed. The staging-copy
+  term IS now separated (measured at 21 % of the 2 KB cost, 0 % at ≤ 1
+  block); what is left at small inputs is *presumed* to be the crate's SIMD
+  single-compress, and that has not been isolated by building one.
+- Not that the bench is rigorous. It is a warm-loop wall-clock A/B, adequate
+  for a 1.3× vs 5× distinction and not for anything finer.

.claude/knowledge/chacha20-vendoring-blast-radius.md

@@ -157,9 +157,18 @@ SIGILL on a runner without AVX-512 silicon.
 `cargo check --target=x86_64-unknown-linux-gnu -p ndarray --features
 approx,serde,rayon` (the second adding `hpc-extras`) — package-scoped to
 `ndarray`, so `crates/encryption` and therefore `vendor/chacha20` are outside
-it. **No CI job compiles the chacha20 AVX-512 backend**, and none of the
-coverage claimed here extends to it. Closing that gap would take an explicit
-step such as:
+it. **No CI job compiles the chacha20 AVX-512 backend.**
+
+**Correction (codex, #268): the wasm arm IS covered, and an earlier version of
+this document said otherwise.** `ci.yaml:141-142`, inside the `wasm_simd` job,
+runs `RUSTFLAGS="-C target-feature=+simd128" cargo build --manifest-path
+vendor/chacha20/Cargo.toml --target wasm32-unknown-unknown --lib` — which
+selects `backends::ndarray_simd` through the cfg's wasm arm, and whose own
+comment calls it "the wasm matryoshka" guard. So the accurate statement is
+**arm-specific, not backend-wide**: wasm32+simd128 is compiled and guarded in
+CI; the x86_64 avx512f arm is compiled by nothing.
+
+Closing the AVX-512 gap would take an explicit step such as:
 
 ```console
 $ CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS="-Ctarget-cpu=x86-64-v4" \