diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 3e7d282f..6df44925 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -5,6 +5,40 @@ > **Spawn protocol:** every agent reads this file before starting, > appends one entry on completion via `tee -a`. +## 2026-07-29 — main thread — oracle repair + chacha20 reach + BLAKE3 gap + +**PRs:** #264 (merged), #265 (merged, `58521c3`) +**Commits:** `87b9b4c`, `de4e4d1`, `3e72c5d`, `3c427ae` +**Docs:** `.claude/knowledge/{blake3-on-ndarray-simd,chacha20-vendoring-blast-radius}.md`, +`.claude/knowledge/simd-codegen-oracle/*`, `td-t22-asm-investigation.md`, +`simd-one-spec-design.md` + +| item | outcome | +|---|---| +| oracle `run.sh` shipped unusable by #264's relocation | repaired, self-contained, 17/17 probes green from a clean master checkout | +| `vendor/chacha20` blast radius | patch reaches 2 repos (ndarray, MedCare-rs), not 7; backend is compile-time gated on `avx512f` | +| `AdaWorldAPI/stream-ciphers` | bare upstream mirror — same head sha, same `chacha20/` tree hash, 0 commits ahead | +| BLAKE3 on `ndarray::simd` | needs one const-generic `exchange` method, no intrinsics, no C | + +**Review:** 3 findings from codex, 6 from CodeRabbit, all addressed in +`3c427ae`. The Major one (transpose probe did not validate a correct +transpose) was a genuine methodological error, not a nit — see EPIPHANIES. + +**Corrected after merge, twice** (#266). Operator: *cargo is CI is github +needs V3; dockerfile is V4*. Pass 1 of the correction concluded +`Dockerfile.avx512` compiles and ships `ndarray_simd` — wrong; codex caught +that both Dockerfiles run bare `cargo build --release` and `default-members` +omits `crates/encryption`. Settled: no image compiles the backend; it is +reached only by `-p encryption` / `--workspace` under an AVX-512 config, or +wasm32+simd128. CI's missing global pin is separately confirmed deliberate +(`ci.yaml:17-22`). EPIPHANIES entry names the two-axis pattern and the fact +that the correction itself was the more confident error. + +**Open:** throughput bench vs `rust_avx2.rs` and vs upstream chacha20's own +`avx512.rs`; the fork-vs-vendored-copy decision; whether the chacha20 cfg +gate should gain a runtime arm. + + ## Fleet manifest | # | Agent | File | Model | Status | diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index e5f483ea..9d29665e 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,5 +1,132 @@ # ndarray — Epiphanies (append-only) +## 2026-07-29 — BLAKE3 needs a method surface, not intrinsics (measured) +**Status:** FINDING +**Scope:** @simd-savant domain:codec +**Cross-ref:** PR #264, PR #265, `.claude/knowledge/blake3-on-ndarray-simd.md`, +`.claude/knowledge/simd-codegen-oracle/` + +C is out of `blake3` (`features = ["pure"]`, #264 — 33 `.o` objects and +`libblake3_avx512_assembly.a` gone). What remained was a second SIMD surface: +2,910 lines of raw `core::arch` in its Rust backends. + +Census + four probes settle what a `ndarray::simd` backend would cost. +`rust_avx2.rs` uses 15 distinct intrinsics; **twelve already exist on +`U32x16`**. The other three families (`unpack{lo,hi}_epi32`, +`unpack{lo,hi}_epi64`, `permute2x128` — 18 call sites) serve one purpose: +`hash_many`'s transpose. + +| probe | packed | scalar on lane data | +|---|---|---| +| `blake3_g_u32x16` | 72 | 0 | +| `interleave_lo_u32x16` | 11 | 0 | +| `transpose_16x16_u32` (index loop) | **0** | 1 | +| `transpose_16x16_composed` (4 stages, correctness-checked) | 79 | 0 | + +The compression core is free. The interleave *primitive* is free — a scalar +index loop compiles to `vpermd` + `vpblendd`, a real two-source cross-lane +permute. Only the **monolithic index-loop spelling** of the transpose fails; +composed from `exchange` at G = 1/2/4/8 it emits `vpunpcklqdq`, `vpermq`, +`vpermpd`, `vinserti128` — the same family the intrinsic backend hand-writes. + +**So no intrinsic override is earned.** The gap is one const-generic method, +~15 lines, no `unsafe` and no `core::arch`. That is the third consecutive +time this oracle has turned "obviously needs intrinsics" into "already free" +(after `saturating_abs`, `widening`, and `cross_lane_reverse`). + +**The methodological catch is the part worth keeping.** The first version of +this finding rested on `transpose_stage_u32x16` — ONE 32-bit stage whose +helpers, by their own doc comment, do not compose into a transpose. Codex and +CodeRabbit independently flagged it. It was an inference wearing a +measurement's clothes, published in the document that cites TD-T22 as the +reason not to do exactly that. Two guards added in response: + +- the composed transpose is **checked against a naive reference and aborts on + mismatch**, control-tested by deleting `stage::<8>` to confirm it fires; +- `run.sh` now **executes** the probe binary, because `--emit asm` links + nothing and the assertion had never actually run. A claim of verification + that does not verify is the same defect one level up. + +Still unmeasured, and stated as such: throughput vs `rust_avx2.rs`. "Emits +packed shuffles" is not "is faster." + + +## 2026-07-29 — A build claim has two axes; the correction erred on the second +**Status:** FINDING +**Scope:** @simd-savant @truth-architect domain:build-tiers +**Cross-ref:** PR #265, `.claude/knowledge/chacha20-vendoring-blast-radius.md`, +`.claude/knowledge/td-t22-asm-investigation.md` + +This repo has **three** build tiers, and an audit that reads only +`.cargo/config.toml` sees one of them and concludes the opposite of the truth. + +| tier | target-cpu | purpose | +|---|---|---| +| CI (`ci.yaml`) | none globally; v4 in `tier4-avx512-check` | one binary, all ISAs, runtime `LazyLock`; the `cross_test` matrix spans i686 and s390x, so a global pin is impossible | +| `Dockerfile` | `x86-64-v3` | the portable image | +| `Dockerfile.avx512` | `x86-64-v4` | the AVX-512 image | + +Operator's formulation: *cargo is CI is github needs V3; dockerfile is V4.* + +**A claim about "what a build does" has TWO axes, and I got each one wrong on +a separate pass** — the second time while correcting the first: + +| axis | question | where the answer lives | +|---|---|---| +| `target-cpu` | which cfg-gated code is *selected*? | `.cargo/config*.toml`, `Dockerfile*` `RUSTFLAGS`, workflow `env:` incl. per-job | +| package selection | is that crate *compiled at all*? | `default-members`, and the `-p` / `--workspace` flags on the actual command | + +Three passes on one paragraph: + +1. **First claim:** *"No default build runs `vendor/chacha20`'s + `ndarray_simd`."* Reasoned only from `.cargo/config.toml` pinning v3. + Right answer, incomplete reason. +2. **First correction:** operator said *cargo is CI is github needs V3; + dockerfile is V4*, so I concluded `Dockerfile.avx512` compiles and ships + the backend. **Wrong.** I fixed the target-cpu axis and immediately erred + on the package-selection axis I still had not checked — both Dockerfiles + run bare `cargo build --release`, `default-members` omits + `crates/encryption`, and nothing else in that set pulls chacha20. Caught by + codex on #266. +3. **Settled:** `ndarray_simd` is reached only by an explicit + `-p encryption` / `--workspace` build under an AVX-512 config, or by + wasm32+`simd128`. No image compiles it. + +A separate instance of the same shape, same week: *"CI has been testing +different machine code than anyone reviews"*, filed as a ⚠ defect. It is the +design, and `ci.yaml:17-22` says so in prose I had not read — a global pin +collides with the non-x86 cross_test matrix and contradicts the +runtime-dispatch intent. + +**The pattern to name:** every individual measurement was correct. The error +was always the *scope quantifier* — "no build", "CI", "ships" — attached to +evidence drawn from one file. `rustc --print cfg` told me what v3 lacks; it +could not tell me which tiers exist. Knowing the tiers still could not tell me +what each tier builds. + +**And the correction is as dangerous as the original claim.** Being handed the +missing piece feels like completion, so pass 2 shipped faster and with more +confidence than pass 1, and was more wrong. A correction is a new claim and +earns no discount on verification. + +Consequence, concretely — before any "no build does X" / "X ships with Y": + +```console +find . -iname 'Dockerfile*' # which tiers exist +grep -n 'RUN cargo' Dockerfile* # what each one actually builds +sed -n '/default-members/,/]/p' Cargo.toml # what a bare build selects +cargo tree -p -i # per package, not per workspace +``` + +A second-order note worth keeping: ndarray's own SIMD upgrades itself at run +time via `LazyLock` even in a v3 build, but `vendor/chacha20`'s gate is +`#[cfg(target_feature = "avx512f")]` — compile-time, no runtime fallback. So a +v3 image on AVX-512 silicon has ndarray's kernels upgrading while the chacha20 +keystream stays on RustCrypto's backends. Compile-time and runtime dispatch +living in one binary is not a contradiction, but it means "this build is v3" +does not settle what any given subsystem selects. + + ## 2026-04-19 — Prompt↔PR ledger is 10⁷× cheaper than code grep **Status:** FINDING **Scope:** @workspace-primer domain:bookkeeping diff --git a/.claude/knowledge/chacha20-vendoring-blast-radius.md b/.claude/knowledge/chacha20-vendoring-blast-radius.md index cfdb90e9..ed66ad76 100644 --- a/.claude/knowledge/chacha20-vendoring-blast-radius.md +++ b/.claude/knowledge/chacha20-vendoring-blast-radius.md @@ -73,24 +73,102 @@ $ rustc --print cfg -Ctarget-cpu=x86-64-v4 | grep avx512f target_feature="avx512f" ``` -| build | target-cpu | `avx512f` | backend compiled | +### The tiers, and what each one actually builds + +**Two independent axes decide this, and reading either one alone gets it +wrong.** Axis one is `target-cpu`, which decides *which backend* is selected. +Axis two is **package selection**, which decides whether `chacha20` is +compiled at all. + +| tier | target-cpu | builds `encryption`? | chacha20 backend | |---|---|---|---| -| ndarray, default | `x86-64-v3` (`.cargo/config.toml`) | absent | RustCrypto soft/avx2/sse2 | -| ndarray, `--config .cargo/config-avx512.toml` | `x86-64-v4` | present | **`ndarray_simd`** | -| MedCare-rs, default | none — no `.cargo/config.toml` at all | absent | RustCrypto soft/avx2/sse2 | -| wasm32 + `simd128` | — | n/a | **`ndarray_simd`** | -| the five unpatched repos | — | — | not present in the source they link | +| **CI** (`ci.yaml`) | none globally | no | not compiled | +| CI `tier4-avx512-check` | v4, per-job | no — `-p ndarray` only | not compiled | +| **`Dockerfile`** | v3 | **no** — bare `cargo build` | **not compiled** | +| **`Dockerfile.avx512`** | v4 | **no** — bare `cargo build` | **not compiled** | +| dev `cargo build -p encryption` | v3 (`.cargo/config.toml`) | yes | RustCrypto avx2/sse2 | +| …with `--config .cargo/config-avx512.toml` | sapphirerapids | yes | **`ndarray_simd`** | +| wasm32 + `simd128` | — | if selected | **`ndarray_simd`** | + +The Dockerfile rows are the ones that surprise. Both images run bare +`cargo build --release`, which selects `default-members` +(`Cargo.toml:436-443`): `.`, `ndarray-rand`, `crates/ndarray-gen`, +`crates/numeric-tests`, `crates/serialization-tests`. **`crates/encryption` +is not among them**, and nothing else in that set depends on chacha20 — +verified per-package: -**No default build of any repo in the workspace runs `ndarray_simd`.** Two -opt-in configurations do: an explicitly-selected AVX-512 x86_64 build, and a -wasm32 build with `simd128`. +```console +$ for p in ndarray ndarray-rand ndarray-gen numeric-tests serialization-tests; do + cargo tree -p "$p" -i chacha20; done +error: package ID specification `chacha20` did not match any packages # x5 + +$ cargo tree -p encryption -i chacha20 +chacha20 v0.9.1 (/workspace/ndarray/vendor/chacha20) +└── chacha20poly1305 v0.10.1 + └── encryption v0.1.0 (/workspace/ndarray/crates/encryption) +``` -This is not a defect — the fallthrough is correct and the comments in -MedCare-rs's manifest describe it accurately ("non-avx512 builds fall through -to RustCrypto's own backends"). It is a scope fact that keeps being stated -one size too large. +`encryption` *is* a workspace member (via `crates/*`), so `--workspace` or +`-p encryption` reaches it — neither Dockerfile passes either. + +**So `ndarray_simd` is reached only by an explicit `-p encryption` / +`--workspace` build under an AVX-512 config, or by a wasm32+`simd128` +build.** No image in this repo compiles it. + +### Correction history — three passes on one paragraph + +Worth keeping visible, because the failure mode repeated: + +1. **First claim:** "no default build runs `ndarray_simd`", reasoned purely + from `.cargo/config.toml` pinning v3. Right answer, incomplete reason. +2. **First correction:** the operator pointed out *cargo is CI is github + needs V3; dockerfile is V4*, so I concluded `Dockerfile.avx512` compiles + and ships the backend. **Wrong** — I fixed the `target-cpu` axis and + introduced a new error on the package-selection axis I still had not + checked. (Caught by codex on PR #266.) +3. **This version:** both axes checked with `cargo tree`, per package. + +The operator's tier statement was correct throughout; what it does not imply +is that either image builds the crate that pulls chacha20. + +Two things none of this changes: + +- The **patch-reach** table above stands. It is about which repos link the + vendored source when they *do* build it. +- The gate is **compile-time**, not runtime. `Dockerfile`'s own comment notes + that ndarray's `simd.rs` detects AVX-512 at run time via `LazyLock` + even in a v3 build — but `vendor/chacha20/src/backends.rs` keys on + `#[cfg(target_feature = "avx512f")]`, resolved by the compiler with no such + fallback. So even in a hypothetical v4 image that *did* build `encryption`, + the two subsystems would dispatch by different mechanisms. + +### The v4 arm is covered in CI, per-job + +CI's global env carries no `target-cpu`, and `ci.yaml:17-22` records why: +it collides with the `cross_test` matrix (`i686` is 32-bit, `s390x` is not +x86 at all) and contradicts the one-binary + runtime-dispatch design intent. +Jobs needing a higher tier opt in individually — `tier4-avx512-check` sets +`CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS: "-D warnings +-Ctarget-cpu=x86-64-v4"`, deliberately the per-target form rather than plain +`RUSTFLAGS`, because `RUSTFLAGS` also applies to host build scripts and those +SIGILL on a runner without AVX-512 silicon. + +**That job covers ndarray's v4 arm and nothing else.** Both its steps are +`cargo check --target=x86_64-unknown-linux-gnu -p ndarray --features +approx,serde,rayon` (the second adding `hpc-extras`) — package-scoped to +`ndarray`, so `crates/encryption` and therefore `vendor/chacha20` are outside +it. **No CI job compiles the chacha20 AVX-512 backend**, and none of the +coverage claimed here extends to it. Closing that gap would take an explicit +step such as: -## One manifest comment overstates it +```console +$ CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS="-Ctarget-cpu=x86-64-v4" \ + cargo check --target=x86_64-unknown-linux-gnu -p encryption +``` + +Not added here — adding a CI job is a product change, and this is a docs PR. + +## The manifest comment, re-read `Cargo.toml:481-482` reads: @@ -98,13 +176,21 @@ one size too large. > keystream … **on any x86_64+avx512f build (the workspace's > `target-cpu=x86-64-v4`)**. -The workspace's pinned target-cpu is **v3**, in `.cargo/config.toml`. `v4` -is the opt-in `.cargo/config-avx512.toml`. The conditional half of the -sentence is right; the parenthetical asserts the default build satisfies it, -and it does not. +An earlier version of this audit filed that parenthetical as a factual error, +on the grounds that the workspace pins v3; a later version withdrew the +filing, on the grounds that `Dockerfile.avx512` is a v4 build of this +workspace. **Both were partly wrong.** + +Settled: "the workspace's `target-cpu=x86-64-v4`" does name a real tier, so +the parenthetical is imprecise rather than false — it is not the tier +`cargo build` selects. But the sentence's *main* clause is the weaker part. +"This transitively accelerates the `encryption` crate's … keystream" holds +only for a build that actually selects `encryption`, and **no image in this +repo does**. The acceleration is real and reachable; it is not automatic, and +nothing ships with it today. -**Not edited here.** Manifest string changes in this repo need an operator -ruling; the finding is recorded instead of applied. +Still **not edited**, for the same reason as before: manifest string changes +need a ruling. Recorded here so the next reader is not misled by it. ## The fork the P0 rule names is not this tree @@ -161,21 +247,27 @@ Three consequences worth stating plainly: ## What is NOT claimed here -- Not that the vendored backend is wrong. It is untested by default builds, - which is a different statement. -- Not that the patch should be removed. That is a decision, and it depends on - (3) above plus the AVX-512 deployment question, neither of which this audit - settles. +- Not that the vendored backend is wrong, or unused. `Dockerfile.avx512` + compiles it; it is the shipped path on v4 silicon. +- Not that the patch should be removed. That depends on (3) above, which is a + measurement nobody has taken. - Not that MedCare-rs's wiring is incorrect. It is the one consumer that declared its patch deliberately and documented the fallthrough honestly. -## Open, for an operator ruling +## Still open -1. **Point `vendor/chacha20` at the fork, or keep the copy?** The P0 rule - says depend on the AdaWorldAPI fork. Today the tree is neither — a copy of +1. **Point `vendor/chacha20` at the fork, or keep the copy?** The P0 rule says + depend on the AdaWorldAPI fork. Today the tree is neither — a hand copy of a version the fork does not carry, with `repository` rewritten to point at - ndarray itself. -2. **Does `ndarray_simd` still beat upstream's own `avx512.rs`?** Answerable - with the oracle. Unmeasured today, on both sides. -3. **The v3/v4 manifest comment** — correct in place, or leave the record and - annotate? + ndarray itself. The fork is provably a bare upstream mirror (same head sha, + same `chacha20/` tree hash), so "track the fork" currently means "track + upstream 0.10.1", which crosses a `cipher` major. +2. **Does `ndarray_simd` still beat upstream's own `avx512.rs`?** This is the + load-bearing question now that upstream ships an AVX-512 backend of its + own, and it is the one that decides (1). Unmeasured on both sides. The + oracle answers the codegen half; the throughput half needs a bench. +3. **Should `vendor/chacha20`'s cfg gate get a runtime arm?** Today it is + compile-time only, so a v3 image on AVX-512 silicon runs RustCrypto's + backends for the keystream while ndarray's own kernels upgrade via + `LazyLock`. Whether that asymmetry is worth closing is a design call, + not a bug report — and it only matters if (2) says the ndarray lane wins. diff --git a/.claude/knowledge/td-t22-asm-investigation.md b/.claude/knowledge/td-t22-asm-investigation.md index 610e548b..07101d59 100644 --- a/.claude/knowledge/td-t22-asm-investigation.md +++ b/.claude/knowledge/td-t22-asm-investigation.md @@ -175,7 +175,7 @@ the ticket originally asked for. --- -## ⚠ CI DOES NOT APPLY THE v3 BASELINE (found 2026-07-29 by the oracle) +## CI DOES NOT APPLY THE v3 BASELINE — BY DESIGN (corrected 2026-07-29) Everything above was measured on a developer host, where `.cargo/config.toml` supplies `-Ctarget-cpu=x86-64-v3`. **On CI it does @@ -195,20 +195,54 @@ $ RUSTFLAGS="-D warnings" cargo build -p ndarray --lib -v | grep -o 'target-cpu= # (empty — flag silently dropped) ``` -### Consequences - -1. **Every CI job that inherits the workflow `env:` compiles at baseline - x86-64 (SSE2)** — `tests/{stable,beta,1.95.0}`, `clippy`, - `native-backend`, `hpc-stream-parallel`. Not the AVX2 tier developers - build and run locally. CI has been testing different machine code than - anyone reviews. +### It is intentional, and the reason is recorded in the workflow + +The first version of this section called that a defect — "CI has been testing +different machine code than anyone reviews." **That framing was wrong.** +`ci.yaml:17-22` states the decision and its rationale explicitly: + +> `-C target-cpu=x86-64-v3` was removed from the global env. It conflicts +> with the cross_test matrix (`i686-unknown-linux-gnu` is 32-bit, `s390x` +> isn't even x86) and contradicts the design intent recorded in +> `.cargo/config.toml`: per-function `#[target_feature]` + runtime +> `LazyLock` detection means one binary, all ISAs. Jobs that +> specifically need a higher target-cpu can opt in via per-job env. + +So the absence of a global pin is the design, not an oversight, and the v4 +arm *is* exercised — `tier4-avx512-check` opts in per-job with +`CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS`, deliberately the +per-target form because plain `RUSTFLAGS` also reaches host build scripts, +which then SIGILL on a runner without AVX-512 silicon. + +### The three tiers + +| tier | target-cpu | what it is for | +|---|---|---| +| CI (`ci.yaml`) | none globally; v4 in `tier4-avx512-check` | one binary, runtime `LazyLock` dispatch; cross_test matrix spans non-x86 | +| `Dockerfile` | `x86-64-v3` | the portable image | +| `Dockerfile.avx512` | `x86-64-v4` | the AVX-512 image; all SIMD inlined, no dispatch | + +Operator's summary: *cargo is CI is github needs V3; dockerfile is V4.* + +### What does still hold + +1. **TD-T22's measurement is baseline-scoped, and that is worth saying.** + "The scalar polyfill compiles to packed AVX2" is true *under a v3-or-above + baseline*. It is the baseline, not the source form, that does the work. + + Be precise about what "no `target-cpu`" then means, because two different + things get conflated: it selects the **generic `x86-64` baseline**, on + which SSE2 is available — it does **not** follow that SSE2 instructions + are emitted. The measurement above says otherwise. `arx_rounds_u32x16` + came back **0 packed / 146 scalar / 491 memory**: LLVM did not vectorize + that source at the generic baseline at all, packed-SSE2 or otherwise. The + available ISA is a ceiling, not a prediction. (Raised by CodeRabbit on + PR #266; the phrasing here previously said "gets SSE2 codegen", which the + histogram directly contradicts.) 2. **The comment in `.cargo/config.toml` — "This is what GitHub CI runs - against" — is false**, and has been since `RUSTFLAGS` was added. -3. **TD-T22's finding is unaffected but narrower than stated.** "The scalar - polyfill compiles to packed AVX2" is true *under the v3 baseline*. It is - the baseline, not the source form, that does the work — and CI never had - it. `tier4-avx512-check` is unaffected because it uses the more specific - `CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS`, which does not collide. + against" — is stale.** CI applies no global pin; `Dockerfile` is what runs + at v3. Left as a record rather than edited (manifest/config string changes + need a ruling), but do not read it as current. ### What was done here @@ -217,8 +251,8 @@ inheriting it, passing `-C target-cpu=x86-64-v3` after the `--` so it lands on the final rustc invocation and wins regardless of ambient `RUSTFLAGS`. Confirmed: with `RUSTFLAGS="-D warnings"` exported, all 13 probes match. -**The wider question — whether CI should build at v3 so tests exercise the -code that ships — is deliberately NOT changed here.** Making every CI job -switch instruction sets is an operator decision with real consequences -(SIGILL risk on runners without AVX2 is what TD-SIMD-1 was about), not a -drive-by fix in a tooling PR. +**The wider question is settled and was never open.** "Should CI build at v3 +so tests exercise the code that ships?" presumes CI and the shipped artifact +are meant to match; they are not. CI proves one binary works across ISAs via +runtime dispatch, and the Dockerfiles pin the two shipping tiers. Changing +CI's global env would break the `cross_test` matrix outright.