blake3: drop the C toolchain requirement; record measured u32/u64 lane codegen

[AdaWorldAPI]

One dependency line changes in the shipped crate. Everything else is documentation under .claude/knowledge/.

The change

blake3's default build compiles hand-written assembly through cc and links it — 33 object files and one static archive. Any vendored or offline build of ndarray therefore requires a C toolchain, and fails without one.

-blake3 = { version = "1", optional = true }
+blake3 = { version = "1", optional = true, default-features = false, features = ["pure"] }

pure routes build.rs to build_sse2_sse41_avx2_rust_intrinsics(), which compiles no C.

	object files	archives	cc invoked
before	33	1	yes
after	0	0	no

Verified after cargo clean -p blake3: the only cfgs set are blake3_{sse2,sse41,avx2}_rust.

Cost: the hand-tuned x86-64 assembly is replaced by Rust intrinsics; the AVX-512 path and the aarch64 NEON C intrinsics are dropped.

cmake, openblas-build and openssl-sys also build C, but are reachable only through the optional blas feature, never through default = ["std", "hpc-extras"]. blake3 was the only C on the default path.

Reviewer note: blake3_*_ffi still appears in build-script output as rustc-check-cfg declarations — cargo enumerating valid cfg names, not setting them. A check must anchor on ^cargo:rustc-cfg= to tell the two apart.

Supporting measurement

Under the -Ctarget-cpu=x86-64-v3 baseline, the macro-generated lane types do not compile to scalar code. 13 probes:

lane operation	packed	scalar (lane data)	lowering
u32 ARX round (ChaCha)	52	0	vpaddd / vpxor / vpshufb
u32 reduce_sum	8	0	logarithmic reduction tree
f32 FMA accumulate	28	0	vfmadd213ps
i8 saturating_abs	4	0	vpxor / vpsubsb / vpblendvb
u16→f32 widening	6	0	vpmovzxwd / vcvtdq2ps
u8x64 cross-lane reverse	9	0	vpshufb / vpermq
u64 rotate	0	8	scalar rorq, one per lane
BLAKE2b G-function	22	~88	packed add, then scalar through all four rotates
gather / table lookup	0	0	movzbl chain, no arithmetic
loop-carried dependency	0	27	inherently serial

The u64 rotate is the outlier. LLVM lowers u32 rotate_left(12)/(7) via vpslld/vpsrld/vpor and folds byte-granular amounts to vpshufb. At 64-bit width it does neither — including for byte-granular 32/24/16 — despite vpsllq/vpsrlq being available.

This is load-bearing for the crypto surface: BLAKE2b is a 64-bit ARX cipher and argon2 uses BLAKE2b. The crate has 8 rotate_left methods, all u32, and no rotate_right at any width, so argon2's kernel cannot currently be expressed. AVX-512 provides _mm512_rorv_epi64 as a single instruction.

The converse also holds: hand-writing intrinsics for the u32 lanes would lose to the optimizer, which already emits 8 vpaddd for 64 u32 lanes — the AVX2 minimum.

Documentation added

• .claude/knowledge/simd-codegen-oracle/ — the measurement instrument (probes, analyzer, baseline, README with the runnable recipe and recorded results). Filed as knowledge rather than as a crate or CI job: the question "does this vectorize?" is answered once during design and recorded, not re-answered on every commit.
• simd-one-spec-design.md — design for consolidating five backend files (31 macro-generated types, 57 hand-written, 13,253 LoC) behind one declaration, with a measured entry criterion for intrinsic overrides.
• crypto-lane-status.md — per-lane status: u32 ARX and f32 FMA need no work; u64 ARX is absent and needs intrinsics.
• td-t22-asm-investigation.md — extended with the CI baseline finding below.

Incidental finding: CI does not apply the v3 baseline

.github/workflows/ci.yaml sets RUSTFLAGS: "-D warnings" at workflow level, and cargo's RUSTFLAGS env var replaces [target.'cfg(…)'].rustflags from .cargo/config.toml rather than merging:

$ cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*'
target-cpu=x86-64-v3
$ RUSTFLAGS="-D warnings" cargo build -p ndarray --lib -v | grep -o 'target-cpu=[a-z0-9-]*'
(empty)

Every job inheriting that env: — tests/{stable,beta,1.95.0}, clippy, native-backend, hpc-stream-parallel — compiles at baseline x86-64 rather than the AVX2 tier. The comment in .cargo/config.toml stating "this is what GitHub CI runs against" is inaccurate. tier4-avx512-check is unaffected; it uses the more specific CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS.

Deliberately not changed here. Switching CI to v3 alters the instruction set every test exercises and carries the SIGILL risk documented in TD-SIMD-1. Separate decision, separate PR.

Out of scope

The u64 rotate implementation, the one-spec migration, and BLAKE3's remaining Rust intrinsics — a second SIMD surface that pure does not remove. Closing that means implementing BLAKE3's compression on ndarray::simd::U32x16, since it is a ChaCha-derived u32 ARX kernel and that lane is already at the instruction floor.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a SIMD codegen oracle with probe kernels, assembly classification, target baselines, CI execution, and documentation describing SIMD lane status and a one-spec multi-backend design.

Changes

SIMD codegen verification

Layer / File(s)	Summary
Lane status and generation design .claude/knowledge/*.md, Cargo.toml	Documents measured SIMD capabilities, defines the proposed lane-generation model, records u64 rotate limitations, and configures the optional pure-Rust blake3 dependency.
Probe crate and SIMD kernels crates/simd-codegen-oracle/Cargo.toml, crates/simd-codegen-oracle/src/main.rs	Adds runtime-seeded SIMD probes covering ARX, FMA, reductions, bitwise operations, scalar-shaped loops, u64 rotates, and BLAKE2b mixing.
Assembly classification and verdicts scripts/codegen_oracle_analyze.py	Parses emitted assembly, classifies instructions, loads TOML expectations, reports probe results, and returns failure bitmasks.
Baseline execution and CI wiring crates/simd-codegen-oracle/baselines/*, scripts/codegen-oracle.sh, .github/workflows/ci.yaml	Adds x86_64 expectations, target-aware assembly generation, analyzer invocation, and CI conclusion dependencies.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• AdaWorldAPI/ndarray#261: Changes AVX2 U32x8/U32x16 rotate implementation directly related to the oracle’s u32 rotate probes.
• AdaWorldAPI/ndarray#262: Adds related SIMD codegen validation for u32x16 ARX and FMA behavior.

Poem

I’m a bunny checking lanes in flight,
Packed little arrows, classified right.
Rotates hop through assembly bright,
Baselines guard each SIMD byte.
CI thumps its paws: “All’s tight!”

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately captures the two main themes: removing the C toolchain dependency via blake3 pure mode and recording measured SIMD lane codegen.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Fix failing CI checks

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c2dbd14b-0f5d-482f-bd30-6c0b85426e5e)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0aea59a44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Return a bounded nonzero status for late probe failures

Use a bounded nonzero exit status rather than returning the full bitmask. On Unix, process exit codes retain only the low 8 bits, so a failure confined to probe index 8 or later produces 256, 512, etc. and is observed by the shell as status 0. With the current sorted probe list, missing rot_u64x4/rot_u64x8 symbols or regressions in saturating_abs_i8x32, serial_dependent_chain, or widening_u16_to_f32 therefore leave the new CI job green despite printing FAILURES.

Useful? React with 👍 / 👎.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/codegen-oracle.sh`:
- Around line 19-47: Update the Cargo invocations in the target-building logic
to execute from $ROOT, such as by wrapping each command with a subshell that
changes to $ROOT before running Cargo. Preserve the existing manifest, target,
and rustc emission arguments while ensuring .cargo/config.toml is resolved from
the repository root.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 47782926-2a41-4027-8f57-71275f9399f9

📥 Commits

Reviewing files that changed from the base of the PR and between 2c8af9a and b0aea59.

⛔ Files ignored due to path filters (1)

• crates/simd-codegen-oracle/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (9)

• .claude/knowledge/crypto-lane-status.md
• .claude/knowledge/simd-one-spec-design.md
• .github/workflows/ci.yaml
• Cargo.toml
• crates/simd-codegen-oracle/Cargo.toml
• crates/simd-codegen-oracle/baselines/x86_64-unknown-linux-gnu.toml
• crates/simd-codegen-oracle/src/main.rs
• scripts/codegen-oracle.sh
• scripts/codegen_oracle_analyze.py

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate target files =="
git ls-files | rg '(^|/)(codegen-oracle\.sh|Cargo\.toml|config\.toml|baselines/|Cargo\.lock)$|cargo' | sed -n '1,160p'

echo "== script =="
if [ -f scripts/codegen-oracle.sh ]; then
  cat -n scripts/codegen-oracle.sh
fi

echo "== oracle cargo files =="
for f in crates/simd-codegen-oracle/Cargo.toml crates/simd-codegen-oracle/.cargo/config.toml crates/simd-codegen-oracle/Cargo.lock; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    cat -n "$f"
  fi
done

echo "== all .cargo/config.toml in repo =="
for f in $(git ls-files '*\.cargo/config.toml'); do
  echo "--- $f ---"
  cat -n "$f"
done

echo "== run from outside manifests using temp target (read-only probe for files only) =="
mkdir -p /tmp/cargo-config-test
cat >/tmp/cargo-config-test/Cargo.toml <<'EOF'
[package]
name = "test"
version = "0.0.0"
edition = "2021"
[profile.dev]
EOF
mkdir -p /tmp/cargo-config-test/.cargo
cat >/tmp/cargo-config-test/.cargo/config.toml <<'EOF'
[build]
rustc-args = ["--crate-name", "fromconfig"]
EOF
cargo metadata --format-version 1 --manifest-path /tmp/cargo-config-test/Cargo.toml >/tmp/metadata.json
python3 - <<'PY'
import json
data=json.load(open('/tmp/metadata.json'))
pk=data['packages']
print([p['name']+":"+p['manifest_path'] for p in pk])
print("root:", data.get('workspace_root'))
PY

echo "== relevant cargo references in repo =="
rg -n "codegen-oracle|simd-codegen-oracle|target-cpu|x86-64-v3|CARGO|cargo rustc|\.cargo" scripts README.md .github crates 2>/dev/null | sed -n '1,240p'

Repository: AdaWorldAPI/ndarray

Length of output: 149

---

🌐 Web query:

Cargo documentation rustc configuration files discovered based on current directory not manifest path .cargo/config.toml

💡 Result:

Cargo configuration files (such as.cargo/config.toml) are discovered by traversing the file system upward, starting from the current working directory (CWD) where the cargo command is executed, not from the directory containing the Cargo.toml manifest file [1][2][3]. This behavior means that if you run a cargo command from a directory other than the project root (e.g., using --manifest-path), Cargo will not automatically locate or respect configuration files located in the project's root directory or its subdirectories [2][3]. This is a long-standing and known limitation in Cargo [4][5][6]. Key aspects of Cargo configuration discovery include: Hierarchy: Cargo searches for.cargo/config.toml (or.cargo/config) files in the current directory and all parent directories up to the root of the file system [1][7][8]. Precedence: Configuration values are merged, with deeper directories (those closer to the current working directory) taking precedence over ancestor directories [1][9][8]. Global configuration files located in $CARGO_HOME (e.g., $HOME/.cargo/config.toml) have the lowest precedence [1][10][8]. Workspaces: When operating within a workspace, Cargo does not automatically read configuration files from individual member crates; it only considers configurations found relative to the CWD [7][9][8]. Recommended Workaround: To ensure Cargo picks up a specific configuration file, the most reliable method is to change your current working directory to the directory containing the desired.cargo/config.toml file before executing the cargo command [3]. Alternatively, you can use the --config CLI flag to provide specific configuration overrides directly [1][11].

Citations:

• 1: https://doc.rust-lang.org/cargo/reference/config.html
• 2: Specifying the manifest path ignores .cargo/config.toml rust-lang/cargo#10302
• 3: Does cargo read .cargo/config in the project directory? rust-lang/cargo#7640
• 4: Cargo does not always search for .cargo/config file in project root rust-lang/cargo#2930
• 5: Cargo config search meta issue. rust-lang/cargo#9769
• 6: https://internals.rust-lang.org/t/problems-of-cargo-config-files-and-possible-solutions/12987
• 7: https://doc.rust-lang.org/cargo/reference/config.html?highlight=git%2Crev
• 8: https://doc.rust-lang.org/1.84.0/cargo/reference/config.html
• 9: https://doc.rust-lang.org/1.63.0/cargo/reference/config.html
• 10: https://dev-doc.rust-lang.org/cargo/reference/config.html
• 11: Merged configuration lists have unexpected precedence ordering rust-lang/cargo#12506

---

🌐 Web query:

Cargo command config file discovered current directory current working directory manifest path

💡 Result:

Cargo discovers configuration files (.cargo/config.toml) by traversing upward from the current working directory [1][2]. It searches the current directory and every parent directory until it reaches the root of the filesystem [1][2]. Additionally, Cargo reads a global configuration file located at $CARGO_HOME/config.toml (which defaults to $HOME/.cargo/config.toml on Unix or %USERPROFILE%.cargo\config.toml on Windows) [1][2]. Crucially, this hierarchical search is based on the current working directory at the time Cargo is invoked, not on the location of the project's Cargo.toml manifest file [3][4]. If you use the --manifest-path option to specify a Cargo.toml file in a different directory, Cargo will still search for configuration files starting from your current working directory, which can lead to unexpected behavior if configuration files exist in those directories [3][4]. For manifest path discovery, Cargo searches upward for a file named Cargo.toml starting from the current working directory [5]. You can use the cargo locate-project command to print the path to the Cargo.toml file that Cargo has resolved [5]. Precedence for configuration values (from lowest to highest) is generally as follows [6][7]: 1. Configuration files in parent directories (hierarchical search) [1][6]. 2. Configuration file in the current working directory [6]. 3. Environment variables (CARGO_*) [6][7]. 4. Command-line overrides (e.g., --config) [6][7]. Note that for configuration files, paths are relative to the parent directory of the directory where the config file is defined [1][2]. For environment variables and --config overrides, paths are relative to the current working directory [1][2].

Citations:

• 1: https://doc.rust-lang.org/cargo/reference/config.html
• 2: https://doc.rust-lang.org/cargo/reference/config.html?highlight=rustflags
• 3: Cargo does not always search for .cargo/config file in project root rust-lang/cargo#2930
• 4: Wrong cargo config resolution when using manifest-path rust-lang/cargo#3808
• 5: https://doc.rust-lang.org/beta/cargo/commands/cargo-locate-project.html
• 6: https://blog.rust-lang.org/inside-rust/2023/08/24/cargo-config-merging/
• 7: Merged configuration lists have unexpected precedence ordering rust-lang/cargo#12506

---

Run the Cargo command from $ROOT before using --manifest-path.

Cargo resolves .cargo/config.toml from the current working directory, not from the manifest path. Since rustc flags like target-cpu are configured there, running this script from outside the repo root or a subdirectory can bypass the intended x86-64-v3 pin and produce incompatible baseline output.

🔧 Proposed fix

 ROOT="$(cd "$(dirname "$0")/.." && pwd)"
 MANIFEST="$ROOT/crates/simd-codegen-oracle/Cargo.toml"
+cd "$ROOT"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	ROOT="$(cd "$(dirname "$0")/.." && pwd)"
	MANIFEST="$ROOT/crates/simd-codegen-oracle/Cargo.toml"
	TARGET="${1:-}"
	case "$TARGET" in
	"" | -*)
	TARGET="$(rustc -vV | sed -n 's/^host: //p')"
	;;
	*)
	shift
	;;
	esac
	# Remaining args (an optional `--` and/or analyzer flags like --verbose) forward as-is.
	if [ "${1:-}" = "--" ]; then
	shift
	fi
	BASELINE="$ROOT/crates/simd-codegen-oracle/baselines/$TARGET.toml"
	if [ ! -f "$BASELINE" ]; then
	echo "==> no baseline for target $TARGET at $BASELINE" >&2
	exit 90
	fi
	echo "==> building simd-codegen-oracle (--emit asm) for $TARGET"
	if [ "$TARGET" = "$(rustc -vV | sed -n 's/^host: //p')" ]; then
	cargo rustc --release --manifest-path "$MANIFEST" -- --emit asm -C debuginfo=0
	else
	cargo rustc --release --manifest-path "$MANIFEST" --target "$TARGET" -- --emit asm -C debuginfo=0
	fi
	ROOT="$(cd "$(dirname "$0")/.." && pwd)"
	MANIFEST="$ROOT/crates/simd-codegen-oracle/Cargo.toml"
	cd "$ROOT"
	TARGET="${1:-}"
	case "$TARGET" in
	"" | -*)
	TARGET="$(rustc -vV | sed -n 's/^host: //p')"
	;;
	*)
	shift
	;;
	esac
	# Remaining args (an optional `--` and/or analyzer flags like --verbose) forward as-is.
	if [ "${1:-}" = "--" ]; then
	shift
	fi
	BASELINE="$ROOT/crates/simd-codegen-oracle/baselines/$TARGET.toml"
	if [ ! -f "$BASELINE" ]; then
	echo "==> no baseline for target $TARGET at $BASELINE" >&2
	exit 90
	fi
	echo "==> building simd-codegen-oracle (--emit asm) for $TARGET"
	if [ "$TARGET" = "$(rustc -vV | sed -n 's/^host: //p')" ]; then
	cargo rustc --release --manifest-path "$MANIFEST" -- --emit asm -C debuginfo=0
	else
	cargo rustc --release --manifest-path "$MANIFEST" --target "$TARGET" -- --emit asm -C debuginfo=0
	fi

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/codegen-oracle.sh` around lines 19 - 47, Update the Cargo invocations
in the target-building logic to execute from $ROOT, such as by wrapping each
command with a subshell that changes to $ROOT before running Cargo. Preserve the
existing manifest, target, and rustc emission arguments while ensuring
.cargo/config.toml is resolved from the repository root.