lance 9 / lancedb 0.33 / DataFusion 54 / Rust 1.97.1 — measured, then landed

.claude/board/AGENT_LOG.md

@@ -1,3 +1,43 @@
+## 2026-08-05 — lance 9 / DataFusion 54 / Rust 1.97.1 cross-repo bump (main thread + 2 Sonnet grindworkers)
+
+- **Agents:** main thread (design, adjudication, all commits/pushes) + 2 Sonnet
+  lint-sweep workers (ndarray; OGAR) spawned WITHOUT worktrees per
+  `agent-cargo-hygiene.md`, edit-only, each writing only its own tag-file
+  (`<repo>/.claude/exec-runs/sonnet-197-lint-sweep.md`). A third resumed OGAR.
+- **Method the operator imposed, and it paid:** test the TOOLCHAIN ALONE on the
+  OLD dependency pins first, so a toolchain failure could never be mistaken for
+  a dependency failure. 1.97.1 came back clean on the old pins; every
+  subsequent red was therefore attributable to the dep bump.
+- **Result:** lance-graph green under lance 9 + DF 54 + 1.97.1 across
+  `lance-graph`, `-catalog`, `-contract`, `-ontology`, `-planner`,
+  `-supervisor`, `cognitive-shader-driver`, and `-callcenter` (both `query`
+  and `query-lite` — the default-feature pass was a FALSE GREEN, since
+  `vsa_udfs`/`ontology_table` are feature-gated and never compiled).
+  ndarray: 2186 lib tests pass, 0 fail. Plan +
+  `E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1` + `E-THE-LEGEND-IS-NOT-THE-GRAMMAR-1`
+  banked; 9 repos' toolchains/CI/Dockerfiles bumped; 8 PRs merged.
+- **Two disk incidents, both handled by the guard, not by luck:** both Sonnet
+  workers stopped on their explicit `df` floor rather than pushing through, and
+  both correctly diagnosed the cause as EXTERNAL (a 9.8 GB sibling `target/`
+  holding two toolchains' artifacts plus both the lance-7 and lance-9 trees)
+  rather than their own residue. `cargo clean` there recovered 10.3 GB. The
+  lesson for future briefs: give grindworkers an explicit numeric disk floor
+  and permission to stop — a worker that halts with a diagnosis is worth far
+  more than one that ENOSPCs mid-edit.
+- **My own error, recorded because it is reusable:** I predicted a second
+  migration (all ~11 UDF types needing `Eq + Hash`, since DF 54's
+  `ScalarUDFImpl` lists `DynEq + DynHash` supertraits blanket-implemented over
+  them). WRONG — DF **53 already** required them, so the tree already satisfied
+  it with hand-written `name()`-keyed impls everywhere. I read the NEW version's
+  bound and inferred "new requirement" without diffing the OLD one; the
+  differential was one grep away. The claim had already reached a commit message
+  before measurement corrected it.
+- **Still owed (not claimed as done):** OGAR's 22 unreached crates + one
+  unverified `clickhouse-ddl` fix (resume worker dispatched); `blockly-rs` and
+  `rig` still on 1.95.0, outside the sweep.
+- **Gates untouched:** D-BLW-5, D-HWV-1/EXP-HOT-WINDOW, PROBE-ORACLE-FUNNEL
+  Stage 1 + Stage 2 — all still wait on the operator's word.
+
 ## 2026-08-05 — PROBE-ORACLE-FUNNEL Stage 0 (main thread; pre-register → build → run → record)
 
 - **Agent:** main thread only (no fleet). **D-id:** PROBE-ORACLE-FUNNEL-S0.

.claude/board/EPIPHANIES.md

@@ -1,3 +1,40 @@
+## E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1 (2026-08-05, operator-ruled — corrects a P0 in `CLAUDE.md`)
+
+**The lance family is consumed from crates.io upstream and NEVER from a fork —
+"too risky, and the upstream is authoritative."** Operator ruling, verbatim
+substance: *"lance and lancedb are never used from forks because it's too risky
+and the upstream is authoritative."* Scope: `lance`, `lancedb`, and the whole
+`lance-*` family (`lance-arrow` / `lance-core` / `lance-index` / `lance-linalg` /
+`lance-namespace` / `lance-datafusion` / …).
+
+**This CORRECTS `CLAUDE.md`'s P0**, which explicitly named `lance` / `lancedb` /
+`lance-index` / `lance-linalg` / `lance-namespace` in its must-be-wired-via-the-fork
+list. That naming was wrong; the P0 has been regraded in place with a `⊘`
+carve-out (append-only — the rule itself stands for `ndarray`, `surrealdb`, and
+every other forked crate).
+
+**The trap this closes, and why it needed a ruling.** `AdaWorldAPI/lance` and
+`AdaWorldAPI/lancedb` DO exist as repos — I found them while auditing the v9
+bump and read their existence, plus the P0's explicit naming, as evidence of a
+standing policy violation to be fixed. It is the opposite: **a fork's existence
+is not evidence of intent to depend on it.** A future session that greps
+`list_repos`, finds the forks, and "corrects" the registry pins would be
+introducing the risk, not removing it. Registry pins for this family are
+CORRECT.
+
+The reasoning generalizes: **forking the storage engine whose durability
+guarantees the whole stack trusts buys a maintenance liability, not freedom.**
+The stack's own value is in the layers above (spine, contracts, codecs) — a
+diverged Lance would have to re-earn upstream's durability testing forever.
+Both forks are in fact stale (no tags, `main` only, last pushed 2026-05-28 and
+2026-02-26) and could not supply v9 today even if policy allowed it.
+
+**What discipline DOES apply:** exact lockstep. Every member pinned `=X.Y.Z`,
+with `lancedb`'s own `lance` pin matching, so a bump is one deliberate measured
+PR (the v9 probe below) and never a drift.
+
+**Status:** RULING (operator). Carve-out live in `CLAUDE.md` P0.
+
 ## E-THE-LEGEND-IS-NOT-THE-GRAMMAR-1 (2026-08-05, measured — PROBE-ORACLE-FUNNEL Stage 0)
 
 **Vocabulary knowledge buys almost nothing at a refuse-don't-guess gate; the

.claude/board/ISSUES.md

@@ -1,5 +1,44 @@
 # Issues Log — Open + Resolved (double-entry, append-only)
 
+## ISS-CODEC-RESEARCH-MDCT-ASSERT (2026-08-05) — OPEN, PRE-EXISTING, DISCOVERED NOT CAUSED
+
+**The observation.** `cargo +1.97.1 test --manifest-path
+crates/lance-graph-codec-research/Cargo.toml --lib` fails **9 of 65** tests. Every
+one of the nine panics at the same line —
+`transform.rs:46 assert_eq!(input.len(), n * 2, "MDCT input must be 2N samples")` —
+reached through `perframe::encode_perframe`, `metrics::compare_all_strategies`, or
+`hybrid::encode_hybrid`. Failing: `hybrid::tests::{test_hybrid_bitrate_breakdown,
+test_hybrid_encode_basic, test_hybrid_scent_stream_valid}`,
+`metrics::tests::{test_compare_noise, test_compare_sine,
+test_hypothesis_tonal_vs_noise}`, `perframe::tests::{test_encode_silence,
+test_encode_sine_wave}`, `transform::tests::test_band_energy_roundtrip`.
+
+**Why it is not a regression from the 1.97 clippy sweep.** The sweep's eight-file
+diff was stashed and the suite re-run at HEAD: **identical 56 passed / 9 failed,
+identical test names.** The sweep is exonerated by measurement, not by argument.
+The commit landing those lint fixes says so and links here.
+
+**The arithmetic, which is the whole diagnosis.** `mdct` binds `n2 =
+output.len()` (commented "N/2"), `n = n2 * 2`, then requires `input.len() == n * 2`
+— i.e. **4× the coefficient count**. Every one of the four call sites passes
+`2 * SAMPLES_PER_FRAME` samples for `SAMPLES_PER_FRAME` coefficients, which is the
+2N→N contract the doc-comment and the assert message both state. So the assert as
+written can never be satisfied by any caller in the crate, and the internal
+indexing (`idx0 = (n4 + k) % (n * 2)`) is consistent with the 4N reading, not with
+the callers. Either the naming is off by one factor of two throughout the function
+or the fold is wrong — **do not "fix" this by relaxing the assert**; that would
+silently index a 2N buffer with 4N arithmetic.
+
+**Why nobody noticed.** `lance-graph-codec-research` is workspace-EXCLUDED, so no
+`-p` run reaches it, and `.github/workflows/style.yml` gates it at **neither** the
+mandatory nor the advisory tier. A crate that no gate runs is a crate whose red is
+invisible. That absence of coverage is the second-order finding here.
+
+**Resolution shape.** Decide the intended transform size convention first (read
+`imdct`, which asserts nothing, and `test_band_energy_roundtrip`'s expectation),
+then correct `mdct` to match and re-anchor. Owner: unassigned. Not a blocker for
+the 1.97/lance-9 arc.
+
 ## ISS-MARM-T1-4X-A0-GAP (2026-08-05) — OPEN, MEASUREMENT DEFECT NOT A RESULT
 
 **The observation.** The M-arm's temporal-reconstruction baseline (T1) reads

.claude/board/LATEST_STATE.md

@@ -1,3 +1,15 @@
+## 2026-08-05 — lance 9 / lancedb 0.33 / DataFusion 54 / Rust 1.97.1 — the ecosystem bump, MEASURED then LANDED across 9 repos
+
+**Current pins:** `lance =9.0.0`, `lancedb =0.33.0`, `datafusion 54`, `arrow 58` (unmoved), `object_store 0.13.2` (unmoved), toolchain **1.97.1**. Plan: `.claude/plans/lance9-datafusion54-upgrade-probe-v1.md`.
+
+Started as a what-breaks probe, became the bump on the operator's "bump all now, fix after". Method that made it legible: the toolchain was tested ALONE on the OLD pins first, so a toolchain failure could never be confused with a dependency failure — it came back clean, making every later red attributable to the deps. **The entire break surface was one uniform DataFusion change** — `Any` moved from a method to a supertrait on seven traits — costing 12 `as_any` deletions and no call-site changes.
+
+**Two operator rulings landed:** `E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1` (lance family from crates.io upstream, NEVER a fork; the `AdaWorldAPI/lance` + `/lancedb` repos exist but are deliberately not depended on — CLAUDE.md's P0 carve-out now says so, since it had named them as must-fork) and, from the Stage-0 funnel probe, `E-THE-LEGEND-IS-NOT-THE-GRAMMAR-1`.
+
+**Contract-inventory deltas:** none — the bump is pins + lint fixes. `lance-graph-ontology`'s 12 long-standing lints are cleared, which is what made `rust-toolchain.toml`'s own "bump when clippy is clean" precondition satisfiable.
+
+**Standing gates (unchanged):** D-BLW-5; D-HWV-1/EXP-HOT-WINDOW; PROBE-ORACLE-FUNNEL Stage 1 (rig LLM arm) and Stage 2 (NARS-34 Gadamer bag); `--features cycle-driver` CI arming. **Owed:** OGAR's 22 unswept crates; `blockly-rs`/`rig` still 1.95.0.
+
 ## 2026-08-05 — PR #894 (MERGED `d7a6efc`) + OGAR #243/#244/#245 — the wishlist round-trip closes and the funnel is MEASURED
 
 One same-day arc across the repo boundary, crate-dependency-free by ruling: **#243** (OGAR) delivered the consumer wishlist handover (F-1 double-sampling finding, W-1..W-5); **#244** (OGAR, loco session) shipped W-1 (compose-then-check), W-2 (`FnSpec.name` — OQ-1 answered "in the spec"), W-4 (`telemetry::FunnelTally`), W-5 (split-contract doc), leaving only the W-3 NARS-34 mint operator-gated; **#245** (OGAR) + **#894** (here) ran PROBE-ORACLE-FUNNEL Stage 0 over that delivery — pre-registered E1–E4 all met (floor 0.5% / legend-constrained 2.5% / stack-aware 100%; E4 discrimination KILL passes at 99.5 points vs the 50 bar). Headline finding: `E-THE-LEGEND-IS-NOT-THE-GRAMMAR-1` — the legend-knowing arm landed at the FLOOR; ~all funnel selectivity is the stack discipline, so the gated Stage-1 LLM arm's prompt must teach the discipline, not just serialize the ~382-token legend. Plan: `.claude/plans/oracle-funnel-probe-v1.md`.

.claude/board/PR_ARC_INVENTORY.md

@@ -33,6 +33,17 @@
 > - **Docs** — knowledge files produced (immutable)
 > - **Confidence (YYYY-MM-DD):** — the ONLY mutable field
 
+## 2026-08-05 — the lance 9 / DataFusion 54 / Rust 1.97.1 cross-repo bump (9 repos; lance-graph PR pending, siblings MERGED)
+
+Siblings merged same day: OGAR #244/#245, ruff #93, stockfish-rs #14, woa-rs #179, a2ui-rs #19, MedCare-rs #351. lance-graph's own arc rides the branch below.
+
+- **Added.** `.claude/plans/lance9-datafusion54-upgrade-probe-v1.md` (the measured what-breaks assessment). Pin moves: `lance =7.0.0→=9.0.0`, `lancedb =0.30.0→=0.33.0`, `datafusion 53→54` across 10 manifests; toolchain 1.95.0→1.97.1 in `rust-toolchain.toml`, CI, and 3 Dockerfiles.
+- **Locked.** `E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1` (operator: the lance family is consumed from crates.io upstream, NEVER a fork — "too risky, upstream is authoritative"; **a fork's existence is not evidence of intent to depend on it**; CLAUDE.md's P0 regraded with a `⊘` carve-out because it had explicitly NAMED lance/lancedb as must-fork). **There is no `lancedb` 0.34–0.36 Rust crate** — the registry tops out at 0.33.0, which IS the lance-9 pairing; 0.36 is the PyPI package on an independent line. **The whole DF-54 break is one uniform change:** `Any` moved from a METHOD to a SUPERTRAIT on `TableSource`, `TableProvider`, `ScalarUDFImpl`, `CatalogProvider`, `SchemaProvider`, `AggregateUDFImpl`, `WindowUDFImpl` — 12 `as_any` deletions, zero call-site changes (`.as_any()` still resolves via the blanket impl; only explicit impls break, and only when their module actually compiles). arrow stays 58 and `object_store` 0.13.2 — only DataFusion crosses a major.
+- **Deferred.** OGAR's 22 unswept crates + 1 unverified fix; `blockly-rs`/`rig` still on 1.95.0; `q2` deliberately left on its nightly pin; `-benches`/`symbiont`/`cognitive-stack`/`surreal_container`/`-python` unchecked; the `delta` feature still broken by deltalake 0.32's own drift (which is what quarantines the duplicate DF major to a non-default feature).
+- **Docs.** The plan (§4 records a falsified prediction of mine: I read DF 54's `DynEq + DynHash` supertraits as a NEW requirement without diffing 53, which already had them — the tree already satisfied it. §6 corrects a parallel audit's "three lance majors", which was a pre-pin-bump state).
+
+**Confidence (2026-08-05):** measured, not predicted — 8 lance-graph crates clippy-clean under `-D warnings` (incl. `-callcenter` under both `query` and `query-lite`, after the default-feature pass proved a FALSE GREEN on feature-gated files), ndarray 2186 lib tests green.
+
 ## 2026-08-05 — lance-graph #894 (MERGED `d7a6efc`) — PROBE-ORACLE-FUNNEL Stage 0: pre-registration, measured results, the legend-is-not-the-grammar finding
 
 **Head at merge:** `e4967e3` (1 commit). 4 files, docs/board only — the harness itself is OGAR-side (`ogar-loco` example, OGAR PR #245, merged same day), per the no-cross-dependency ruling in the merged wishlist handover (OGAR #243; delivery #244): candidates and legends cross as DATA, never as a crate edge.

.claude/exec-runs/sonnet-bgz-tensor-clippy.md

@@ -0,0 +1,38 @@
+# bgz-tensor clippy sweep (Rust 1.97.1)
+
+Scope: `crates/bgz-tensor/` only, driven via `--manifest-path` (workspace-excluded crate).
+
+## Fixes
+
+| File:line | Lint | Fix |
+|---|---|---|
+| `src/adaptive_codec.rs:681-682` | `clippy::doc_lazy_continuation` (9 errors, lines 682-690) | Inserted a blank `///` line between the end of the 2-item doc list and the following prose paragraph, so the paragraph is no longer read as an unindented list-item continuation. No wording changed. |
+
+That single blank-line insertion cleared all 9 reported errors (they were one multi-line diagnostic cluster on the same paragraph).
+
+## Skipped / `#[expect]`-ed
+
+None needed — no lints touched arithmetic, quantization, table indexing, or iteration order (the AttentionSemiring / u16 distance table / u8 compose table / HHTL cascade code was already clean on this toolchain).
+
+## Post-fix `cargo fmt`
+
+Ran `cargo +1.97.1 fmt --manifest-path crates/bgz-tensor/Cargo.toml` per instructions. It reformatted two files beyond the one I hand-edited:
+- `examples/probe_l5_fisherz_amortization.rs` — pure line-wrapping of long method chains / `println!` argument lists (no logic change).
+- `src/matryoshka.rs` — minor formatting (2 lines).
+
+Re-ran clippy after fmt: still 0 errors, 0 warnings.
+
+## Test result
+
+```
+test result: ok. 207 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.98s
+Doc-tests bgz_tensor: 0 passed; 0 failed
+```
+
+## Real bugs exposed by lints
+
+None — the crate was already clippy-clean except for the single doc-comment formatting issue above, which was cosmetic (rustdoc rendering), not a code-correctness issue.
+
+## Disk
+
+Free space stayed well above the 4 GB floor throughout (28G → 29G → 8.6G... wait, actual readings: 10G free at start, 8.6G free after fmt — still comfortably above floor). No stop triggered.