diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 63c3bd79..58a8432b 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,57 @@ +## 2026-07-30 — E-WORDNET-MAKES-THE-4-ARY-ADDRESS-SEMANTIC-1 — the 4⁴ fold turns HHTL adjacency into a *semantic* prior, and the sub-nibble rung is load-bearing (2.47 hops the 16-ary router cannot address) + +**Status:** FINDING (measured, 5/5 gates green on real WordNet 3.1). **Confidence:** High for the structure claim; explicitly NOT extended to embedding-trained codebooks (see Boundaries). Probe: `crates/lance-graph-contract/examples/probe_wordnet_44_activation.rs`; record: `.claude/board/exec-runs/wordnet-44-activation.md`. + +**Operator anchors (verbatim, arrival order):** +1. *"And it feeds 4^4 micro traversal decision"* / *"Not sure if that's a thing in our substrate"* +2. *"4^4 would be a sparse adjacent if the sentence refers to out of bounds meaning"* +3. *"besides wordnet makes CLAM HHTL spacial activation via 4^4"* + +**What the third anchor changed.** The first plan was to derive a 4-ary hierarchy by k-means over embeddings and hope ancestry emerged — the same shape as `PROBE-CODEBOOK-44`, and gated behind the same Base17 fold ceiling (ρ≈0.26 on single words). Anchor 3 inverts it: **WordNet's `@` hypernym relation IS ground-truth ancestry**, so the address does not have to *discover* structure, it *encodes* structure. That removes embeddings, k-means, and the API key from the critical path and leaves the sharper question — given real ancestry, does a fixed 4-ary depth-4 fold preserve enough semantic distance to be a usable search prior? + +**Measured (82,192 noun synsets, root `entity`, 65,292 leaves, max depth 19):** + +| gate | result | +|---|---| +| W1 ancestry-by-construction | corr(shared address levels, LCA depth) real **+0.494**, shuffled **−0.036** | +| W2 monotone ladder | mean path distance **15.78 → 12.76 → 11.15 → 8.69 → 7.05** across rungs 0→4; strictly decreasing, spread **8.73 hops** | +| W3 spatial activation | **out-of-cell, FULL corpus**: band **0.763** vs random **0.031** = **24.71×**; silent half = the random arm's inertness (cover guard dropped as inert — see correction) | +| W4 sub-nibble structure | inside one top nibble the 16-ary router sees ONE bucket (mean 10.55); 4-ary splits it **11.15 vs 8.69** = **2.47 hops** | +| W5 fold balance | 256/256 cells used, occupancy min 29 / median 255 / max 1270 | + +**The headline is W4.** It answers "not sure if that's a thing" with a number: the sub-nibble decision is not decoration. `NiblePath` (`FAN_OUT = 16`, the shipped router) can express exactly two levels in a byte; inside one top nibble it sees a single undifferentiated bucket. The 4-ary address splits that same population into two rungs that differ by 2.47 WordNet hops. **That is real structure the current router is blind to** — and it is what makes anchor 2's "sparse adjacent" implementable: 3 siblings × 4 levels = 12 of 256 cells (4.7 %), a graded escalation ladder, where 16-ary offers 15 nibble-siblings or nothing. + +**The three-rung reference taxonomy this licenses** (in-cell / sparse-adjacent / beyond-adjacency) now has its middle rung measured. W3 is the load-bearing half: the band is a *discriminating prior*, beating random 1.50× while deliberately NOT covering everything (0.855 < 0.95). Both halves were required — a band capturing ~everything would carry exactly as much information as one capturing nothing (`E-ANTI-EIGENVALUE-MACHINERY-CAN-ITSELF-BECOME-THE-EIGENVALUE-1`). + +**Two failures on the first run are part of the finding, not noise:** +1. **The first fold reproduced the le-contract's own warning.** Expanding each level to exactly `ARITY` roots before balancing gave one cell **15,769 leaves against a median of 20** — "lacking proper bucket rollover … saturates silently", in my own code. LPT can only isolate a giant subtree and hope the next level splits it; at the terminal level there is no next level. Fix: expand to `ARITY·24` roots so the balancer has fine-grained items. **Consequence for the substrate: a 4-ary fold of any real taxonomy needs an explicit granularity knob — the arity alone does not balance it.** +2. **The W1 falsifier was mis-designed and the number exposed it.** A label permutation is a bijection, so same-cell pairs stay same-cell; the shuffle arm scored **+0.645** on pure cell identity while appearing to validate prefix structure. Restricting both arms to different-cell pairs collapsed it to −0.036. **A falsifier that cannot destroy the thing it tests is the vacuous-assertion pattern wearing a lab coat.** + +**⊘ W3 CORRECTED IN PLACE (pre-merge, codex/CodeRabbit review of #875) — the original W3 was BOTH mislabelled and arithmetically mis-specified, and the corrected result is ~10× stronger:** + +1. **Mislabelled metric.** The candidate pool was sampled WITH replacement and the 32 nearest taken without dedup, so one synset could occupy several slots — a weighted sampled-entry recall, not "recall of the 32 nearest neighbours". Both arms shared the bias so the *ratio* survived, but the label was false. Pool now deduped. +2. **The twin gate could barely be satisfied.** Both arms credited the anchor's OWN cell, so same-cell neighbours inflated the baseline to 0.621 — capping the achievable ratio at 1/0.621 = **1.61** while the gate demanded > 1.5 and the cover guard demanded < 0.95. That is a **0.018-wide window**: the original 1.50× pass was luck, not evidence. After dedup it fell to 1.44× and correctly FAILED. +3. **The fix is to measure the actual claim, not to move the bar.** Sparse adjacency is about references landing OUT OF CELL (*"if the sentence refers to out of bounds meaning"*); a neighbour already in the home cell needs no band to reach it. Restricting to out-of-cell neighbours: **band 0.754 vs random 0.052 = 14.43×**. The null validates itself — random-12 scores 5.2 %, against an expected 12/255 ≈ 4.71 % (the home cell is excluded from the out-of-cell draw, so the denominator is 255, not the 256 used for the band's share of the whole codebook). **Home-cell inflation was MASKING the effect, not creating it.** +4. **The `< 0.95` cover guard is now calibrated rather than asserted.** The same measurement against a deliberately coarse 2-level address (6+1 of 16 cells) yields **0.998**, which the guard rejects — so 0.95 demonstrably separates a prior from a cover on this data, satisfying the workspace inertness rule (a threshold that never bites is decoration). + +The all-neighbour figure is retained as SECONDARY and labelled saturating: 0.895 vs 0.621 = 1.44×. + +**⊘ W3 ROUND TWO (codex P1 on #875) — the sampled pool, and the twin gate's THIRD mis-specification:** + +- **Sampled pool.** 20,000 of 65,292 addressed leaves (31 %), so "the 32 nearest neighbours" meant "within a sample" — an omitted leaf can be strictly closer, moving both the cutoff and the cells. Fixed by scoring the **full corpus** (~20 M tree walks, seconds) rather than relabelling; dedup becomes moot since `addressed` is distinct by construction. Result strengthens: **band 0.763 vs random 0.031 = 24.71×**, null tightening to 0.031 against 12/255 ≈ 0.047. +- **The gate was still incoherent.** The fire half had moved to out-of-cell recall while the cover guard stayed on the all-neighbour statistic just declared wrong — and that secondary drifted to 0.941, nine thousandths from tripping a guard that should never have gated it. **Both halves must measure the same quantity on the same basis.** +- **Recomputed like-for-like, the cover guard is INERT → DROPPED.** A coarse 2-level address scores **0.840** on the out-of-cell basis, not >0.95: the threshold separated nothing, and only the degenerate band-is-everything case would trip it. Structurally unnecessary too — a band fixed at 4.7 % of the codebook with no home-cell credit cannot be secretly dense. **The honest silent half was already there: the random arm (0.031) — same statistic, same basis, wrong cells, nothing found.** Keeping an inert threshold is the exact defect this probe exists to catch. + +**Generalized:** THREE defects in this one probe were mis-specified falsifiers (non-falsifying shuffle, hair-wide twin window, inert cover guard), and zero were wrong measurements. **The falsifiers deserve more scrutiny than the findings, because a broken falsifier fails silently — it reports PASS.** + +**Division of labour, operator-fixed same session** (*"and CLAM to calculate, that's established"* / *"alternative is using HHTL+ helix residue"*): the 4⁴ fold is the **ADDRESS** (which cell, which 12-cell band); **CLAM is the CALCULATOR and is established — do not re-derive it**; **HHTL + helix residue** (place deterministic, residue stored) is the named alternative calculator, unmeasured against CLAM. This probe's LCA walk is an **ORACLE** — scoring only, never a runtime path, exactly as tesseract-rs oracles link libtesseract to produce ground truth and never ship. Nothing here should be read as "the substrate computes distance by walking to an LCA." The address says WHERE to look; CLAM computes. The fork matters *because* of W4: a finer address makes more of the answer deterministic-place and less of it stored-residue, so 2.47 hops is precisely the granularity a residue would no longer carry — **hypothesis, not result**; the head-to-head (`PROBE-CLAM-VS-HELIX-RESIDUE`) is queued. + +**Boundaries (do not cite past these):** +- STRUCTURE probe, not a codec probe. Says nothing about whether an embedding-trained 4⁴ codebook inherits these properties — that remains Phase B, still gated by the Base17 fold ceiling. +- WordNet nouns are a DAG; this takes the FIRST `@` parent only. Both comparison arms see the same tree so it cannot flatter results, but **a concept with two genuine parents is representable at only one address** — a real limit of any fixed-arity address. Visible in the worked example: `dog` lands among `man`/`foster-brother`/`macho`/`sirrah` (WordNet's informal-term-for-a-man sense), not the animal. +- Path length is a crude metric (Resnik/Lin information content would weight by corpus frequency). Used because it is what the address approximates — both are pure structure. +- 4-ary is measured BETTER here; it is not measured CHEAPER. No traversal-cost benchmark was run, and `RouteAction`'s four variants (Skip/Attend/Compose/Escalate = exactly 2 bits) being the same width as a rung is an observed consonance, NOT a wired mechanism. + ## 2026-07-30 — E-64K-THOUGHTS-DONT-DO-QUORUM-PLASTICITY-BREATHES-FROM-TENSION-1 — operator ruling: the 64k parallel thoughts do NOT do quorum; inconsistency is plasticity's work; contradictions are the substrate's fuel; and the honesty line runs between EPISODIC and EPISTEMIC causality **Status:** OPERATOR-RULED (mid-W7-review, verbatim anchors below). **Confidence:** High as a direction ruling; consequences applied same-commit (the quorum-binder recommendation in `E-A-CHIP-BEARS-LOAD-OR-IT-IS-A-JOKE-1` is withdrawn in place, pre-merge). diff --git a/.claude/board/exec-runs/wordnet-44-activation.md b/.claude/board/exec-runs/wordnet-44-activation.md new file mode 100644 index 00000000..04273ab6 --- /dev/null +++ b/.claude/board/exec-runs/wordnet-44-activation.md @@ -0,0 +1,207 @@ +# PROBE-WORDNET-44-ACTIVATION — execution record + +Operator directive: *"Feel free to document your ideas and test it, besides +wordnet makes CLAM HHTL spacial activation via 4^4"* (2026-07-30). Probe-first +rule applied: the deliverable is the falsifier, not more synthesis. + +## Files + +- NEW `crates/lance-graph-contract/examples/probe_wordnet_44_activation.rs` + (the probe; zero-dep crate, ~470 lines, 5 gates). +- NEW `crates/lance-graph-planner/examples/data/wordnet/build_isa_tree.py` + (the corpus generator; lives beside its `build_wordnet_rail.py` sibling per + this directory's convention — generator committed, data gitignored). +- Board: `EPIPHANIES.md` prepend (`E-WORDNET-MAKES-THE-4-ARY-ADDRESS-SEMANTIC-1`), + this file, `bf16-hhtl-terrain.md` routing row. Same commit (board-hygiene rule). + +## Corpus (LOCAL-ONLY, gitignored) + +WordNet 3.1 `dict` from Princeton → `/tmp/wn31/`. Generator emits +`child_offset / parent_offset / depth / lemma`: **82,192 noun synsets, one root +(`entity`), 65,292 leaves, max depth 19.** First `@`/`@i` parent only (DAG→tree; +declared, and both comparison arms see the same tree). + +## What the probe asks + +NOT "can k-means find hierarchy" (that is `PROBE-CODEBOOK-44`, and its real-data +leg is capped by a Base17 fold ceiling ρ≈0.26 on single words). WordNet supplies +ancestry as ground truth, so the question sharpens to: **does folding real +taxonomic ancestry into a fixed 4-ary depth-4 address preserve semantic distance +well enough that address-adjacency is a usable search prior — and does 4-ary buy +anything the shipped 16-ary `NiblePath` cannot express?** + +## Gates — 5/5 GREEN + +| gate | result | +|---|---| +| W1 ancestry-by-construction (shuffle falsifier) | real **+0.4938**, shuffled **−0.0356** | +| W2 monotone ladder | 15.78 → 12.76 → 11.15 → 8.69 → 7.05; strictly decreasing, spread **8.73 hops** | +| W3 spatial activation (twin-tested) | **out-of-cell, FULL corpus**: band **0.763** vs random **0.031** = **24.71×**; silent half = the random arm's inertness (see § W3 correction) | +| W4 sub-nibble structure | nibble sees ONE bucket (10.55); 4-ary splits 11.15 vs 8.69 = **2.47 hops** | +| W5 fold balance | 256/256 cells, occupancy 29 / 255 / 1270 (min/median/max) | + +Worked example (nearest in-band by hops): `dog` @ cell `0x5a` (`01|01|10|10`) → +`foster-brother(3), macho(3), man(3), mother's_son(3), sirrah(3), Adam(4)`; +best distance OUTSIDE the band = 4 hops. + +## The two first-run failures (kept — they are part of the finding) + +1. **W5 max cell 15,769 vs median 20.** Expanding each level to exactly `ARITY` + roots before balancing reproduces the le-contract's own warning — "lacking + proper bucket rollover … saturates silently" — in this probe's code. LPT can + only isolate a giant subtree and hope the next level splits it; at the + terminal level there is none. Fixed by expanding to `ARITY·24` roots + (`GRAIN`) so the balancer has fine-grained items. **Substrate consequence: a + 4-ary fold of a real taxonomy needs an explicit granularity knob; arity alone + does not balance it.** +2. **W1's falsifier could not falsify.** A cell-label permutation is a + bijection, so same-cell pairs stay same-cell and the shuffle arm scored + **+0.645** on cell identity alone — while looking like validation. Both arms + now exclude same-cell pairs (the claim is about levels 1–3); shuffled + collapses to −0.036. Cross-ref `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`: + this is that pattern inside a falsifier. + +Also corrected pre-commit: the worked example printed the first six band +residents in FILE ORDER, which reads as a semantic result while being +arbitrary. Now prints nearest-by-path plus the best out-of-band distance. + +## Verification + +- `cargo run --release -p lance-graph-contract --example probe_wordnet_44_activation` + → ALL GATES GREEN (re-run after fmt: still green). +- `cargo clippy -p lance-graph-contract --example probe_wordnet_44_activation + -- -D warnings` → exit 0. +- `cargo fmt -p lance-graph-contract` → applied. +- Scoped `-p` per repo practice; the workspace-wide sweep is + `TD-WORKSPACE-FMT-DRIFT` (measured: 1,094 hunks across 64 files in 9 crates, + none in these files). + +## Honest boundaries + +- STRUCTURE probe, not a codec probe — nothing here transfers automatically to + an embedding-trained 4⁴ codebook (Phase B, still fold-ceiling-gated). +- First-parent-only means a genuinely polysemous concept gets ONE address. The + `dog` example shows it: WordNet's informal-term-for-a-man sense, not the + animal. This is a real limit of any fixed-arity address, reported not hidden. +- Path length ignores information content (Resnik/Lin would weight by corpus + frequency). Used because it is what the address approximates. +- **4-ary measured BETTER, not CHEAPER.** No traversal-cost benchmark ran. + `RouteAction`'s four variants (Skip/Attend/Compose/Escalate = 2 bits) matching + a rung's width is an observed consonance, NOT a wired mechanism. + +## W3 correction (pre-merge, from the #875 review) — two defects, and the fixed result is ~10× stronger + +Review found the metric mislabelled and the gate arithmetically mis-specified. +Both are confirmed; neither was fixed by moving a threshold. + +1. **Mislabelled.** The candidate pool was sampled WITH replacement and the + nearest 32 taken without dedup, so one synset could fill several slots — + a weighted sampled-entry recall, not "recall of the 32 nearest neighbours". + Both arms shared the bias so the ratio survived, but the LABEL was false. + Pool is now deduplicated. +2. **The twin gate left a 0.018-wide window.** Both arms credited the anchor's + OWN cell, inflating the baseline to 0.621 — which caps the achievable ratio + at `1/0.621 = 1.61` while the fire-half demanded `> 1.5` and the cover guard + demanded `< 0.95`. Passing that the first time was luck. After dedup it read + 1.44× and correctly FAILED. +3. **Fix = measure the claim, not lower the bar.** Sparse adjacency is about + references landing OUT OF CELL (the operator's *"if the sentence refers to + out of bounds meaning"*); a neighbour already in the home cell needs no band + to reach. Out-of-cell only: **band 0.754 vs random 0.052 = 14.43×**. The null + validates itself — random-12 scores 5.2 %, against an expected 12/255 ≈ 4.71 %. + (255, not 256: the home cell is excluded from the out-of-cell draw. 12/256 is a + different quantity — the band's share of the WHOLE codebook. Both round to + 4.7 %, which is why the conflation is easy and worth pinning.) **Home-cell inflation was MASKING the effect.** +4. **The cover guard is now calibrated.** The same measurement against a + deliberately coarse 2-level address (6+1 of 16 cells) yields **0.998**, which + the guard rejects — so `0.95` demonstrably separates a prior from a cover on + this data. Satisfies the inertness rule: a threshold that never bites is + decoration. + +All-neighbour figure retained as SECONDARY and labelled saturating (0.895 vs +0.621 = 1.44×). **Lesson for the next twin gate: check that the two halves are +mutually satisfiable by more than a hair BEFORE running — compute the maximum +achievable value of the fire statistic under the silent guard.** + +## W3, round two (codex P1 on #875) — sampled pool, and an incoherent twin gate + +**P1: the pool was 20,000 of 65,292 addressed leaves (31 %).** "The 32 nearest +WordNet neighbours" therefore meant "the 32 nearest *within a sample*" — an +omitted leaf can be strictly closer than every sampled one, moving both the +distance cutoff and which cells the neighbours occupy. Same class of defect as +the dedup one: the label claimed more than the measurement. **Fixed by scoring +all 65,292** (~20 M tree walks, seconds in release) rather than relabelling — +which also makes dedup moot, since `addressed` is distinct by construction. + +Full-corpus result: **band 0.763 vs random 0.031 = 24.71×** (was 14.43× on the +sample). The null tightens to 0.031 against the 12/255 ≈ 0.047 expectation. + +**Then the full-corpus run exposed that the twin gate was STILL incoherent** — +the third instance of the same mistake in this probe. The fire half had moved to +out-of-cell recall while the cover guard stayed on the all-neighbour statistic +that had just been declared the wrong one; and that secondary had drifted to +0.941, nine thousandths from tripping a guard it should never have been gating. +**Both halves must measure the same quantity on the same basis.** + +**Recomputed like-for-like, the cover guard is INERT and was DROPPED.** On the +out-of-cell basis a coarse 2-level address scores **0.840**, not >0.95 — so the +threshold separated nothing, and only the degenerate band-is-the-whole-codebook +case would trip it. It is also structurally unnecessary for the primary: a band +fixed at 4.7 % of the codebook with no home-cell credit *cannot* be secretly +dense. The saturation worry was only ever real for the secondary statistic. + +**The honest silent half was already present:** the random arm. Same statistic, +same basis, same machinery, wrong cells → 0.031. That is a real can-it-stay- +silent demonstration; the cover guard was a second, redundant, non-firing one. + +Per the inertness rule (*a threshold that never bites is decoration*), keeping it +would have been the very defect this probe exists to catch. + +## Division of labour — ADDRESS vs CALCULATOR vs ORACLE (operator, 2026-07-30) + +Operator ruling, mid-run: *"and CLAM to calculate, that's established"* / +*"alternative is using HHTL+ helix residue"*. This fixes a boundary the probe +must not blur: + +| role | who | status | +|---|---|---| +| **ADDRESS / activation** | the 4⁴ fold — which cell, which 12-cell band | measured here, 5/5 green | +| **CALCULATOR** | **CLAM** (ndarray: build + search + `rho_nn`, 46 tests) | **ESTABLISHED — do not re-derive** | +| *alternative calculator* | **HHTL + helix residue** (place deterministic, residue stored) | the named fork; unmeasured against CLAM | +| **ORACLE** | this probe's `path_distance` LCA walk | scoring only — NEVER a runtime path | + +**The correction this makes:** nothing in this probe should be read as "the +substrate computes semantic distance by walking to an LCA." It does not. The +LCA walk exists here for the same reason the tesseract-rs oracles link +libtesseract — to produce ground truth a transcode can be scored against, never +to ship. The address says WHERE to look; CLAM (or HHTL+helix residue) computes. + +**Why the fork is a real choice, not a preference.** CLAM calculates over a +built tree — its cost is tree traversal plus `rho_nn` at the leaf. HHTL+helix +splits differently: PLACE is regenerated deterministically from the address +(never stored) and only the RESIDUE is read, so the calculation rides the +address rather than a second structure. The 4⁴ result matters to that fork +because a finer address makes a larger share of the answer deterministic-place +and a smaller share stored-residue — W4's 2.47 hops is exactly the granularity +the residue would no longer have to carry. **Unmeasured; the head-to-head is +the next probe, not an assertion here.** + +## Follow-ons (not done here) + +- **PROBE-CLAM-VS-HELIX-RESIDUE** (the fork above): same 4⁴-addressed corpus, + two calculators — CLAM tree search vs HHTL+helix place/residue — scored + against this probe's oracle. Gates: agreement with the oracle (both must + reproduce it, else one is simply wrong), and cost per resolved query. The + interesting hypothesis is that a finer address shifts work from stored + residue to deterministic place; the falsifier is that residue size does not + move with address granularity, which would make 4⁴ irrelevant to the + calculator and leave it an addressing-only win. +- Feed the three-rung reference taxonomy (in-cell / sparse-adjacent / + beyond-adjacency) into `PROBE-FREE-ENERGY-DESCENT` Phase A as the escalation + classifier — metonymy should resolve in-band, metaphor should not. CLAM is + the calculator there too; the descent measures F per tier, it does not invent + a distance function. +- Phase B: nested 4×4 codebook build (two 4-way splits per 16-way level) so + sub-nibble ancestry exists for EMBEDDING-trained codebooks too. W4's 2.47-hop + gap is the justification; the Base17 fold ceiling is the standing blocker. +- A traversal-cost benchmark, to convert "better" into "cheaper" or refute it. diff --git a/crates/lance-graph-contract/examples/probe_wordnet_44_activation.rs b/crates/lance-graph-contract/examples/probe_wordnet_44_activation.rs new file mode 100644 index 00000000..00df28c5 --- /dev/null +++ b/crates/lance-graph-contract/examples/probe_wordnet_44_activation.rs @@ -0,0 +1,789 @@ +//! PROBE-WORDNET-44-ACTIVATION — does a 4⁴ address make HHTL adjacency +//! *semantic*, and does 4-ary buy resolution the nibble cannot? +//! +//! ## The claim under test (operator, 2026-07-30) +//! +//! > "wordnet makes CLAM HHTL spatial activation via 4^4" +//! +//! and, one message earlier: +//! +//! > "4^4 would be a sparse adjacent if the sentence refers to out of bounds +//! > meaning" +//! +//! Read together: when a reference lands outside the current cell, the first +//! hypothesis set should NOT be "the whole codebook" and should NOT be an +//! immediate cross-domain jump — it should be the **sparse adjacent band**, the +//! ancestry-neighbours reachable by flipping the last 2-bit group, then the one +//! above it, and so on. At 4-ary that band is `3 siblings × 4 levels = 12` cells +//! out of 256 (**4.7 %**) — a graded ladder. At 16-ary ([`NiblePath`], the +//! shipped router, `FAN_OUT = 16`) a byte has only TWO levels, so the ladder has +//! two rungs and no sub-nibble structure exists at all. +//! +//! The reason WordNet is the right corpus is that it removes the hard part: +//! [`Palette::build_hierarchical`]-style codebooks must *discover* ancestry with +//! k-means and hope it survives, whereas WordNet's `@` hypernym relation IS +//! ground-truth ancestry. So this probe does not ask "did clustering find +//! structure?" — it asks the sharper question: **given real taxonomic ancestry, +//! does folding it into a fixed 4-ary depth-4 address preserve semantic +//! distance well enough that address-adjacency is a usable search prior?** +//! +//! ## Honest boundaries (read before citing) +//! +//! - **This is a STRUCTURE probe, not a codec probe.** No embeddings, no +//! Base17, no k-means. It measures the fold, nothing downstream of it. It +//! says nothing about whether an *embedding-trained* 4⁴ codebook inherits +//! these properties — that is the separate Phase-B question, and +//! `PROBE-CODEBOOK-44`'s real-data leg already warns that a Base17 fold +//! ceiling (ρ≈0.26 on single words) caps codebook fidelity independently. +//! - **WordNet's noun graph is a DAG; this takes the FIRST `@` parent only**, +//! turning it into a tree. Multiple inheritance is real and is dropped. Both +//! sides of every comparison below see the same tree, so the simplification +//! cannot flatter the result — but a concept with two genuine parents is +//! representable at only one address, which is a real limit of any +//! fixed-arity address and is reported, not hidden. +//! - **Path length is a crude semantic metric** (it ignores information +//! content — Resnik/Lin would weight by corpus frequency). It is used because +//! it is the metric the ADDRESS is trying to approximate: both are pure +//! structure. A frequency-weighted rerun is future work. +//! +//! ## Gates +//! +//! - **W1 ancestry-is-by-construction** — a prefix of the address is a genuine +//! fold-ancestor, and a SHUFFLED address assignment (same cell sizes, permuted +//! labels) destroys it. Anti-vacuity: the shuffle arm must fail. +//! - **W2 monotone ladder** — mean WordNet path distance must increase strictly +//! with adjacency rung: same-cell < share-3 < share-2 < share-1 < share-0. +//! This IS the sparse-adjacency claim. Falsifier: any inversion, or a flat +//! profile (address distance carrying no semantic signal). +//! - **W3 spatial activation** — for an anchor concept, do its 12 ancestry- +//! adjacent cells actually hold its semantic neighbours? +//! - **PRIMARY statistic: OUT-OF-CELL recall, scored over the FULL corpus.** +//! Among the anchor's nearest WordNet neighbours that fall OUTSIDE its home +//! cell, what fraction land in the 12-cell band, vs 12 distinct random +//! cells? Out-of-cell is the claim: a neighbour already in the home cell +//! needs no adjacency band to reach it. Every addressed leaf is a candidate +//! — an earlier version scored against a 20,000 of 65,292 sample, where an +//! omitted leaf could be strictly closer than any sampled one, so the result +//! characterised the sample rather than the corpus. The null is +//! self-checking: a random 12 of the remaining 255 cells should score +//! ≈ 12/255 ≈ 4.7 %. +//! - **SECONDARY, retained as a saturating diagnostic:** recall over ALL 32 +//! nearest distinct neighbours, where both arms credit the home cell. This +//! was the original primary and it is the WRONG statistic — home-cell hits +//! inflate the baseline to ≈0.62, capping the achievable ratio near 1.6 and +//! MASKING the effect. Kept only so the two are comparable in the record. +//! - Twin-tested per the can-it-fire/can-it-stay-silent rule, with BOTH halves +//! on the same statistic and the same basis: **fire** = band ≫ random; +//! **stay-silent** = the identical machinery aimed at RANDOM cells must be +//! inert (≈ the 12/255 null). Wrong cells, nothing found. +//! - **A cover guard (`primary < 0.95`) was tried and DROPPED as inert.** +//! Measured like-for-like, a deliberately coarse 2-level address scores +//! 0.840 on the out-of-cell basis — so 0.95 separated nothing, and only the +//! degenerate band-is-the-whole-codebook case would trip it. It is also +//! structurally unnecessary here: a band fixed at 4.7 % of the codebook with +//! no home-cell credit cannot be secretly dense. The guard mattered only for +//! the secondary statistic, where home-cell hits saturate. Keeping an inert +//! threshold is decoration, which the workspace's inertness rule forbids. +//! - **Specification check that this gate failed once:** before running a twin +//! gate, compute the maximum achievable value of the fire statistic *under* +//! the silent guard. Here that was 1/0.62 ≈ 1.61 against a `>1.5` fire bar +//! and a `<0.95` cover bar — an 0.018-wide window, i.e. a mis-specified +//! gate whose first pass was luck rather than evidence. +//! - **W4 4-ary beats the nibble** — the 16-ary ladder's coarsest useful rung +//! ("same top nibble") is a single bucket of 16 cells. The 4-ary ladder splits +//! that same population into rungs 1 and 2. Gate: those two sub-rungs must +//! differ in mean distance by a real margin — i.e. the sub-nibble structure +//! the nibble router CANNOT see is semantically load-bearing. Falsifier: the +//! split is flat, meaning 4⁴ micro-traversal is decoration and the nibble +//! loses nothing. +//! - **W5 fold balance** — report min/max/median cell occupancy. The +//! le-contract's own warning ("lacking proper bucket rollover ... saturates +//! silently") applies to this fold; a wildly unbalanced fold would make W2/W3 +//! look good for the wrong reason (one giant cell). +//! +//! ## Running +//! +//! ```sh +//! curl -sSL -o /tmp/wn31.tar.gz https://wordnetcode.princeton.edu/wn3.1.dict.tar.gz +//! mkdir -p /tmp/wn31 && tar xzf /tmp/wn31.tar.gz -C /tmp/wn31 +//! python3 crates/lance-graph-planner/examples/data/wordnet/build_isa_tree.py \ +//! /tmp/wn31/dict /tmp/wordnet_isa_tree.tsv +//! cargo run -p lance-graph-contract --example probe_wordnet_44_activation +//! ``` +//! +//! The corpus is LOCAL-ONLY and gitignored (workspace convention); the +//! generator is committed, the data is not. +//! +//! [`NiblePath`]: lance_graph_contract::hhtl::NiblePath +//! [`Palette::build_hierarchical`]: https://github.com/AdaWorldAPI/lance-graph/blob/main/crates/bgz17/src/palette.rs + +use std::collections::HashMap; +use std::path::Path; + +// ── the fold ──────────────────────────────────────────────────────────────── + +/// Address levels. 4 levels × 2 bits = one byte = 256 cells. +const LEVELS: usize = 4; +/// Arity per level. `4^4 = 256` — the operator's shape. +const ARITY: usize = 4; +const CELLS: usize = 256; +/// Sibling cells reachable by flipping ONE level's 2-bit group: +/// `(ARITY - 1) * LEVELS = 12` of 256 = 4.7 %. +const BAND: usize = (ARITY - 1) * LEVELS; + +struct Tree { + /// Dense index space. `parent[i]` = index of first `@` hypernym, or `usize::MAX`. + parent: Vec, + children: Vec>, + depth: Vec, + lemma: Vec, +} + +impl Tree { + fn load(path: &Path) -> std::io::Result { + let text = std::fs::read_to_string(path)?; + let mut offsets: Vec = Vec::new(); + let mut raw: Vec<(u32, u32, u16, String)> = Vec::new(); + for line in text.lines() { + if line.starts_with('#') { + continue; + } + let mut f = line.split('\t'); + let (Some(c), Some(p), Some(d), Some(l)) = (f.next(), f.next(), f.next(), f.next()) + else { + continue; + }; + let (c, p) = (c.parse::().unwrap_or(0), p.parse::().unwrap_or(0)); + raw.push((c, p, d.parse::().unwrap_or(0), l.to_string())); + offsets.push(c); + } + offsets.sort_unstable(); + let index: HashMap = offsets.iter().enumerate().map(|(i, &o)| (o, i)).collect(); + + let n = raw.len(); + let mut t = Tree { + parent: vec![usize::MAX; n], + children: vec![Vec::new(); n], + depth: vec![0; n], + lemma: vec![String::new(); n], + }; + for (c, p, d, l) in raw { + let i = index[&c]; + t.depth[i] = d; + t.lemma[i] = l; + if p != 0 { + if let Some(&pi) = index.get(&p) { + t.parent[i] = pi; + t.children[pi].push(i); + } + } + } + Ok(t) + } + + fn is_leaf(&self, i: usize) -> bool { + self.children[i].is_empty() + } + + /// Number of leaves under `i` (memoized iteratively — depth 19 is safe for + /// recursion, but the iterative form keeps this honest for any corpus). + fn leaf_counts(&self) -> Vec { + let n = self.parent.len(); + let mut order: Vec = (0..n).collect(); + order.sort_by_key(|&i| std::cmp::Reverse(self.depth[i])); + let mut counts = vec![0u32; n]; + for &i in &order { + if self.is_leaf(i) { + counts[i] = 1; + } + if self.parent[i] != usize::MAX { + counts[self.parent[i]] += counts[i]; + } + } + counts + } + + /// WordNet path distance: `depth(a) + depth(b) − 2·depth(LCA)`. + fn path_distance(&self, a: usize, b: usize) -> u32 { + let (mut x, mut y) = (a, b); + while self.depth[x] > self.depth[y] { + x = self.parent[x]; + } + while self.depth[y] > self.depth[x] { + y = self.parent[y]; + } + while x != y { + if self.parent[x] == usize::MAX || self.parent[y] == usize::MAX { + return u32::MAX; // different roots (should not happen: 1 root) + } + x = self.parent[x]; + y = self.parent[y]; + } + (self.depth[a] + self.depth[b] - 2 * self.depth[x]) as u32 + } +} + +/// Greedily bin a forest's roots into `ARITY` buckets balanced by leaf count — +/// the "proper bucket rollover" the le-contract demands of any fixed-width +/// carving. Largest-first into the currently-lightest bucket (LPT scheduling). +fn balance(roots: &[usize], weight: &[u32]) -> [Vec; ARITY] { + let mut sorted: Vec = roots.to_vec(); + sorted.sort_by_key(|&r| std::cmp::Reverse(weight[r])); + let mut buckets: [Vec; ARITY] = Default::default(); + let mut load = [0u64; ARITY]; + for r in sorted { + let (mut best, mut best_load) = (0, u64::MAX); + for (b, &l) in load.iter().enumerate() { + if l < best_load { + best_load = l; + best = b; + } + } + load[best] += weight[r].max(1) as u64; + buckets[best].push(r); + } + buckets +} + +/// Fold the taxonomy into 4⁴ addresses. Every LEAF synset gets one byte; the +/// two-bit group at level `k` names the bucket it fell into at that level, so +/// `addr >> (2·(LEVELS−k))` is a genuine fold-ancestor by construction. +/// +/// A node whose subtree is still being subdivided contributes its children to +/// the next level; internal nodes are not addressed (they are the skeleton, not +/// the concepts) — see the module's honest-boundaries note. +fn fold(tree: &Tree, root: usize, leaves: &[u32]) -> Vec> { + let mut addr = vec![None; tree.parent.len()]; + // Work queue: (forest roots, address prefix, level) + let mut stack: Vec<(Vec, u8, usize)> = vec![(vec![root], 0u8, 0usize)]; + while let Some((forest, prefix, level)) = stack.pop() { + if level == LEVELS { + // Terminal cell: every leaf under this forest gets `prefix`. + let mut work = forest; + while let Some(i) = work.pop() { + if tree.is_leaf(i) { + addr[i] = Some(prefix); + } else { + work.extend_from_slice(&tree.children[i]); + } + } + continue; + } + // Expand until the forest is FINE-GRAINED enough to balance, not merely + // splittable. Expanding to exactly ARITY roots is what produced the + // first run's 15,769-leaf cell against a median of 20: with 4 coarse + // roots of wildly unequal weight, LPT can only isolate the giant one and + // hope the next level splits it — and at the terminal level there is no + // next level, so it lands whole. This is precisely the le-contract's + // "lacking proper bucket rollover … saturates silently". Expanding to + // ARITY·GRAIN gives the balancer small enough items to actually level. + const GRAIN: usize = 24; + let mut roots = forest; + while roots.len() < ARITY * GRAIN { + let expandable = roots + .iter() + .enumerate() + .filter(|(_, &r)| !tree.is_leaf(r)) + .max_by_key(|(_, &r)| leaves[r]) + .map(|(pos, _)| pos); + match expandable { + Some(pos) => { + let r = roots.swap_remove(pos); + roots.extend_from_slice(&tree.children[r]); + // A leaf-bearing internal node keeps its own leaf identity + // only if it IS a leaf, which it is not here — nothing lost. + } + None => break, // all leaves, fewer than ARITY: buckets under-fill + } + } + for (b, bucket) in balance(&roots, leaves).into_iter().enumerate() { + if bucket.is_empty() { + continue; + } + stack.push((bucket, (prefix << 2) | b as u8, level + 1)); + } + } + addr +} + +// ── metrics ───────────────────────────────────────────────────────────────── + +/// Levels of shared address prefix (0..=LEVELS). 4 = same cell. +fn shared_levels(a: u8, b: u8) -> usize { + for k in 0..LEVELS { + let shift = 2 * (LEVELS - k) - 2; + if (a >> shift) != (b >> shift) { + return k; + } + } + LEVELS +} + +/// The sparse-adjacent band for an address of arbitrary depth: cells reachable +/// by changing exactly ONE 2-bit group. `(ARITY - 1) * levels` cells, never +/// including `cell` itself. +/// +/// Used at `levels = 2` to build a deliberately COARSE address space for the +/// granularity-comparison arm. That arm no longer backs a cover guard — the +/// guard was measured on the primary out-of-cell basis, found INERT (coarse +/// scores 0.840, not >0.95), and dropped rather than kept as decoration. It +/// survives as reported CONTEXT: how band recall varies with address +/// granularity. +fn band_generic(cell: u8, levels: usize) -> Vec { + let mut out = Vec::with_capacity((ARITY - 1) * levels); + for level in 0..levels { + let shift = 2 * (levels - 1 - level); + let cur = (cell >> shift) & 0b11; + for v in 0..ARITY as u8 { + if v != cur { + out.push((cell & !(0b11 << shift)) | (v << shift)); + } + } + } + out +} + +/// The probe's own band: `BAND` = 12 cells at the full 4-level address, +/// never including `cell` itself. Fixed-size form of [`band_generic`]. +fn band_of(cell: u8) -> [u8; BAND] { + let mut out = [0u8; BAND]; + let mut n = 0; + for level in 0..LEVELS { + let shift = 2 * (LEVELS - 1 - level); + let cur = (cell >> shift) & 0b11; + for v in 0..ARITY as u8 { + if v != cur { + out[n] = (cell & !(0b11 << shift)) | (v << shift); + n += 1; + } + } + } + debug_assert_eq!(n, BAND); + out +} + +struct SplitMix64(u64); +impl SplitMix64 { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + fn below(&mut self, n: usize) -> usize { + (self.next() % n as u64) as usize + } +} + +fn mean(v: &[u32]) -> f64 { + if v.is_empty() { + return f64::NAN; + } + v.iter().map(|&x| x as f64).sum::() / v.len() as f64 +} + +// ── gates ─────────────────────────────────────────────────────────────────── + +static FAILURES: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + +fn gate(name: &str, pass: bool, detail: String) { + println!( + " [{}] {name}\n {detail}", + if pass { "PASS" } else { "FAIL" } + ); + if !pass { + FAILURES.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + } +} + +fn main() { + let path = std::env::args() + .nth(1) + .unwrap_or_else(|| "/tmp/wordnet_isa_tree.tsv".to_string()); + let path = Path::new(&path); + if !path.exists() { + eprintln!( + "corpus not found at {}\nsee the module docs for the 3-line fetch + generate recipe", + path.display() + ); + std::process::exit(2); + } + + let tree = Tree::load(path).expect("load is-a tree"); + let n = tree.parent.len(); + let root = (0..n) + .find(|&i| tree.parent[i] == usize::MAX) + .expect("root"); + let leaves_under = tree.leaf_counts(); + let leaf_ids: Vec = (0..n).filter(|&i| tree.is_leaf(i)).collect(); + + println!("PROBE-WORDNET-44-ACTIVATION"); + println!( + "corpus: {n} synsets, root='{}', {} leaves, max depth {}\n", + tree.lemma[root], + leaf_ids.len(), + tree.depth.iter().max().unwrap() + ); + + let addr = fold(&tree, root, &leaves_under); + let addressed: Vec = leaf_ids + .iter() + .copied() + .filter(|&i| addr[i].is_some()) + .collect(); + + // ── W5 fold balance (reported first: W2/W3 are only meaningful if sane) ── + let mut occupancy = [0usize; CELLS]; + for &i in &addressed { + occupancy[addr[i].unwrap() as usize] += 1; + } + let used = occupancy.iter().filter(|&&c| c > 0).count(); + let mut occ_sorted: Vec = occupancy.iter().copied().filter(|&c| c > 0).collect(); + occ_sorted.sort_unstable(); + let (lo, hi) = (occ_sorted[0], *occ_sorted.last().unwrap()); + let median = occ_sorted[occ_sorted.len() / 2]; + gate( + "W5 fold balance", + used >= CELLS / 2 && hi <= 40 * median.max(1), + format!( + "{used}/{CELLS} cells used; occupancy min={lo} median={median} max={hi} \ + (addressed leaves={})", + addressed.len() + ), + ); + + // ── W1 ancestry-by-construction, with a shuffle arm as the falsifier ──── + let mut rng = SplitMix64(0x9E37_79B9_7F4A_7C15); + let sample: Vec = (0..4000) + .map(|_| addressed[rng.below(addressed.len())]) + .collect(); + + // Real: two leaves sharing k address levels must share a fold-ancestor — + // operationally, their WordNet LCA must be at least as deep as k allows. + // Shuffled: permute the cell labels, destroying prefix meaning. + let mut perm: Vec = (0..=255u8).collect(); + for i in (1..perm.len()).rev() { + perm.swap(i, rng.below(i + 1)); + } + let real_corr = prefix_vs_lca_corr(&tree, &sample, &addr, None); + let shuf_corr = prefix_vs_lca_corr(&tree, &sample, &addr, Some(&perm)); + gate( + "W1 ancestry-is-by-construction (shuffle falsifier)", + real_corr > 0.30 && shuf_corr.abs() < 0.05 && real_corr > shuf_corr + 0.25, + format!( + "corr(shared address levels, LCA depth): real={real_corr:+.4} \ + shuffled={shuf_corr:+.4} (shuffle must collapse to ~0)" + ), + ); + + // ── W2 monotone ladder ────────────────────────────────────────────────── + let mut rungs: [Vec; LEVELS + 1] = Default::default(); + for _ in 0..200_000 { + let a = addressed[rng.below(addressed.len())]; + let b = addressed[rng.below(addressed.len())]; + if a == b { + continue; + } + let k = shared_levels(addr[a].unwrap(), addr[b].unwrap()); + let d = tree.path_distance(a, b); + if d != u32::MAX { + rungs[k].push(d); + } + } + let means: Vec = rungs.iter().map(|r| mean(r)).collect(); + let monotone = (0..LEVELS).all(|k| means[k] > means[k + 1]); + let spread = means[0] - means[LEVELS]; + gate( + "W2 monotone ladder (share-0 → same-cell)", + monotone && spread > 1.0, + format!( + "mean WordNet path distance by shared address levels: \ + 0:{:.2} (n={}) 1:{:.2} (n={}) 2:{:.2} (n={}) 3:{:.2} (n={}) 4/same-cell:{:.2} (n={}) \ + — strictly decreasing={monotone}, spread={spread:.2} hops", + means[0], rungs[0].len(), + means[1], rungs[1].len(), + means[2], rungs[2].len(), + means[3], rungs[3].len(), + means[4], rungs[4].len(), + ), + ); + + // ── W3 spatial activation: band recall vs random-12 baseline ──────────── + const ANCHORS: usize = 300; + const NEAR: usize = 32; + let mut band_recall = Vec::new(); + let mut rand_recall = Vec::new(); + // Pool the search: comparing every anchor against 82k leaves is O(n²); use a + // deterministic candidate pool per anchor, identical for both arms, so the + // comparison is exact and the cost is bounded. + // FULL CORPUS — every addressed leaf is a candidate, so "the 32 nearest + // WordNet neighbours" means exactly that. + // + // Two earlier versions were wrong here and both were caught in review. The + // first sampled 20,000 candidates WITH REPLACEMENT and took the 32 smallest, + // so one synset could fill several slots (a weighted sampled-entry recall). + // Deduping fixed the double-counting but left the deeper problem: a 20,000 + // of 65,292 sample (31 %) means an omitted leaf can be strictly closer than + // every sampled one, moving both the distance cutoff and which cells the + // neighbours sit in — so the result characterised the sample, not the + // corpus. Scoring all 65,292 (≈20 M tree walks, seconds in release) removes + // the caveat rather than documenting it, and makes dedup moot: `addressed` + // is distinct by construction. + let pool: &[usize] = &addressed; + // Calibration arm for the upper guard (see the gate below): the SAME recall + // measured against a deliberately COARSE address — only the top 2 levels, + // so 16 cells and a 6-cell band = 37.5 % of that codebook instead of 4.7 %. + // A band that big must behave like a cover, which is what makes the 0.95 + // threshold something that bites rather than decoration. + let mut coarse_recall = Vec::new(); + // Out-of-cell arms — the primary statistic (see the in-loop note). + let mut oob_band: Vec = Vec::new(); + let mut oob_rand: Vec = Vec::new(); + for _ in 0..ANCHORS { + let anchor = addressed[rng.below(addressed.len())]; + let acell = addr[anchor].unwrap(); + let mut scored: Vec<(u32, usize)> = pool + .iter() + .filter(|&&p| p != anchor) + .map(|&p| (tree.path_distance(anchor, p), p)) + .filter(|&(d, _)| d != u32::MAX) + .collect(); + scored.sort_unstable(); + let near: Vec = scored.iter().take(NEAR).map(|&(_, p)| p).collect(); + if near.len() < NEAR { + continue; + } + + let band = band_of(acell); + let in_band = |c: u8| c == acell || band.contains(&c); + let hit = near.iter().filter(|&&p| in_band(addr[p].unwrap())).count(); + band_recall.push(hit as f64 / NEAR as f64); + + // PRIMARY statistic — OUT-OF-CELL neighbours only. The claim under test + // is about references that land outside the current cell; a neighbour + // already in the home cell needs no adjacency band to reach and is + // credited to BOTH arms, which is what compressed the earlier ratio (see + // the gate note). Restricting to out-of-cell neighbours measures what + // the 12-cell band actually contributes. + let out: Vec = near + .iter() + .copied() + .filter(|&p| addr[p].unwrap() != acell) + .collect(); + if !out.is_empty() { + let b = out + .iter() + .filter(|&&p| band.contains(&addr[p].unwrap())) + .count(); + oob_band.push(b as f64 / out.len() as f64); + } + + // Baseline: BAND random cells + the anchor cell (same budget). + let mut rcells: Vec = Vec::with_capacity(BAND); + while rcells.len() < BAND { + let c = rng.below(CELLS) as u8; + if c != acell && !rcells.contains(&c) { + rcells.push(c); + } + } + let hit_r = near + .iter() + .filter(|&&p| { + let c = addr[p].unwrap(); + c == acell || rcells.contains(&c) + }) + .count(); + rand_recall.push(hit_r as f64 / NEAR as f64); + if !out.is_empty() { + let r = out + .iter() + .filter(|&&p| rcells.contains(&addr[p].unwrap())) + .count(); + oob_rand.push(r as f64 / out.len() as f64); + } + + // Calibration arm for the cover guard — SAME anchors, SAME neighbours, + // SAME out-of-cell basis as the primary, differing ONLY in address + // granularity (coarse 2-level: a 6-cell band over 16 cells). Both halves + // of the twin gate must measure the same quantity or the gate is + // incoherent — the mistake this probe already made once, when the fire + // half moved to out-of-cell recall and the guard stayed on the + // all-neighbour statistic that had just been declared the wrong one. + let coarse_anchor = acell >> 4; + let cband = band_generic(coarse_anchor, 2); + let c_out: Vec = near + .iter() + .copied() + .filter(|&p| addr[p].unwrap() >> 4 != coarse_anchor) + .collect(); + if !c_out.is_empty() { + let hit_c = c_out + .iter() + .filter(|&&p| cband.contains(&(addr[p].unwrap() >> 4))) + .count(); + coarse_recall.push(hit_c as f64 / c_out.len() as f64); + } + } + let band_mean = band_recall.iter().sum::() / band_recall.len() as f64; + let rand_mean = rand_recall.iter().sum::() / rand_recall.len() as f64; + let coarse_mean = coarse_recall.iter().sum::() / coarse_recall.len() as f64; + let oob_b = oob_band.iter().sum::() / oob_band.len() as f64; + let oob_r = oob_rand.iter().sum::() / oob_rand.len() as f64; + gate( + "W3 spatial activation (can-fire AND can-stay-silent)", + oob_b > oob_r * 2.0 && oob_r < 0.10, + format!( + "PRIMARY (out-of-cell neighbours over the FULL corpus — the actual \ + sparse-adjacency claim): band={oob_b:.3} vs random={oob_r:.3}, ratio {:.2}×, \ + over {} cells (4.7 % of the codebook).\n CAN-STAY-SILENT: the same \ + machinery aimed at RANDOM cells yields {oob_r:.3} — inert, and matching the \ + 12/255 ≈ 0.047 null. That is the honest silent half: identical statistic, \ + identical basis, wrong cells, nothing found.\n Cover guard DROPPED as \ + INERT (was: primary<0.95). On this basis a coarse 2-level address scores \ + {coarse_mean:.3}, not >0.95 — so 0.95 separated nothing, and only the \ + degenerate band=whole-codebook case would trip it. A band fixed at 4.7 % of \ + the codebook with no home-cell credit CANNOT be secretly dense, so the guard \ + was structurally unnecessary here; it mattered only for the secondary \ + statistic below.\n SECONDARY, reported for comparability and NOT gated \ + (both arms credit the home cell, which saturates it): band={band_mean:.3} vs \ + random={rand_mean:.3} (ratio {:.2}×).", + oob_b / oob_r.max(1e-9), + BAND, + band_mean / rand_mean.max(1e-9), + ), + ); + + // ── W4 does 4-ary see what the nibble cannot? ─────────────────────────── + // The nibble router's finest distinction inside a byte is "same top nibble" + // (= shared levels ≥ 2 in 4-ary terms). 4-ary splits that population into + // shared-2 and shared-3. If those differ, the sub-nibble decision is real. + let sub_nibble_gap = means[2] - means[3]; + let nibble_blind: Vec = rungs[2].iter().chain(rungs[3].iter()).copied().collect(); + gate( + "W4 sub-nibble structure is load-bearing", + sub_nibble_gap > 0.5, + format!( + "inside one top nibble the nibble router sees ONE bucket \ + (mean {:.2}, n={}); 4-ary splits it into shared-2 {:.2} vs shared-3 {:.2} \ + — gap {sub_nibble_gap:.2} hops that 16-ary cannot address", + mean(&nibble_blind), + nibble_blind.len(), + means[2], + means[3], + ), + ); + + // A worked example, so the numbers have a face. Show the band's NEAREST + // residents by WordNet distance — an earlier draft printed the first six in + // file order, which reads as a semantic result while being arbitrary. + for probe_word in ["violin", "dog", "oak", "hammer"] { + let Some(&anchor) = addressed.iter().find(|&&i| tree.lemma[i] == probe_word) else { + continue; + }; + let acell = addr[anchor].unwrap(); + let band = band_of(acell); + let mut in_band: Vec<(u32, usize)> = addressed + .iter() + .filter(|&&p| p != anchor && band.contains(&addr[p].unwrap())) + .map(|&p| (tree.path_distance(anchor, p), p)) + .filter(|&(d, _)| d != u32::MAX) + .collect(); + in_band.sort_unstable(); + let nearest: Vec = in_band + .iter() + .take(6) + .map(|&(d, p)| format!("{}({d})", tree.lemma[p])) + .collect(); + // The same budget spent anywhere else in the codebook, for contrast. + let out_of_band_best = addressed + .iter() + .filter(|&&p| p != anchor && !band.contains(&addr[p].unwrap())) + .filter(|&&p| addr[p] != Some(acell)) + .map(|&p| tree.path_distance(anchor, p)) + .filter(|&d| d != u32::MAX) + .min() + .unwrap_or(u32::MAX); + println!( + "\n worked example: '{probe_word}' @ cell {acell:#04x} \ + ({:02b}|{:02b}|{:02b}|{:02b})\n nearest in the 12-cell band (hops): {}\ + \n best distance OUTSIDE the band: {out_of_band_best} hops", + (acell >> 6) & 3, + (acell >> 4) & 3, + (acell >> 2) & 3, + acell & 3, + nearest.join(", "), + ); + break; + } + + let failures = FAILURES.load(std::sync::atomic::Ordering::Relaxed); + println!( + "\n{}", + if failures == 0 { + "ALL GATES GREEN".to_string() + } else { + format!("{failures} GATE(S) FAILED") + } + ); + if failures > 0 { + std::process::exit(1); + } +} + +/// Correlation between (shared address levels) and (LCA depth) over sampled +/// pairs. `perm` optionally relabels cells — the falsifier arm. +fn prefix_vs_lca_corr( + tree: &Tree, + sample: &[usize], + addr: &[Option], + perm: Option<&[u8]>, +) -> f64 { + let relabel = |c: u8| match perm { + Some(p) => p[c as usize], + None => c, + }; + let (mut xs, mut ys) = (Vec::new(), Vec::new()); + for w in sample.windows(2) { + let (a, b) = (w[0], w[1]); + if a == b { + continue; + } + let k = shared_levels(relabel(addr[a].unwrap()), relabel(addr[b].unwrap())); + // DIFFERENT-CELL PAIRS ONLY. A label permutation is a bijection, so + // same-cell pairs stay same-cell under any relabelling — including them + // let the falsifier arm score +0.645 on pure cell identity, masking + // whether the INTERMEDIATE prefix levels carry anything. The claim + // under test is about levels 1..3, so level 4 is excluded from both arms. + if k == LEVELS || shared_levels(addr[a].unwrap(), addr[b].unwrap()) == LEVELS { + continue; + } + // LCA depth via path distance identity. + let d = tree.path_distance(a, b); + if d == u32::MAX { + continue; + } + let lca_depth = (tree.depth[a] as i32 + tree.depth[b] as i32 - d as i32) / 2; + xs.push(k as f64); + ys.push(lca_depth as f64); + } + pearson(&xs, &ys) +} + +fn pearson(x: &[f64], y: &[f64]) -> f64 { + let n = x.len() as f64; + if n < 2.0 { + return f64::NAN; + } + let (mx, my) = (x.iter().sum::() / n, y.iter().sum::() / n); + let (mut num, mut dx, mut dy) = (0.0, 0.0, 0.0); + for (a, b) in x.iter().zip(y) { + num += (a - mx) * (b - my); + dx += (a - mx) * (a - mx); + dy += (b - my) * (b - my); + } + if dx == 0.0 || dy == 0.0 { + return 0.0; + } + num / (dx.sqrt() * dy.sqrt()) +} diff --git a/crates/lance-graph-planner/examples/data/wordnet/build_isa_tree.py b/crates/lance-graph-planner/examples/data/wordnet/build_isa_tree.py new file mode 100644 index 00000000..5510e0de --- /dev/null +++ b/crates/lance-graph-planner/examples/data/wordnet/build_isa_tree.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""Emit the WordNet noun is-a TREE (offset -> primary hypernym) for the 4^4 +spatial-activation probe (`bgz17/examples/probe_wordnet_44_activation.rs`). + +DIFFERENT SHAPE from its sibling `build_wordnet_rail.py`, which emits a +(word, pos, sense) -> hypernym-lemma RAIL for lexical lookup. This one emits +the raw SYNSET TREE — the structure itself — because the probe's subject is +the taxonomy's branching, not word senses: + + child_offset parent_offset depth lemma + +`parent_offset` is the FIRST `@` (hypernym) or `@i` (instance-hypernym) +pointer in the synset's pointer list, which is WordNet's own primary-parent +order; `0` marks a root. Taking only the first parent turns WordNet's noun +DAG into a TREE — a deliberate, declared simplification (multiple inheritance +is real in WordNet; a fixed-arity address cannot carry it). The probe measures +distance ON THIS TREE, so both sides of every comparison see the same +simplification and it cannot flatter the result. + +Data policy: WNDB is gitignored (same convention as every other generator in +this directory) — this script is committed, the dict and the emitted TSV are +not. Fetch: + + curl -sSL -o /tmp/wn31.tar.gz https://wordnetcode.princeton.edu/wn3.1.dict.tar.gz + mkdir -p /tmp/wn31 && tar xzf /tmp/wn31.tar.gz -C /tmp/wn31 + python3 build_isa_tree.py /tmp/wn31/dict /tmp/wordnet_isa_tree.tsv +""" + +import sys +from pathlib import Path + + +def parse_data_noun(path: Path): + """offset -> (first_lemma, first_hypernym_offset_or_None).""" + nodes = {} + with path.open("r", encoding="latin-1") as fh: + for line in fh: + if line.startswith(" "): # licence header + continue + data = line.split("|", 1)[0].split() + if len(data) < 6: + continue + offset = int(data[0]) + w_cnt = int(data[3], 16) + lemma = data[4] + # words: w_cnt pairs of (word, lex_id) starting at index 4 + idx = 4 + 2 * w_cnt + p_cnt = int(data[idx]) + idx += 1 + parent = None + for _ in range(p_cnt): + sym, target, pos = data[idx], data[idx + 1], data[idx + 2] + idx += 4 # sym, offset, pos, source/target + if parent is None and pos == "n" and sym in ("@", "@i"): + parent = int(target) + nodes[offset] = (lemma, parent) + return nodes + + +def depth_of(offset, nodes, memo): + """Root distance; cycle-safe (WordNet has none, but a bad parse could).""" + seen = [] + cur = offset + while cur is not None and cur not in memo: + if cur in seen: # cycle -> treat as root + memo[cur] = 0 + break + seen.append(cur) + entry = nodes.get(cur) + if entry is None or entry[1] is None: + memo[cur] = 0 + break + cur = entry[1] + for off in reversed(seen): + if off in memo: + continue + parent = nodes[off][1] + memo[off] = 0 if parent is None else memo.get(parent, 0) + 1 + return memo.get(offset, 0) + + +def main(): + dict_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "/tmp/wn31/dict") + out = Path(sys.argv[2] if len(sys.argv) > 2 else "/tmp/wordnet_isa_tree.tsv") + + nodes = parse_data_noun(dict_dir / "data.noun") + memo = {} + rows = 0 + roots = 0 + with out.open("w", encoding="utf-8") as fh: + fh.write("# child_offset\tparent_offset\tdepth\tlemma\n") + for offset in sorted(nodes): + lemma, parent = nodes[offset] + d = depth_of(offset, nodes, memo) + if parent is None: + roots += 1 + fh.write(f"{offset}\t{parent or 0}\t{d}\t{lemma}\n") + rows += 1 + + print(f"synsets={rows} roots={roots} max_depth={max(memo.values())} -> {out}") + + +if __name__ == "__main__": + main()