Repository navigation
refactor(ast): role tables, their gates, and the Halstead keyword roles (R01–R03) - #1705
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1705 +/- ##
==========================================
+ Coverage 98.29% 98.30% +0.01%
==========================================
Files 371 380 +9
Lines 109012 110026 +1014
Branches 108577 109591 +1014
==========================================
+ Hits 107151 108159 +1008
- Misses 1195 1202 +7
+ Partials 666 665 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
dekobon
force-pushed
the
refactor/1691-role-table-proof
branch
from
October 23, 2026 22:28
658cc30 to
cb3561e
Compare
This was referenced Oct 23, 2026
Closed
Adds `big-code-analysis-ast/src/roles/`: the `role_table!` macro, the fixed set of twenty-nine roles a classifier or metric dispatcher can ask of a node, and one table each for Java, Python and Elixir proving every row shape against the pinned grammars — plain, gated, projected, self-paying, text-gated, space-mapping, `wraps`, `excludes`, `pays never`, `absent`, `pending`. Nothing reads the tables yet; #1693 wires the first consumer and removes the module's dead-code allow. Production output is unchanged. Shapes settled by measurement rather than as the issue spelled them: rows are `Role { … }` blocks with parenthesised owner attributes, and each kind expands to an `if` clause rather than a `match` arm, because an indented `=>` row and a `match` inside a `$( … )*` repetition are both permanent rustfmt bails under `check-rustfmt-bail.py`; `Excluded` is a row-level `excludes [K "why"]` attribute rather than a standalone entry, since a standalone one needs a second per-role name table for nothing it adds. Three paying modes — a `pays` list (split projections), none (owners pay under both), `pays never` (neither) — make a consumer's choice of occurrence visible only where a row declares a split. Six `compile_fail` doctests pin the rejections: a role declared twice or alongside `absent`, a kind in both lists of one row, and `pays_itself` where it contradicts `pays never` or is redundant. Parsed fixtures per shape, each verified by perturbation: the Java `<` / `>` gate, the `instanceof` self-payer, the string-template `wraps` child, the anonymous-class body gate, the lambda closure-vs- space split; Python's wrapper-vs-leaf comparison projections and the `not in` compound; Elixir's `def` / `defmodule` text gates and the `quote` chain gate. Closes the implementation half of #1691; the issue records the evidence.
Every `LANG` variant now has a `roles/<lang>.rs`: 22 new tables beside R01's three, each role `pending Rnn` for the package that will declare it, or `absent "why"` where the language has no such construct — the two C-family helper grammars included, which recognise comments and string literals and are pending on those rows rather than absent wholesale. The `enums/` generator emits a `<LANG>_KIND_FLAGS` sidecar into each `language_<lang>.rs`: tree-sitter's named and visible flags per runtime kind id, labels for the ids the enum already identifies, with no entry for the generator's ERROR sentinel. `utils/role_tables.py` is the one reader of the declaration text for the gates the next commit adds: it reads `mk_langs!`, the `@named` role set, every `role_table!` row and every generated enum without a Rust build, fails closed on any shape it does not know, and is held to the compiled constants by `python_reader_agrees_with_the_compiled_tables`, which compares its `--dump` against the `canonical()` every table now emits. Nothing reads any of this in production. Part of #1692.
Three gates in `make lint`, `make pre-commit`, `make ci`, the pre-commit hooks and the CI lint job, each with a self-test suite and a baseline it may only shrink: - `check-role-census.py`: every (language, role) cell is declared rows, `absent "why"` or `pending Rnn`; a hole, an unreadable table or a missing file fails naming what to add. The pending cells are an exact set in `.role-census-baseline.txt` (547 at bootstrap). - `check-role-coverage.py`: every runtime kind id of every generated enum is claimed by a row, excluded with a reason under the role that rejected it, or debt in `.role-coverage-baseline.txt` (7842 of 7927 ids at bootstrap). Identity is `(language, id)` under the pinned grammar version, never a spelling; the flags sidecar labels an entry and forgives nothing; a baselined id that comes to spell another kind fails, and a moved pin fails until `--migrate-pin` confirms the table. `.role-coverage-observed.txt` records what the integration corpus emits — written by `--observe`, which builds its own `bca`, reads only node headers, and records corpus SHAs, pins and per-language file counts — and `--require-observations` fails on a stale, missing or broken record. "Not observed" is never "unreachable". - `check-role-bypass.py`: inside the consumer boundary — the per-language dispatch files, their dispatcher modules and the macros that generate them — a kind a row claims must not be restated in a `matches!`, a match arm (read whole across lines), a `kind_id()` equality, a `kind() == "…"` compare or a `const` array, with imports resolved in lexical scope and three macro shapes resolved from their headers; any other kind-reading macro is reported as unsupported. `.role-bypass-baseline.txt` holds the 125 pre-migration sites, keyed per site within an item; six naming and doc-attachment sites carry reasoned `// role-bypass: allow(<Role>) — …` exemptions instead. Imports resolve in lexical scope (inner bindings shadow outer), out-of-line `#[cfg(test)] mod x;` files are left out of the boundary walk, and a `kind_id()` held in a local is a documented blind spot. All three baselines are exact record sets: `--update` only removes and refuses an increase; `--bootstrap` refuses a file that exists or is tracked. Corpus staleness is judged against the index gitlinks, so a corpus bump staged with its re-observation passes; a grammar-pin move (every release bumps the five vendored pins) is cleared with `--migrate-pin all`, which `RELEASING.md` now lists. `make fmt` / `fmt-check` reach the excluded `enums/` and `fuzz/` crates by manifest path — the sidecar's test was the first excluded-crate file rustfmt had not seen. Every run prints the two scope limits #1692 requires: these gates account; none judges a row's membership, and the coverage ratchet does not replace #1458's measured sweeps. Closes #1692.
A row's `owns` list may hold `..name!()`, a hand-written `macro_rules!` that appends its kinds through continuation passing in both the row and the `opens_space` fold. `utils/role_tables.py` expands the same fragments from the `@splice` arm, so the census, coverage and bypass gates read a spliced row as the compiler does. Groundwork for #1693, where the four C-family dialects share one modifier, declaration-keyword and primitive list.
The `Modifier`, `DeclKeyword` and `Primitive` rows of C, Cpp, Mozcpp and Objc are declared: the kinds all four bill live once, in the `roles/c_family.rs` fragments, and each dialect's row splices them beside its own. The four `get_op_type` arms stop listing a keyword. `Getter::get_keyword_role` is the new classifier a keyword row feeds; `keyword_or_op_type` is the one fallback every spelling of `get_op_type_with_code` ends in — the default forwarder, the five language overrides and `braced_word_op_type` — so a row is read on every path the walk takes and the byte-less `get_op_type` no longer knows a keyword. `keywords_are_operators!` emits the C-family reading (every row's keyword bills as an operator); PHP will bill its primitives as operands through the same hook. Zero drift: `bca ops` and `bca metrics` JSON over every corpus file plus a keyword fixture per language, 25 languages, byte-identical before and after. `tests/parity/cpp_mozcpp_parity.rs` pins Mozcpp's rows to an independently written vocabulary, since nothing routes to it. Dropping `Volatile2` from the fragment fails the coverage gate naming the kind in all four dialects and three spelling tests. The bypass sites a claim creates outside classification take reasoned exemptions: `Checker::is_primitive` selects the vocabulary key after the row has classified; `npm` / `npa` ask which access label, not whether the token is a modifier; ObjC's declarator naming and ungated-identifier operand arm. The census, coverage and bypass baselines ratchet down by 12, 199 and 0 entries. `assert_keywords_are_operands_only` and `assert_modifier_spellings_bill` now share `assert_keyword_role` behind the existing `Role` enum, as #1598 asked; every fixture row is kept. `TokenRole` derives `Clone`, `Copy`, `Debug`, `PartialEq` and `Eq`. Refs #1693, #1598.
…d once The compiled crate reads a fragment's `@splice` and `@opens_splice` arms independently, so a kind dropped from one left the other and nothing compared them; `utils/role_tables.py` now parses both and fails on a difference, with self-tests for a mismatch and a missing arm. `owns_kind` / `pays_at_leaf_kind` take the node's kind id so a consumer asking three rows about one node reads it once rather than six times; on the corpus Halstead walk the migration measures inside the run-to-run noise (interleaved before/after: 13.6/12.6, 12.43/12.48, 14.0/13.5 ms). CHANGELOG entries for the `TokenRole` derives and the C-family rows. Refs #1693.
…bles
JavaScript, MozJS, TypeScript and TSX declare `Modifier`,
`DeclKeyword` and `Primitive`: the shared `static` and the five
declaration keywords live once in `roles/js_family.rs`, TypeScript and
TSX add their member modifiers, type-declaration keywords and
`predefined_type` beside them, and `impl_js_family_get_op_type!` takes
the grammar's roles module and emits `get_keyword_role` from it. The
two text-free gates the macro carried — `export { type }`'s recovered
keyword leaf (#1557) and the `void`-wrapped `predefined_type` (#453)
— are `gated_by` functions on the rows, emitted per grammar by
`ts_family_gates!`; the `predefined_void:` and `export_type_name:`
macro parameters are gone. An `absent` row now emits constant-false
`owns_kind` / `pays_at_leaf_kind`, so a consumer shared by several
grammars reads JavaScript's absent `Primitive` as it reads a declared
row; a `pending` row still emits nothing, so a consumer cannot read
one by mistake.
Zero drift: `bca ops` and `bca metrics` JSON byte-identical over the
corpora and fixtures, 25 languages. Perturbing either gate or the
shared `Static` fails the existing #453 / #1557 / #1563 tests. The
`const` keyword read by the ABC `const`-binding walk and the two
`is_primitive` key selectors take reasoned exemptions. Census and
coverage baselines ratchet down by 10 and 72 entries.
Refs #1693.
Java, Groovy, C#, Kotlin, Rust, PHP and Go declare `Modifier`, `DeclKeyword` and `Primitive`; Bash, Lua, Tcl, iRules, Perl, Python, Ruby and Elixir declare `DeclKeyword`, the one of the three their grammars have. Every gate a getter carried on a keyword moves to its row: Java's `import static` / `requires static` and class-literal `class`, Groovy's contextual `trait` / `record` / `sealed` / `permits` and text-gated primitives, C#'s childless `modifier` alias, Kotlin's soft-keyword recovery test, TypeScript's two from the previous commit. PHP bills its primitives as operands and says so in its own `get_keyword_role`; the shared `keywords_are_operators!` serves the other twenty-three. Rust's `Checker::is_primitive` reads the row's `OWNERS` rather than restating the seventeen `primitive_type` aliases. Zero drift: `bca ops` and `bca metrics` JSON byte-identical over the corpora and a keyword fixture per language, 25 languages. Dropping any kind or gate from a row fails an existing Halstead test, measured across twenty-two perturbations; the three that did not — PHP's `bottom_type`, Elixir's `fn` — gain a test each, and the C# aliased `this` is documented as held by call order rather than by the list. Rust's sixteen suffixed `primitive_type` aliases are unreachable from written source and stay pinned by `rust_aliased_primitive_type_classification`. #1600 is decided on the Java and Groovy `Modifier` rows: the directive guard stays, both options measured, C# recorded as the outlier. Key-selection, which-modifier and naming sites take reasoned exemptions; the census and coverage baselines ratchet down by 28 and 217 entries, and every getter's `halstead.effort` baseline entry falls. Refs #1693, #1600.
The `export_type_name` and `predefined_void` macro parameters are gone; the three comments that still credited them now name the `declares_a_type` / `names_a_type_not_void` gates on the rows, the TS/TSX invocation comments point at `roles/typescript.rs` rather than repeating its derivations, and Rust's `Primitive` row says what the seventeen `primitive_type` ids are (one per spelling, canonicalised by the parser) rather than one per position. The book's Halstead rules name C# as the directive-keyword outlier (#1600). Refs #1693, #1600.
rustdoc under -D warnings rejects a public doc that links a private item; the two links to keyword_or_op_type now describe it instead. Refs #1693.
Groovy's `def` joins its `DeclKeyword` row, as every sibling's
declarator did. `utils/role_tables.py` no longer resolves a fragment
defined inside a `#[cfg(test)]` module, which rustc would reject and
the differential test could not see; self-test added. A spelling test
covers every row kind no corpus file or other fixture spells (Bash
`declare` / `typeset` / `readonly`, Ruby `undef`, Rust `dyn` / `async`,
PHP `global` / `var`, Tcl and iRules `global`, C# `unsafe` / `record`),
verified to fail by name on each. The Java and Tcl fallback tests now
discriminate — their rows own `class` / `int` / `proc` — and their
docs say so instead of the reverse. Stale headers in `roles/{cpp,
mozcpp,irules}.rs`, `roles/mod.rs`, `getter.rs` and `getter/tcl.rs`
describe the tree as it is; the CHANGELOG entry covers all 25
languages rather than the C family.
Refs #1693.
…tive role A fragment carried its kind list twice because the row walk and the `opens_space` fold each had an accumulator walker; the two continuations differ in shape, so one `@owners` walk with two terminal arms serves both and a fragment is one `@splice` arm. The reader's two-arm comparison and its self-tests go with it. `keywords_are_operators!` becomes `keyword_roles!`, taking the `TokenRole` a primitive bills as (`Operator` by default, `Operand` for PHP), so PHP's hand-written body is gone. The two spelling helpers in `metrics/halstead.rs` fold into `assert_spellings_bill_as_operators`, and the Python reader keeps a fragment's entries split once. Refs #1693.
… language
A synthetic `Conflict` getter whose row and byte-less arm disagree on
every node shows the row's answer winning; swapping the two in
`keyword_or_op_type` fails it, where no shipped language could. The
Tcl braced-word test names the byte of the one withdrawn `{`. The
spelling test for row kinds no other fixture spells is one test per
language, so a single-language build still checks its own rows.
Refs #1693.
A hand-written `impl Getter for Conflict` shares its method names with the trait methods the C test calls on `CCode`, and the per-language test gate's call-by-name derivation read that test as their caller and gated them on `c` — so a cpp-only build lost the overrides and the conflict test saw an empty stream. Generated through a macro the methods are invisible to that derivation and need no gate. Refs #1693.
An `absent` row emitted only the `_kind` spellings, so a consumer calling `owns` / `pays_at_leaf` / `pays_at_wrapper` on one — the documented contract — would not compile; it now emits all five, and a test asks each of JavaScript's absent `Primitive`. The Python reader's `#[cfg(test)]` test admits the whitespace Rust does and rejects a predicate that is not exactly `test`, both self-tested. Comments that still described the two-arm fragment are rewritten, the conflict test names the shipped nodes where the row and the arm do disagree, and `grammar-dispatch.md` points a shared kind list at the roles directory. Refs #1693.
…ifier C# `required` / `fixed` / `implicit`, Perl `state` and Rust `move` are row kinds no corpus file or other fixture spelled; each now has a spelling row that fails by name when dropped. `type_specifier`, the hidden C-family supertype the shared `Primitive` fragment lists defensively, gets the lesson-34 drift marker in all four dialects. Two comments that still pointed at `is_primitive` and the Cpp arm as the home of a list now point at the rows. Refs #1693.
…rows Groovy's `var` and Python's `global` / `nonlocal` were left in the coverage residue with the reasoning in an issue comment, where the next reader of the row would not find it. Groovy's `DeclKeyword` row now excludes `Var` naming #1704 — it is a declaration keyword no arm ever billed, so claiming it moves `n1` and is a fix of its own — and Python's excludes `Global` / `Nonlocal` as scope statements rather than declarations. Three residue entries leave the baseline; output is unchanged. Refs #1693, #1704.
The advisory landed after main's last cargo-deny run: ctutils 0.4.3's const-fn mask generation could branch, fixed in 0.4.4. A transitive dependency through digest (hmac in the root crate, sha1 in actix-http for the web crate); no manifest changes. The workspace-excluded fuzz crate's lockfile moves with it.
Codecov's patch report named the trait's default `get_keyword_role` as uncovered: every language overrides it, so the default was a hook nothing used — and one a future language could inherit silently, billing no keyword with nothing to say so. The method is now required; the comment-stripping and preprocessor grammars read their three `absent` rows through `keyword_roles!` like everyone else, which answers `Unknown` on purpose rather than by omission. The other uncovered patch lines are the `const fn`s that run only at compile time under the macro's `const` assertions, the conflict getter's never-returned arm (the proof working), and a test's panic-message formatting; none is a gap a test would honestly close. `Role::of`'s "in neither vocabulary" arm now has one: a keyword the C# fixture does not spell contributes no row. Refs #1693.
dekobon
force-pushed
the
refactor/1691-role-table-proof
branch
from
October 23, 2026 22:55
80adf17 to
bdd70eb
Compare
15 tasks done
Open
7 of 11 tasks
This was referenced Oct 24, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1691, #1692, #1693, #1598, #1484, #1600. Part of #1690.
The first three packages of the role-table series, in order: the DSL and its proof (R01), the three gates that keep it honest (R02), and the first consumer migration — the Halstead keyword roles for all 25 languages (R03). Eighteen commits; the package boundaries are
0f49f387–b9ea546f(R01),9c03629a–92d7ff74(R02),7fbef3b6–bef8600d(R03). Each issue carries its own completion comment with the evidence; this body is the summary.What
R01 (#1691).
role_table!inbig-code-analysis-ast/src/roles/declares, per language, which kinds own a construct and which pay for it, with gates, projections,opensspace kinds,wraps,excludes,absent "why"andpending Rnn. Every row shape is proven against a real grammar (Java, Python, Elixir). Also a quadratic Elixirdef-under-quoteclimb folded down the walk, found while proving the text-gated shape.R02 (#1692). Every language has a
roles/<lang>.rs; three gates inmake pre-commit/make ci:role-census(every language × role cell declared),role-coverage(every runtime kind id claimed, excluded with a reason, or baselined — keyed by id under the pinned grammar, with--migrate-pinfor a bump),role-bypass(no row-claimed kind restated as classification inside the consumer boundary). One Python reader,utils/role_tables.py, held to the compiled tables by a differential test.R03 (#1693).
Modifier,DeclKeywordandPrimitiveare read from the rows.Getter::get_keyword_roleis the classifier a row feeds (keyword_roles!(lang[, primitive: Role])),keyword_or_op_type::<G>prefers it and falls back to the byte-lessget_op_type, and every spelling ofget_op_type_with_code— the default, the five overrides, the Tcl-family braced-word revision — ends in it. Noget_op_typelists a keyword a row owns. Every gate those arms carried (Javaimport static, Groovy contextual keywords and text-told primitives, C# aliased parameter modifiers, Kotlin soft-keyword recovery, TSvoidwrapper andexport { type }) is agated_by/text_gated_byfunction beside its row. Shared lists are..name!()fragments (roles/c_family.rs,roles/js_family.rs), read by the compiler and the Python reader alike.Zero drift
bca opsandbca metricsJSON over 2,938 files — every corpus file plus one keyword fixture per language, 25 languages, Mozcpp over the Cpp corpus — byte-identical toorigin/mainafter each migration commit and at HEAD. Mozcpp (no file extension) is additionally pinned to an independently written keyword vocabulary intests/parity/cpp_mozcpp_parity.rs.Guards
Volatile2from the C-family fragment (the refactor(getter/c-family): one kind table for the four copied modifier lists #1598 case) failsmake role-coveragenaming__volatile__in all four dialects, plus three spelling tests.role-bypass: allowexemptions, none for membership (key selection after classification, which-modifier reads, naming, the operand side of a gated spelling), each with its reason on the line.make bench-scalingpasses; each language's predicates compile to oneswitch i16.make pre-commit→BCA_GATE: passat HEAD;make chain-audit,make check-test-lang-gates-compare COMPARE_REF=origin/mainpass; no pending snapshots.Reviews
astra-review on the plan and at three checkpoints, sol-review, the
reviewskill checklist (twice), a performance pass,simplify-rust,audit-tests, the security agent, and a claims-vs-code audit of the commit messages, CHANGELOG, STABILITY, baselines and issue comments. Every finding is in the branch except two recorded out of scope: Perl's$h{sub}auto-quoted key billing (pre-existing, output-moving) and the TS/TSX + Cpp/Mozcpp row copies (contract 5 of #1690 assigns them to R04). Groovy's unbilledvaris #1704, excluded on its row.Decisions recorded
import static/requires staticguard stays; Java and Groovy bill no directive keyword, C# billsusingand is the outlier, now said in the book's Halstead rules. Measured both ways in the issue.22a10d7a; the correct claim is on PHP'sPrimitiverow.c_family_*!fragments rather thankind_sets.rs; the two test helpers shareassert_keyword_role.Also
658cc305bumpsctutils0.4.3 → 0.4.5 (Cargo.lockandfuzz/Cargo.lock, no manifest change) for RUSTSEC-2026-0385, which landed in the advisory database aftermain's lastcargo-denyrun and fails that job on any branch until bumped.Not squashed
The history is the review trail — several later commits amend an earlier one (
8744f4eb,667fb6d0,85e85c8b,18a2a6ae,bcc46ae1,de773579). A squash into six is straightforward but.bca-baseline.tomlis-merge, so it needs therebase --execrefresh from AGENTS.md rather than plain fixups; happy to do it on request.Coverage
Project coverage is unchanged (98.29%). The patch report's 34 uncovered lines were read one by one rather than chased: 27 are
const fns the macro evaluates at compile time underconst _: () = assert!(…)(pinned bycompile_faildoctests; llvm-cov cannot see const evaluation), a test's panic-message path, and the conflict getter's never-returned arm. The two that were real are in80adf17a: the trait's defaultget_keyword_role, which every language overrode and a future one could have inherited silently, is now required; andRole::of's "in neither vocabulary" arm has an honest assertion.