Ledger rules classify diffs they do not describe, and a rule that stops explaining anything is invisible

[derek73]

tools/differential/compare.py classifies a diff by the first ledger rule that matches, where fields matches by subset (compare.py:469) and name_regex rules sort ahead of fields-only ones (compare.py:114). Nothing checks that the rule which claims a diff actually describes it, so a broad rule silently explains changes it says nothing about — and the run exits 0.

Found while reviewing #370, which hit it twice: once as a discovery, once by reproducing it.

Confirmed instances

1. fix(suffix-routing) claims the whole corpus. expected_since_1.4.0.toml:166 — the only fields-only rule in any ledger, fields = ["given","family","suffix"], no regex, so it matches all 751 names. Tallying classifier-of-record over every (name × field) pair, it owns 1639 of 5257 — 31% of the space. #370's diff landed on it before a rule was written:

classify('Mr. Van Nguyen', {'given','family'})
  without a #367 rule -> fix(suffix-routing) "two-token name with unambiguous trailing suffix stays suffix"

2. fix(comma-family) matches on a bare comma. expected_since_1.4.0.toml:118 — name_regex = ",", fields = ["given","title","suffix"], reaching 236 corpus names and 715 (name × field) pairs. It latently absorbs #367's own comma-bearing class:

'Dr. Do Van Johnson, MD'   moved ['given','title']  -> classified fix(comma-family)

No such name is in the corpus today, so this is latent — it becomes a green-on-regression the day one is added. That rule's prose already concedes it absorbed five CJK honorific rows it "has nothing to do with", which is documented; the bare-comma reach is not.

3. A blanket CJK rule shadows five to seven specific ones. expected_since_1.4.0.toml:62 and the 2.0.0 twin — name_regex is a bare CJK/Hangul/Kana character class, fields = ["given","middle","family"], matching all 97 CJK corpus names. It mechanically shadows fix(cjk-maiden-marker), fix(cjk-comma-compound), fix(cjk-honorific-suffix), fix(cjk-delimited-nickname), fix(cjk-fullwidth-paren-nickname), and in the 2.0.0 ledger also fix(#308/#312/#319/#320) and fix(#307/#308/#320). A future regression in Japanese or Korean given/family splitting is labelled with 2.1-era issue numbers and exits 0.

The structural gap: a rule that stops explaining anything is invisible

_CORPUS_CLAIMS in tests/v2/test_ledger_guards.py records what a rule's regex matches, which is independent of the parser. Reverting #370's fix and re-running showed 5 unit tests failing (good) but no ledger or differential failure — the 2.1.0 baseline reports 0 diffs, 0 unexplained, exit 0, with its only rule now explaining nothing. Deleting a rule is caught (test_every_rule_claims_the_recorded_share_of_the_corpus); a rule going inert is not.

Two cheap mechanical checks

Full over-claim detection is human judgement — no code decides whether "lone post-comma piece routes to suffix/title" describes Dr. Do Van Johnson, MD. But two checks are not judgement, and either would have surfaced instances 1 and 2 at authoring time:

1. Report ambiguity. classify() returns on first match. Have it collect every matching rule and print AMBIGUOUS 'Mr. Van Nguyen' -> [fix(#367), fix(suffix-routing)] alongside the classification. Exit code unchanged; the operator sees the overlap without having to delete a rule to discover it.
2. A specificity floor for fields-only rules. validate_rules (compare.py:362) already rejects a fields list naming all seven roles, but permits a fields-only rule with six. A fields-only rule reaching every corpus name is the shape most worth a warning, or an explicit declaration of unbounded reach.

A --strict mode asserting every rule fired at least once would close the third gap.

Note

#370 tightened its own fix(#367) rule after this was found — it originally reached 11 corpus names including Vincent van Gogh, which AGENTS.md:148 names as a standard regression canary — so the new rule is not an instance. The three above are pre-existing.

[derek73]

Closing — all three confirmed instances resolved, both proposals declined on measurement.

Instance 1, fix(suffix-routing) — #376. It claimed 25 names in four shapes while its prose described four. Split into fix(cjk-glued-honorific-peel) (17), a widened fix(cjk-honorific-suffix) (11), and fix(comma-precomma-family) (3, which move no suffix at all). Now claims exactly the 4 its original prose names.

It was never too broad. It is the only fields-only rule in any ledger, so it sorts after every name_regex rule and takes whatever nothing narrower named. When it grows, the question is which rule is missing.

Instance 2, fix(comma-family) — #375. Bare , reached every comma in every script; 7 of its 8 names were CJK. Latin-anchored at U+0250; now claims Andrews, M.D. alone. The 7 went to fix(cjk-comma-honorific-peel), added because the first fix merely relocated them onto a rule that also didn't describe them.

Instance 3, the blanket CJK rule — not live, and not quite as filed. fix(#271/#272/#298) carries fields = ["given", "middle", "family"], and every rule it was said to shadow includes a field outside that set (maiden, nickname, suffix, title). Fields separate them, not order, so it cannot claim their characteristic diffs. Verified: all seven CJK-specific rules fire at 1.4.0 and all four at 2.0.0. If one ever were shadowed, #373 now reports it as EXPLAINED NOTHING ... shadowed by <issue>.

The structural gap — #373. A rule explaining nothing now fails the run, in both directions, with three diagnoses. #374 made the diagnosis a kind rather than prose. The --strict suggestion is implemented as always-on rather than behind a flag: a check nobody passes a flag to is a check that doesn't exist.

Report ambiguity — declined. Measured: 28% of claimed (name × role) pairs already have two or more matching rules, 432 of them fix(comma-family) under fix(suffix-routing) alone. A report firing on 28% of what it inspects is wallpaper. The actionable slice ships instead as the shadowed by <issue> diagnosis, which speaks only when a rule is fully shadowed.

Specificity floor — declined. There is exactly one fields-only rule in any ledger, naming 3 of 7 roles. A six-of-seven floor matches nothing, and nothing would reveal it was vacuous.

Also added: _CROSS_RULE_WINNERS pins which rule wins for contested names. Measured — a pure file reorder, no regex or fields change, fails that pin and nothing else in 3218 tests.