diff --git a/AGENTS.md b/AGENTS.md index 98cf4953..16a45834 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -306,12 +306,12 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. - **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment in `post_rules` decides that fork on the same path and reports it (#405), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and in `post_rules` when P6's attachment takes a trailing particle into the family after a comma, so all three report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; decisions.md's Excluded block records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. -- **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. +- **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). - **The shim TRANSLATES; it never raises on a config v1 accepted, and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. Six exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`). Note that last one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. -- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT rules.md, which is implementation-free by its own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. +- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT rules.md, which is implementation-free by its own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`), which cannot be wrong the way a sentence can. - **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited. - **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content. diff --git a/docs/design/decisions.md b/docs/design/decisions.md index e065204f..5eed4346 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -328,7 +328,7 @@ Declined (rc1 arc; the full argument is AGENTS.md's gotcha): OPENING THE NAME IS NOT OPENING A PIECE LIST, and the first draft got that wrong: a credential run legitimately opens segment 1 after a family comma (`Smith, Ph. D. Jr.`), which C1 reads as a listing rather than a name beginning. Testing `k == 0` alone broke fifteen tests and took the gate red at all three baselines. The caller passes the distinction in, since `_group_segment` cannot see which segment it holds. ACCEPTED, and found by review rather than by design: a TITLE before the pair keeps the credential reading, because `k` is a piece index and a title is a piece. `Sir Ph. D. Van Johnson` still reads given 'Van Johnson' with an empty family — #371's own symptom, surviving one word to the left. It is 1.4.0 parity, and the rule states it as a boundary rather than deviating: "the first piece of the name" cannot be computed before this merge, since H2's abbreviation test is true of `Ph.` itself, so a scan that stepped over titles would step over the very piece being judged. rules.md#S2 therefore says OPENS THE STRING, which is what the code holds and what v1 held. THE FIRST DRAFT GOT "THE HEAD" WRONG IN THE OTHER DIRECTION, and the corpus could not see it. Testing the PIECE index alone declined the merge for `"Bob" Ph. D. John Smith`: `extract_delimited` removes a quoted or bracketed clause before grouping, so the pair reaches the merge at `k == 0` with a word standing before it in the INPUT. v1 merges there (its regex needed only a preceding space), and 112 measured names broke parity — every one opening with a quote or a bracket, and not one of them in any corpus. The test is now `a[0] == 0`, the first TOKEN, which is v1's boundary exactly. - ACCEPTED: `Parser.revise(suffix="Ph. D.")` now renders 'Ph., D.'. revise() runs a full sub-parse of the string it is given, and a field value has no head for a head-position rule to consult; a second draft carved that out by requiring a name to displace, and the carve-out made the head reading depend on what FOLLOWED it — appending a maiden clause changed whether 'Ph.' was a title, which `test_a_maiden_clause_changes_nothing_else` caught. Dropping the carve-out removed the inconsistency with it. The merge exists for a credential someone TYPED after a name; a caller who writes the spaced form into the suffix field is taken at their word (Derek's call, 2026-08-31). + ACCEPTED: `Parser.revise(suffix="Ph. D.")` now renders 'Ph., D.'. revise() runs a full sub-parse of the string it is given, and a field value has no head for a head-position rule to consult; a second draft carved that out by requiring a name to displace, and the carve-out made the head reading depend on what FOLLOWED it — appending a maiden clause changed whether 'Ph.' was a title, which `test_a_maiden_clause_changes_nothing_else` caught. Dropping the carve-out removed the inconsistency with it. The merge exists for a credential someone TYPED after a name; a caller who writes the spaced form into the suffix field is taken at their word (Derek's call, 2026-08-31). SUPERSEDED 2026-09-06 by #511, and half of the reasoning here still holds: the MERGE is a head-position rule and still does not fire in a field value. The other half — a caller who writes the spaced form is taken at their word — is what #511 reverses (Derek's call, 2026-09-06, choosing the entry rule over the accepted split when the round-trip limit was measured). What changed is that rules.md#R1's entry pass now runs inside `revise()` over the forced roles, and `Ph.` and `D.` share a comma bucket with nothing between them, so the ENTRY pass joins the pair for the reason it joins `MD` and `PhD` — a different rule reaching the same two words. `Parser.revise(suffix="Ph. D.")` renders 'Ph. D.' (decisions.md#C1, 2026-09-06 #511). What the leading `Ph.` becomes is H2's business, not this rule's: an abbreviation before a name is almost always a title, which is the same clause that reads `Esq. van Gogh` as title 'Esq.' — so the pair reads title 'Ph.', given 'D.'. The issue proposed given 'Ph.', middle 'D.'; H2 claims the first word before the positional read sees it, and the two answers differ only in which field holds `Ph.`. ### O4 — positional assignment and declared order @@ -581,8 +581,9 @@ Excluded (MAIDEN_MARKERS, per nameparser/config/maiden_markers.py): VIEWS, over the 1117 corpus names — the 1116 that were there plus the R1 example: exactly 13 of the 1116 pre-existing names move, and the new example `John Smith MD PhD` makes 14; each of them moves in `suffix_list` and in the v2 dictionary and token-tag list and nowhere else. No other `*_list` view and no v2 role's token list moves on any name in the corpora. Recipe: for every corpus name dump the facade's eleven `*_list` properties plus `initials_list()` (a method, not a property) and, from `parse(name)`, `as_dict()`, `tokens_for(role)` for each role and every token's tags, before and after, and diff key by key. The tags are dumped beside the views because the change IS a tag change, and a view that renders the same over a token whose tags moved is what a view-only snapshot cannot see. Every mover goes from a comma-joined render to a space-joined one; none goes the other way, and none has a comma in the original between the two words. RE-PINNED, nine `cases.py` rows and not the two the spec expected: `title_plus_one_word_two_suffixes`, `family_segment_multiple_suffixes`, `suffix_run`, `family_comma_segment_zero_is_not_the_run`, `family_comma_numeral_behind_a_suffix_is_not_an_initial` and the four CJK honorific rows. Eight assertions outside the case table moved with them, four of those in the v1-parity suite (`tests/test_capitalization.py`, `tests/test_suffixes.py`), and they are a deliberate v1 deviation rather than a parity break: v1 inserted a comma into a run the writer had spaced, its own comments calling that "not ideal but at least its in the right bucket", and the release-log bullet names the deviation. Four rules.md example lines outside R1 — P2's `John Smith Mc V`, P5's `abdul Smith Jr Ma`, C1's `Smith, John PhD I.` and W3's `田中さん 様.` — carried the comma too; all four names are differential corpus entries, the last of them in `corpus_cjk_tolerated.jsonl`, so the movement was observed by the gate and not only by the doc test that executes them. `family_comma_segment_zero_is_not_the_run` keeps its NAME and its point — segment 0 is the family segment even when wholly credential-shaped, which is what the mutation matrix found and what the row still guards — and only its separator moves. The comma-joined values the nine carried were the shape machinery's artifact and never a reading anyone chose. ROUND-TRIPPING is stable on the shapes #429 fixed, which is what that entry recorded as the one load-bearing consequence of leaving this open: `str()` of a fixed parse is a no-comma string, and the no-comma path put the comma back. `test_a_suffix_run_survives_a_render_and_a_reparse` in tests/v2/test_render.py asserts `parse(str(parse(s))).suffix == parse(s).suffix` over the three writings. NOT stable in general, and the general claim is the one to resist — measured 2026-09-06 by evaluating that same equality over every corpus name, at the parent tree and here: 35 of 1116 names failed at the parent and 23 of 1117 fail here, thirteen fixed and ONE newly failing. The one is `Smith, John PhD I.`, and it is this same artifact seen from the other side: it round-tripped only because the render emitted 'John Smith PhD, I.' with a comma the writer never typed, which the no-comma path read back as a run. The render writes 'John Smith PhD I.' now, and that string reads middle 'Smith PhD', family 'I.' at the parent and here alike — a bare trailing initial behind a name is name material, #430's own boundary — so the two spellings genuinely disagree and the comma had been hiding it. The 23 that remain are the standing class: `str()` is a rendering, not a canonical form, and no rule in this file promises it re-parses. - NOT FIXED, and named because this change makes it reachable on more inputs: `revise(n, suffix=n.suffix)` is not the identity on a space-joined run. `Parser.revise` classifies each value by a full sub-parse of that value alone — its own docstring records the limit, the value being "classified ON ITS OWN" — and a bare 'MD PhD' carries no comma for the entry rule to route by, so the two words come back as two entries and the field re-renders 'MD, PhD'. Pre-existing, and the same shape as the `str()` limit above rather than a new one. Measured 2026-09-06 over every corpus name, comparing `revise(parse(n), suffix=parse(n).suffix).suffix` against `parse(n).suffix` at the parent tree and here: 24 of the parent's 1116 names fail and 38 of the 1117 here do, none of the parent's 24 recovering, and the 14 added are exactly the 14 movers — the 13 pre-existing names plus the R1 example. Not this change's to fix: the entry rule reads the commas in a whole name, and what `revise` hands its sub-parse is a field, so closing it means deciding what a field value's separators mean, which is a `revise` question. + NOT FIXED, and named because this change makes it reachable on more inputs: `revise(n, suffix=n.suffix)` is not the identity on a space-joined run. `Parser.revise` classifies each value by a full sub-parse of that value alone — its own docstring records the limit, the value being "classified ON ITS OWN" — and a bare 'MD PhD' carries no comma for the entry rule to route by, so the two words come back as two entries and the field re-renders 'MD, PhD'. Pre-existing, and the same shape as the `str()` limit above rather than a new one. Measured 2026-09-06 over every corpus name, comparing `revise(parse(n), suffix=parse(n).suffix).suffix` against `parse(n).suffix` at the parent tree and here: 24 of the parent's 1116 names fail and 38 of the 1117 here do, none of the parent's 24 recovering, and the 14 added are exactly the 14 movers — the 13 pre-existing names plus the R1 example. Not this change's to fix: the entry rule reads the commas in a whole name, and what `revise` hands its sub-parse is a field, so closing it means deciding what a field value's separators mean, which is a `revise` question. Answered, and FIXED, by #511 — the next bullet. DECLINED, all four with the evidence: marking the boundary at the core-drop site (#437's MARK-DONT-STRIP shape) — `dropped` already holds the fact with its span, and a second recording of it is the duplication that mechanism exists to prevent, one level up; making the `"joined"` tag role-aware — within a piece it is role-blind and correct for every role, `Smith, Ph. D. Smith` giving `first_list == ['Ph. D.']`, and only the between-piece half was ever a suffix concept; a render-time span scan instead of the recorded tag — `_facade.__setstate__` and `ParsedName.replace()` synthesise span-less tokens, so an unpickled name has nothing to scan and the tag IS the entry structure a pickle carries; and fixing the no-comma path inside group's block as a third branch beside `tail` and `reading`, which is the shape #429 took. +- 2026-09-06 #511 — a suffix value handed to `revise()` derives its entries from its own commas, by the rule a whole name uses. `Parser.revise` sub-parsed each value and forced every harvested token to the named role AFTER the sub-parse had run R1's entry pass, which keys on Role.SUFFIX; a bare 'MD PhD' reads there as a title and a family name, so the pass joined nothing and the field rendered 'MD, PhD'. The fix runs the same pass again: `revise` now sub-parses to a ParseState, forces the role on every non-dropped token, and calls `suffix_entries` — the pass, lifted unedited out of post_rules' tail into a function the two callers share (mechanisms.md#ONE-PREDICATE-PER-QUESTION) — over the forced state, then assembles and harvests as before. So a comma in the value parts two credentials and a space joins them. TWO SPELLINGS of the lifted pass, and the reason is the call budget: a first draft had post_rules call the state-in/state-out wrapper, and the second ParseState build cost three more calls per parse against the band tests/v2/test_benchmark.py holds (py3.11, 2026-09-06, tools/perf/call_count.py: 450 calls/name before the move, 451 with the in-place worker `_mark_suffix_entries` that post_rules now calls, 454 with the draft; the facade band tops at 455.9), so the worker writes in place as every other post rule does and the wrapper exists for the one caller with no token list of its own. NO `_run` HELPER for the same reason: a draft routed `parse()` through a private state builder shared with `revise`, and that cost one frame per parse on the hot path (py3.11, 2026-09-06: 415 calls/name against 414 without it, in a 402-418 band), so the four-line construction is spelled twice and `parse()` is untouched. MEASURED 2026-09-06 over the 1117 distinct names in `tools/differential/corpus*.jsonl`, comparing `revise(p, suffix=p.suffix).suffix` against `p.suffix` under the default parser for every name with a non-empty suffix (368 of the 1117): 38 differed before and 1 after; the differential gate is byte-identical at all four baselines, `revise` not being on the compare path. RECOMPUTE: parse each corpus name, skip an empty suffix, revise the parse with its own suffix, count the names whose suffix moved (the script is in the #511 issue body). THE ONE LEFT is '김민준씨, J.씨', and it is not entry structure: the whole-name parse keeps 'J.씨' one glued suffix token, suffix '씨, J.씨', while the sub-parse of the bare value '씨, J.씨' peels the honorific off the initial, so the revised field renders '씨, J. 씨' — right entries, the spurious comma of before ('씨, J., 씨') gone, one word read differently by the value's own parse than by the whole name's. That is the "classified ON ITS OWN" limit `revise`'s docstring has always recorded, and CJK honorific peeling is a W-rule question this change does not move; pinned in `test_revise_reads_a_glued_honorific_on_its_own`. SUPERSEDES the phd-merge acceptance of 'Ph., D.' on this path (its bullet says how). STALE TAGS, tried and backed out: a draft cleared the sub-parse's own "joined" with the forcing and re-derived it, because the pass only ADDS the tag and `revise(n, family="Jones MD PhD")` carries the sub-parse's between-piece suffix mark on 'PhD' onto a FAMILY token. Measured 2026-09-06 against `330ee55`, the clear also destroyed every WITHIN-piece mark on a non-suffix value — 'D.' of `revise(n, family="John Ph. D. Smith")` lost the merge mark the #436 bullet's DECLINED list calls role-blind and correct for every role — while on a ParsedName only the suffix string view reads "joined" (`_text_for`'s suffix_join gate) — the facade's `_list_for` heals it for every role, but no path puts a revised name into a HumanName, the v1 setters going through `replace()`, so wiring those setters onto `revise` is the change that would show the stale mark, as `last_list == ['Jones', 'MD PhD']` — `initials()` and every field string being identical at both trees for every shape measured. So the tags are kept minus FOLDED_TAG as before; the between-piece mark on a forced non-suffix role is a tag-only oddity that predates this change and stays, and for a suffix value nothing depends on the sub-parse's marks, every pair it joined sharing a bucket with no parting token and the pass setting it again; pinned in `test_revise_keeps_the_sub_parses_within_piece_mark`. Dropped tokens keep their role and tags through the forcing, because the pass filters them by index and assemble omits them. ONE MORE LIMIT, pinned in `test_revise_leaves_a_policy_delimiter_unparted_without_a_tail_segment`, and it is the "classified ON ITS OWN" limit again rather than a rule of revise's: a delimiter the policy names through `extra_suffix_delimiters` is dropped, and so parts entries, only on a segment after a comma that the reading of the words makes a tail, and a value with no comma of its own has none — under `Policy(extra_suffix_delimiters=frozenset({" - "}))` the whole name 'Doe, John, MD PhD - FACS' renders 'MD PhD, FACS' while `revise(n, suffix="MD PhD - FACS")` renders 'MD PhD - FACS', the dash surviving as a token and the forced role making it a suffix word; measured 2026-09-06, a value whose own words read with a tail segment does part ('John Doe, MD - FACS' revises to 'John Doe, MD, FACS') and one after a suffix comma does not ('MD, PhD - FACS' stays), which is how the value's words read as a name deciding it. The round-trip is unaffected, the whole-name view having already rendered that boundary as a comma. DECLINED: a list-valued `revise(p, suffix=["MD PhD", "FACS"])`, which widens the API and leaves the string path where it was; documenting the limit with pins alone, the docstring already recording it and #511's measurement showing it reachable on every space-joined run #436 produced; and a text-only read of the delimiter cores inside `revise`, a second rule for a case no round-trip reaches. Out of scope and left as is: `ParsedName.replace(suffix="MD, PhD")` renders 'MD,, PhD', `replace` whitespace-splitting by contract and the facade setters riding on it for v1 parity. ### T1 — separators, not joiners diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 240a52ec..42a3671d 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it diff --git a/docs/design/rules.md b/docs/design/rules.md index 73ccd979..56644205 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1235,7 +1235,20 @@ R1. Rationale: a field is a way of reading the parse, not a stored "Smith, MD PhD" → suffix="MD PhD" "John Smith MD PhD" → suffix="MD PhD" "John Smith, MD, Bart" → suffix="MD, Bart" - history: decisions.md#C1 · interacts: O3, P6, R3 · implemented: nameparser/_pipeline/_post_rules.py, nameparser/_types.py + Accepted: a suffix value handed to revise() derives its entries the + same way, from the value's own commas, so a name's rendered suffix + revises back to itself wherever the value's words read as the + whole name read them — a glued CJK honorific peels off an initial + in a bare value where the whole name kept it glued, one corpus + name of the 368 with a suffix (decisions.md#C1, 2026-09-06). A + delimiter the configuration names parts a value only where the + value's own words, read as a name, give it a tail segment for the + core to be dropped on; a run of post-nominals has none, with or + without a comma of its own, so there the delimiter stays a word of + the run — write a comma at the boundary instead. Stated without an + example line because every line here names an input string, and + this shape needs a field revised after the parse. + history: decisions.md#C1 · interacts: O3, P6, R3 · implemented: nameparser/_parser.py, nameparser/_pipeline/_post_rules.py, nameparser/_types.py R2. Rationale: callers need the surname with and without its particles — sorting wants "Vega", display wants "de la Vega". diff --git a/docs/release_log.rst b/docs/release_log.rst index a4b2885f..6a5b7779 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -10,7 +10,9 @@ Release Log - **Record a 2.0.0 change to HumanName.initials() that no release note had classified:** since 2.0.0 the facade initials each WORD of a name part, where 1.4.0 initialed a joined run as one group -- ``HumanName("Juan Velasquez y Garcia").initials()`` is ``J. V. G.`` and was ``J. V G.``; ``Abdul Salam Hassan`` is ``A. S. H.`` and was ``A S. H.``. Nothing changes in 2.3.0; the differential gate now compares ``initials()`` (#484) and this is what it found. See the ``differential-ledger, the initials view`` entry of ``docs/design/decisions.md`` - - **Fix a space-separated run of post-nominals rendering with a comma the writer never typed.** ``HumanName("John Smith MD PhD").suffix`` is ``MD PhD`` and was ``MD, PhD`` at every release since 1.4.0; ``Kenneth Clarke QC MP`` gives ``QC MP``, and the CJK honorific runs (``田中さん II``, ``김민준 박사 씨``) follow the same rule. This is a deliberate deviation from 1.4.0, which inserted a comma into a run the writer had spaced; the v1-parity suite is re-pinned to match. The comma forms are unchanged -- ``HumanName("Smith, MD, PhD").suffix`` is still ``MD, PhD`` -- because the separator is now the comma the writer typed rather than the shape of the comma segments, and a configured suffix delimiter still parts a run, as does a name word standing between two post-nominals. Round-tripping is fixed for these runs, which the 2.2.0 note below recorded as broken: ``str(HumanName("Smith, MD PhD"))`` is ``Smith MD PhD`` and re-parses to suffix ``MD PhD``, where the comma-written and space-written spellings of one run used to disagree. ``str()`` is still a rendering rather than a canonical form, and one name goes the other way: ``HumanName("Smith, John PhD I.")`` renders ``John Smith PhD I.``, which reads middle ``Smith PhD`` and family ``I.`` as it always has, so that name round-tripped only on the strength of the comma this fix removes. Thirteen names that predate this change move in the differential corpora (fourteen with the example this change adds), all in the same direction, and no other field view moves. One limit is not fixed here: ``Parser.revise(n, suffix=n.suffix)`` is not the identity on a space-joined run, because the bare string has no comma to route by and the words come back as separate entries — pre-existing, ``revise``'s own docstring recording that a value is classified on its own, and reachable on more inputs now. See the ``C1`` entry of ``docs/design/decisions.md`` (closes #436, closes #437) + - **Fix a space-separated run of post-nominals rendering with a comma the writer never typed.** ``HumanName("John Smith MD PhD").suffix`` is ``MD PhD`` and was ``MD, PhD`` at every release since 1.4.0; ``Kenneth Clarke QC MP`` gives ``QC MP``, and the CJK honorific runs (``田中さん II``, ``김민준 박사 씨``) follow the same rule. This is a deliberate deviation from 1.4.0, which inserted a comma into a run the writer had spaced; the v1-parity suite is re-pinned to match. The comma forms are unchanged -- ``HumanName("Smith, MD, PhD").suffix`` is still ``MD, PhD`` -- because the separator is now the comma the writer typed rather than the shape of the comma segments, and a configured suffix delimiter still parts a run, as does a name word standing between two post-nominals. Round-tripping is fixed for these runs, which the 2.2.0 note below recorded as broken: ``str(HumanName("Smith, MD PhD"))`` is ``Smith MD PhD`` and re-parses to suffix ``MD PhD``, where the comma-written and space-written spellings of one run used to disagree. ``str()`` is still a rendering rather than a canonical form, and one name goes the other way: ``HumanName("Smith, John PhD I.")`` renders ``John Smith PhD I.``, which reads middle ``Smith PhD`` and family ``I.`` as it always has, so that name round-tripped only on the strength of the comma this fix removes. Thirteen names that predate this change move in the differential corpora (fourteen with the example this change adds), all in the same direction, and no other field view moves. See the ``C1`` entry of ``docs/design/decisions.md`` (closes #436, closes #437) + + - **Fix Parser.revise() splitting a space-separated suffix value into comma-separated entries.** ``Parser().revise(n, suffix="MD PhD").suffix`` is ``MD PhD`` and was ``MD, PhD``, and a name's own rendered suffix now revises back to itself for all but one of the 38 differential-corpus names that failed to on 2026-09-06 (368 of the 1117 carry a suffix) (recipe in the ``C1`` entry of ``docs/design/decisions.md``; the one left is a Korean honorific glued to an initial, which the value's own parse peels where the whole name kept it glued -- a word read on its own, not an entry boundary). A suffix value's entries are now derived from the value's own commas by the same rule a whole name uses: a comma parts two credentials and a space joins them, so ``revise(n, suffix="MD, PhD")`` is still two entries. ``revise(n, suffix="Ph. D.")`` renders ``Ph. D.`` where the 2.2.0 note below accepted ``Ph., D.``; the head-position merge that note describes still does not fire, the pair joining under the entry rule instead. One limit: a delimiter configured through ``extra_suffix_delimiters`` parts a value only where the value's own words read as a name with a tail segment, so in a run of post-nominals it stays a word; write a comma at the boundary instead. ``ParsedName.replace()`` is unchanged (closes #511) * 2.2.0 - August 31, 2026 diff --git a/nameparser/_parser.py b/nameparser/_parser.py index dd64c928..5405f337 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -17,11 +17,12 @@ from nameparser._locale import Locale from nameparser._pipeline import run from nameparser._pipeline._assemble import assemble +from nameparser._pipeline._post_rules import suffix_entries from nameparser._pipeline._state import ParseState from nameparser._pipeline._vocab import _SCRIPT_MATCHERS from nameparser._policy import UNSET, Policy, PolicyPatch, _Unset, apply_patch from nameparser._types import ( - FOLDED_TAG, ParsedName, Segmenter, Token, _guarded_getstate, + FOLDED_TAG, ParsedName, Role, Segmenter, Token, _guarded_getstate, _guarded_setstate, _validated_field_strings, ) @@ -165,16 +166,30 @@ def revise(self, name: ParsedName, **fields: str) -> ParsedName: capitalized()) behave as if the text had been parsed. The value is classified ON ITS OWN, though -- a word whose reading depends on surrounding context may classify differently than - it would in place (a standalone "B. S." reads as initials, not - a suffix run). The sub-parse's role choices and ambiguities - are discarded -- every harvested token takes the named field's - role -- and its structural behavior applies: delimiter - characters do not become tokens, and a maiden marker is - consumed as in parsing -- mid-value always, and leading a - DELIMITED value under a policy routing that pair to maiden, - where "(née Jones)" revises to "Jones" while the bare - "née Jones" keeps its marker, a leading marker in an + it would in place, and a glued CJK honorific the whole name + kept on an initial may peel in the value. The sub-parse's role + choices and ambiguities are discarded -- every harvested token + takes the named field's role -- and its structural behavior + applies: delimiter characters do not become tokens, and a + maiden marker is consumed as in parsing -- mid-value always, + and leading a DELIMITED value under a policy routing that pair + to maiden, where "(née Jones)" revises to "Jones" while the + bare "née Jones" keeps its marker, a leading marker in an undelimited value being no marker at all (#329). + A suffix value's ENTRY structure is derived from the value's + own commas after the role is forced, by the rule a whole name + uses: a comma parts two credentials and a space joins them, so + ``revise(n, suffix="MD PhD")`` is one entry and a name's + rendered suffix revises back to itself wherever the value's + words read as the whole name read them -- the honorific peel + above is the one corpus exception of 368 suffix-bearing names, + 2026-09-06 (#511). A delimiter the policy names through + ``extra_suffix_delimiters`` parts a value only where the + value's own words, read as a name, give it a tail segment for + the core to be dropped on; a run of post-nominals has none, + with or without a comma of its own (``"MD PhD - FACS"`` and + ``"MD, PhD - FACS"`` both keep the dash as a word), so write a + comma at the boundary you want rather than the delimiter. Tokens are synthetic (span=None); original is unchanged; a value with no name content (empty, whitespace, or punctuation only) clears the field; ambiguities referencing replaced @@ -182,12 +197,46 @@ def revise(self, name: ParsedName, **fields: str) -> ParsedName: if not isinstance(name, ParsedName): raise TypeError(f"revise() takes a ParsedName, got {name!r}") replaced = _validated_field_strings(fields) - harvested = { - role: tuple( + harvested: dict[Role, tuple[Token, ...]] = {} + for role, value in replaced.items(): + # the same construction parse() makes, spelled twice + # rather than through a helper: routing parse() through + # one cost a frame per parse on the hot path (py3.11, + # 2026-09-06: 415 calls/name against 414 without it, in a + # 402-418 band), the same trade _mark_suffix_entries + # refused, one row over + state = run(ParseState(original=value, lexicon=self.lexicon, + policy=self.policy, + segmenter=self.segmenter)) + dropped = set(state.dropped) + # Force the role BEFORE the entry pass: it keys on + # Role.SUFFIX, and a bare value's sub-parse reads its words + # as a name ('MD PhD' is a title and a family there), so + # inside the sub-parse it joined nothing. The sub-parse's + # tags are KEPT, "joined" included: a within-piece mark + # (the Ph. D. merge) is role-blind and right for every + # role, and a clear was measured and backed out -- + # decisions.md#C1 (2026-09-06 #511) carries the rest: the + # between-piece mark that rides onto a non-suffix role, and + # which views read it. Dropped tokens keep their role: the + # pass filters them by index and assemble omits them. + forced = tuple( + tok if i in dropped + else dataclasses.replace(tok, role=role) + for i, tok in enumerate(state.tokens)) + # rules.md#R1: "a run of post-nominals written with spaces + # renders with spaces, and one written with commas keeps + # them" -- the same pass post_rules runs last, over the + # value's own comma offsets, so the harvest carries exactly + # the entry structure the value's commas describe. Run for + # every role: for a non-suffix role it is a no-op, kept + # unconditional as the simpler contract, revise() being off + # the call-count band. + entried = suffix_entries( + dataclasses.replace(state, tokens=forced)) + harvested[role] = tuple( Token(t.text, None, role, t.tags - {FOLDED_TAG}) - for t in self.parse(value).tokens) - for role, value in replaced.items() - } + for t in assemble(entried).tokens) return name._with_field_tokens(harvested) # -- comparison ------------------------------------------------------- diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index cf61e7eb..1b94b845 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -14,7 +14,9 @@ Implements rules H1, M4, P1, O1, O2, O3 and R1 of docs/design/rules.md; each is cited at its code below, and H1/P1/O1/O2's history lives in -docs/design/decisions.md. +docs/design/decisions.md. `suffix_entries` is the R1 entry pass as a +state-in/state-out function, for Parser.revise to run over a +forced-role sub-parse (#511); `post_rules` runs the same worker last. """ from __future__ import annotations @@ -63,6 +65,145 @@ _RENDERS_ELSEWHERE = frozenset({Role.TITLE, Role.NICKNAME, Role.MAIDEN}) +def _mark_suffix_entries(tokens: list[WorkToken], state: ParseState) -> None: + # In place over the caller's token list, the way every other rule + # in post_rules writes: a state-in/state-out spelling here cost + # three calls per parse for the second ParseState build, against + # the call-count band tests/v2/test_benchmark.py holds (measured + # 2026-09-06 with tools/perf/call_count.py, py3.11: 450 calls/name + # before the move, 451 with this worker, 454 with the state-wrapper + # draft; the facade band tops at 455.9 and parse, the tighter row, + # sits at 414 in a 402-418 band). suffix_entries() below is the + # state wrapper; its docstring says for whom. + # Reads dropped, comma_offsets and the policy off `state`, which + # post_rules does not change, so the un-rebuilt state is current. + # + # rules.md#R1: "a run of post-nominals written with spaces renders + # with spaces, and one written with commas keeps them" + # + # The entry boundary, read off the text the writer typed rather + # than off the shape of the segments (#436/#437). Two consecutive + # SUFFIX tokens are one entry iff they sit in the same comma + # bucket AND nothing between them parts the run -- what parts it + # and what does not is spelled out below. The + # comma is the separator the rule names -- comma_bucket is the + # function segment BUILDS segments with and classify asks about + # boundaries, so "same part" here is an identity with segment's + # answer rather than a resemblance to it + # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). + # + # What parts a run: a name word between the two post-nominals + # (GIVEN/MIDDLE/FAMILY), or a dropped delimiter core (#206). What + # does NOT part it: a token whose role is in _RENDERS_ELSEWHERE, + # because it renders into another field entirely and so is not + # standing in the run at all -- 'Smith, MD Dr. PhD', + # 'Smith, MD "Doc" PhD' and 'Smith, MD (nee Jones) PhD' are each + # one entry; and a dropped token that is not a core, which is the + # maiden MARKER of 'Smith, MD nee Jones PhD' (the marker is + # dropped with no role at all, so the role test cannot see it). + # That is why the dropped arm reads the core set instead of + # treating every dropped index as a boundary: a core is the one + # dropped token the writer typed AS a separator, and the set is + # `delimiter_cores` -- group's own derivation off + # Policy.extra_suffix_delimiters, imported rather than repeated + # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). The two sites read + # ONE derivation and differ only in a gate: group drops a core + # only on a `tail` segment, through `seg_cores`, while this arm + # reads `delimiter_cores` whole and asks by TEXT alone. So a + # dropped token whose text the policy names as a delimiter parts + # the run whatever dropped it. The gate is not needed here: a core + # dropped BY GROUP was on a tail segment by construction, and the + # only case the ungated read adds is a maiden marker the policy + # ALSO lists -- under + # `Policy(extra_suffix_delimiters=frozenset({" nee "}))`, + # 'John Doe, MD nee Jones PhD' renders 'MD, PhD' where the default + # policy renders 'MD PhD' (measured 2026-09-06). That is the + # policy's own declaration deciding it: the writer's configuration + # named that text a separator, so the run parts there. + # + # Four shapes were declined, recorded in decisions.md#C1 by the + # bundle that landed this pass: marking the boundary at the + # core-drop site (`dropped` already holds the fact with its + # span, so a second recording of it is the duplication + # MARK-DONT-STRIP exists to prevent); making the "joined" tag + # role-aware (within a piece it is role-blind and right for every + # role -- 'Smith, Ph. D. Smith' gives first_list ['Ph. D.']); + # scanning spans at render time instead of reading the tag; and + # adding a third shape-derived branch inside group's block. + # + # AFTER assign, and THAT is the load-bearing constraint: this keys + # on Role.SUFFIX, and the same span rule run role-blind would join + # the A and B of 'John A B Smith' into one middle_list element + # (test_the_pass_runs_after_roles_are_settled pins it). + # + # Within post_rules the pass runs last by convention, not by + # necessity. It reads SUFFIX and _RENDERS_ELSEWHERE, and no rule + # in post_rules writes either: every retag in post_rules targets a + # NAME role and nothing else -- `_retag` is called with + # Role.FAMILY, Role.GIVEN, Role.MIDDLE, and with + # `_name_positions`' return, which is those same three; the three + # `role=Role.FAMILY` replaces (P6's attachment on both arms, O3's + # fold) are FAMILY as well. So no rule here + # moves a token into or out of SUFFIX, or into or out of + # {TITLE, NICKNAME, MAIDEN}, and this predicate reads the same + # answer wherever in the stage it stands (measured 2026-09-06 by + # reading the stage's retag targets). + # + # RECORDED as a tag rather than recomputed by the render, because + # the render cannot see a span: _facade.__setstate__ and + # ParsedName.replace() build span-less tokens AFTER the pipeline, + # so an unpickled name has nothing to scan and the tag IS the + # entry structure the pickle carries + # (mechanisms.md#MARK-DONT-STRIP). Every token here has a span -- + # tokenize is the sole producer of a WorkToken, and WorkToken.span + # is not Optional -- so `span.start` is read unguarded. + # + # The `i not in dropped` filter is belt-and-braces: a dropped token + # never carries a SUFFIX role. Cores leave `pieces` before assign + # runs, so assign gives them no role at all; the one class of + # dropped token that arrives already roled is the MAIDEN one that + # tokenize roles from an extracted clause ('Smith, MD (nee Jones) + # PhD' drops index 2, the marker, and it is Role.MAIDEN). The + # filter is here so that `suffixes` and the `parted` scan below + # cannot disagree about what a dropped index is. + dropped = set(state.dropped) + cores = delimiter_cores(state.policy.extra_suffix_delimiters) + suffixes = [i for i, tok in enumerate(tokens) + if tok.role is Role.SUFFIX and i not in dropped] + for previous, current in zip(suffixes, suffixes[1:]): + same_part = (comma_bucket(tokens[previous].span.start, + state.comma_offsets) + == comma_bucket(tokens[current].span.start, + state.comma_offsets)) + parted = any( + tokens[between].text in cores if between in dropped + else tokens[between].role not in _RENDERS_ELSEWHERE + for between in range(previous + 1, current)) + if same_part and not parted: + tokens[current] = dataclasses.replace( + tokens[current], tags=tokens[current].tags | {"joined"}) + + +def suffix_entries(state: ParseState) -> ParseState: + """rules.md#R1's entry pass over a whole state, wrapping + _mark_suffix_entries: tag a SUFFIX token "joined" when it + continues the entry of the SUFFIX token before it. post_rules runs + the worker last, over the roles assign settled; this wrapper exists + for Parser.revise (#511), which runs it over a SUB-PARSE of a field + value whose every non-dropped token it has forced to the named + role, so a suffix value derives its entries from its own commas by + the rule a whole name uses. The pass ADDS the tag and never removes + one; a caller forcing roles keeps what the sub-parse marked, a + within-piece mark being role-blind (decisions.md#C1, the #436 + DECLINED list) and every between-piece mark on a suffix value + being one this pass sets again. Reads comma_offsets, dropped and + Policy.extra_suffix_delimiters; writes the "joined" tag and + nothing else.""" + tokens = list(state.tokens) + _mark_suffix_entries(tokens, state) + return dataclasses.replace(state, tokens=tuple(tokens)) + + def _idx(tokens: list[WorkToken], role: Role) -> list[int]: return [i for i, t in enumerate(tokens) if t.role is role] @@ -637,108 +778,6 @@ def post_rules(state: ParseState) -> ParseState: for i in part: tokens[i] = dataclasses.replace( tokens[i], tags=tokens[i].tags | {UNJOINED_TAG}) - # rules.md#R1: "a run of post-nominals written with spaces renders - # with spaces, and one written with commas keeps them" - # - # The entry boundary, read off the text the writer typed rather - # than off the shape of the segments (#436/#437). Two consecutive - # SUFFIX tokens are one entry iff they sit in the same comma - # bucket AND nothing between them parts the run -- what parts it - # and what does not is spelled out below. The - # comma is the separator the rule names -- comma_bucket is the - # function segment BUILDS segments with and classify asks about - # boundaries, so "same part" here is an identity with segment's - # answer rather than a resemblance to it - # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). - # - # What parts a run: a name word between the two post-nominals - # (GIVEN/MIDDLE/FAMILY), or a dropped delimiter core (#206). What - # does NOT part it: a token whose role is in _RENDERS_ELSEWHERE, - # because it renders into another field entirely and so is not - # standing in the run at all -- 'Smith, MD Dr. PhD', - # 'Smith, MD "Doc" PhD' and 'Smith, MD (nee Jones) PhD' are each - # one entry; and a dropped token that is not a core, which is the - # maiden MARKER of 'Smith, MD nee Jones PhD' (the marker is - # dropped with no role at all, so the role test cannot see it). - # That is why the dropped arm reads the core set instead of - # treating every dropped index as a boundary: a core is the one - # dropped token the writer typed AS a separator, and the set is - # `delimiter_cores` -- group's own derivation off - # Policy.extra_suffix_delimiters, imported rather than repeated - # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). The two sites read - # ONE derivation and differ only in a gate: group drops a core - # only on a `tail` segment, through `seg_cores`, while this arm - # reads `delimiter_cores` whole and asks by TEXT alone. So a - # dropped token whose text the policy names as a delimiter parts - # the run whatever dropped it. The gate is not needed here: a core - # dropped BY GROUP was on a tail segment by construction, and the - # only case the ungated read adds is a maiden marker the policy - # ALSO lists -- under - # `Policy(extra_suffix_delimiters=frozenset({" nee "}))`, - # 'John Doe, MD nee Jones PhD' renders 'MD, PhD' where the default - # policy renders 'MD PhD' (measured 2026-09-06). That is the - # policy's own declaration deciding it: the writer's configuration - # named that text a separator, so the run parts there. - # - # Four shapes were declined, recorded in decisions.md#C1 by the - # bundle that landed this pass: marking the boundary at the - # core-drop site (`dropped` already holds the fact with its - # span, so a second recording of it is the duplication - # MARK-DONT-STRIP exists to prevent); making the "joined" tag - # role-aware (within a piece it is role-blind and right for every - # role -- 'Smith, Ph. D. Smith' gives first_list ['Ph. D.']); - # scanning spans at render time instead of reading the tag; and - # adding a third shape-derived branch inside group's block. - # - # AFTER assign, and THAT is the load-bearing constraint: this keys - # on Role.SUFFIX, and the same span rule run role-blind would join - # the A and B of 'John A B Smith' into one middle_list element - # (test_the_pass_runs_after_roles_are_settled pins it). - # - # Within this stage the pass sits last by convention, not by - # necessity. It reads SUFFIX and _RENDERS_ELSEWHERE, and no rule - # in post_rules writes either: every retag above targets a NAME - # role and nothing else -- `_retag` is called with Role.FAMILY, - # Role.GIVEN, Role.MIDDLE, and with `_name_positions`' return, - # which is those same three; the two `role=Role.FAMILY` replaces - # (P6's attachment, O3's fold) are FAMILY as well. So no rule here - # moves a token into or out of SUFFIX, or into or out of - # {TITLE, NICKNAME, MAIDEN}, and this predicate reads the same - # answer wherever in the stage it stands (measured 2026-09-06 by - # reading the stage's retag targets). - # - # RECORDED as a tag rather than recomputed by the render, because - # the render cannot see a span: _facade.__setstate__ and - # ParsedName.replace() build span-less tokens AFTER the pipeline, - # so an unpickled name has nothing to scan and the tag IS the - # entry structure the pickle carries - # (mechanisms.md#MARK-DONT-STRIP). Every token here has a span -- - # tokenize is the sole producer of a WorkToken, and WorkToken.span - # is not Optional -- so `span.start` is read unguarded. - # - # The `i not in dropped` filter is belt-and-braces: a dropped token - # never carries a SUFFIX role. Cores leave `pieces` before assign - # runs, so assign gives them no role at all; the one class of - # dropped token that arrives already roled is the MAIDEN one that - # tokenize roles from an extracted clause ('Smith, MD (nee Jones) - # PhD' drops index 2, the marker, and it is Role.MAIDEN). The - # filter is here so that `suffixes` and the `parted` scan below - # cannot disagree about what a dropped index is. - dropped = set(state.dropped) - cores = delimiter_cores(state.policy.extra_suffix_delimiters) - suffixes = [i for i, tok in enumerate(tokens) - if tok.role is Role.SUFFIX and i not in dropped] - for previous, current in zip(suffixes, suffixes[1:]): - same_part = (comma_bucket(tokens[previous].span.start, - state.comma_offsets) - == comma_bucket(tokens[current].span.start, - state.comma_offsets)) - parted = any( - tokens[between].text in cores if between in dropped - else tokens[between].role not in _RENDERS_ELSEWHERE - for between in range(previous + 1, current)) - if same_part and not parted: - tokens[current] = dataclasses.replace( - tokens[current], tags=tokens[current].tags | {"joined"}) + _mark_suffix_entries(tokens, state) return dataclasses.replace(state, tokens=tuple(tokens), ambiguities=tuple(ambiguities)) diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 7ca7b164..2ffc3980 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -5,6 +5,7 @@ from nameparser._lexicon import Lexicon from nameparser._pipeline import run +from nameparser._pipeline._post_rules import suffix_entries from nameparser._pipeline._state import ParseState from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, PatronymicRule, Policy, @@ -954,3 +955,65 @@ def test_the_pass_runs_after_roles_are_settled() -> None: middles = [tok for tok in state.tokens if tok.role is Role.MIDDLE] assert [tok.text for tok in middles] == ["A", "B"] assert not any("joined" in tok.tags for tok in middles) + + +# -- suffix_entries as a function of its own (#511) ------------------- +# +# Parser.revise forces the named role on a sub-parse's tokens and then +# runs THIS pass over the forced state, so the pass is exercised here +# on states whose roles were set by hand rather than by assign. + + +def _forced(text: str, role: Role, + policy: Policy | None = None) -> ParseState: + """A full parse of `text` with every non-dropped token re-roled to + `role`, which is what revise() hands the pass. The forcing keeps + the sub-parse's tags, as revise() keeps them. Dropped tokens keep + their role, as revise() leaves them.""" + state = run(ParseState(original=text, lexicon=Lexicon.default(), + policy=policy or Policy())) + dropped = set(state.dropped) + return dataclasses.replace(state, tokens=tuple( + tok if i in dropped + else dataclasses.replace(tok, role=role) + for i, tok in enumerate(state.tokens))) + + +def _forced_entry_tags(text: str, role: Role, + policy: Policy | None = None) -> list[tuple[str, bool]]: + state = suffix_entries(_forced(text, role, policy)) + return [(tok.text, "joined" in tok.tags) + for i, tok in enumerate(state.tokens) + if tok.role is role and i not in set(state.dropped)] + + +def test_suffix_entries_joins_a_forced_role_run() -> None: + """A bare 'MD PhD' parses as a title and a family name, so the + pass inside that parse marks nothing; over the FORCED state it + reads two suffix tokens in one comma bucket and joins them.""" + assert _forced_entry_tags("MD PhD", Role.SUFFIX) == [ + ("MD", False), ("PhD", True)] + + +def test_suffix_entries_reads_the_values_own_comma() -> None: + """The pass reads the comma offsets of the STATE it is handed -- + for revise() that is the value's own string, not the outer name's + -- so a comma in the value parts the entries.""" + assert _forced_entry_tags("MD, PhD", Role.SUFFIX) == [ + ("MD", False), ("PhD", False)] + + +def test_suffix_entries_heals_a_spaced_credential_by_the_comma_rule() -> None: + """The Ph. D. MERGE is a head-position rule (#371) and does not + fire in a bare value; the ENTRY pass joins the pair anyway, for the + reason it joins MD and PhD -- same bucket, nothing between.""" + assert _forced_entry_tags("Ph. D.", Role.SUFFIX) == [ + ("Ph.", False), ("D.", True)] + + +def test_suffix_entries_keys_on_the_suffix_role() -> None: + """A forced MIDDLE value has no suffix token, so the pass writes + nothing there: revise(middle='A B') stays two middle tokens, + neither a continuation.""" + assert _forced_entry_tags("A B", Role.MIDDLE) == [ + ("A", False), ("B", False)] diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index f5e4d332..30fd64e1 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -829,19 +829,26 @@ def test_revise_preserves_particle_tags() -> None: assert r.initials() == "J. V. S." # particles contribute no initial -def test_revise_takes_a_spaced_credential_literally() -> None: - # The Ph./D. merge is a HEAD-POSITION rule (#371) and a field value - # has no head: revise() runs a full sub-parse of the string it is - # given, so "Ph. D." there is two separate suffix pieces -- `Ph.` - # by vocabulary, `D.` as an initial -- which the suffix view - # renders comma-joined, rather than one healed credential. - # Deliberate -- the merge exists for a credential someone TYPED - # after a name, and a caller who writes the spaced form into the - # suffix field gets it read as two suffix items. +def test_revise_derives_suffix_entries_from_the_values_commas() -> None: + # A suffix value's entry structure is the value's own commas, read + # by rules.md#R1's pass after the named role is forced (#511). The + # sub-parse of a bare "MD PhD" reads a title and a family name and + # so joins nothing on its own; revise() runs the pass again over + # the forced roles, where the two share a comma bucket. p = Parser() - n = p.parse("John Smith Ph.D.") - r = p.revise(n, suffix="Ph. D.") - assert r.suffix == "Ph., D." + n = p.parse("John Smith") + assert p.revise(n, suffix="MD PhD").suffix == "MD PhD" + assert p.revise(n, suffix="MD, PhD").suffix == "MD, PhD" + # The Ph. D. MERGE is a head-position rule (#371) with no head to + # consult in a field value, and it still does not fire here; the + # ENTRY pass joins the pair for the reason it joins MD and PhD. + # Supersedes the 2026-08-31 acceptance of 'Ph., D.' on this path + # (decisions.md#phd-merge). + assert p.revise(n, suffix="Ph. D.").suffix == "Ph. D." + # a dash is a token under the default policy, and a suffix word + # once the role is forced + assert (p.revise(n, suffix="MD PhD - FACS Fellow").suffix + == "MD PhD - FACS Fellow") def test_revise_views_match_a_fresh_parse() -> None: @@ -960,6 +967,114 @@ def test_revise_forces_the_named_role_on_every_harvested_token() -> None: assert all(t.role is Role.FAMILY for t in r.tokens_for(Role.FAMILY)) +def test_revise_round_trips_a_rendered_suffix() -> None: + # the view renders entries comma-joined and a comma is what the + # pass reads, so a name's own suffix revises back to itself -- + # the three shapes #511 reported, none of which round-tripped, + # and a fourth from the corpus -- the only value in the set whose + # own suffix CONTAINS A COMMA ('Jr., Col. Ret'), so it is the one + # pinning that the pass does not over-join two entries into one + p = Parser() + for text in ("John Smith MD PhD", "John Smith, Ph. D.", + "Doe, John, MD PhD - FACS Fellow", + "Andrew Perkins, Jr., Col. (Ret)"): + n = p.parse(text) + assert p.revise(n, suffix=n.suffix).suffix == n.suffix, text + + +def test_revise_leaves_a_policy_delimiter_unparted_without_a_tail_segment() -> None: + # The pinned limit, and the rule behind it: a configured delimiter + # core is dropped -- and so parts entries -- only on a TAIL + # segment, which a value gets only where its own words read as a + # name. A run of post-nominals has none, with or without a comma + # of its own, so the dash stays a word; a value that reads as a + # name does part. The round-trip is unaffected because the + # whole-name view already rendered that boundary as a comma. + p = Parser(policy=Policy(extra_suffix_delimiters=frozenset({" - "}))) + n = p.parse("Doe, John, MD PhD - FACS") + assert n.suffix == "MD PhD, FACS" + assert p.revise(n, suffix="MD PhD - FACS").suffix == "MD PhD - FACS" + assert p.revise(n, suffix="MD, PhD - FACS").suffix == "MD, PhD - FACS" + assert p.revise(n, suffix=n.suffix).suffix == "MD PhD, FACS" + assert (p.revise(n, suffix="John Doe, MD - FACS").suffix + == "John Doe, MD, FACS") + + +def test_revise_reads_a_glued_honorific_on_its_own() -> None: + # The docstring's "classified ON ITS OWN" limit, on the one corpus + # name of 1117 whose suffix does not revise back to itself after + # #511 (measured 2026-09-06). The whole-name parse keeps 'J.씨' one + # glued suffix token; the sub-parse of the bare value peels the + # honorific off the initial, and the entry pass then joins the + # two it made. Right entries, the spurious comma of before gone + # ('씨, J., 씨'), one word read differently in the value than in + # the name. Honorific peeling is a W-rule question, not R1's. + p = Parser() + n = p.parse("김민준씨, J.씨") + assert n.suffix == "씨, J.씨" + assert p.revise(n, suffix=n.suffix).suffix == "씨, J. 씨" + + +#: The one suffix-bearing corpus name whose suffix does not revise back +#: to itself: the honorific peel pinned just above. Named here so the +#: guard below fails on a NEW exception and not on the known one. +_HONORIFIC_PEEL = frozenset({"김민준씨, J.씨"}) + + +def _suffix_bearing_corpus_names() -> list[str]: + from ._differential_fixtures import _CORPUS_NAMES + p = Parser() + return [n for n in _CORPUS_NAMES + if p.parse(n).suffix and n not in _HONORIFIC_PEEL] + + +def test_the_suffix_bearing_corpus_is_not_empty() -> None: + # 367 on 2026-09-06; a floor, so a filter that empties the + # parametrization below fails here instead of skipping there + assert len(_suffix_bearing_corpus_names()) > 300 + + +@pytest.mark.parametrize("name", _suffix_bearing_corpus_names()) +def test_a_rendered_suffix_revises_back_to_itself(name: str) -> None: + # The #511 property over the corpus rather than by example, because + # revise() is off the differential gate's compare path and a change + # to group's marks, delimiter_cores or the entry predicate could + # reopen it with the gate green and the four pins above passing. + p = Parser() + n = p.parse(name) + assert p.revise(n, suffix=n.suffix).suffix == n.suffix + + +def test_revise_keeps_the_sub_parses_within_piece_mark() -> None: + # The forcing keeps the sub-parse's tags: a "joined" made WITHIN a + # piece (the Ph. D. merge, mid-value where the head rule lets it + # fire) is role-blind and right for every role (decisions.md#C1's + # #436 DECLINED list), and a draft that cleared it with the role + # lost it on every non-suffix value. For a suffix value the pass + # writes the marks itself ('Ph. D. MD' sub-parses with none), so + # nothing there depends on the sub-parse's own. The between-piece + # mark a sub-parse leaves on a token forced to another role + # predates #511; on a ParsedName only the suffix string view reads + # the tag, and no path puts a revised name into a HumanName, whose + # list views would heal it. + p = Parser() + r = p.revise(p.parse("John Smith"), family="John Ph. D. Smith") + family = r.tokens_for(Role.FAMILY) + assert [t.text for t in family] == ["John", "Ph.", "D.", "Smith"] + assert "joined" in family[2].tags + assert r.family == "John Ph. D. Smith" + s = p.revise(p.parse("John Smith"), suffix="Ph. D. MD") + assert s.suffix == "Ph. D. MD" + assert [("joined" in t.tags) for t in s.tokens_for(Role.SUFFIX)] == [ + False, True, True] + # the stale between-piece mark, and that no ParsedName view reads it + stale = p.revise(p.parse("John Smith"), family="Jones MD PhD") + fam = stale.tokens_for(Role.FAMILY) + assert "joined" in fam[2].tags + assert stale.family == "Jones MD PhD" + assert stale.family_base == stale.family + + def test_wholly_cjk_names_read_family_first_by_default() -> None: # the 2026-07-27 amendment: script determines the convention, so # no pack is needed -- release-log-classified fix (#271)