Compare range-element parents under NFC - #28
Draft
imnasnainaec wants to merge 9 commits into
Draft
Conversation
imnasnainaec
force-pushed
the
range-parent-nfc
branch
from
August 13, 2026 13:15
b8ab20a to
2860d04
Compare
FLEx normalizes strings to NFC on export, but a few writes emit the NFD it holds in memory -- among them the grammatical-info and lexical-relation range-element ids, while the parent attribute on those same elements is normalized. Comparing the two as exact strings reported a sound parent link as dangling: both range-parent errors in the sango fixture are that encoding difference, not defects. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validation notes described the .lift as NFC and the .lift-ranges as NFD. It is narrower than that: the normalizer is applied per write, so a range-element id can be NFD beside a label, a parent, and a .lift value that are all NFC. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normalizing both sides of the range comparisons hid a real property of FLEx output: an id and the names referring to it can differ in encoding, and any consumer comparing raw strings -- a Send/Receive merge, say -- fails to resolve them. Report that as a warning, once per range-element id however many references differ (6 rather than 82 in the sango fixture), naming both spellings by code point because they render identically, and addressing it to the file the id lives in. The new negative pair carries the shape end to end: NFD ids, NFC parents and grammatical-info value, one parent that dangles in every normalization, and bytes that must survive the round trip unchanged -- normalization belongs to the comparison, not to what gets written. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keying each range's lookup by both the ids as written and their NFC forms, with an exact id mapping to itself, makes a reference that matches a sibling exactly stay an exact match. Before, the first spelling of an NFC-equal pair won, so a range carrying one name in both normalizations -- distinct ids as strings, and FLEx normalizes some writes and not others -- would resolve the second element's references to the first and report them as mismatched. The lookup is now built once per range and read by the value checks, which were rebuilding it per trait: thousands of traits over a few ranges of hundreds of elements each. The sango fixture's semantic pass drops from 1.54s to 0.84s. The module docstring's semantic checks fed the API reference as one sentence running nine checks and their parentheticals together. They are a bullet each now, named by the Problem code the check reports, matching how layer 1 lays out its deviations; the comments around the normalization helpers drop the mechanism they were re-telling from that list. Also escapes the list markers in the new PROVENANCE sentence, per that file's convention for hard-wrapped prose, and drops a Markdown escape that had no business in an XML comment. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file the mismatch warning reports against is worked out for that warning alone, three lines under a range-parent error about the very same range-element that still reports against the .lift. Both belong to the document the element lives in, so the lookup moves above the check that walks the ranges and answers for both. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comparison of a name against a range-element id was normalized while the comparison that reaches the range in the first place -- a trait name, a header range id -- stayed exact. A range whose id FLEx wrote in NFD is therefore never found by the NFC trait names keyed on it, and its values go unchecked: an unresolved name is indistinguishable from a trait no range defines, which is silently accepted, so the whole range drops out of validation with nothing to show for it. Both lookups are now built by one helper over a collection of ids, and a range reached only by normalizing is a normalization-mismatch warning like an element is. The value checks name their range through that resolution, which also drops the allowed-ids set from the per-check tuples: they carry the resolved range id, and the lookup kept for it answers by that key. The validate guide's normalization entry, five sentences beside four one-liners once this landed, keeps the mechanism on the bullet and gives the rest a sub-bullet each. It moves last as well, so the list runs schema deviations first and the semantic check after them, as the layers above are ordered. The Sango notes counted its 6 warnings as 6 ids. They are 5: FLEx aliases the part-of-speech list under two range ids, and the id under both is warned about under each, since each range defines its own element. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolving the header's range id against the companion's own id was done inside the check for a dangling href, which runs only for a lexicon read from disk. Whether those two spellings agree is a property of the document, so a lexicon held in memory reported nothing where the same document on disk warned -- and validation is supposed to answer for what save() would write, wherever the document came from. Every header range id resolves before that check now, and the check reads the answer. Ranges that carry their own elements resolve to themselves and record nothing, so covering them all costs a lookup each. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
imnasnainaec
force-pushed
the
range-parent-nfc
branch
from
August 14, 2026 18:41
0685bf1 to
94ec467
Compare
The existing coverage for reporting a normalization mismatch once per id uses references that are all spelled the same way, so it pins only the collapse of identical references. Add a case where two references differ from the id and from each other -- a parent link in canonical mark order and a grammatical-info value with the same marks reversed -- and assert one warning. Which spelling the message names is deliberately left unasserted, so the order references are reached in stays free. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Devin review: https://app.devin.ai/review/sillsdev/python-sil-lift/pull/28
Fixes #14. Fixes #27.
The
range-parentcheck compared arange-element'sparentagainst its siblings'ids as exact strings. FLEx writes those two attributes with different Unicodenormalization — the
grammatical-infoandlexical-relationrange-elementidsbypass its normalizing helper and come out NFD, while the
parentattribute on thesame element goes through it and comes out NFC — so a sound parent link in a real
export was reported as an error. #14 reports this from a Sena export, #27 from
Sango; they are the same defect, and #27 adds the exporter-side mechanism.
What changes
One policy for resolving a name to an id. Every name that reaches a range or a
range-element now resolves the same way: by its exact spelling first, then under NFC.
That covers
parentlinks, range values, and — new here — the name that keys a rangeat all, a
traitname or a headerrangeid. Onlyundefined-range-valuenormalizedbefore, which is why one encoding split was an error in one check and invisible in
another. A range whose own id FLEx wrote in NFD was worse than invisible: the NFC trait
names keyed on it never found it, so every value in it went unchecked, an unresolved
name being indistinguishable from a trait no range defines.
A resolution that needed normalizing is reported, not swallowed. Suppressing the
false error alone would hide a real property of the data: sil-lift never rewrites the
ids (fidelity), so a consumer comparing raw strings still fails to resolve those
references, and nothing else would tell the user. New
normalization-mismatchwarning — once per id however many references differ, for range ids and
range-element ids alike, naming both spellings by code point (
'Comple\u0301ments'vs'Compl\u00e9ments') because they render identically.range-parentstays an errorfor a parent no sibling defines in any normalization.
Findings about a range are addressed to the file that defines it. Both the mismatch
warning and
range-parentnow report against the companion holding the element, ratherthan one of them naming the
.liftthat merely references it. Header range ids areresolved before the dangling-href check rather than inside it, so whether a header id
and a companion's own id agree is answered for a lexicon held in memory too, not only
one read from disk.
The prose said something the corpus doesn't support. Three sites and the changelog
described the
.liftas NFC and the.lift-rangesas NFD. The normalizer is appliedper write, not per file: in
sango.lift-rangesevery label, abbrev, description andevery
parentis NFC, and so are thesemantic-domain-ddp4,translation-type, andusage-typeids. Rewritten to name the mechanism, with the new warning documented inthe module docstring's list of semantic checks, the FLEx-output policy section, and the
exporter guide.
Coverage. A hand-authored
negative/nfd-range-ids.lift+.lift-rangespaircarries the shape end to end — NFD ids, NFC
parents andgrammatical-infovalue, oneparent that dangles in every normalization — asserting one error, one deduplicated
warning, and byte-identical save of both files, since normalization belongs to the
comparison and not to what gets written. In-memory regressions cover the range-keying
name, the header id, and a range holding one name in two normalizations, where each
reference must resolve to the sibling it matches exactly.
PROVENANCE.mdrecords whatis NFD and what is NFC in the Sango fixture.
Each range's id lookup is built once and read by the value checks, which rebuilt it
per trait before: Sango's semantic pass drops from 1.54s to 0.84s over the same
findings.
Effect on the corpus
sango.liftrange-parentsango.liftnormalization-mismatchsango.liftundefined-range-valueprenom)prenom)AllFLExFields, all other fixturesDeliberately not changed
The
dangling-refsweep #14 asks for came up empty: all 7 findings in the corpus(
negative/dangling-ref.lift,spec-examples/0.13/{fields any order,hugal-ideal,lexical-relations}.lift) are ASCII and none resolve under NFC, so entry ids andrelation/@refkeep comparing exactly.duplicate-guidcompares guids, ASCII byconstruction. Neither gets a normalization pass without a case to justify it.
Verification
ruff check,ruff format --check,mypy, and the full suite (543 passed).Recommendations for the upstream FieldWorks exporter — including a separate, more
serious escaping gap on the
morph-typeid — are in a comment on #27, and are nowcarried by sillsdev/FieldWorks#1063.
This change is