Skip to content

Compare range-element parents under NFC - #28

Draft
imnasnainaec wants to merge 9 commits into
mainfrom
range-parent-nfc
Draft

Compare range-element parents under NFC#28
imnasnainaec wants to merge 9 commits into
mainfrom
range-parent-nfc

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Devin review: https://app.devin.ai/review/sillsdev/python-sil-lift/pull/28

Fixes #14. Fixes #27.

The range-parent check compared a range-element's parent against its siblings'
ids as exact strings. FLEx writes those two attributes with different Unicode
normalization — the grammatical-info and lexical-relation range-element ids
bypass its normalizing helper and come out NFD, while the parent attribute on the
same element goes through it and comes out NFC — so a sound parent link in a real
export was reported as an error. #14 reports this from a Sena export, #27 from
Sango; they are the same defect, and #27 adds the exporter-side mechanism.

What changes

One policy for resolving a name to an id. Every name that reaches a range or a
range-element now resolves the same way: by its exact spelling first, then under NFC.
That covers parent links, range values, and — new here — the name that keys a range
at all, a trait name or a header range id. Only undefined-range-value normalized
before, which is why one encoding split was an error in one check and invisible in
another. A range whose own id FLEx wrote in NFD was worse than invisible: the NFC trait
names keyed on it never found it, so every value in it went unchecked, an unresolved
name being indistinguishable from a trait no range defines.

A resolution that needed normalizing is reported, not swallowed. Suppressing the
false error alone would hide a real property of the data: sil-lift never rewrites the
ids (fidelity), so a consumer comparing raw strings still fails to resolve those
references, and nothing else would tell the user. New normalization-mismatch
warning — once per id however many references differ, for range ids and
range-element ids alike, naming both spellings by code point ('Comple\u0301ments' vs
'Compl\u00e9ments') because they render identically. range-parent stays an error
for a parent no sibling defines in any normalization.

Findings about a range are addressed to the file that defines it. Both the mismatch
warning and range-parent now report against the companion holding the element, rather
than one of them naming the .lift that merely references it. Header range ids are
resolved before the dangling-href check rather than inside it, so whether a header id
and a companion's own id agree is answered for a lexicon held in memory too, not only
one read from disk.

The prose said something the corpus doesn't support. Three sites and the changelog
described the .lift as NFC and the .lift-ranges as NFD. The normalizer is applied
per write, not per file: in sango.lift-ranges every label, abbrev, description and
every parent is NFC, and so are the semantic-domain-ddp4, translation-type, and
usage-type ids. Rewritten to name the mechanism, with the new warning documented in
the module docstring's list of semantic checks, the FLEx-output policy section, and the
exporter guide.

Coverage. A hand-authored negative/nfd-range-ids.lift + .lift-ranges pair
carries the shape end to end — NFD ids, NFC parents and grammatical-info value, one
parent that dangles in every normalization — asserting one error, one deduplicated
warning, and byte-identical save of both files, since normalization belongs to the
comparison and not to what gets written. In-memory regressions cover the range-keying
name, the header id, and a range holding one name in two normalizations, where each
reference must resolve to the sibling it matches exactly. PROVENANCE.md records what
is NFD and what is NFC in the Sango fixture.

Each range's id lookup is built once and read by the value checks, which rebuilt it
per trait before: Sango's semantic pass drops from 1.54s to 0.84s over the same
findings.

Effect on the corpus

before after
sango.lift range-parent 2 errors none
sango.lift normalization-mismatch 6 warnings — 82 references reaching 5 ids, one of them under both aliases of the POS list
sango.lift undefined-range-value 1 (prenom) 1 (prenom)
AllFLExFields, all other fixtures no new findings

Deliberately not changed

The dangling-ref sweep #14 asks for came up empty: all 7 findings in the corpus
(negative/dangling-ref.lift, spec-examples/0.13/{fields any order,hugal-ideal,
lexical-relations}.lift) are ASCII and none resolve under NFC, so entry ids and
relation/@ref keep comparing exactly. duplicate-guid compares guids, ASCII by
construction. Neither gets a normalization pass without a case to justify it.

Verification

ruff check, ruff format --check, mypy, and the full suite (543 passed).

Recommendations for the upstream FieldWorks exporter — including a separate, more
serious escaping gap on the morph-type id — are in a comment on #27, and are now
carried by sillsdev/FieldWorks#1063.


This change is Reviewable

imnasnainaec and others added 8 commits August 14, 2026 14:41
FLEx normalizes strings to NFC on export, but a few writes emit the NFD
it holds in memory -- among them the grammatical-info and lexical-relation
range-element ids, while the parent attribute on those same elements is
normalized. Comparing the two as exact strings reported a sound parent
link as dangling: both range-parent errors in the sango fixture are that
encoding difference, not defects.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validation notes described the .lift as NFC and the .lift-ranges as
NFD. It is narrower than that: the normalizer is applied per write, so a
range-element id can be NFD beside a label, a parent, and a .lift value
that are all NFC.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normalizing both sides of the range comparisons hid a real property of
FLEx output: an id and the names referring to it can differ in encoding,
and any consumer comparing raw strings -- a Send/Receive merge, say --
fails to resolve them. Report that as a warning, once per range-element
id however many references differ (6 rather than 82 in the sango
fixture), naming both spellings by code point because they render
identically, and addressing it to the file the id lives in.

The new negative pair carries the shape end to end: NFD ids, NFC parents
and grammatical-info value, one parent that dangles in every
normalization, and bytes that must survive the round trip unchanged --
normalization belongs to the comparison, not to what gets written.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keying each range's lookup by both the ids as written and their NFC forms,
with an exact id mapping to itself, makes a reference that matches a
sibling exactly stay an exact match. Before, the first spelling of an
NFC-equal pair won, so a range carrying one name in both normalizations --
distinct ids as strings, and FLEx normalizes some writes and not others --
would resolve the second element's references to the first and report them
as mismatched.

The lookup is now built once per range and read by the value checks, which
were rebuilding it per trait: thousands of traits over a few ranges of
hundreds of elements each. The sango fixture's semantic pass drops from
1.54s to 0.84s.

The module docstring's semantic checks fed the API reference as one
sentence running nine checks and their parentheticals together. They are a
bullet each now, named by the Problem code the check reports, matching how
layer 1 lays out its deviations; the comments around the normalization
helpers drop the mechanism they were re-telling from that list.

Also escapes the list markers in the new PROVENANCE sentence, per that
file's convention for hard-wrapped prose, and drops a Markdown escape that
had no business in an XML comment.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file the mismatch warning reports against is worked out for that
warning alone, three lines under a range-parent error about the very same
range-element that still reports against the .lift. Both belong to the
document the element lives in, so the lookup moves above the check that
walks the ranges and answers for both.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comparison of a name against a range-element id was normalized while
the comparison that reaches the range in the first place -- a trait name,
a header range id -- stayed exact. A range whose id FLEx wrote in NFD is
therefore never found by the NFC trait names keyed on it, and its values
go unchecked: an unresolved name is indistinguishable from a trait no
range defines, which is silently accepted, so the whole range drops out
of validation with nothing to show for it.

Both lookups are now built by one helper over a collection of ids, and a
range reached only by normalizing is a normalization-mismatch warning
like an element is. The value checks name their range through that
resolution, which also drops the allowed-ids set from the per-check
tuples: they carry the resolved range id, and the lookup kept for it
answers by that key.

The validate guide's normalization entry, five sentences beside four
one-liners once this landed, keeps the mechanism on the bullet and gives
the rest a sub-bullet each. It moves last as well, so the list runs
schema deviations first and the semantic check after them, as the layers
above are ordered.

The Sango notes counted its 6 warnings as 6 ids. They are 5: FLEx aliases
the part-of-speech list under two range ids, and the id under both is
warned about under each, since each range defines its own element.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolving the header's range id against the companion's own id was done
inside the check for a dangling href, which runs only for a lexicon read
from disk. Whether those two spellings agree is a property of the
document, so a lexicon held in memory reported nothing where the same
document on disk warned -- and validation is supposed to answer for what
save() would write, wherever the document came from.

Every header range id resolves before that check now, and the check reads
the answer. Ranges that carry their own elements resolve to themselves and
record nothing, so covering them all costs a lookup each.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The existing coverage for reporting a normalization mismatch once per id
uses references that are all spelled the same way, so it pins only the
collapse of identical references. Add a case where two references differ
from the id and from each other -- a parent link in canonical mark order
and a grammatical-info value with the same marks reversed -- and assert
one warning. Which spelling the message names is deliberately left
unasserted, so the order references are reached in stays free.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant