range-parent false positives: FLEx writes range-element ids in NFD but their parent attributes in NFC

[imnasnainaec]

Summary

The semantic range-parent check compares a range-element's parent against its
siblings' ids as exact strings. FieldWorks (FLEx) writes those two attributes with
different Unicode normalization, so a perfectly well-formed parent link in a real
FLEx export is reported as an error.

Evidence (corpus)

tests/corpus/large/sango/sango.lift-ranges — both hits are the same element, aliased
under FLEx's two names for its part-of-speech list:

range	child @id	@parent	matches after NFC
grammatical-info	Complément du lieu (NFD)	Compléments (NFC)	yes
from-part-of-speech	Complément du lieu (NFD)	Compléments (NFC)	yes

The sibling those parents refer to exists; its id is Compléments (NFD).
tests/test_validate.py currently locks these in as "two genuinely dangling
range-element parents" — they are not dangling.

Root cause

FLEx holds strings as NFD in memory and normalizes to NFC on export, but a handful of
writes bypass the normalizing helpers (MakeSafeAndNormalizedXml /
MakeSafeAndNormalizedAttribute in Src/LexText/LexTextControls/LiftExporter.cs) and
emit raw NFD. The grammatical-info range-element id is one of them
(LiftExporter.cs:1738) while the parent written two lines later
(LiftExporter.cs:1744) goes through the normalizer — hence the asymmetry within a
single element. Same for lexical-relation (:1775/:1781).

Other known bypass sites, for completeness: morph-type range-element id (:2173,
which also skips XML escaping), leading-symbol / trailing-symbol traits
(:2180/:2185), stem-name feature-set trait (:2294), reversal-type id
(:1954).

Our docs state this imprecisely

Three places say FLEx writes the .lift in NFC and the .lift-ranges in NFD:

• docs/en/csharp-differences.md (Validation)
• docs/en/guides/validate.md (Real-world FieldWorks output)
• the comment above the undefined-range-value check in src/sil_lift/_validate.py

That is not what the corpus shows. In sango.lift-ranges every label, abbrev,
description, and every @parent is NFC; only certain range-element @ids are NFD
(grammatical-info, from-part-of-speech, lexical-relation — while
semantic-domain-ddp4, translation-type, and usage-type ids are NFC). It is
specific writes that bypass the normalizer, not the file.

Fix

• NFC-normalize the range-parent comparison, the way the undefined-range-value
  check already normalizes range values (report the original spelling in the message).
• Correct the three prose/comment sites to describe the mechanism.
• Update the sango expectation and add a regression test for an NFD id with an NFC
  parent; note the asymmetry in tests/corpus/PROVENANCE.md next to the existing
  duplicate-guid finding.

[imnasnainaec]

Recommendations for the FieldWorks exporter

Line numbers checked against main at aec7a69c2 in Src/LexText/LexTextControls/LiftExporter.cs — pointers, not anchors.

1. Route the bypassed writes through the normalizing helper.
MakeSafeAndNormalizedXml / MakeSafeAndNormalizedAttribute (:607-621) exist exactly
for this and are used for everything else, so these are omissions rather than deliberate
choices. Affected: grammatical-info range-element id (:1738), lexical-relation
range-element id (:1775), morph-type range-element id (:2173),
leading-symbol / trailing-symbol traits (:2180, :2185), stem-name feature-set
trait (:2294), reversal-type id (:1954). The writing-system trait (:2122) is
ASCII in practice — same treatment for consistency, no urgency.

2. morph-type's id (:2173) is the more serious one: it is not XML-escaped either.
Normalization only makes ids compare unequal; missing escaping makes the file
unparseable. A user-renamed morph type containing &, <, or " — the standard list
editor accepts any of them — yields a .lift-ranges no conforming XML parser will read.
Worth fixing on its own merits, ahead of the normalization question.

3. The symptom is cross-file, not just intra-file. The .lift writes
<grammatical-info value="Idéophone"/> in NFC (it goes through the helper) while the
companion .lift-ranges declares that same part of speech with id="Idéophone" in NFD.
Any consumer that resolves a value against its range by string equality has to normalize
first or conclude the value is undefined. Confirmed in a real 3507-entry export
(SIL.FLEx 9.1.15.658), and it is the reason this library normalizes both sides.

4. The import side needs no change for this. AddToPossibilityMap keys the merger's
possibility maps with tss.Text.Normalize() — NFC — so normalized ids are what the
importer already expects, and nothing inside FieldWorks is keyed on the old NFD
spelling. (Its raw-string insert and lookup sites are worth tightening anyway, so that
LIFT written in NFD by some other exporter cannot create duplicate list items, but that
is an independent defect rather than a prerequisite.)

5. Regression coverage. Two exporter tests: a list item whose name carries a
combining diacritic, asserting id, parent, and label agree; and a morph type whose
name contains &, asserting the file parses. Neither existed before.

---

Jira: https://jira.sil.org/browse/LT-22697
PR: sillsdev/FieldWorks#1063