Verify that surrogate pairs in LIFT files don't break the tooling

[imnasnainaec]

Related: Do we need UTF-16 support?

[imnasnainaec]

I tested this empirically against current main rather than just reading code.

Non-BMP characters — what "surrogate pairs" mean in UTF-16 terms (emoji, CJK Extension B, etc.) — already round-trip correctly. Loading and saving a LIFT entry containing 🎉 (U+1F389) and 𠀀 (U+20000) produces byte-identical output, and reloading it matches semantically. Python strings are codepoint sequences, not UTF-16 code units, so there's no actual pairing concern in this codebase. The byte-span scanner (src/sil_lift/_scan.py) only looks for ASCII structural bytes (<, >, quotes), which never collide with UTF-8 continuation bytes, so 4-byte UTF-8 sequences pass through untouched.

Malformed/invalid surrogate encodings are rejected cleanly, not silently corrupted. Tried two known bug patterns from tools that mishandle UTF-16 internally:

• A numeric character reference to a lone surrogate (�) — invalid XML.
• Raw CESU-8/WTF-8-style bytes (each UTF-16 surrogate half individually UTF-8-encoded instead of paired first) — invalid UTF-8.

Both raise lxml.etree.XMLSyntaxError at parse time, which sil-lift wraps into LiftParseError. No hang, no silent data loss.

UTF-16-encoded LIFT files are already an explicitly documented case. docs/en/fidelity.md lists a non-ASCII-compatible source encoding as one of the exceptions to the byte-identity guarantee: such files load fine (libxml2 handles the encoding transparently) but fall back to canonical UTF-8 re-serialization instead of byte-for-byte passthrough. Covered by tests/test_reader.py's test_untrustworthy_bytes_load_but_save_canonically (id "utf-16").

Gap: none of the above is pinned down by a deterministic test. The existing UTF-16 fixtures don't contain surrogate-pair/non-BMP content, and the Hypothesis property fuzzer (tests/test_property_roundtrip.py) excludes lone surrogates but doesn't specifically target non-BMP codepoints — so today's correct behavior isn't locked in by the suite.

Bottom line: current tooling already handles surrogate pairs and UTF-16 correctly; the open work, if any, is adding regression coverage — not fixing a defect.

---

Investigated by Claude (Sonnet 5, model ID claude-sonnet-5) via Claude Code.