Skip to content

LT-22674: fix quadratic NFC offset-map build that made opening decomposed texts take 10+ seconds - #1056

Open
johnml1135 wants to merge 5 commits into
mainfrom
lt-22674-nfc-offset-map-quadratic
Open

LT-22674: fix quadratic NFC offset-map build that made opening decomposed texts take 10+ seconds#1056
johnml1135 wants to merge 5 commits into
mainfrom
lt-22674-nfc-offset-map-quadratic

Conversation

@johnml1135

@johnml1135 johnml1135 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Fixes LT-22674.

Opening a text in Texts & Words took 10+ seconds when the baseline was a single long paragraph containing more than one writing system. Reproduced on Maasai-Parser with camus4-D (1 paragraph, 236 segments, 7,571 characters, two writing systems on two different font families).

Cause

BuildNfcOffsetMaps in Src/views/lib/UniscribeSegment.cpp normalised every prefix of a range from scratch:

for (int ich = 1; ich <= cchOrig; ++ich)
{
    StrUni stuPrefix(stuOrig.Chars(), ich);
    StrUtil::NormalizeStrUni(stuPrefix, UNORM_NFC);
    vichOrigToNfc[ich] = stuPrefix.Length();
}

Quadratic in the range length, and called once per analysed range, so the aggregate cost grows with the square of the paragraph length. For a 7,571-character paragraph that is tens of millions of characters normalised per layout pass.

The maps exist to translate between original and normalised offsets — Uniscribe shapes the normalised text while selections, cursor offsets and hit-testing live in the original text. They are only needed when normalisation actually changes the string. It fires when the text is not already NFC and the layout-pass cache is present (UniscribeSegment.cpp:3157 before this change). Text reaching the Views layer is typically NFD, so for vernacular data that was the normal path, not an edge case.

This is a regression. BuildNfcOffsetMaps, vichOrigToNfc and LayoutPassCache all arrived together in 502b832 ("perf: Views engine render optimizations", #724, 2026-05-08). Before it there was no eager construction: translation was on demand via OffsetInNfc/OffsetToOrig, whose comment notes it "is not called with high frequency".

Fix

Text that normalisation leaves unchanged needs no maps at all, because the mapping is the identity. So the analysis cache now stores only that text, and anything normalisation would rewrite falls back to the pre-existing on-demand translation. The eager map machinery is removed rather than rewritten.

The fix is net -21 lines. ShapeRunCache — the genuine ~3x improvement from the same commit — is untouched.

An earlier attempt kept caching non-NFC text and made BuildNfcOffsetMaps linear by splitting at ICU normalisation boundaries. It was measured and rejected: the decomposed proxy still took 3,468 / 2,174 ms against a 298 / 269 ms control, because a linear build still makes roughly one ICU call per character and is still invoked once per range. Fixing the constant factor could not fix the aggregate.

Numbers

Cold render, milliseconds, 2 replicates. "cache-OFF control" is the same binary with FW_PERF_P125_PATH2=0, which is the configuration confirmed to resolve the problem on the real project.

scenario before after cache-OFF control
single-para-mixed-ws (decomposed, camus-shaped) 3,468 / 2,174 325 / 334 277 / 289
multi-line-wrap-single-ws (decomposed) 832 / 835 168 / 409 175 / 180
multi-ws 487 / 458 / 452 148 / 167 143 / 300
rtl-script 116 / 121 / 122 78 / 77 78 / 93
many-paragraphs 147 / 158 82 / 90 80 / 162
suite cold average 383 / 315 145 / 154 130 / 157

After the fix, cache-on matches the cache-off control, i.e. the penalty is gone rather than reduced.

Verification

  • Native tests: 301 passed, 0 failed, 0 errors. The reported exit code -1 is a pre-existing post-completion hang that Invoke-CppTest.ps1 already has explicit handling for; it is unrelated to this change.
  • Render transparency. These caches must not change output. The six scenarios with pre-existing pixel differences are unchanged — complex 10, footnote-heavy 16, many-paragraphs 50, custom-heavy 5, medium 5, multi-book 5 — across four runs in both cache configurations, and no passing scenario began failing.
  • Build freshness confirmed by timestamp, because Views.mak has no header-dependency tracking and a stale Views.dll reports success (see below).

Why the existing tests did not catch this

Three safety nets were disconnected, all introduced or left broken by 502b832 itself.

  1. The benchmark scenarios could not reach the regressed path. The suite sets IsGraphiteEnabled = false and uses ASCII almost throughout, and ASCII is identical under NFD and NFC, so the offset-map branch never executed. multi-ws and rtl-script reached it only incidentally, because Arabic harakat are canonically reordered. This PR adds three scenarios built from decomposed text — nfc-composable-diacritics, and decomposed forms of single-para-mixed-ws and multi-line-wrap-single-ws — which reproduce the defect at seconds scale.

  2. The cache unit tests had never run. Src/views/Test/TestViewCaches.h was added by the same commit with 9 test methods across TestColorStateCache, TestFontHandleCache and TestShapeRunCache, but was never listed as a dependency of Collection.cpp in testViews.mak — and that list is the input to test collection. This PR registers it; all 9 pass.

  3. Header-only native edits silently produce stale binaries. Views.mak via Bld/_rule.mak uses {srcdir}.cpp{objdir}.obj: rules with no header-dependency tracking, so editing a header leaves the previous Views.dll in place while the build reports success. Hit twice during this investigation. Not addressed here — it is a broader build-system defect and needs its own issue.

Deliberately not included

  • The FW_PERF_P125_PATH2 default is unchanged. With the quadratic construction gone the cache is no longer harmful, so flipping a default would only obscure what this PR claims. The environment variable remains available.
  • TextAnalysisCache capacity and retention changes. Measured no reproducible benefit and were aimed at a cost that turned out not to be the problem.
  • Re-enabling the cache for non-NFC text with lazily built maps. That is the right long-term design — cache the expensive parts eagerly, build offset maps only when a translation is actually requested — but it changes the entry's contract and needs to prove it beats these numbers. It belongs with the optimisation work, not with a regression fix.

The three new scenarios have no accepted pixel baseline yet, so they report "Missing verified render baseline" until someone reviews and accepts the .received.png files. Their timings are still recorded.


This change is Reviewable

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown

Render comparison artifacts

Render snapshot failures were reported in a0e16e5b722a run 31584662718.1, but the latest run 2d00ee01141f run 31784647591.1 is clean.

This comment will be replaced if a future run produces render snapshot failures again.

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown

NUnit Tests

    1 files   -     1      1 suites   - 1   11m 11s ⏱️ - 3m 20s
5 767 tests +    6  5 686 ✅ +   44  81 💤 ± 0  0 ❌  - 38 
5 776 runs   - 5 764  5 695 ✅  - 5 609  81 💤  - 81  0 ❌  - 74 

Results for commit a666ea0. ± Comparison against base commit 7f93348.

♻️ This comment has been updated with latest results.

@codecov-commenter

codecov-commenter commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.04%. Comparing base (7f93348) to head (a666ea0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1056      +/-   ##
==========================================
+ Coverage   37.91%   38.04%   +0.13%     
==========================================
  Files        1499     1499              
  Lines      350117   350101      -16     
  Branches    40233    40233              
==========================================
+ Hits       132747   133200     +453     
+ Misses     188043   187616     -427     
+ Partials    29327    29285      -42     
Files with missing lines Coverage Δ
Src/views/lib/UniscribeSegment.cpp 70.41% <100.00%> (-0.38%) ⬇️

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

johnml1135 and others added 5 commits August 14, 2026 04:35
RealDataTestsBase refused to delete its reusable test project unless the
sentinel file was present, but CreateNewLangProj creates that directory
seconds before the sentinel can be written. A run interrupted inside project
creation therefore left a directory that no later run would delete, failing
every render benchmark scenario with "Refusing to delete" until someone wrote
the dotfile by hand.

Deletion now restores a missing sentinel when the directory holds no project
data beyond the fixture's own generated file, and still refuses when it finds
project data the fixture did not generate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Opening a text whose baseline is a single long paragraph with more than one
writing system took over ten seconds. The analysis cache built NFC offset maps
for every analysed range by normalizing each prefix of the range from scratch,
which is quadratic in the range length and repeated per range.

The maps exist only to translate between original and normalized offsets, which
is needed only when normalization changes the text. Text that normalizes
unchanged needs no maps, so the cache now stores only that text and everything
else falls back to the existing on-demand translation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestViewCaches.h was never listed as a dependency of Collection.cpp, and that
list is the input to test collection, so its nine tests were never run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The existing scenarios use text that is identical under NFD and NFC, so they
cannot reach the offset-map path at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed the .received.png output for single-para-mixed-ws,
nfc-composable-diacritics, and multi-line-wrap-single-ws and
confirmed correct rendering, so accepting them as the new
verified baselines these scenarios were missing.
@johnml1135
johnml1135 force-pushed the lt-22674-nfc-offset-map-quadratic branch from b9e6e39 to a666ea0 Compare August 14, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants