Add Tier 2 shaping regression against external harfrust corpus - #7
Merged
Merged
Conversation
Introduces an opt-in regression mode that shapes every test case in a local harfrust checkout through pyharfrust.shape and compares against the expected output embedded in harfrust's own tests. Enabled by setting HARFRUST_SOURCE; default-denied in pytest config so local dev and the main-branch CI stay fast. CI runs Tier 2 only on push/PR targeting the release branch, against harfrust pinned at commit efdae31 (0.5.2) to match the hr-shape dep.
The `_collect_external_cases()` function was incorrectly parsing `.tests` files from harfrust's `tests/custom/` directory. These files are source inputs for harfrust's test generator (`gen-shaping-tests.py`), not actual test cases to run. They contain expected values that may intentionally differ from harfrust's current behavior, as noted in the files themselves: "the expected values for the shaping process will be ignored and sometimes wrong." This caused false test failures, such as the `--language=pl` BigCaslon test which expects `cacute.polish` but harfrust outputs `cacute` (confirmed by running hr-shape CLI directly). Now we only parse the generated `.rs` files in `tests/shaping/`, which represent the actual tests that `cargo test` runs in harfrust. Test count: 6145 → 6139 external cases (removed 6 invalid cases from .tests)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Get a high-confidence signal that
pyharfrust.shapestays faithful to upstream harfrust output — not just on the handful of cases we bundle, but on the full shaping corpus harfrust itself uses to guard against regressions.Bundled Tier 1 tests are fast and self-contained, but they cover a tiny slice of scripts and features. Every harfrust version bump is a chance for subtle output drift that our ~6 bundled cases would never catch. Tier 2 closes that gap.
Approach
Opt-in, not always-on. A harfrust checkout is a heavy dependency (fonts, generated test files, a specific commit pin), and the suite parametrizes into ~6 k cases. Making it the default would punish every local
pytestrun and everymain-branch CI job for a signal that only matters at release time.The mechanism:
HARFRUST_SOURCE=/path/to/harfrustto enable.@pytest.mark.externalat collection time.pyproject.tomldefault-denies viaaddopts = "-m 'not external'", so Tier 2 only runs when the marker filter is explicitly overridden (pytest -m externalor-m "").MIN_EXTERNAL_CASESfloor ensures a silently-broken parser can't pass by collecting zero cases.Parse, don't re-run. harfrust's corpus lives in generated
tests/shaping/*.rsfiles containing literalshape(...)assertions. We extract the(font, text, options, expected)tuples with a targeted regex (handles\u{XXXX}escapes and line continuations), then compare againstpyharfrust.shapeoutput. We intentionally skip thetests/custom/*.testssource files — those are generator inputs that may contain stale expected values.CI pinned to the shipped version. Tier 2 runs on push/PR targeting
release, against harfrust checked out at commitefdae31(the 0.5.2 tag, matching thehr-shapeversion inCargo.toml). Bumping the dep means bumping the SHA in the same PR — corpus and shaper stay in lockstep. Main-branch CI is unchanged and still fast.Why regex over an AST
The
.rsfiles are machine-generated with a rigid, uniformshape("...", "...", "..."), "..."shape. A full Rust parser (tree-sitter-rust, syn) would be overkill, add a heavy dep, and gain nothing on content this regular. If harfrust's generator format ever changes materially, the regex will fail loudly (via the min-case guard) rather than silently drift.Out of scope
hr-shape— manual step, intentional coupling.