Skip to content

Add Tier 2 shaping regression against external harfrust corpus - #7

Merged
hasanzakeri merged 2 commits into
mainfrom
feat/test-external-corpus
Apr 16, 2026
Merged

hasanzakeri merged 2 commits into
mainfrom
feat/test-external-corpus

Conversation

@hasanzakeri

Copy link
Copy Markdown
Owner

Goal

Get a high-confidence signal that pyharfrust.shape stays faithful to upstream harfrust output — not just on the handful of cases we bundle, but on the full shaping corpus harfrust itself uses to guard against regressions.

Bundled Tier 1 tests are fast and self-contained, but they cover a tiny slice of scripts and features. Every harfrust version bump is a chance for subtle output drift that our ~6 bundled cases would never catch. Tier 2 closes that gap.

Approach

Opt-in, not always-on. A harfrust checkout is a heavy dependency (fonts, generated test files, a specific commit pin), and the suite parametrizes into ~6 k cases. Making it the default would punish every local pytest run and every main-branch CI job for a signal that only matters at release time.

The mechanism:

  • Set HARFRUST_SOURCE=/path/to/harfrust to enable.
  • Cases are marked @pytest.mark.external at collection time.
  • pyproject.toml default-denies via addopts = "-m 'not external'", so Tier 2 only runs when the marker filter is explicitly overridden (pytest -m external or -m "").
  • A MIN_EXTERNAL_CASES floor ensures a silently-broken parser can't pass by collecting zero cases.

Parse, don't re-run. harfrust's corpus lives in generated tests/shaping/*.rs files containing literal shape(...) assertions. We extract the (font, text, options, expected) tuples with a targeted regex (handles \u{XXXX} escapes and line continuations), then compare against pyharfrust.shape output. We intentionally skip the tests/custom/*.tests source files — those are generator inputs that may contain stale expected values.

CI pinned to the shipped version. Tier 2 runs on push/PR targeting release, against harfrust checked out at commit efdae31 (the 0.5.2 tag, matching the hr-shape version in Cargo.toml). Bumping the dep means bumping the SHA in the same PR — corpus and shaper stay in lockstep. Main-branch CI is unchanged and still fast.

Why regex over an AST

The .rs files are machine-generated with a rigid, uniform shape("...", "...", "..."), "..." shape. A full Rust parser (tree-sitter-rust, syn) would be overkill, add a heavy dep, and gain nothing on content this regular. If harfrust's generator format ever changes materially, the regex will fail loudly (via the min-case guard) rather than silently drift.

Out of scope

  • Vendoring the harfrust corpus into this repo (too large, and would defeat the point of testing against upstream).
  • Automating the harfrust SHA bump alongside hr-shape — manual step, intentional coupling.

  Introduces an opt-in regression mode that shapes every test case in a
  local harfrust checkout through pyharfrust.shape and compares against
  the expected output embedded in harfrust's own tests. Enabled by
  setting HARFRUST_SOURCE; default-denied in pytest config so local
  dev and the main-branch CI stay fast.

  CI runs Tier 2 only on push/PR targeting the release branch, against
  harfrust pinned at commit efdae31 (0.5.2) to match the hr-shape dep.
The `_collect_external_cases()` function was incorrectly parsing `.tests`
files from harfrust's `tests/custom/` directory. These files are source
inputs for harfrust's test generator (`gen-shaping-tests.py`), not actual
test cases to run. They contain expected values that may intentionally
differ from harfrust's current behavior, as noted in the files themselves:
"the expected values for the shaping process will be ignored and sometimes
wrong."

This caused false test failures, such as the `--language=pl` BigCaslon test
which expects `cacute.polish` but harfrust outputs `cacute` (confirmed by
running hr-shape CLI directly).

Now we only parse the generated `.rs` files in `tests/shaping/`, which
represent the actual tests that `cargo test` runs in harfrust.

Test count: 6145 → 6139 external cases (removed 6 invalid cases from .tests)
@hasanzakeri
hasanzakeri merged commit e670a6b into main Apr 16, 2026
2 checks passed
@hasanzakeri
hasanzakeri deleted the feat/test-external-corpus branch April 16, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant