Skip to content

feat: golden baseline regression gate for composition snapshots - #3756

Open
mvanhorn wants to merge 4 commits into
heygen-com:mainfrom
mvanhorn:feat/golden-baseline-gate
Open

feat: golden baseline regression gate for composition snapshots#3756
mvanhorn wants to merge 4 commits into
heygen-com:mainfrom
mvanhorn:feat/golden-baseline-gate

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What

A committed-baseline visual regression gate for compositions:

  • Convention: reference frames live at golden/<compositionId>/<timeMs>.png with an optional golden/<compositionId>/golden.json manifest (times, threshold, maxDiffRatio, ignoreAntialiasing).
  • hyperframes check --golden re-captures the composition at the manifest times, pixel-diffs each frame against its baseline (configurable per-channel threshold, 1px anti-aliasing shift detection), fails the check on regression, and writes golden-diff/ with per-frame red-highlight diff PNGs plus a baseline | current | diff contact sheet.
  • hyperframes check --update-golden / hyperframes snapshot --update-golden refresh the baselines and manifest.
  • Agent-readable JSON summary embedded in the check --json envelope: {ok, failed:[{id, time, timeMs, maxDelta, diffRatio, reason}], diffSheet, ...}.
  • Runnable example with committed baselines under examples/golden-baseline/; docs at docs/reference/cli-golden.mdx and docs/guides/visual-regression.mdx.

Why

Agents regress layouts silently: a refactor nudges a headline, a token change recolors a button — and every command still exits 0. HyperFrames already has snapshot and compare, but no committed golden baseline and no fail-on-diff CI gate. This turns "the video still looks right" into a hard, committable contract with one exit code.

How

  • packages/cli/src/golden/pixelDiff.ts — pure diff math on raw RGBA buffers via sharp (already a CLI dependency; no new dependencies): per-channel threshold, neighborhood-based anti-aliasing classification, red/amber/pale-gray diff visualization.
  • packages/cli/src/golden/baseline.ts — the golden/ convention, manifest parsing/validation, sample-time resolution (--at → manifest → baseline filenames → default spread), and the gate/update orchestration. Capture reuses the same font-localized, settled-page path as snapshot/compare (openSettledCompositionPage + seekCompositionTimeline).
  • packages/cli/src/capture/contactSheet.ts — new createGoldenDiffContactSheet (one baseline | current | diff row per failed time).
  • check/snapshot wiring: --golden, --update-golden, --golden-threshold; golden failures combine with the normal check exit code; a missing baseline fails with instructions rather than silently passing.
  • Dimension changes always fail regardless of the pixel budget; golden-diff/ is derived output and git-ignored in the example.

Walkthrough (HyperFrames-rendered)

The walkthrough is itself a HyperFrames composition rendered to MP4 with this branch's CLI — storyboarded first, authored as HTML, validated with hyperframes lint + hyperframes check (0 errors, 33/33 WCAG AA text checks), then rendered with:

npx hyperframes render . --quality high --no-browser-gpu -o renders/golden-gate-walkthrough.mp4
# ◇ 2.8 MB · 48.0s video · rendered in 50.5s (1440 frames @ 30fps)

Every terminal line and every frame image in the film is real CLI output from examples/golden-baseline: the three committed golden PNGs, the failing run's current frame, its red-highlight diff, and the generated contact sheet.

Watch the walkthrough (MP4, 48s, 2.8 MB)

Stills from the rendered film:

baselines frozen → gate passes one hex of drift → gate fails arrival: the diff sheet
title scene failing gate scene with red diff output and Check failed exit 1 chip diff sheet arrival with red callout ring on the diff column

The real diff contact sheet produced by the failing run (golden-diff/golden-baseline/contact-sheet.jpg):

golden-diff contact sheet: baseline, current, diff rows for the three regressed sample times

Test plan

  • Unit tests added/updated — packages/cli/src/golden/pixelDiff.test.ts (diff math on synthetic PNGs, no Chrome: thresholds, AA classification, dimension mismatch, PNG round trip), packages/cli/src/golden/baseline.test.ts (path conventions, manifest validation, sample-time precedence, gate verdicts), check.test.ts (flag grammar + command wiring/exit codes). Full @hyperframes/cli suite: 213 files, 3034 tests passing (bun run test).
  • Manual testing performed — on examples/golden-baseline: created baselines (snapshot --update-golden), verified check --golden passes across independent captures, tweaked the accent color #f5a623 → #e0245e, verified the gate fails with exit 1 + diff sheet + JSON summary, then refreshed with check --update-golden.
  • Documentation updated — docs/reference/cli-golden.mdx, docs/guides/visual-regression.mdx, registered in docs/docs.json; example README.
  • bunx oxlint / oxfmt clean; tsc --noEmit clean; fallow audit clean (no new dead code, cycles, duplication, or complexity findings).

AI disclosure

This PR was authored by an AI agent (Claude Fable 5 via Cursor Cloud Agents), including the code, tests, docs, example, and the HyperFrames-rendered walkthrough video. It was developed and verified end-to-end as described in the test plan; per CONTRIBUTING, correctness responsibility is accepted by the submitter.

Adds a committed-baseline visual regression gate:

- golden/<compositionId>/<timeMs>.png convention with an optional
  golden.json manifest (times, threshold, maxDiffRatio, ignoreAntialiasing)
- 'hyperframes check --golden' re-captures at the manifest times,
  pixel-diffs against the baselines (sharp raw RGBA, configurable
  per-channel threshold, 1px anti-aliasing shift detection), fails the
  check on regression, and writes golden-diff/ with per-frame red-highlight
  diffs plus a baseline|current|diff contact sheet
- 'hyperframes check --update-golden' / 'hyperframes snapshot
  --update-golden' refresh the baselines and manifest
- agent-readable JSON summary {ok, failed:[{id,time,maxDelta}], diffSheet}
  embedded in the check --json envelope
- unit tests for the diff math (synthetic PNGs, no Chrome), the golden
  conventions, and the check command wiring
- reference/cli-golden: directory convention, flags, golden.json manifest
  fields, failure artifacts, and the agent-readable JSON summary
- guides/visual-regression: the commit-baselines workflow, agent loop
  usage, CI wiring, determinism guidance, and sample-time selection
A minimal 640x360 composition plus committed golden/ baselines and a
manifest, demonstrating the check --golden / --update-golden loop and a
scripted regression to try. Un-ignores the new example directory.
The repo-wide '*-demo/' ignore rule silently swallowed
golden/golden-baseline-demo/, so the example shipped without its
baselines. Rename the composition id to golden-baseline and commit the
three baseline PNGs plus the manifest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants