feat: golden baseline regression gate for composition snapshots - #3756
Open
mvanhorn wants to merge 4 commits into
Open
feat: golden baseline regression gate for composition snapshots#3756mvanhorn wants to merge 4 commits into
mvanhorn wants to merge 4 commits into
Conversation
Adds a committed-baseline visual regression gate:
- golden/<compositionId>/<timeMs>.png convention with an optional
golden.json manifest (times, threshold, maxDiffRatio, ignoreAntialiasing)
- 'hyperframes check --golden' re-captures at the manifest times,
pixel-diffs against the baselines (sharp raw RGBA, configurable
per-channel threshold, 1px anti-aliasing shift detection), fails the
check on regression, and writes golden-diff/ with per-frame red-highlight
diffs plus a baseline|current|diff contact sheet
- 'hyperframes check --update-golden' / 'hyperframes snapshot
--update-golden' refresh the baselines and manifest
- agent-readable JSON summary {ok, failed:[{id,time,maxDelta}], diffSheet}
embedded in the check --json envelope
- unit tests for the diff math (synthetic PNGs, no Chrome), the golden
conventions, and the check command wiring
- reference/cli-golden: directory convention, flags, golden.json manifest fields, failure artifacts, and the agent-readable JSON summary - guides/visual-regression: the commit-baselines workflow, agent loop usage, CI wiring, determinism guidance, and sample-time selection
A minimal 640x360 composition plus committed golden/ baselines and a manifest, demonstrating the check --golden / --update-golden loop and a scripted regression to try. Un-ignores the new example directory.
The repo-wide '*-demo/' ignore rule silently swallowed golden/golden-baseline-demo/, so the example shipped without its baselines. Rename the composition id to golden-baseline and commit the three baseline PNGs plus the manifest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A committed-baseline visual regression gate for compositions:
golden/<compositionId>/<timeMs>.pngwith an optionalgolden/<compositionId>/golden.jsonmanifest (times,threshold,maxDiffRatio,ignoreAntialiasing).hyperframes check --goldenre-captures the composition at the manifest times, pixel-diffs each frame against its baseline (configurable per-channel threshold, 1px anti-aliasing shift detection), fails the check on regression, and writesgolden-diff/with per-frame red-highlight diff PNGs plus abaseline | current | diffcontact sheet.hyperframes check --update-golden/hyperframes snapshot --update-goldenrefresh the baselines and manifest.check --jsonenvelope:{ok, failed:[{id, time, timeMs, maxDelta, diffRatio, reason}], diffSheet, ...}.examples/golden-baseline/; docs atdocs/reference/cli-golden.mdxanddocs/guides/visual-regression.mdx.Why
Agents regress layouts silently: a refactor nudges a headline, a token change recolors a button — and every command still exits 0. HyperFrames already has
snapshotandcompare, but no committed golden baseline and no fail-on-diff CI gate. This turns "the video still looks right" into a hard, committable contract with one exit code.How
packages/cli/src/golden/pixelDiff.ts— pure diff math on raw RGBA buffers viasharp(already a CLI dependency; no new dependencies): per-channel threshold, neighborhood-based anti-aliasing classification, red/amber/pale-gray diff visualization.packages/cli/src/golden/baseline.ts— thegolden/convention, manifest parsing/validation, sample-time resolution (--at→ manifest → baseline filenames → default spread), and the gate/update orchestration. Capture reuses the same font-localized, settled-page path assnapshot/compare(openSettledCompositionPage+seekCompositionTimeline).packages/cli/src/capture/contactSheet.ts— newcreateGoldenDiffContactSheet(onebaseline | current | diffrow per failed time).check/snapshotwiring:--golden,--update-golden,--golden-threshold; golden failures combine with the normal check exit code; a missing baseline fails with instructions rather than silently passing.golden-diff/is derived output and git-ignored in the example.Walkthrough (HyperFrames-rendered)
The walkthrough is itself a HyperFrames composition rendered to MP4 with this branch's CLI — storyboarded first, authored as HTML, validated with
hyperframes lint+hyperframes check(0 errors, 33/33 WCAG AA text checks), then rendered with:Every terminal line and every frame image in the film is real CLI output from
examples/golden-baseline: the three committed golden PNGs, the failing run's current frame, its red-highlight diff, and the generated contact sheet.▶ Watch the walkthrough (MP4, 48s, 2.8 MB)
Stills from the rendered film:
The real diff contact sheet produced by the failing run (
golden-diff/golden-baseline/contact-sheet.jpg):Test plan
packages/cli/src/golden/pixelDiff.test.ts(diff math on synthetic PNGs, no Chrome: thresholds, AA classification, dimension mismatch, PNG round trip),packages/cli/src/golden/baseline.test.ts(path conventions, manifest validation, sample-time precedence, gate verdicts),check.test.ts(flag grammar + command wiring/exit codes). Full@hyperframes/clisuite: 213 files, 3034 tests passing (bun run test).examples/golden-baseline: created baselines (snapshot --update-golden), verifiedcheck --goldenpasses across independent captures, tweaked the accent color#f5a623 → #e0245e, verified the gate fails with exit 1 + diff sheet + JSON summary, then refreshed withcheck --update-golden.docs/reference/cli-golden.mdx,docs/guides/visual-regression.mdx, registered indocs/docs.json; example README.bunx oxlint/oxfmtclean;tsc --noEmitclean;fallow auditclean (no new dead code, cycles, duplication, or complexity findings).AI disclosure
This PR was authored by an AI agent (Claude Fable 5 via Cursor Cloud Agents), including the code, tests, docs, example, and the HyperFrames-rendered walkthrough video. It was developed and verified end-to-end as described in the test plan; per CONTRIBUTING, correctness responsibility is accepted by the submitter.