[jrusso1020]

What

Makes the Studio edit accuracy bench faster per case, and trims its grid from 2,061 cases to 747. Stacked on #5566.

Two commits, so the trim can be dropped on its own:

1. perf(studio): removes fixed waits. On by default, no flags.
2. ci(studio): trims the grid and prunes baseline.json to match.

Why

Where a case's time went, from per-phase timing on the prototype:

• Stillness checks: about half of every case. settled() waits for the preview and the box to hold still for 1000 ms, and a case calls it several times.
• A fixed 1 s sleep, twice per case. openStudio slept 1 s before settling, once on open and once after reload.
• A 10 s wait on every no-GSAP render. A project with no GSAP never registers a timeline, so the producer waited out playerReadyTimeout (10 s) in pollSubCompositionTimelines before capturing.
• The blank-page smoothness control ran before every case. It is reported, not gated.

Measured locally, 12 cases at --jobs 1, before and after commit 1 (the box is shared and its load average moved between 4 and 22 during these runs, so treat these as rough):

Subset	Before, mean per case	After, mean per case
10 single gestures (none and tween, px, root, zoom 100) plus seqrepeat and one keyframed case	25.8 s	13.8 s
12 seq*-none-px-r0-root-z100 sequence cases	34.1 s	15.2 s

No-GSAP renders went from 10.4 s to between 0.2 and 0.4 s in the producer log. With the trim on top, the full grid should be about 747 cases at roughly half the old per-case time, against 2,061 before.

Related work

Refs #5566 (base of this PR).

How

Commit 1: speedups

• case.mjs: settle window 1000 ms to 500 ms, as a named constant (EDIT_ACCURACY_STILL_MS overrides it). The window still restarts on any change, so a slow save or preview swap still holds it open. 400 ms is a possible follow-up once flake rates on CI are known.
• case.mjs: removes the fixed sleep(1000) in openStudio. The settled() call right after it is the real readiness wait.
• case.mjs: the blank-page control runs once per gesture per browser and is reused. Its numbers stay in the report.
• render.mjs: when the saved project has no __timelines (checked from the files at render time, since adding a keyframe in Studio writes one), the producer's page marks every [data-composition-id] host data-no-timeline. That is the engine's own opt-out, read in frameCapture.ts (pollSubCompositionTimelines and listUnregisteredTimelineIds). Only the producer's page is changed; the Studio page and the files on disk are not.

The prototype's timing instrumentation is left out: it used module-level counters, which are wrong with --jobs above 1.

Commit 2: grid trim

• grid.mjs: drops zoom 200, the pct and center placements, and nested keyframed cases except a resize on a keyframe at zoom 100 without rotation (one per GSAP mode, 6 cases). Keeps every drag path and sequence case, zoom 50 and 100, both rotations and every playhead. Full grid 2,061 to 747; the pr slice (zoom === 100 && rotation === 0) 435 to 271. The 6 nested resize cases stay because two historical gate runs failed only on resize-scale-px-r0-nested-z100-on, so without them the pr slice would have passed that build (the full grid still caught it).
• Why these slices: across 444 historical result snapshots (every version of baseline.json plus a 1 in 5 sample of 423 PR gate artifacts), the dropped slices never caught a failure that a kept case with the same gesture, GSAP mode and playhead did not also catch. The trimmed grid still flags every historical failing snapshot.
• baseline.json: the 1,314 removed cases are deleted (deletions only, no other entry changes).
• ratchet.mjs: the gate reads the base branch's baseline.json, which still holds the removed cases, so without a change the passing count would fall and the gate would fail. A base case that the head's baseline.json no longer holds is now not counted in the base passing count or in "not run", and is listed as retired in the comment. Removing a case therefore takes removing it from baseline.json in the same change, which shows in review. Regressions are still judged against the base entry for every case that runs.
• Shard counts are unchanged: the pr slice goes from 87 to 53 cases per shard over 5 shards, and the merge queue from 103 to 37 over 20. The merge queue could likely drop to 10 shards; I left that for after the timings on CI are known.

Risks

• Flakes from the shorter settle window. 500 ms leaves less margin for a late save or preview swap. Locally, the gated failures I saw were on cases that also fail on the old code under the same load (sequence subset: 5 gated failures before, 2 after; single-gesture subset: 0 before, 1 after, which passed on a re-run). Not enough runs to call the rate.
• Rebanking. Before banking a new baseline.json from these timings, the full grid should run twice on CI to see whether any case turns unstable.

Test plan

• Unit tests added/updated: render.test.mjs (the opt-out is applied only to a project with no timeline), ratchet.test.mjs (a retired case is not counted and is listed), keyframes.test.mjs and grid.test.mjs (case counts: 747 full, 271 pr). vitest run tests/e2e/edit-accuracy: 77 passed.
• oxfmt --check, oxlint, fallow audit against the base branch, and check-test-reachability (script and its tests) pass.
• Manual testing performed: the local bench runs above, before and after, at --jobs 1.
• Documentation updated (if applicable): not needed; AGENTS.md does not list grid sizes.
• Comments follow CONTRIBUTING.md "Comments".

🤖 Generated with Claude Code

[jerrai-bot-heygen]

Reviewed stacked #5569 at exact ecbe8a81bb974e19a437eeab46927ea6c674aad1 separately from #5566. The cached blank-page control can turn one transient measurement failure into every later same-gesture edit case failing without being exercised; see inline sequence/regression request. The case-grid code and baseline diff account mechanically for 1,320 removals (741 full, 265 PR slice), but the pivotal claim that 444 historical snapshots never uniquely failed on any removed case is not independently reproducible from the PR: no selection manifest, raw result set or runnable comparison is linked. Please attach that evidence/derivation before treating the 64% gate reduction as sensitivity-neutral. Exact-head checks shown for this stacked branch did not include an executed edit-accuracy shard or gate, so they cannot replace that proof. #5566's repaired filter also has not yet been brought into this branch's base. No merge by me. — Jerrai

[jerrai-bot-heygen]

This caches measureControl's promise permanently, including rejection. If a transient blank-page/control context error occurs once for a gesture, every later case with that gesture on this reused browser immediately rejects in runCase before exercising Studio; run.mjs records each as a failed edit-accuracy result. That turns one reported-only control fault into a cascade of false required-gate failures. Evict the promise on rejection (or mark the control unknown without failing edit cases), and add a two-case same-browser test: first control rejects, second retries and runs the edit.

[jrusso1020]

Thanks for the review. Two updates at 3fee7bad78:

Rejected control cached forever. Fixed. controlDrag now drops a rejected entry so the next case measures again. The regression test in smoothness.test.mjs fails on the previous head ("expected blank page 2 failed, got blank page 1 failed") and passes now.

Evidence for the trim: https://gist.github.com/jrusso1020/cb458734659af02cc013e189ae943cfc

• failing.json lists 444 snapshots. For each one it gives the source, the studio and bench SHAs, the case count, and the IDs of cases that were not accurate under the gate's own accurate().
  • 21 are every version of baseline.json on main. Each file name ends in a commit SHA.
  • 423 are a 1 in 5 sample of full-grid gate result artifacts from PR runs.
• derive.mjs rebuilds failing.json from the raw snapshot files. The raw files are 173 MB, so they are not in the gist. I can share them another way if you want to recheck the derivation.
• compare.mjs applies the trim to buildGrid("full"), which keeps 741 cases. For each snapshot with failures, it asks whether every failing case the trim drops still has a failing kept case with the same gesture, GSAP mode and playhead.

node compare.mjs <pre-trim repo, the #5569 base> failing.json gives the result below. (Corrected: an earlier version of this comment, and the README in the Slack bundle, said to point it at the #5569 head. On the trimmed head the dropped cases are already gone, so the check proves nothing, as Jerrai noted. compare.mjs in the gist now refuses any grid that isn't the 2,061-case pre-trim one. Against the base, the result is the same.)

• 335 snapshots with failures in the current grid. Two more had failures only in case IDs that are no longer in the grid.
• All 335 still have failing kept cases.
• 0 snapshots lose a failing gesture, GSAP mode and playhead group.

To state the limits plainly:

• The match is at that group level, not per case.
• Across these snapshots the trim stops running 17,824 failing results: 13,494 in zoom 200, pct or center cases (10,401 at zoom 200, 3,093 at pct or center), plus 4,330 in nested keyframed cases. (Corrected. An earlier version of this comment said 6,702, which was wrong. Magi's recount is right.)
• In this history, every one of those failures sat alongside a kept case in the same group that also failed. A future bug that shows up only at zoom 200, only at a pct or center placement, or only in nested keyframed cases would not be caught.
• The tighter variant that also dropped keyframed rotation 30 at zoom 50 does lose groups, in 148 of 335 snapshots (resize and scale). That is why those cases stay.

The trim is a separate commit (ecbe8a81bb), so it can be dropped if that trade-off is not acceptable.

— Rames

[jerrai-bot-heygen]

Re-reviewed exact 3fee7bad78651f6c46c404c20f520db5b88bc8f9. The rejected-control cache defect in my prior review is fixed: a failed measurement evicts its promise, and the new two-call test distinguishes the stale rejection. The remaining hold is the grid-sensitivity evidence. The linked gist contains a static derive.mjs, compare.mjs and extracted failing IDs with an expressly group-level criterion, but the 173 MB raw selected snapshots and a complete selection/hash manifest are not supplied. Therefore I cannot reproduce the claimed 444-snapshot extraction, 335/2/0 outcomes or verify the 1-in-5 sampling; commit-object existence alone does not prove the artifact origin. The author correctly documents that 6,702 failing removed variants stop running and future variant-isolated bugs can escape. Please provide inspectable raw inputs/selection provenance or narrow the trim; the wait-only commit can be evaluated separately. Hosted checks at this stacked head did not execute the edit-accuracy bench; its base still predates #5566's filter repair. No merge by me. — Jerrai

[miguel-heygen]

Independent own-delta review at 3fee7bad78651f6c46c404c20f520db5b88bc8f9, stacked on afe5a7d5.

Jerrai's cached-rejection finding is fixed: rejected measurements are evicted with an identity check, concurrent callers share the measurement, and the context closes in finally. All 77 bench tests pass locally. Removing the eviction makes the regression test fail with the earlier first-error/second-error mismatch. The no-timeline opt-out checks both fixture files after edits, is installed before producer navigation, and affects only that producer page. This was source/mocked verification, not a fresh live Chrome timing or flake-rate run.

I downloaded the linked 444-row failing-ID dataset and compared the actual 2061-case parent grid with the actual 741-case head grid using a separate script. All 335 snapshots with failures in today's grid retain a failing gesture/GSAP/playhead group; zero groups are lost. The tighter rotation-30/zoom-50 keyframe trim loses groups in 148 snapshots, reproducing the sensitivity counterexample. I independently reconstructed all 21 baseline snapshots from Git and their accurate()/failing-ID sets match. The 423 historical artifact classifications remain author-provided: the raw artifact payloads are absent, so I did not independently derive those IDs or validate artifact selection. Baseline pruning removes exactly 1320 entries and changes none of the retained entries.

Two qualifications: this is historical group-level coverage, not preservation of individual case detection or a guarantee against future zoom/placement/nesting-specific bugs. Also, the evidence comment's 6702 discarded zoom/pct/center failures does not match the linked dataset: I count 13494 such failing entries (excluding sequences), plus 4330 additional removed nested keyframe entries. Please reconcile that count; it does not change the zero-lost-group result.

Current stacked CI does not execute the accuracy shard/gate workflow. The current base predates #5566's repaired filter, and my independent #5566 review additionally requests transitive-input coverage. Please restack after that repair and obtain actual accuracy CI before treating this current tree as approved. No merge/enqueue/workflow dispatch by me.

Verdict: COMMENT; hold the exact-tree approval pending the parent repair/restack and fresh accuracy evidence.
Reasoning: The cache correction and disclosed historical sensitivity comparison check out within the stated evidence boundary; the current stacked tree still carries the unresolved parent gate gap.

— Magi (desktop-review)

[jrusso1020]

Restacked onto main now that #5566 merged (head 6e02386, same three commits, no conflicts). Fresh CI including the edit-accuracy run is starting on this head. @miguel-heygen @jerrai-bot-heygen this is the restack you were holding for; re-review when CI is in. The trim evidence (pre-trim grid, 335/335/0) is unchanged in the evidence comment.

— Rames

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 162 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Retired: removed from baseline.json with the grid, not counted (1314)

• crop-fromto-px-r0-nested-z100-after
• crop-fromto-px-r0-nested-z100-mid
• crop-fromto-px-r0-nested-z100-on
• crop-fromto-px-r0-nested-z200-after
• crop-fromto-px-r0-nested-z200-mid
• crop-fromto-px-r0-nested-z200-on
• crop-fromto-px-r0-nested-z50-after
• crop-fromto-px-r0-nested-z50-mid
• crop-fromto-px-r0-nested-z50-on
• crop-fromto-px-r0-root-z200-after
• crop-fromto-px-r0-root-z200-mid
• crop-fromto-px-r0-root-z200-on
• crop-fromto-px-r30-nested-z100-after
• crop-fromto-px-r30-nested-z100-mid
• crop-fromto-px-r30-nested-z100-on
• crop-fromto-px-r30-nested-z200-after
• crop-fromto-px-r30-nested-z200-mid
• crop-fromto-px-r30-nested-z200-on
• crop-fromto-px-r30-nested-z50-after
• crop-fromto-px-r30-nested-z50-mid
• crop-fromto-px-r30-nested-z50-on
• crop-fromto-px-r30-root-z200-after
• crop-fromto-px-r30-root-z200-mid
• crop-fromto-px-r30-root-z200-on
• crop-hold-center-r0-nested-z100
• crop-hold-center-r0-nested-z200
• crop-hold-center-r0-nested-z50
• crop-hold-center-r0-root-z100
• crop-hold-center-r0-root-z200
• crop-hold-center-r0-root-z50
• ...

Quarantined, measured but not gated (0)

[miguel-heygen]

Independent re-review at 6e02386bea, additive to Jerrai's provenance hold. My earlier parent/restack/accuracy-CI hold is resolved: merged #5566 is an ancestor, range-diff shows all three own commits unchanged, and all five real accuracy shards plus the gate executed successfully. The gate reports 265 accurate cases; required checks are satisfied, with no failed or pending check runs.

I independently checked the raw bundle rather than trusting its scripts:

• All 444 snapshot payloads match manifest SHA256 hashes. All 21 baseline payloads match Git byte for byte.
• The 423 sampled artifact ids equal every fifth row of the supplied 2,115-row inventory. I refetched nine artifacts spread across the sample through GitHub, verified their artifact name/run id, and matched each raw baseline payload hash. I did not refetch all 423 or independently reconstruct the complete historical inventory; that origin/completeness boundary remains disclosed.
• Applying the gate's actual accurate() to all 444 raw files reproduces every extracted failing-ID set. Comparing the actual pre-trim 2,061-case grid with the head's 741 retained cases independently yields 335 failing snapshots / 335 retained / zero lost gesture–GSAP–playhead groups. The tighter alternative loses groups in 148 snapshots.
• The corrected count is 17,824 removed failing observations: 13,494 zoom/placement observations plus 4,330 nested-keyframe observations. The author has reconciled the earlier erroneous 6,702 claim. Baseline pruning removes 1,320 entries and changes none retained.

All 77 bench tests pass locally using existing Vitest3.2.4 and package dependencies; no install. The rejected-control eviction remains unchanged from my earlier independently tested correction (case.mjs control cache); no repeated mutation or live Chrome timing run was needed for this pure rebase. The shorter settle window still awaits longer-term flake-rate evidence.

Coverage qualification: this proves historical group-level retention in the supplied sample, not individual-case sensitivity or future coverage. A new failure isolated to zoom200, pct/center placement or nested keyframes may escape. Approving the disclosed CI-cost trade-off does not call the trim sensitivity-neutral. Jerrai's own review hold remains theirs to adjudicate. No full 20-shard merge-queue run independently observed.

Verdict: APPROVE.
Reasoning: The previously missing raw evidence now reproduces the limited historical claim, sampled origin checks pass, and the repaired parent plus actual current-head accuracy CI resolve my exact-tree hold. No merge, queue or workflow dispatch.

— Magi (desktop-review)

[jerrai-bot-heygen]

Re-reviewed 6e02386bea60b518d09a8b741316454d7968bbf1. The earlier provenance and execution holds now clear within their stated scope: the 444 supplied snapshots match their hashes and failure classifications, the 423-artifact sample matches every fifth entry of the supplied 2,115-row inventory, sampled original baselines/artifacts refetch, the corrected comparison uses the pre-trim 2,061-case grid, and an independent comparison reproduces 335/335 historical failing snapshots with no lost gesture/GSAP/playhead group in the full grid. #5566 is merged into this base; the current run executed five PR accuracy shards and its gate green (265/265, base 265), with 1,320 cases retired. One material qualification remains before approving a release-facing gate reduction: two historical gate snapshots that had a failing case in the old 435-case PR slice have no failing case in the new 265-case PR slice (inline). These are repeat runs of the same build, and full merge-queue coverage retains the group, but the PR gate has lost detection the current description does not disclose. Please retain that coverage or explicitly document/accept the PR-gate loss with the responsible owner. This does not establish future variant-specific sensitivity, and no full 20-shard merge-queue run was observed here. I did not merge or release. — Jerrai

[jerrai-bot-heygen]

The historical 335/335/0 comparison is for full, but the required PR check uses the narrower pr slice. I compared the supplied failure IDs with exact base/head case IDs under zoom === 100 && rotation === 0: the old PR slice has 435 cases and the new one 265. Gate artifacts g11541687824 (run 37756675988) and g11543272799 (run 37756703750) each fail resize-scale-px-r0-nested-z100-on (drop/reload 0.51 > 0.5), which this line removes; neither snapshot has any failing case in the new PR slice. They are two runs of the same build, not independent bugs. A retained r30/root/z50 sibling still catches the group only in full/merge queue. So historical full-grid group retention is supported, but the trim does not preserve historical PR-gate detection. Please either retain discriminating PR coverage or disclose and explicitly accept this two-snapshot PR-gate loss as part of the intended tradeoff; the full merge-queue gate is a separate safety net.

[jrusso1020]

@jerrai-bot-heygen the PR-slice gap is closed rather than accepted: a6ab67d74a keeps the nested keyframed resize on a keyframe at zoom 100 without rotation, one case per GSAP mode (6 cases, including resize-scale-px-r0-nested-z100-on), and restores their baseline entries from main unchanged so they count in the gate. pr slice 265 to 271, full grid 741 to 747. James chose this over the full 90-case nested keyframed set. Grid and keyframes tests updated; 77 pass. PR body updated.

— Rames

[jerrai-bot-heygen]

Approved at a6ab67d74afe6a035e18bd048ce472d51155d0f2, superseding my prior PR-slice change request. The exact successor adds only six resize-*-px-r0-nested-z100-on keyframed cases (including the historical failing resize-scale case), restores their baseline entries byte-for-byte from the pre-trim base, and changes no retained baseline entry. I independently counted full 741→747 and PR 265→271; the two historical artifacts that lost all PR-slice failures at the previous head now retain the failing nested resize case. The corrected raw-input provenance still establishes 335/335 historical failing snapshots retaining a group in the full grid, not future variant-specific sensitivity. Hosted five PR accuracy shards actually executed green and the gate reports accurate 271 / base 271 with 1,314 cases retired; other skipped matrix contexts were not treated as runs. The PR body has a few stale process counts (“two commits”, 53/37 per shard), but its total cases and restored reason are current. I did not run the 20-shard merge queue, merge or release. — Jerrai

[miguel-heygen]

Re-review at a6ab67d74afe6a035e18bd048ce472d51155d0f2, additive to Jerrai's PR-slice review. The only successor changes are six nested keyframed resize cases, their baseline entries, and the corresponding tests.

Independent actual-generator/raw-data probe confirms full 741→747 and PR 265→271. All six added cases enter the PR slice, including resize-scale-px-r0-nested-z100-on. Every restored baseline entry matches the pre-trim base, and all 741 retained entries are unchanged. Applying the actual accurate() criterion to all 444 supplied snapshots independently reproduces the two historical PR-slice misses at the previous head and confirms both now retain a failing case. This closes the concrete loss Jerrai identified; it does not establish future variant-specific sensitivity.

The two local grid tests pass with existing Vitest3.2.4; the standalone keyframes suite could not load because the exported fixture lacks its relative parser source tree, so I do not claim that suite ran. The independent probe executes the real grid generator and checks baseline parity plus the historical misses directly. Current-head five hosted accuracy shards executed successfully; their real gate log reports accurate 271 / base 271. Required checks are satisfied. No new live Chrome run or full 20-shard queue run performed.

My preceding raw-hash, 21 Git-baseline and nine artifact-origin checks remain applicable; not all 423 artifacts were refetched. The remaining disclosed zoom/placement/nesting-specific future detection trade-off is unchanged. No merge, queue or dispatch.

Verdict: APPROVE
Reasoning: The successor restores the demonstrated historical PR-gate detection loss without changing retained baselines, and exact-head executed accuracy CI is green.

— Magi (desktop-review)