Skip to content

test(inbox): pixel and DOM parity harness for the thread - #2222

Merged
backnotprop merged 5 commits into
mainfrom
test/inbox-thread-parity-harness
Oct 27, 2026
Merged

backnotprop merged 5 commits into
mainfrom
test/inbox-thread-parity-harness

Conversation

@backnotprop

@backnotprop backnotprop commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Slice 0 of the Inbox thread extract (plan record, not in this repo): a reproducible pixel and DOM parity harness for the thread. Test-only: no product code changes. Not part of test:e2e:inbox or CI.

One command

bun run test:parity:inbox-thread            # origin/main (reference) vs this checkout (candidate)
#   --base <ref> | --base-binary <path>      # the reference
#   --candidate-binary <path>               # skip building this checkout
#   --out <dir>  --keep
  1. Binaries. The reference is a dev build (__CLI_VERSION__="0.0.0-dev", built review then hook) of --base. It is built in a temporary git worktree and cached in .local/parity/bin/base-<sha>/plannotator. The candidate is a dev build of this checkout. Each binary must be named plannotator, because provenInbox requires it.

  2. One seed. The reference binary seeds one scratch data dir (tests/e2e/inbox-thread-parity-seed.ts) through the real routes, then stops. The seed has:

    • the 7-message folding thread (answered questions, an annotated file, a guided review, one new message);
    • open, picked and text questions, with a reply draft;
    • a replaced question and a Re-ask;
    • image, SVG, markdown, HTML and video files;
    • action items, and a 👍 settled and delivered to Pi;
    • a reply that tags a second agent (per-recipient delivery lines);
    • right-to-left and long-token text;
    • a link choice answered (today's raw markdown, which Inbox: show a markdown choice label's words, not its markdown, in folded lines and answer rows #2219 will change) beside picture choices.

    The seed also records the sessions that asked, for the live captures (step 3).

  3. Captures (tests/e2e/inbox-thread-parity.config.ts). Every project copies the seed and starts its own Inbox on the copy. The projects:

    • desktop 1440×900 in light and in dark;
    • the thread pane at 360 px in light and in dark, and at 320 px (a host's sidebar, for the shared parts' container rules);
    • an iPhone 15 in WebKit, in light and in dark;
    • a Pixel 7 in Chromium.

    The page clock is fixed at the seed's frozenAt. The locale is en-US and the time zone is America/New_York. Motion is reduced, and animations and the caret are off. Fonts and images are loaded and decoded before each capture; WebKit also waits out its half-second high-quality image repaint. Chromium runs without the GPU and without partial raster. PLANNOTATOR_INBOX_NOTES=0 keeps the note offer away, because the scratch HOME has a ~/.claude (the plugin registry, for the restart line). In order, the harness captures:

    • the list's rows;
    • per thread: the landing, every page from the top, its tooltips on a desktop (a time's full moment, a folded line's time and answers, the 👍 pill), the Files band open, Expand all, the open questions list and the Artifacts view;
    • the list again, with Quiet open and each delivery tick's tooltip (Delivered included);
    • the list and each thread 3 days and 400 days later (the other-day and other-year time branches);
    • a 👍 inside its Undo window (its real-clock times rewritten to 12:00 PM for that capture);
    • live delivery, last: the spec polls the bridge as three seeded sessions (Claude Code busy with Send now: Undelivered and Send now; Claude Code busy on plugin 0.28.23 with 0.28.25 installed: the restart line; Pi busy). It sets the page clock 90 s after the Inbox's own wait clock, so both runs read "Undelivered 1 min". This step runs last because the Inbox remembers a connection for a while after it stops.
  4. Comparison. Screenshots must match at zero pixels (toHaveScreenshot, threshold: 0, maxDiffPixels: 0). The outerHTML of what each capture shows is normalized for the port, the server session, blob URLs, the HTML asset token and the scratch folder, and diffed file by file. Each step's page errors and console.error lines are diffed too (the sandboxed HTML attachment's "Blocked script execution" line is the sandbox working, and is left out). An actionTimeout makes a blocked click fail with its reason. The script writes report.md and report.json, with dom-diff/ and the actual, expected and diff PNGs. It exits 0 only when every capture is identical.

The sandbox is the same as the e2e specs: scratch HOME and PLANNOTATOR_DATA_DIR, NO_BROWSER, PLANNOTATOR_INBOX_CONNECT=0 and PLANNOTATOR_UPDATE_CHECK=0, with phones, the relay and update URLs pointed nowhere. It never touches ~/.plannotator. It is documented in docs/agent-guides/inbox-phones-and-security.md.

Proof

  • main vs main, a dev build of 86538b3 (0.28.32) against itself: 709 screenshots, 0 differing; 829 DOM and error files, 0 differing. On e789a3d it was identical twice with 708 and 828. (The first version, with 199 and 239, was identical three times, once under load.)
  • Known WebKit noise. WebKit now and then blends the one pixel row where the thread's scroller clips a picture tile a few colour levels differently, even main against itself. It shows as 12 to 132 pixels on that row only, with the DOM matching. Repainting the scrollers before each capture made it rarer. The guide says to rerun before reading such a capture as a change.
  • Deliberate change (a temporary, uncommitted build: .ib-newdiv gap 10px → 11px plus a data-parity-probe attribute on the divider): 91 screenshots differing (510 px each on desktop) and 133 DOM files differing. Exit 1. The first version caught 21 and 46.
  • Noise found and fixed in the second version: the note offer dialog, the 👍's real-clock times, the sandboxed srcdoc console line, and WebKit's image repaint (see above).
  • The first version caught real noise, which is now fixed: a per-run HTML asset token (now normalized), and anti-aliasing on an image tile's corners (Chromium now runs without the GPU and without partial raster, and images are decoded before each capture).
  • Guard tests pass over the new files: never-touch-the-real-inbox, no-browser-sentinel, child-env-guard and agents-md-size.

One command (bun run test:parity:inbox-thread) builds a dev binary of
origin/main as the reference and one of this checkout as the candidate,
seeds one scratch data dir with the reference binary, and captures every
seeded thread with both, each project on its own copy of the seed: desktop
light and dark, the thread pane at 360 px, an iPhone 15 in WebKit and a
Pixel 7 in Chromium. Zero-pixel toHaveScreenshot plus a normalized
outerHTML diff; report.md and report.json under .local/parity/.

Test-only: no product code changes.
…ps and later clocks

- Live delivery, last: the spec polls the bridge as three seeded sessions
  (Claude Code busy with Send now; Claude Code busy on an older plugin, with a
  plugin registry in the scratch HOME; Pi busy) and sets the page clock from
  the Inbox's own wait clock, so Undelivered, Send now and the restart line
  read the same minutes in both runs.
- The list's rows (start, end with Quiet open, later clocks) are captured too.
- Tooltips on a desktop: a time's whole moment, a folded line's time and
  answers, the thumbs-up pill, each delivery tick in the list.
- The Files band open; a thumbs-up inside its Undo window (its real-clock
  times rewritten to 12:00 PM for the capture).
- Later clocks: the list and each thread 3 and 400 days on.
- Seed: a link choice answered (today's raw markdown) beside picture choices;
  the thumbs-up delivered to Pi.
- Projects: + the pane at 360 px dark, at 320 px, iPhone 15 dark.
- console.error is recorded (the sandboxed srcdoc's blocked-script line is
  the sandbox working and is left out); an actionTimeout so a blocked click
  fails with its reason; PLANNOTATOR_INBOX_NOTES=0 (the scratch HOME now has a
  ~/.claude); WebKit waits out its low-quality image repaint.

main vs main: 708 screenshots and 828 DOM files, 0 differing.
…ure; names its clip-row noise

WebKit can blend the one pixel row where the thread's scroller clips a picture
tile a few colour levels differently from run to run (main against itself too).
A full repaint of the scrollers after the image-quality wait makes it rarer;
the guide says what it looks like (that row only, the DOM matching).
@backnotprop
backnotprop merged commit 7e4ab4c into main Oct 27, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant