diff --git a/docs/agent-guides/inbox-phones-and-security.md b/docs/agent-guides/inbox-phones-and-security.md index df5a3166e..15d7ce43e 100644 --- a/docs/agent-guides/inbox-phones-and-security.md +++ b/docs/agent-guides/inbox-phones-and-security.md @@ -74,6 +74,7 @@ PNGs land in `.local/proof/`. Every spec makes its temp folders with `inboxScratch` and ends with `removeInboxScratch` in a `finally` (`tests/e2e/inbox-scratch.ts`); a spec that stops its own Inbox mid-test uses `stopScratchInbox`. Both signal only an Inbox proven to be the one the folder started (`provenInbox` in `tests/helpers/inbox-process.ts`, shared by every test): an entry remembered after each start or found in an `inbox.json` under the folder, whose command line is still `plannotator inbox` / `index.ts inbox`, and whose port answers health with that pid and `serverSession` (or, with nothing answering, whose working directory, `/proc` or `lsof`, is inside the folder). SIGTERM, then SIGKILL after re-proving, then the folder goes. +- The thread parity harness (`bun run test:parity:inbox-thread`, `scripts/inbox-thread-parity.ts`; not part of `test:e2e:inbox` or CI): proves that a change to how the thread draws (moving its parts into `@plannotator/ui` or `@plannotator/core`) changes no pixel and no DOM. It builds a dev binary (`__CLI_VERSION__="0.0.0-dev"`, the build order, review then hook) of `--base` (default `origin/main`, in a temporary git worktree, cached in `.local/parity/bin/base-/plannotator`) as the reference, and one of this checkout as the candidate (`--base-binary` / `--candidate-binary` take a built binary instead; each must be named `plannotator`, as `provenInbox` requires). The reference binary seeds one scratch data dir (`tests/e2e/inbox-thread-parity-seed.ts`: a folding thread with answered questions, an annotated file and a guided review; open, picked and text questions with a reply draft; a replaced question and a Re-ask; image, SVG, markdown, HTML and video files; action items; a ๐Ÿ‘ settled and delivered; a reply tagging a second agent; right-to-left and long-token text; a link choice answered beside picture choices). Then `tests/e2e/inbox-thread-parity.config.ts` runs twice. Each project copies the seed and starts its own Inbox on the copy, with `PLANNOTATOR_INBOX_NOTES=0` and a Claude Code plugin registry in its scratch HOME: desktop 1440 by 900 light and dark, the thread pane at 360 px light and dark and at 320 px (a host's sidebar), an iPhone 15 in WebKit light and dark, and a Pixel 7 in Chromium. The page clock is fixed at the seed's `frozenAt`, with `en-US` and `America/New_York`, reduced motion, animations and the caret off, and fonts and images loaded and decoded. It captures the list's rows, then for each thread the landing, every page from the top, its tooltips on a desktop (a time's whole moment, a folded line's time and answers, the ๐Ÿ‘ pill), the Files band open, Expand all, the open questions list and the Artifacts view; the list again with the Quiet band open and each delivery tick's tooltip; the list and each thread three days and 400 days later; and a ๐Ÿ‘ inside its Undo window. Last, live delivery, which lives in the Inbox's memory and no copy carries: the spec polls the bridge as three seeded sessions (Claude Code busy with Send now: Undelivered and Send now; Claude Code busy on an older plugin: the restart line; Pi busy), reads the Inbox's own wait clock and sets the page clock 90 s after it, so the minutes read the same in both runs. The reference run writes the PNGs; the candidate run compares them at zero pixels (`threshold: 0`). Both write the outerHTML of what each capture shows, normalized for the port, the server session, blob URLs, the HTML asset token and the scratch folder, and each step's page errors and `console.error` lines. The script diffs those files and writes `report.md` and `report.json`, with `dom-diff/` and the candidate's actual, expected and diff PNGs, to `.local/parity//` (or `--out`). It exits 0 only when everything is identical. Known noise: WebKit (the iPhone projects) now and then blends the one pixel row where the thread's scroller clips a picture tile a few colour levels differently, even main against itself; such a capture differs by a dozen to about 130 pixels on that row while its DOM matches, so rerun before reading it as a change. `--keep` keeps the seed folder. It runs under the same sandbox as the specs: a scratch HOME and data dir, `NO_BROWSER`, `PLANNOTATOR_INBOX_CONNECT=0`, `PLANNOTATOR_UPDATE_CHECK=0`, and phones and the relay off. - The connections: - Claude Code: `apps/hook/hooks/mod/inbox.test.ts`, run by `.github/workflows/inbox-claude-code.yml`. - Pi: `apps/pi-extension/inbox.test.ts`, the real Pi in RPC mode, plus the 0.79.1 floor in `.github/workflows/inbox-pi.yml`. diff --git a/package.json b/package.json index 00ad7dd83..03eba7305 100644 --- a/package.json +++ b/package.json @@ -40,7 +40,8 @@ "test": "bun test", "typecheck": "bash apps/pi-extension/vendor.sh && tsc --noEmit -p packages/core/tsconfig.json && tsc --noEmit -p packages/shared/tsconfig.json && tsc --noEmit -p packages/ai/tsconfig.json && tsc --noEmit -p packages/server/tsconfig.json && tsc --noEmit -p packages/ui/tsconfig.json && tsc --noEmit -p packages/guide-viewer/tsconfig.json && tsc --noEmit -p apps/guides-show/tsconfig.json && tsc --noEmit -p apps/relay/tsconfig.json && tsc --noEmit -p apps/telemetry/tsconfig.json && tsc --noEmit -p packages/ui/tsconfig.strict-consumer.json && tsc --noEmit -p apps/pi-extension/tsconfig.json && tsc --noEmit -p apps/hook/hooks/mod/tsconfig.json && tsc --noEmit -p apps/hook/tsconfig.server.json && tsc --noEmit -p packages/inbox/tsconfig.json && tsc --noEmit -p apps/inbox/tsconfig.json && tsc --noEmit -p packages/t3-code/tsconfig.json", "build:ui-css": "bun run --cwd packages/ui build:css", - "test:e2e:inbox": "playwright test -c tests/e2e/playwright.config.ts" + "test:e2e:inbox": "playwright test -c tests/e2e/playwright.config.ts", + "test:parity:inbox-thread": "bun scripts/inbox-thread-parity.ts" }, "dependencies": { "@anthropic-ai/claude-agent-sdk": "^0.3.303", diff --git a/scripts/inbox-thread-parity.ts b/scripts/inbox-thread-parity.ts new file mode 100644 index 000000000..5adc69599 --- /dev/null +++ b/scripts/inbox-thread-parity.ts @@ -0,0 +1,226 @@ +/** + * The Inbox thread parity harness, one command: + * + * bun run test:parity:inbox-thread [--base ] [--base-binary ] [--candidate-binary ] [--out ] [--keep] + * + * 1. The reference binary: a dev build (`__CLI_VERSION__="0.0.0-dev"`) of + * `--base` (default origin/main) in a temporary git worktree, cached in + * .local/parity/bin/base-/plannotator; or `--base-binary`. + * 2. The candidate binary: a dev build of this checkout (review, hook, then + * compile; .local/parity/bin/candidate/plannotator); or `--candidate-binary`. + * 3. One seeded data dir (tests/e2e/inbox-thread-parity-seed.ts), written by + * the reference binary in a scratch folder. + * 4. tests/e2e/inbox-thread-parity.config.ts twice: the reference run writes + * the PNGs and the threads' outerHTML; the candidate run compares at zero + * pixels and writes its own outerHTML. Each project copies the seed and + * starts its own Inbox on the copy. + * 5. The outerHTML diffed file by file; report.md and report.json in the + * evidence folder (default .local/parity//). Exit 0 only when + * every screenshot and every DOM file is identical. + * + * Sandboxed like every Inbox test: scratch HOME and PLANNOTATOR_DATA_DIR, + * PLANNOTATOR_BROWSER from tests/helpers/no-browser.ts, agent connect, + * update checks, phones and the relay off. Never the real ~/.plannotator. + */ + +import { spawnSync, type SpawnSyncOptions } from 'node:child_process'; +import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join, relative, resolve } from 'node:path'; +import { seedThreadParity } from '../tests/e2e/inbox-thread-parity-seed'; +import { inboxScratch, rememberInboxOf, removeInboxScratch, stopScratchInbox } from '../tests/e2e/inbox-scratch'; + +const repo = resolve(import.meta.dir, '..'); +const args = process.argv.slice(2); +const flag = (name: string) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const keep = args.includes('--keep'); + +function sh(cmd: string[], options: SpawnSyncOptions = {}): string { + const result = spawnSync(cmd[0]!, cmd.slice(1), { cwd: repo, encoding: 'utf8', maxBuffer: 1 << 28, ...options }); + if (result.status !== 0) throw new Error(`${cmd.join(' ')} failed (${result.status}):\n${result.stderr ?? ''}${result.stdout ?? ''}`); + return String(result.stdout ?? '').trim(); +} + +function log(line: string): void { + process.stdout.write(`[parity] ${line}\n`); +} + +/** The build order (review, then hook), then a dev compile, in `dir`. */ +function build(dir: string, outfile: string): void { + mkdirSync(dirname(outfile), { recursive: true }); + const loud: SpawnSyncOptions = { cwd: dir, stdio: ['ignore', 'ignore', 'inherit'] }; + sh(['bun', 'run', '--cwd', 'apps/review', 'build'], loud); + sh(['bun', 'run', 'build:hook'], loud); + sh(['bun', 'build', 'apps/hook/server/index.ts', '--compile', '--no-compile-autoload-bunfig', '--define', '__CLI_VERSION__="0.0.0-dev"', '--outfile', outfile], loud); +} + +function baseBinary(): { binary: string; label: string } { + const given = flag('--base-binary'); + if (given) return { binary: resolve(given), label: given }; + const ref = flag('--base') ?? 'origin/main'; + if (ref.startsWith('origin/')) sh(['git', 'fetch', '-q', 'origin']); + const sha = sh(['git', 'rev-parse', `${ref}^{commit}`]); + const binary = join(repo, '.local', 'parity', 'bin', `base-${sha.slice(0, 12)}`, 'plannotator'); + if (existsSync(binary)) { + log(`reference: ${ref} ${sha.slice(0, 12)} (cached ${relative(repo, binary)})`); + return { binary, label: `${ref} ${sha}` }; + } + const src = join(tmpdir(), `plannotator-parity-src-${sha.slice(0, 12)}`); + rmSync(src, { recursive: true, force: true }); + log(`reference: building ${ref} ${sha.slice(0, 12)} in ${src}`); + sh(['git', 'worktree', 'add', '-q', '--detach', src, sha]); + try { + sh(['bun', 'install', '--frozen-lockfile'], { cwd: src, stdio: ['ignore', 'ignore', 'inherit'] }); + build(src, binary); + } finally { + sh(['git', 'worktree', 'remove', '--force', src]); + } + return { binary, label: `${ref} ${sha}` }; +} + +function candidateBinary(): { binary: string; label: string } { + const given = flag('--candidate-binary'); + if (given) return { binary: resolve(given), label: given }; + const binary = join(repo, '.local', 'parity', 'bin', 'candidate', 'plannotator'); + const head = sh(['git', 'rev-parse', 'HEAD']); + const dirty = sh(['git', 'status', '--porcelain']) !== ''; + log(`candidate: building this checkout (${head.slice(0, 12)}${dirty ? ', with uncommitted changes' : ''})`); + build(repo, binary); + return { binary, label: `this checkout ${head}${dirty ? ' (uncommitted changes)' : ''}` }; +} + +function capture(role: 'reference' | 'candidate', binary: string, seedRoot: string, out: string): number { + log(`${role}: capturing with ${binary}`); + const result = spawnSync(join(repo, 'node_modules', '.bin', 'playwright'), ['test', '-c', 'tests/e2e/inbox-thread-parity.config.ts'], { + cwd: repo, + stdio: 'inherit', + env: { ...process.env, PLANNOTATOR_PARITY_SEED: seedRoot, PLANNOTATOR_PARITY_OUT: out, PLANNOTATOR_PARITY_ROLE: role, PLANNOTATOR_E2E_BINARY: binary }, + }); + return result.status ?? 1; +} + +function files(dir: string, prefix = ''): string[] { + if (!existsSync(dir)) return []; + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => (entry.isDirectory() ? files(join(dir, entry.name), `${prefix}${entry.name}/`) : [`${prefix}${entry.name}`])); +} + +interface Failure { + test: string; + project: string; + message: string; +} + +/** The candidate run's failed expectations, from Playwright's JSON report. */ +function failures(results: string): Failure[] { + if (!existsSync(results)) return [{ test: '(the run)', project: '', message: 'no results.json' }]; + const report = JSON.parse(readFileSync(results, 'utf8')) as { suites: Suite[]; errors?: { message?: string }[] }; + type Suite = { title: string; specs?: { title: string; tests: { projectName: string; results: { status: string; errors?: { message?: string }[] }[] }[] }[]; suites?: Suite[] }; + const out: Failure[] = (report.errors ?? []).map((e) => ({ test: '(global)', project: '', message: e.message ?? '' })); + const walk = (suite: Suite) => { + for (const spec of suite.specs ?? []) { + for (const t of spec.tests) { + for (const r of t.results) { + for (const e of r.errors ?? []) out.push({ test: spec.title, project: t.projectName, message: (e.message ?? '').replace(/\u001b\[[0-9;]*m/g, '') }); + if (r.status !== 'passed' && (r.errors ?? []).length === 0) out.push({ test: spec.title, project: t.projectName, message: r.status }); + } + } + } + for (const child of suite.suites ?? []) walk(child); + }; + for (const suite of report.suites) walk(suite); + return out; +} + +/** The snapshot a screenshot failure names ("Snapshot: folding-land.png", or the missing file's path). */ +function shotOf(message: string): string { + return /Snapshot: (\S+)/.exec(message)?.[1] ?? /png\/[^/]+\/(\S+?\.png)/.exec(message)?.[1] ?? '(unknown)'; +} + +/** "510 pixels (ratio 0.01 of all image pixels) are different", or the first line. */ +function pixelsOf(message: string): string { + return /\d+ pixels \(ratio [^)]*\) are different/.exec(message)?.[0] ?? /Expected an image \d+px by \d+px, received \d+px by \d+px[^.]*/.exec(message)?.[0] ?? message.split('\n')[0]!; +} + +async function main(): Promise { + const stamp = new Date().toISOString().replace(/[:.]/g, '-'); + const out = resolve(flag('--out') ?? join(repo, '.local', 'parity', stamp)); + if (existsSync(out)) rmSync(out, { recursive: true }); + mkdirSync(out, { recursive: true }); + const base = baseBinary(); + const candidate = candidateBinary(); + + const seedRoot = inboxScratch('plannotator-thread-parity-seed-'); + try { + log(`seeding ${seedRoot} with the reference binary`); + const seed = await seedThreadParity(base.binary, seedRoot, async (dataDir) => { + rememberInboxOf(dataDir); + await stopScratchInbox(dataDir); + }); + writeFileSync(join(out, 'seed.json'), JSON.stringify(seed, null, 2)); + const referenceStatus = capture('reference', base.binary, seedRoot, out); + if (referenceStatus !== 0) throw new Error(`the reference run failed (exit ${referenceStatus}); see above. Nothing was compared.`); + capture('candidate', candidate.binary, seedRoot, out); + } finally { + if (keep) log(`kept the seed at ${seedRoot}`); + else await removeInboxScratch(seedRoot); + } + + // Screenshots: every reference PNG was compared (or is missing in the candidate run). + const pngs = files(join(out, 'reference', 'png')).filter((f) => f.endsWith('.png')); + const failed = failures(join(out, 'candidate', 'results.json')); + const pixel = failed.filter((f) => /toHaveScreenshot|Screenshot comparison failed|snapshot doesn't exist|pixels/.test(f.message)); + const other = failed.filter((f) => !pixel.includes(f)); + + // outerHTML (and each thread's page errors): every file, both ways. + const refDom = join(out, 'reference', 'dom'); + const candDom = join(out, 'candidate', 'dom'); + const names = [...new Set([...files(refDom), ...files(candDom)])].sort(); + const domDiffs: string[] = []; + const diffDir = join(out, 'dom-diff'); + for (const name of names) { + const a = join(refDom, name); + const b = join(candDom, name); + if (existsSync(a) && existsSync(b) && readFileSync(a, 'utf8') === readFileSync(b, 'utf8')) continue; + domDiffs.push(name); + const diff = spawnSync('diff', ['-u', existsSync(a) ? a : '/dev/null', existsSync(b) ? b : '/dev/null'], { encoding: 'utf8', maxBuffer: 1 << 28 }).stdout; + mkdirSync(dirname(join(diffDir, `${name}.diff`)), { recursive: true }); + writeFileSync(join(diffDir, `${name}.diff`), diff); + } + + const ok = pixel.length === 0 && other.length === 0 && domDiffs.length === 0; + const summary = { + ok, + reference: base.label, + candidate: candidate.label, + screenshots: { compared: pngs.length, differing: pixel.length }, + dom: { compared: names.length, differing: domDiffs.length }, + otherFailures: other.length, + }; + const pixelShots = pixel.map((f) => ({ project: f.project, snapshot: shotOf(f.message), difference: pixelsOf(f.message) })); + writeFileSync(join(out, 'report.json'), JSON.stringify({ ...summary, pixel: pixelShots, other, domDiffs }, null, 2)); + const lines = [ + '# Inbox thread parity', + '', + `- Reference: ${base.label}`, + `- Candidate: ${candidate.label}`, + `- Screenshots: ${pngs.length} compared at zero pixels, ${pixel.length} differing`, + `- outerHTML and page errors: ${names.length} files compared, ${domDiffs.length} differing`, + `- Other failures: ${other.length}`, + `- Result: ${ok ? 'IDENTICAL' : 'DIFFERENT'}`, + '', + ...(pixel.length ? ['## Differing screenshots (actual, expected and diff PNGs in candidate/test-results/)', '', ...pixel.map((f) => `- ${f.project}/${shotOf(f.message)}: ${pixelsOf(f.message)}`), ''] : []), + ...(domDiffs.length ? ['## Differing DOM files (unified diffs in dom-diff/)', '', ...domDiffs.map((n) => `- ${n}`), ''] : []), + ...(other.length ? ['## Other failures', '', ...other.map((f) => `- ${f.project} โ€บ ${f.test}: ${f.message.split('\n')[0]}`), ''] : []), + ]; + writeFileSync(join(out, 'report.md'), `${lines.join('\n')}\n`); + process.stdout.write(`\n${lines.join('\n')}\nEvidence: ${out}\n`); + process.exit(ok ? 0 : 1); +} + +main().catch((error) => { + process.stderr.write(`${error instanceof Error ? error.stack : String(error)}\n`); + process.exit(2); +}); diff --git a/tests/e2e/inbox-thread-parity-seed.ts b/tests/e2e/inbox-thread-parity-seed.ts new file mode 100644 index 000000000..f58a35b84 --- /dev/null +++ b/tests/e2e/inbox-thread-parity-seed.ts @@ -0,0 +1,353 @@ +/** + * The seeded data dir of the thread parity harness (scripts/inbox-thread-parity.ts): + * one Inbox, filled once through its real routes (agents through `plannotator + * inbox mcp`, the person through the window's own POSTs), then stopped. Each + * capture run copies `data/` and starts its own Inbox on the copy, so the + * reference binary and the branch binary read the same ids, timestamps, + * looks and picks. + * + * The threads cover what the message parts draw (the seed text comes from the + * specs that prove each part): a seven-message folding thread with answered + * questions, an annotated file, a guided review and one new message; open, + * picked and text questions with a reply draft; a replaced question and a + * Re-ask; image, SVG, markdown, HTML and video files; action items; a settled + * ๐Ÿ‘; a reply that tags a second agent; worst-case text (right to left, + * long tokens, a table, emoji); and a link choice answered (its raw markdown + * in the folded line and "You answered", as today) beside picture choices. + * + * Live connection state (busy, Undelivered, Send now, restart) lives in the + * Inbox's memory, never in the data dir, so a copy cannot carry it: the + * capture polls as the seeded sessions itself (`sessions` in seed.json). + * + * Writes `/seed.json`: the thread ids by name and `frozenAt`, the + * instant every capture's page clock is fixed to. + */ + +import { spawnSync } from 'node:child_process'; +import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import { SimAgent, scratchProject } from '../../scripts/inbox-sim'; +import { NO_BROWSER } from '../helpers/no-browser'; +import { refuseRealDataDir } from '../helpers/real-data-dir'; +import { markTourDecided } from '../helpers/tour-decided'; +import { mp4 } from '../helpers/video-fixtures'; + +const repo = resolve(__dirname, '../..'); + +export interface ParitySeed { + /** The scratch root: `data/` (the seeded data dir) and `src/` (the projects, read by absolute path). */ + root: string; + dataDir: string; + /** The page clock of every capture (ISO): two minutes after the last write. */ + frozenAt: string; + /** Thread ids by name, in capture order. */ + threads: Record; + /** The agent session that asked in a thread, by thread name (the live captures poll as it). */ + sessions: Record; + /** The binary that wrote the seed. */ + seededBy: string; +} + +/** The environment of every process the harness starts: sandboxed, no browser, no network, no agents. */ +export function parityEnv(home: string, dataDir: string): Record { + refuseRealDataDir(dataDir, 'parityEnv'); + return { + PATH: process.env.PATH ?? '/usr/bin:/bin', + HOME: home, + TMPDIR: process.env.TMPDIR ?? '/tmp', + PLANNOTATOR_DATA_DIR: dataDir, + PLANNOTATOR_BROWSER: NO_BROWSER, + PLANNOTATOR_INBOX_CONNECT: '0', + PLANNOTATOR_UPDATE_CHECK: '0', + PLANNOTATOR_INBOX_PHONES: '0', + // No offer to add a note to an agent's instructions (the scratch HOME has a ~/.claude for the restart line). + PLANNOTATOR_INBOX_NOTES: '0', + PLANNOTATOR_RELAY_URL: 'http://127.0.0.1:9', + PLANNOTATOR_UPDATE_API_URL: 'http://127.0.0.1:9/releases/latest', + PLANNOTATOR_INSTALL_SCRIPT_URL: 'http://127.0.0.1:9/install.sh', + }; +} + +const q = (prompt: string, choices: string[], extra: string[] = []) => [':::question', prompt, ...extra, '', ...choices.map((c) => `- [ ] ${c}`), ':::'].join('\n'); +const text = (prompt: string) => [':::question-text', prompt, ':::'].join('\n'); + +// The folding thread (inbox-thread-folding.spec.ts). +const ASK = [ + 'I looked at three ways to add search to the docs site. Short version: the site is static, about 240 pages, and every option works with the Astro build we already have.', + '', + '| Option | How it runs | Cost |', + '| --- | --- | --- |', + '| Pagefind | Index built at deploy, searched in the browser | ~1.6 MB index, no service |', + '| Algolia DocSearch | Hosted crawler and API | Free for open source, needs approval |', + '', + 'Two calls before I start:', + '', + ':::question', + 'Which search index should the docs site use for its first public release, given the **1.6 MB** index, the `astro build` step in CI and the review deadline this Friday?', + '', + '- [ ] Pagefind', + '- [ ] Algolia DocSearch', + '', + 'Recommended: Pagefind', + ':::', + '', + ':::question-multi', + 'Which sections go in the index?', + '', + '- [ ] Guides', + '- [ ] API reference', + '- [ ] Blog', + ':::', +].join('\n'); +const PLAN = ['Pagefind is wired into the build on a branch. Before I polish it, here is the plan. Annotate it if something is off.', '', q('Which shortcut opens search?', ['/ and โŒ˜K', 'โŒ˜K only'])].join('\n'); +const LATEST = [ + 'CI is green. The empty state now offers the closest section and a GitHub search link.', + '', + 'One number changed my mind about deploys: the index is **1.8 MB** across 312 files, and building it on every deploy adds about **40 seconds** to each preview build.', + '', + '- The index is uploaded with the site, never served from elsewhere.', + '- The panel keeps the last query when it reopens.', + '', + q('When should the search index build?', ['On main and release tags', 'On every deploy'], ['', 'Stopped: the docs-search branch cannot merge without this.']), +].join('\n'); + +// Replaced and Re-ask (inbox-replaced.spec.ts, inbox-reask.spec.ts). +const WORKER = [ + 'I checked the 409 row against the Stripe docs. Two things before I go on.', + '', + q('Which way should the worker go on a Stripe 409?', ['Retry with the same idempotency key', 'Fail the job and send it to the dead-letter queue'], ['Stopped: the retry worker cannot ship without this.']), + '', + q('Ship the retry behind a flag?', ['Yes', 'No'], ['', 'The flag would be `billing.retry_409`, off by default.']), +].join('\n'); +const QUEUE = [ + 'You moved the 409 handling into the jobs queue in our session, so my worker question no longer fits. This is the one the queue needs.', + '', + q('How many times should the jobs queue retry a Stripe 409 before it dead-letters the job?', ['3 times over 10 minutes', 'Once, then dead-letter', 'Never retry; dead-letter at once'], ['', 'Stopped: the queue change cannot merge without this.']), +].join('\n'); + +// Action items (inbox-actions.spec.ts). +const ACTIONS = [ + 'These are secrets, so only you can do them.', + '', + ':::action', + 'Set the GitHub App client secret on staging', + 'On the App\'s page, click "Generate a new client secret" and copy it.', + 'Where: https://github.com/organizations/acme/settings/apps/appnotes-staging', + 'Run: `pbpaste | npx wrangler secret put APPNOTES_GITHUB_APP_CLIENT_SECRET --env staging`', + 'Holds up: staging sign-in with GitHub', + "Watching: the staging secret list; I'll see it appear", + ':::', + '', + ':::action', + 'Approve the staging deploy on Vercel', + 'Where: [Vercel](https://vercel.com/acme/appnotes/deployments)', + 'Stopped: release 4.2 cannot go out until it is approved', + ':::', + '', + q('Publish 0.29 as latest, or as next first?', ['latest', 'next first'], ['', 'Recommended: next first']), +].join('\n'); + +// Worst-case text (inbox-mobile.spec.ts, inbox-worst-case.spec.ts). Made-up strings, nothing like a real key. +const LONG_URL = 'https://hooks.example.com/stripe/v1/refunds/reconcile/billing-svc/production/us-east-1/account-0001-placeholder-0002-placeholder-0003?sig=9f8e7d6c5b4a3928170615243342516071829304a5b6c7d8e9f0&replay=true'; +const WORST = [ + `The webhook endpoint is ${LONG_URL}`, + '', + `Rotate the key \`example_key_${'a1B2c3D4e5F6g7H8'.repeat(5)}\` before Friday. Raw token: example_token_${'0123456789abcdef'.repeat(6)}`, + '', + 'ืฉืœื•ื! ื”ืกื ื›ืจื•ืŸ ื”ืœื™ืœื™ ืฉืœ ืกืคืจ ื”ื—ืฉื‘ื•ื ื•ืช ื ื›ืฉืœ ืคืขืžื™ื™ื ื”ืœื™ืœื”. ู…ุฑุญุจุงุŒ ูุดู„ุช ุงู„ู…ุฒุงู…ู†ุฉ ุงู„ู„ูŠู„ูŠุฉ ู…ุฑุชูŠู†. ๐Ÿš€๐Ÿ”ฅโœ… ๐Ÿ‘ฉ๐Ÿฝโ€๐Ÿ’ป', + '', + '| Region | Failed payouts | Oldest |', + '| --- | ---: | --- |', + '| us-east-1 | 1,204 | 3 days |', + '| eu-central-1 | 17 | 2 hours |', + '', + '```ts', + "export const RETRY_POLICY = { attempts: 5, backoff: 'exponential', maxDelayMs: 3_600_000, deadLetter: 'refunds-dlq-production-us-east-1' } as const;", + '```', + '', + q('Which idempotency key format should the retry worker use?', [`idem_01HZX9QW3E5R7T9Y1U3I5O7P9A1S3D5F7G9H1J3K5L7Z9X1C3V5B7N9M1Q3W5E7R9T1Y3U5I7O9P1A3S5D7F9G1H3J5K7L9`, 'charge id plus attempt number'], ['Stopped: the retry worker cannot ship without this.']), +].join('\n'); + +// Link choices (answered: today's folded line and "You answered" keep the raw markdown) and picture choices +// (inbox-picture-choices.spec.ts). +const LINKS = ['Two libraries fit. Pick one and I will wire it in.', '', q('Which search library should the docs use?', ['[Pagefind](https://pagefind.app)', '[Algolia DocSearch](https://docsearch.algolia.com)'])].join('\n'); +const DRAWERS = [ + 'Two drawer designs for checkout.', + '', + ':::question', + 'Which checkout drawer should ship?', + 'Stakes: ships to all of checkout on Thursday; changing it later costs a week of QA.', + 'Shows: ![Both drawers side by side](shots/both.png)', + 'Reversible: no โ€” the old drawer code is deleted with the release', + '', + '- [ ] A ยท Total on top โ€” ![Drawer A](shots/drawer-a.png) closest to today', + '- [ ] B ยท Total pinned with Pay โ€” ![Drawer B](shots/drawer-b.png) the total never scrolls away', + '', + 'Recommended: B ยท Total pinned with Pay', + '', + 'Why: on a 320 px screen only B keeps the total next to the button.', + ':::', +].join('\n'); + +type Sent = { message_id: string; thread_id: string; questions: { key: string; prompt: string; kind: string }[]; attachments?: { id: string }[] }; +const answer = (question: Sent['questions'][number], selected: string[]) => ({ key: question.key, revision: 0, answer: { v: 1, key: question.key, kind: question.kind, prompt: question.prompt, selected } }); + +async function until(what: string, check: () => Promise, ms = 60_000): Promise { + const deadline = Date.now() + ms; + while (Date.now() < deadline) { + if (await check()) return; + await new Promise((r) => setTimeout(r, 250)); + } + throw new Error(`seed: timed out waiting for ${what}`); +} + +/** Fill a new Inbox under `root/data` with `binary`; stop it; return the manifest (also written to root/seed.json). */ +export async function seedThreadParity(binary: string, root: string, stop: (dataDir: string) => Promise): Promise { + const dataDir = join(root, 'data'); + const home = join(root, 'seed-home'); + mkdirSync(home, { recursive: true }); + markTourDecided(dataDir); + const env = parityEnv(home, dataDir); + const started = spawnSync(binary, ['inbox', '--background'], { env, cwd: root, encoding: 'utf8', timeout: 60_000 }); + if (started.status !== 0) throw new Error(`seed: inbox --background failed: ${started.stderr}`); + const url = started.stdout.trim().replace(/\/$/, ''); + const post = async (path: string, body: unknown): Promise> => { + const response = await fetch(`${url}${path}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); + if (!response.ok) throw new Error(`seed: POST ${path} answered ${response.status}: ${await response.text()}`); + return (await response.json()) as Record; + }; + const get = async (path: string): Promise> => (await (await fetch(`${url}${path}`)).json()) as Record; + // The one-time notification line is not what these captures are about. + await post('/api/inbox/settings', { notifications: { dismissed: true } }); + + const docs = scratchProject(join(root, 'src'), 'docs-site'); + const billing = scratchProject(join(root, 'src'), 'billing-svc'); + writeFileSync(join(docs, 'search-plan.md'), '# Search plan\n\nThe index builds after `astro build`, in CI only.\n\nResults are grouped by section.\n'); + mkdirSync(join(billing, 'docs'), { recursive: true }); + mkdirSync(join(billing, 'shots'), { recursive: true }); + writeFileSync(join(billing, 'docs', 'retry-policy.md'), '# Retry policy\n\nFailed refunds are retried with exponential backoff, five times at most.\n\n| Attempt | Delay |\n| --- | --- |\n| 1 | 1 min |\n| 2 | 5 min |\n'); + writeFileSync(join(billing, 'docs', 'retry-dashboard.html'), 'Retry dashboard

Retry dashboard

1,204 failed payouts in us-east-1.

\n'); + writeFileSync(join(billing, 'docs', 'refund-states.svg'), '\n'); + for (const name of ['retry-policy-desktop.png', 'both.png', 'drawer-a.png', 'drawer-b.png']) copyFileSync(join(repo, 'packages/review-editor/assets/pr-artifact-icons/image.png'), join(billing, 'shots', name)); + writeFileSync(join(billing, 'shots', 'retry-walkthrough.mp4'), mp4()); + + const agents: SimAgent[] = []; + const agent = async (name: string, host: string) => { + const made = await SimAgent.connect({ binary, env, name, host }); + agents.push(made); + return made; + }; + const threads: Record = {}; + const sessions: Record = {}; + const sessionOf = async (threadId: string) => ((await get(`/api/inbox/threads/${threadId}`)).thread.messages[0].author.session as string) ?? ''; + try { + // 1. Folding: seven messages, questions answered, an annotated file, a guided review, one new. + const claude = await agent('Claude Code', 'claude-code'); + const ask = (await claude.send({ project_path: docs, subject: 'Search for the docs site', body: ASK })) as Sent; + threads.folding = ask.thread_id; + const [index, sections] = ask.questions; + await post(`/api/inbox/messages/${ask.message_id}/reply`, { idempotency_key: 'r1', questions: [answer(index!, ['Pagefind']), answer(sections!, ['Guides', 'API reference'])] }); + const plan = (await claude.send({ project_path: docs, body: PLAN, reply_to: ask.message_id, attachments: ['search-plan.md'] })) as Sent; + for (const id of ['ann-1', 'ann-2']) { + await post('/api/inbox/annotations', { + attachment_id: plan.attachments![0]!.id, + version: 'current', + annotation: { id, type: 'COMMENT', text: 'Show the section path on each result.', originalText: 'Results are grouped by section.', blockId: 'block-1', startOffset: 0, endOffset: 0, createdA: 1 }, + }); + } + await post(`/api/inbox/messages/${plan.message_id}/reply`, { + idempotency_key: 'r2', + words: 'The plan is close. Two notes on the file.', + questions: [answer(plan.questions[0]!, ['/ and โŒ˜K'])], + feedback: '# Feedback on the attached files\n\nShow the section path on each result.', + annotation_ids: ['ann-1', 'ann-2'], + }); + const { example } = await claude.guideBrief(); + const guide = (await claude.submitGuide({ project_path: docs, ...example, body: 'Both notes are in: results show the section path. Ready for a read, file by file.', reply_to: ask.message_id })) as Sent; + await post(`/api/inbox/messages/${guide.message_id}/reply`, { idempotency_key: 'r3', words: 'Read it, looks good. One thing: an empty result says only "No results".' }); + await claude.readThread(ask.thread_id); + await post(`/api/inbox/threads/${ask.thread_id}/seen`, {}); + await claude.send({ project_path: docs, body: LATEST, reply_to: ask.message_id }); + + // 2. Questions: open (choice, multi, text), one picked into the draft, and a reply draft with words. + const asker = await agent('Claude Code', 'claude-code'); + const first = (await asker.send({ project_path: billing, subject: 'Retry worker for failed Stripe charges', body: `I'm adding a retry worker.\n\n${q('Which queue backs the retry worker?', ['Postgres (pg-boss)', 'Redis'])}` })) as Sent; + threads.questions = first.thread_id; + await asker.send({ project_path: billing, reply_to: first.message_id, body: `Rate limits next.\n\n${q('How many times should a rate-limited charge be retried?', ['3', '5', '10'], ['Holds up: the retry worker; the admin page'])}` }); + const flag = (await asker.send({ project_path: billing, reply_to: first.message_id, body: `The worker runs end to end in staging.\n\n${q('Ship the retry worker behind a flag?', ['Yes', 'No'])}\n\n${text('Anything the on-call runbook should say about the retry worker?')}` })) as Sent; + await post(`/api/inbox/messages/${flag.message_id}/picks`, { questions: [answer(flag.questions[0]!, ['Yes'])] }); + await post(`/api/inbox/threads/${first.thread_id}/draft`, { body: 'Postgres is fine for now; we already run pg-boss for the mailer.', base: null }); + + // 3. Replaced and Re-ask: question 1 replaced by the queue question; Re-ask on what stays open. + const replacer = await agent('Claude Code', 'claude-code'); + const worker = (await replacer.send({ project_path: billing, subject: 'Which way on a Stripe 409?', body: WORKER })) as Sent; + threads.replaced = worker.thread_id; + await replacer.send({ project_path: billing, body: QUEUE, reply_to: worker.message_id, supersedes: [`${worker.message_id}/${worker.questions[0]!.key}`] }); + await post(`/api/inbox/messages/${worker.message_id}/reask`, { idempotency_key: 'parity-reask' }); + + // 4. Files: an image, an SVG, markdown, HTML and a video, plus the image in the text. + const filer = await agent('OpenCode', 'opencode'); + const files = (await filer.send({ + project_path: billing, + subject: 'The retry policy, written down', + body: 'The policy and the dashboard are attached. Today\'s page, for reference:\n\n![Today\'s retry page](shots/retry-policy-desktop.png)\n\nThe walkthrough is the short video.', + attachments: ['docs/retry-policy.md', 'docs/retry-dashboard.html', 'docs/refund-states.svg', 'shots/retry-policy-desktop.png', 'shots/retry-walkthrough.mp4'], + })) as Sent; + threads.files = files.thread_id; + + // 5. Action items and a question beside them. + const actor = await agent('Claude Code', 'claude-code'); + threads.actions = ((await actor.send({ project_path: billing, subject: 'Staging secrets and the deploy', body: ACTIONS })) as Sent).thread_id; + + // 6. ๐Ÿ‘ Looks good, settled (sent to the agent once its Undo window passed) and delivered (below). + const pi = await agent('Pi', 'pi'); + const merged = (await pi.send({ project_path: docs, subject: 'Toast fix merged; CI is green', body: "Merged #1937. CI is green on `main` and the release branch picked it up.\n\nNothing needs you here. Next I'm looking at the stepping bug in full screen." })) as Sent; + threads.ack = merged.thread_id; + await post(`/api/inbox/messages/${merged.message_id}/ack`, { ack: true }); + + // 7. Delivery and tags: a reply the agent read (Delivered), one it has not (Sent) that tags another agent. + const talker = await agent('Claude Code', 'claude-code'); + const talk = (await talker.send({ project_path: billing, subject: 'Nightly reconciliation', body: 'Starting the nightly reconciliation work. I will post progress here.' })) as Sent; + threads.delivery = talk.thread_id; + await post(`/api/inbox/messages/${talk.message_id}/reply`, { idempotency_key: 'd1', words: 'Keep going, and tell me when batch 1 matches.' }); + await talker.readThread(talk.thread_id); + const batch = (await talker.send({ project_path: billing, reply_to: talk.message_id, body: 'Batch 1 done: 120 payouts checked, 1 mismatch, all timing.' })) as Sent; + const piSession = ((await get(`/api/inbox/threads/${merged.thread_id}`)).thread.messages[0].author.session as string) ?? ''; + await post(`/api/inbox/messages/${batch.message_id}/reply`, { idempotency_key: 'd2', words: 'Thanks. Pi, can you look at the one mismatch?', tags: [{ host: 'pi', session: piSession, label: 'Pi' }] }); + + // 8. Worst case: right to left, long tokens, a table, code, emoji, a long choice. + const worst = await agent('Codex', 'codex'); + threads.worst = ((await worst.send({ project_path: billing, subject: 'Webhook URL, key rotation and the nightly sync (ื ื›ืฉืœ / ูุดู„ุช)', body: WORST })) as Sent).thread_id; + + // 9. Choices: a link choice answered, then picture choices open. + const chooser = await agent('Claude Code', 'claude-code'); + const links = (await chooser.send({ project_path: billing, subject: 'Search library and the checkout drawer', body: LINKS })) as Sent; + threads.choices = links.thread_id; + const linkQuestion = ((await get(`/api/inbox/threads/${links.thread_id}`)).thread.messages[0].questions as { key: string; choices: { label: string }[] }[])[0]!; + await post(`/api/inbox/messages/${links.message_id}/reply`, { idempotency_key: 'c1', questions: [answer(links.questions[0]!, [linkQuestion.choices[0]!.label])] }); + await chooser.readThread(links.thread_id); + await chooser.send({ project_path: billing, body: DRAWERS, reply_to: links.message_id }); + + for (const name of ['delivery', 'replaced', 'ack']) sessions[name] = await sessionOf(threads[name]!); + + // The ๐Ÿ‘ settles once its Undo window has passed (the Inbox's sweep); wait so no copy settles it later. + await until('the ๐Ÿ‘ to settle', async () => { + const t = (await get(`/api/inbox/threads/${merged.thread_id}`)).thread as { messages: { id: string; ack?: { settled_at: string | null } | null }[] }; + return Boolean(t.messages.find((m) => m.id === merged.message_id)?.ack?.settled_at); + }); + // Pi picks it up: the pill reads "Delivered to Pi" and the row's tick is Delivered. + await pi.waitForReply(merged.thread_id, 20); + } finally { + await Promise.all(agents.map((a) => a.close().catch(() => {}))); + await stop(dataDir); + } + + const frozenAt = new Date(Math.ceil((Date.now() + 120_000) / 60_000) * 60_000).toISOString(); + const seed: ParitySeed = { root, dataDir, frozenAt, threads, sessions, seededBy: binary }; + writeFileSync(join(root, 'seed.json'), JSON.stringify(seed, null, 2)); + return seed; +} + +export function readParitySeed(root: string): ParitySeed { + return JSON.parse(readFileSync(join(root, 'seed.json'), 'utf8')) as ParitySeed; +} diff --git a/tests/e2e/inbox-thread-parity.config.ts b/tests/e2e/inbox-thread-parity.config.ts new file mode 100644 index 000000000..f8d23590c --- /dev/null +++ b/tests/e2e/inbox-thread-parity.config.ts @@ -0,0 +1,63 @@ +/** + * The thread parity harness's Playwright config (scripts/inbox-thread-parity.ts + * runs it twice: the reference binary writes the PNGs, the candidate binary is + * compared with them at zero pixels). Not part of `bun run test:e2e:inbox`. + * + * Env (set by the script): PLANNOTATOR_PARITY_SEED (the seeded scratch root), + * PLANNOTATOR_PARITY_OUT (the evidence folder), PLANNOTATOR_PARITY_ROLE + * (`reference` or `candidate`) and PLANNOTATOR_E2E_BINARY. + * + * Projects: desktop 1440 by 900 light and dark, the thread pane at 360 px + * light and dark and at 320 px (a host's sidebar, for the shared parts' + * container rules), an iPhone 15 in WebKit light and dark, and a Pixel 7 in + * Chromium. + */ +import { defineConfig, devices } from '@playwright/test'; +import { join, resolve } from 'node:path'; + +const out = resolve(process.env.PLANNOTATOR_PARITY_OUT ?? join(__dirname, '../../.local/parity/scratch')); +const role = process.env.PLANNOTATOR_PARITY_ROLE === 'reference' ? 'reference' : 'candidate'; +/** + * Chromium flags for repeatable pixels: no GPU, one colour profile, no LCD text, and whole-tile raster + * (partial raster let an image tile's rounded corners anti-alias differently from run to run). + */ +const CHROMIUM_ARGS = ['--use-mock-keychain', '--password-store=basic', '--font-render-hinting=none', '--disable-gpu', '--force-color-profile=srgb', '--disable-lcd-text', '--disable-partial-raster', '--disable-skia-runtime-opts']; +const desktop = { browserName: 'chromium' as const, viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1, launchOptions: { args: CHROMIUM_ARGS } }; + +export default defineConfig({ + testDir: '.', + testMatch: ['inbox-thread-parity.spec.ts'], + outputDir: join(out, role, 'test-results'), + // Both runs read the reference run's PNGs; only the reference run writes them. + snapshotPathTemplate: join(out, 'reference', 'png', '{projectName}', '{arg}{ext}'), + updateSnapshots: role === 'reference' ? 'all' : 'none', + fullyParallel: false, + workers: 1, + retries: 0, + timeout: 600_000, + expect: { + timeout: 30_000, + // Zero tolerance: any pixel of any colour counts. + toHaveScreenshot: { maxDiffPixels: 0, threshold: 0, animations: 'disabled', caret: 'hide', scale: 'device' }, + }, + reporter: [['list'], ['json', { outputFile: join(out, role, 'results.json') }]], + use: { + headless: true, + locale: 'en-US', + timezoneId: 'America/New_York', + reducedMotion: 'reduce', + serviceWorkers: 'block', + // A click that cannot happen fails with its reason instead of using up the whole test. + actionTimeout: 30_000, + }, + projects: [ + { name: 'desktop-light', use: { ...desktop, colorScheme: 'light' } }, + { name: 'desktop-dark', use: { ...desktop, colorScheme: 'dark' } }, + { name: 'narrow-360', use: { ...desktop, colorScheme: 'light' } }, + { name: 'narrow-360-dark', use: { ...desktop, colorScheme: 'dark' } }, + { name: 'narrow-320', use: { ...desktop, colorScheme: 'light' } }, + { name: 'iphone-15', use: { ...devices['iPhone 15'], browserName: 'webkit', colorScheme: 'light' } }, + { name: 'iphone-15-dark', use: { ...devices['iPhone 15'], browserName: 'webkit', colorScheme: 'dark' } }, + { name: 'pixel-7', use: { ...devices['Pixel 7'], browserName: 'chromium', colorScheme: 'light', launchOptions: { args: CHROMIUM_ARGS } } }, + ], +}); diff --git a/tests/e2e/inbox-thread-parity.spec.ts b/tests/e2e/inbox-thread-parity.spec.ts new file mode 100644 index 000000000..1fe81f44f --- /dev/null +++ b/tests/e2e/inbox-thread-parity.spec.ts @@ -0,0 +1,434 @@ +/** + * The thread parity harness's captures (run by scripts/inbox-thread-parity.ts + * through inbox-thread-parity.config.ts, never by `bun run test:e2e:inbox`). + * + * Each project (desktop light and dark, the pane at 360 px light and dark and + * at 320 px, iPhone 15 light and dark, Pixel 7) copies the seeded data dir + * (inbox-thread-parity-seed.ts) into its own scratch folder and starts its + * own Inbox on the copy with PLANNOTATOR_E2E_BINARY, so the reference binary + * and the candidate binary read the same ids, timestamps, looks and picks. + * The page clock is fixed at the seed's `frozenAt`, motion is reduced and + * animations and the caret are off; fonts and images are loaded and decoded + * before a capture. + * + * In order, every capture a zero-pixel `toHaveScreenshot` (the reference run + * writes the PNGs) plus the outerHTML of what it shows (normalized for the + * port, the server session, blob URLs and the HTML asset token), written for + * the script to diff: + * + * 1. the list's rows before any thread is opened; + * 2. per seeded thread: as it lands, page by page from the top, its hover + * tooltips (a desktop only: the whole moment of a time, a folded line's + * answers, the ๐Ÿ‘ pill), the Files band open, Expand all page by page, + * the open questions list, the Artifacts view; + * 3. the list's rows after every thread was looked at, and (a desktop only) + * each delivery tick's tooltip; + * 4. the list and each thread as it lands three days later and a year later + * (the other-day and other-year time branches); + * 5. a ๐Ÿ‘ inside its Undo window; + * 6. live delivery, last: the seeded sessions poll as the Claude Code mod and + * Pi do (busy with Send now: Undelivered and Send now; busy on an older + * plugin: the restart line; busy without Send now: queued), with the page + * clock set from the Inbox's own wait clock so the minutes read the same. + * + * Each thread's page errors and console errors are written too, and diffed. + */ + +import { expect, test, type Page, type TestInfo } from '@playwright/test'; +import { spawnSync } from 'node:child_process'; +import { cpSync, existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import { INBOX_BRIDGE_POLL_PATH, INBOX_SEND_NOW_POLL_FIELD } from '../../packages/shared/inbox/connection'; +import { parityEnv, readParitySeed, type ParitySeed } from './inbox-thread-parity-seed'; +import { inboxScratch, rememberInboxOf, removeInboxScratch } from './inbox-scratch'; + +const seedRoot = process.env.PLANNOTATOR_PARITY_SEED; +const out = resolve(process.env.PLANNOTATOR_PARITY_OUT ?? ''); +const role = process.env.PLANNOTATOR_PARITY_ROLE === 'reference' ? 'reference' : 'candidate'; +const binary = resolve(process.env.PLANNOTATOR_E2E_BINARY ?? ''); + +/** At most this many pages of one thread (a page is the scroller's height less an overlap). */ +const MAX_PAGES = 12; +/** The thread pane's width in the projects that stand for a host's sidebar. */ +const NARROW: Record = { 'narrow-360': 360, 'narrow-360-dark': 360, 'narrow-320': 320 }; +/** The seeded threads, in capture order (the list reads earlier threads as looked at). */ +const THREADS = ['folding', 'questions', 'replaced', 'files', 'actions', 'ack', 'delivery', 'worst', 'choices']; +/** The later clocks: another day this year, and another year. */ +const LATER = [ + { name: 'days-later', ms: 3 * 24 * 60 * 60_000 }, + { name: 'year-later', ms: 400 * 24 * 60 * 60_000 }, +]; +/** What a desktop hovers for its tooltip, in a thread at the top. */ +const TIPS = [ + { name: 'time', selector: 'section.ib-pane [data-full-when]' }, + { name: 'folded-time', selector: 'section.ib-pane button.ib-fline .ib-fl-tm' }, + { name: 'answers', selector: 'section.ib-pane button.ib-fline [data-snippet="answers"]' }, + { name: 'ack', selector: 'section.ib-pane .ib-ackwhen' }, +]; +/** A list row's delivery tick, by state: "Delivered โ€ฆ", "Sent โ€ฆ" (the list's tooltips). */ +const LIST_TIPS = ['delivered', 'sent', 'saved', 'undelivered', 'queued'].map((state) => ({ name: `tick-${state}`, selector: `.ib-lbody .ib-tick[data-tick="${state}"]` })); +const TOOLTIP = '[data-pn-tooltip]'; + +interface Run { + seed: ParitySeed; + url: string; + /** This project's scratch folder (its HOME and data dir). */ + root: string; + plugins: { installed: string; loaded: string }; + port: number; + token: string; + serverSession: string; +} + +let run: Run; + +test.beforeAll(async ({}, info) => { + expect(seedRoot, 'PLANNOTATOR_PARITY_SEED: run scripts/inbox-thread-parity.ts').toBeTruthy(); + expect(existsSync(binary), `the binary: ${binary}`).toBe(true); + const seed = readParitySeed(seedRoot!); + const root = inboxScratch(`plannotator-thread-parity-${role}-${info.project.name}-`); + const dataDir = join(root, 'data'); + const home = join(root, 'home'); + // Claude Code's plugins in the scratch HOME (the restart line, inbox-restart.spec.ts): 0.28.25 installed, + // and 0.28.23's folder, which a working session loaded before the update. + const pluginFolder = (version: string) => join(home, '.claude', 'plugins', 'cache', 'plannotator', 'plannotator', version); + const plugins = { installed: pluginFolder('0.28.25'), loaded: pluginFolder('0.28.23') }; + for (const folder of Object.values(plugins)) mkdirSync(folder, { recursive: true }); + writeFileSync( + join(home, '.claude', 'plugins', 'installed_plugins.json'), + JSON.stringify({ + version: 2, + plugins: { 'plannotator@plannotator': [{ scope: 'user', installPath: plugins.installed, version: '0.28.25', installedAt: '2026-10-04T22:07:24.658Z', lastUpdated: '2026-10-17T05:27:07.299Z' }] }, + }), + ); + // The seeded data, never the seeding Inbox's registry or log. + cpSync(seed.dataDir, dataDir, { recursive: true, filter: (src) => !/[/\\]inbox[/\\]inbox\.(json|log)$/.test(src) }); + const started = spawnSync(binary, ['inbox', '--background'], { env: parityEnv(home, dataDir), cwd: root, encoding: 'utf8', timeout: 60_000 }); + expect(started.status, started.stderr).toBe(0); + rememberInboxOf(dataDir); + const url = started.stdout.trim().replace(/\/?$/, '/'); + const health = (await (await fetch(`${url}api/inbox/health`)).json()) as { serverSession: string }; + const registry = JSON.parse(readFileSync(join(dataDir, 'inbox', 'inbox.json'), 'utf8')) as { port: number; token: string }; + run = { seed, url, root, plugins, port: registry.port, token: registry.token, serverSession: health.serverSession }; +}); + +test.afterAll(async () => { + // Every Inbox this project started, then its folder (inbox-scratch.ts). + await removeInboxScratch(); +}); + +function normalize(html: string): string { + return html + .replace(/http:\/\/(127\.0\.0\.1|localhost):\d+/g, 'http://127.0.0.1:PORT') + .replace(/blob:[^"'\s)]+/g, 'blob:URL') + // An HTML attachment's asset base carries a token minted per Inbox run. + .replace(/\/api\/html-assets\/[0-9a-f]+\//g, '/api/html-assets/TOKEN/') + .split(run.serverSession) + .join('SERVER_SESSION') + .split(run.root) + .join('SCRATCH') + .replace(/>\n<'); +} + +/** Fonts, images and two frames: what a capture waits for beyond toHaveScreenshot's own two equal shots. */ +async function settle(page: Page): Promise { + await page.evaluate(async () => { + await document.fonts.ready; + await Promise.all( + [...document.images].filter((img) => !img.complete).map((img) => new Promise((done) => { + img.addEventListener('load', done, { once: true }); + img.addEventListener('error', done, { once: true }); + })), + ); + // Decoded too, so no image paints in a later frame than its tile. + await Promise.all([...document.images].map((img) => img.decode().catch(() => {}))); + await new Promise((done) => requestAnimationFrame(() => requestAnimationFrame(done))); + }); + // WebKit draws an image it just scaled at low quality and repaints it at high quality half a second later + // (its ImageQualityController): wait that out, then repaint the scrollers whole (a layer for two frames, + // then none; nothing moves, nothing takes focus). Even so, WebKit now and then blends the one pixel row + // where a scroller clips a picture tile a few colour levels differently from run to run (main against + // itself too); the report names such a capture, and its DOM still matches. + if (page.context().browser()?.browserType().name() === 'webkit') { + await page.waitForTimeout(700); + await page.evaluate(async () => { + const frames = () => new Promise((done) => requestAnimationFrame(() => requestAnimationFrame(done))); + // Each scroller's style attribute is put back exactly as it was, so the captured DOM is untouched. + const scrollers = [...document.querySelectorAll('.ib-pbody, .ib-lbody')].map((el) => ({ el, style: el.getAttribute('style') })); + for (const { el } of scrollers) el.style.setProperty('transform', 'translateZ(0)'); + await frames(); + for (const { el, style } of scrollers) { + if (style === null) el.removeAttribute('style'); + else el.setAttribute('style', style); + } + await frames(); + }); + } +} + +function domDir(info: TestInfo): string { + const dir = join(out, role, 'dom', info.project.name); + mkdirSync(dir, { recursive: true }); + return dir; +} + +/** + * `realTimes`: what in the capture shows a time the Inbox stamped from its own clock while the capture ran + * (not the seed's). Its clock times are rewritten to 12:00 PM in the page just before the capture (a mask + * would be as wide as the time it hides), so the rest of that text is still compared. + */ +async function capture(page: Page, info: TestInfo, thread: string, state: string, dom = 'section.ib-pane', realTimes: string[] = []): Promise { + for (const selector of realTimes) { + await page.locator(selector).evaluateAll((els) => { + for (const el of els) { + const walker = document.createTreeWalker(el, NodeFilter.SHOW_TEXT); + for (let node = walker.nextNode(); node; node = walker.nextNode()) node.nodeValue = node.nodeValue!.replace(/\b\d{1,2}:\d{2}(\s?[AP]M)\b/g, '12:00$1'); + } + }); + } + await settle(page); + await expect.soft(page).toHaveScreenshot(`${thread}-${state}.png`); + const html = await page.locator(dom).evaluateAll((els) => els.map((el) => el.outerHTML).join('\n')); + const metrics = await page.evaluate(() => { + const scroller = document.querySelector('section.ib-pane .ib-pbody'); + return scroller ? { scrollTop: Math.round(scroller.scrollTop), scrollHeight: scroller.scrollHeight, clientHeight: scroller.clientHeight } : null; + }); + writeFileSync(join(domDir(info), `${thread}-${state}.html`), `\n${normalize(html)}\n`); +} + +/** The thread from the top, a page at a time. */ +async function pages(page: Page, info: TestInfo, thread: string, prefix: string): Promise { + const scroller = page.locator('section.ib-pane .ib-pbody'); + if ((await scroller.count()) === 0) return; + const { height, client } = await scroller.evaluate((el) => ({ height: el.scrollHeight, client: el.clientHeight })); + const step = Math.max(200, client - 80); + for (let i = 0, top = 0; i < MAX_PAGES; i++, top += step) { + await scroller.evaluate((el, y) => el.scrollTo({ top: y, behavior: 'instant' }), Math.min(top, height - client)); + await capture(page, info, thread, `${prefix}${i}`); + if (top + client >= height) break; + } +} + +async function scrollTop(page: Page): Promise { + await page.locator('section.ib-pane .ib-pbody').evaluate((el) => el.scrollTo({ top: 0, behavior: 'instant' })); +} + +async function open(page: Page, hash: string): Promise { + await page.goto('about:blank'); + await page.goto(`${run.url}#${hash}`); + await expect(page.locator('section.ib-pane')).toBeVisible(); +} + +/** The list at rest, no thread open (on a phone, the list screen). */ +async function list(page: Page, info: TestInfo, state: string): Promise { + await page.goto('about:blank'); + await page.goto(run.url); + await expect(page.locator('.ib-lbody [data-thread-id]').first()).toBeVisible(); + await capture(page, info, 'list', state, '.ib-lbody'); +} + +/** Each tooltip a desktop shows here: hovered, captured with its popup, then let go. */ +async function tooltips(page: Page, info: TestInfo, thread: string, tips = TIPS): Promise { + for (const tip of tips) { + const target = page.locator(tip.selector).first(); + if ((await target.count()) === 0 || !(await target.isVisible())) continue; + await target.scrollIntoViewIfNeeded(); + await target.hover(); + await expect(page.locator(TOOLTIP)).toBeVisible(); + await capture(page, info, thread, `tip-${tip.name}`, `${tip.selector}, ${TOOLTIP}`); + await page.mouse.move(0, 0); + await expect(page.locator(TOOLTIP)).toHaveCount(0); + } +} + +/** One agent session's connection, polling as the Claude Code mod (or Pi) polls: busy, and what it loaded. */ +class Connection { + private stopped = false; + private readonly loop: Promise; + constructor(private readonly body: Record) { + this.loop = this.run(); + } + + private async run(): Promise { + while (!this.stopped) { + // It never reports a delivery: what waits keeps waiting. + await fetch(`http://127.0.0.1:${run.port}${INBOX_BRIDGE_POLL_PATH}`, { + method: 'POST', + headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${run.token}` }, + body: JSON.stringify({ waitMs: 1_000, busy: true, started_at: Date.parse('2026-10-01T09:00:00Z'), idle_since: null, ...this.body }), + }).catch(() => null); + await new Promise((r) => setTimeout(r, 200)); + } + } + + async stop(): Promise { + this.stopped = true; + await this.loop; + } +} + +type Waits = Record; + +/** The Inbox's wait clocks for `session`'s items in `thread` (GET โ€ฆ/sessions?asker=), once its poll was seen. */ +async function waitsOf(thread: string, session: string): Promise { + let waits: Waits = {}; + await expect + .poll(async () => { + const model = (await (await fetch(`${run.url}api/inbox/threads/${thread}/sessions?asker=${session}`)).json()) as { asker?: { connection?: { busy: boolean | null; waits?: Waits } | null } }; + waits = model.asker?.connection?.waits ?? {}; + return model.asker?.connection?.busy === true && Object.keys(waits).length > 0; + }, { timeout: 15_000 }) + .toBe(true); + return waits; +} + +// One test per project: a capture that differs fails softly and every later capture still runs in the +// same Inbox, so each capture reads the same looks as in the reference run. +test('every seeded thread', async ({ page }, info) => { + const errors: string[] = []; + page.on('pageerror', (error) => errors.push(`pageerror: ${error.message}`)); + page.on('console', (message) => { + // The browser refusing a script in a sandboxed HTML attachment is the sandbox working; when it logs + // depends on when the frame loads. + if (message.type() === 'error' && !/^Blocked script execution in 'about:srcdoc'/.test(message.text())) errors.push(`console.error: ${message.text()}`); + }); + const hover = !info.project.use.hasTouch; + await page.clock.setFixedTime(new Date(run.seed.frozenAt)); + const narrow = NARROW[info.project.name]; + if (narrow) { + await page.addInitScript((css) => { + document.addEventListener('DOMContentLoaded', () => { + const style = document.createElement('style'); + style.dataset.parity = 'narrow'; + style.textContent = css; + document.head.appendChild(style); + }); + }, `section.ib-pane { flex: 0 0 ${narrow}px !important; width: ${narrow}px !important; max-width: ${narrow}px !important; }`); + } + const step = async (name: string, body: () => Promise) => + test.step(name, async () => { + errors.length = 0; + await body(); + writeFileSync(join(domDir(info), `${name}-errors.txt`), `${errors.map((e) => normalize(e)).join('\n')}\n`); + }); + const id = (name: string) => { + const value = run.seed.threads[name]; + expect(value, `the seed has no ${name} thread`).toBeTruthy(); + return value!; + }; + + await step('list-start', () => list(page, info, 'start')); + + for (const name of THREADS) { + await step(name, async () => { + // As it lands (a new message: at "New since you looked"), then page by page. + await open(page, `thread=${id(name)}`); + await capture(page, info, name, 'land'); + await pages(page, info, name, 'page-'); + await scrollTop(page); + if (hover) { + await tooltips(page, info, name); + await scrollTop(page); + } + + // The Files band, open. + const files = page.locator('section.ib-pane [data-header-files]').first(); + if ((await files.count()) > 0 && (await files.isVisible())) { + await files.click(); + await page.mouse.move(0, 0); + await capture(page, info, name, 'files-band'); + await files.click(); + await page.mouse.move(0, 0); + } + + // Expand all, when the thread folds. + const expand = page.locator('section.ib-pane [data-expand-all]'); + if ((await expand.count()) > 0 && (await expand.getAttribute('aria-label')) === 'Expand all messages') { + await expand.click(); + await page.mouse.move(0, 0); + await pages(page, info, name, 'expanded-'); + } + + // The open questions list. + const questions = page.locator('section.ib-pane .ib-pfoot [data-open-questions]'); + if ((await questions.count()) > 0 && (await questions.isVisible())) { + await questions.click(); + await expect(page.locator('[data-open-questions-list]')).toBeVisible(); + await capture(page, info, name, 'open-questions', 'section.ib-pane, [data-open-questions-list]'); + await page.keyboard.press('Escape'); + } + + // The Artifacts view, when the thread has one. + if ((await page.locator('section.ib-pane [data-artifacts-button]').count()) > 0) { + await open(page, `thread=${id(name)}&artifacts`); + await capture(page, info, name, 'artifacts'); + } + }); + } + + await step('list-end', async () => { + await list(page, info, 'end'); + // The Quiet band open (the ๐Ÿ‘ thread), then each delivery tick's tooltip. + const quiet = page.locator('[data-section-id="quiet"] .ib-band'); + if ((await quiet.count()) > 0 && (await quiet.getAttribute('aria-expanded')) !== 'true') { + await quiet.click(); + await page.mouse.move(0, 0); + await capture(page, info, 'list', 'end-quiet', '.ib-lbody'); + } + if (hover) await tooltips(page, info, 'list', LIST_TIPS); + }); + + // Later clocks: the other-day and other-year branches of every time. + for (const later of LATER) { + await step(later.name, async () => { + await page.clock.setFixedTime(new Date(Date.parse(run.seed.frozenAt) + later.ms)); + await list(page, info, later.name); + for (const name of THREADS) { + await open(page, `thread=${id(name)}`); + await capture(page, info, name, later.name); + } + }); + } + + // A ๐Ÿ‘ inside its Undo window (the Inbox sends it once the window passes), the toast hidden: its timer is + // the page's, not the clock's. + await step('ack-pending', async () => { + await page.clock.setFixedTime(new Date(run.seed.frozenAt)); + await open(page, `thread=${id('files')}`); + await page.addStyleTag({ content: '.ib-toast { visibility: hidden !important; }' }); + await page.locator('section.ib-pane [data-ack]').click(); + await page.mouse.move(0, 0); + await expect(page.locator('section.ib-pane [data-ack-pill="pending"]')).toBeVisible(); + // The ๐Ÿ‘ and the Mark read it brings are stamped by the Inbox's clock as the capture runs. + await capture(page, info, 'files', 'ack-pending', 'section.ib-pane', ['section.ib-pane .ib-ackwhen', 'section.ib-pane [data-read-line] > div > span:first-child']); + }); + + // Live delivery, last (the Inbox remembers a connection for a while after it stops, so a later capture would + // depend on timing). Claude Code on the delivery thread works with Send now: Undelivered and Send now. The + // replaced thread's Claude Code works on plugin 0.28.23 with 0.28.25 installed: the restart line. Pi on the + // ๐Ÿ‘ thread works with no Send now: queued. + await step('live', async () => { + const { sessions } = run.seed; + const { plugins } = run; + const project = join(run.seed.root, 'src', 'billing-svc'); + const connections = [ + new Connection({ session: sessions.delivery, host: 'claude-code', project_path: project, session_folder: project, [INBOX_SEND_NOW_POLL_FIELD]: true, plugin: { version: '0.28.25', root: plugins.installed, loaded_at: Date.parse('2026-10-20T09:00:00Z') }, engine: '2.1.305' }), + new Connection({ session: sessions.replaced, host: 'claude-code', project_path: project, session_folder: project, plugin: { version: '0.28.23', root: plugins.loaded, loaded_at: Date.parse('2026-10-16T09:14:00Z') }, engine: '2.1.305' }), + new Connection({ session: sessions.ack, host: 'pi', project_path: join(run.seed.root, 'src', 'docs-site') }), + ]; + try { + // The minutes on a waiting reply count from the Inbox's own clock (when it first saw the session busy): + // read it, and set the page clock 90 s after it, so every run reads the same minutes. + const waits = { ...(await waitsOf(id('delivery'), sessions.delivery)), ...(await waitsOf(id('replaced'), sessions.replaced)) }; + const since = Math.max(...Object.values(waits).map((w) => Date.parse(w.since))); + await page.clock.setFixedTime(new Date(since + 90_000)); + for (const name of ['delivery', 'replaced', 'ack']) { + await open(page, `thread=${id(name)}`); + await page.locator('section.ib-pane .ib-pbody').evaluate((el) => el.scrollTo({ top: el.scrollHeight, behavior: 'instant' })); + await capture(page, info, name, 'live'); + } + } finally { + await Promise.all(connections.map((c) => c.stop())); + } + }); +});