scripts: add a browser verification loop for the admin console by ac-pplx · Pull Request #1272 · perplexityai/SPACE

ac-pplx · 2026-10-31T20:00:22Z

Summary

Adds scripts/console-loop/, a Playwright loop for the admin console (ui/) on the local Kind stack. It seeds synthetic organizations, users, API keys and sandboxes through the public and admin APIs. It signs in through the real login page, visits every console route and runs the operator flows. For each step it records a screenshot, console and page errors, failed and slow requests, error toasts and the step's own checks, and it compares two runs (for example main against a branch), failing on new error types. It is development and review evidence, not a CI gate. Behavior checks that need no browser belong in typed space_smoke scenarios (the space-validation smoke-testing reference); loop steps are for what only a browser shows: rendering, navigation, dialogs and console errors. CI runs the tool's own type check and unit tests in the existing SPACE Guide job, never the loop itself. The step installs from the job's restored npm cache: the tool's lockfile is an input of the npm cache family in ci/scripts/package-cache.py, next to docs/api/package-lock.json, and the trim before main saves the cache keeps both lockfiles' packages.

Safety and cleanup:

One stack, on this host. seed, clean and every loop run that creates things refuse a non-loopback API or console URL unless --allow-remote is passed. They also read the stack's identity, the cluster row's id and created_at (generated with the database; managing_org_id is the same on every stack), from /admin/api/v1/cluster and /auth/v1/cluster, through both the API URL and the console's origin. They refuse when the readings differ, or differ from the identity the seed recorded, and the refusal names the state file and the remedy for a recreated stack. The tool's own API requests and the browser requests it intercepts never follow a redirect; the console's own requests and navigations do. The README says what this cannot prove: the console's /api and data-plane routes, browser requests to other origins, proxies that route differently later, and stacks that share a database.
State per prefix. The state is out/seed-<prefix>.json, and seed, loop and clean refuse a state recorded for another prefix. Mutating loop runs require a seed state.
Recovery. The stack is disposable. Each created object is recorded with its ID as soon as the API answers, and each sandbox also with its create operation, because a create's answer names the sandbox while that operation can still be queued. A rerun reuses fixtures by slug, email, key name, or sandbox ID or name; a create interrupted before its answer is not recorded. clean and a reseed wait, up to ten minutes, for a recorded create operation to finish before they treat a missing sandbox as gone; on timeout the record stays and counts as a failure. A pending create is never replaced by name, and an absent sandbox is never force-deleted. If a rerun does not recover, delete out/seed-<prefix>.json and reseed, or recreate the stack; the README and the error messages say so. clean deletes only what was recorded, waits for each delete, and exits 1 on failures.
Seeded organizations are kept by clean, which the next seed reuses by slug; the organizations loop runs create are deleted. --delete-orgs also deletes the seeded ones. Deleting an organization cascades to its API keys, credentials, credential references, master keys, template grants and more (listed in the README). clean refuses an organization that still holds a child organization, a user, an API key or a sandbox that is not deleted; the other cascaded rows are not checked and go with it.
No live API key in the browser. During the API-key step every POST goes through page.route, with maxRedirects: 0 and any 3xx refused. A create records its key's ID, and every secret in a JSON answer is replaced, so the dialog shows console-loop-placeholder-not-a-key. It fails closed: if a secret could have reached the page another way, the dialog opened before the route answered, or the dialog shows anything key-shaped, the step opens about:blank before any screenshot and fails, and the run stops after it (exit 1); the browser is closed and a --video recording deleted in a finally path, and the summary says so. A run killed or interrupted while it closes can still leave a raw recording under its run directory's .video/; the README says to delete that before sharing a run. The recorder also redacts 64-hex strings in everything it writes.
Fixtures are verified. Seeding fails when a fixture cannot be discovered or set up. It verifies the full required set by ID: orgs, users, API keys, and (unless --skip-sandboxes) each sandbox at rest and the deleted one gone. The loop's fixtures step checks the same set through the console.
Evidence stays with its step. A late response, including a body that finishes reading after its step ended, is a "late" note, not an event of the next step.
--only and --skip take comma-separated exact step names or tags; no regular expression is built from them.

The fixtures are synthetic, but screenshots show everything the console account can see on the stack. The README asks for a disposable stack and a look at each image before sharing it.

Validation

On a local Kind stack (infra/space/kind/scripts/up.sh), at the head commit:

npm run typecheck passes, and npm test passes 13 unit tests (selection, normalizer, secret replacement, per-prefix state, state merging, stack identity, pending creates). The CI step, run as the workflow runs it (clean checkout, Node 22.23.3, the registry blocked), installs from a filled cache and passes; package-cache.py's tests pass, including the two-lockfile trim. The pplx-space plugin moves to 0.2.46 for the skill-reference change, and claude plugin validate passes on it.
ci/scripts/kind-pull-needed.sh returns false for this head and for a change only to the tool, so editing it no longer selects the Kind smoke.
npm run seed created and verified 18 fixtures, a full npm run loop (47 steps) ran, and npm run clean deleted everything recorded, including the --delete-orgs refusal for an organization holding an unrecorded user. The previous version of the tool gave the same per-step results on the same stack, and compare found 0 new error types; the 17 steps with events are existing console findings (snapshots 404 on sandbox pages, 403s for tenant roles, the unknown-org page, the users list's default org).
The API-key step's dialog shows the placeholder, and no file under out/ holds a 64-hex string.
Refusals, all exit 2: a non-loopback target, a console on another stack, a state recorded on another stack, and another prefix's state.
With the cluster controller scaled to zero, so accepted creates stay queued and read as 404: a reseed after a killed seed waited on the recorded create and then took over the sandbox it made, with no duplicate and the same ID; clean waited, then deleted the sandbox once it appeared. The timeout case is covered by a unit test.

Visual confirmation

A short run against main: sign-in, then the console's routes and the sandbox search steps:

Video: loop-run.webm.

The API-key step answers the create request itself, so the dialog shows a placeholder and the browser never holds a live key:

Severity and risk

Severity: does not apply. This is a development tool.

Risk: low. Nothing in CI or in any image runs the loop, and it changes no service, chart or ui/ file; CI only type-checks and unit-tests it in the existing SPACE Guide job. The seed writes only through the public and admin APIs, as the console account, and only to a stack on this host unless --allow-remote is passed; clean deletes only what the state file recorded, and keeps the seeded organizations unless --delete-orgs is passed. It refuses to delete an organization that still holds a child organization, a user, an API key or a sandbox that is not deleted; anything else the organization holds (credentials, credential references, master keys, template grants) is deleted with it. Mutating runs refuse a target whose stack identity differs between the API, the console and the seed state. The tool lives outside the paths in ci/kind-pull-paths.txt, so changes to it do not require the Kind smoke. Roll back by reverting the commits.

Not covered

CI. This is not a smoke test. A follow-up PR adds typed space_smoke console scenarios: operator invariants (lookup by every ID form, detail completeness per state, cross-org actions, destructive-action confirmation, role gating, link integrity), an allowlisted error baseline next to the scenarios, and selection so the Kind smoke shards run them only when ui/ or the admin routes change.
The steps cover the console as it is on main. Steps for behavior a PR adds belong with that PR.

AI assistance: Claude Code wrote the code, the README and this description, and ran the validation; a maintainer reviews it.

github-actions · 2026-10-31T20:00:31Z

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

zibi-pplx

A useful review aid. Questions about scope and cost:

[medium] Should the "add a step" guidance point to typed space_smoke scenarios?
infra/space/kind/console-loop/README.md:282
steps.ts already has 43 c.rec.check assertions against a running Kind stack, for example "the role change persists", "searching a full sbx_ ID finds the sandbox", and lifecycle actions reaching a state. The README asks contributors to "add a step to src/steps.ts for each behavior a change introduces … and keep it if it fails before and passes after", which is regression-test guidance. The same README says smoke tests against a deployment are typed space_smoke scenarios, with console scenarios as a follow-up. Should this bullet send behaviour checks that don't need a browser to those scenarios? Steps would then stay for what only a browser can show. That would keep the follow-up from re-implementing the same checks.
[low] Editing the tool runs the full required Kind smoke
ci/kind-pull-paths.txt:10
The tool lives under infra/space/kind/**, so any change to it runs the six Kind shards and the producer in pull CI. If #1225 merges, the Direct lane runs too. Meanwhile nothing in CI runs its typecheck or npm test. Would a location outside that glob fit better (for example beside ui/), with npm run typecheck added to an existing lint job?
[low] The seed's recovery scope
infra/space/kind/console-loop/src/seed.ts
About 780 lines handle idempotent replay of interrupted creates, 409 and 422 classification, and operations the API garbage-collects after a week. The README asks for a disposable stack. Could recovery fall back to "delete out/seed-<prefix>.json and reseed", so the seeder shrinks to what a fresh stack needs?

Withdrawing this approval for now. Any feedback above still stands.

infra/space/kind/console-loop/ is a small Node package that checks the admin console (ui/) in a browser against the local Kind stack: - `npm run seed` creates synthetic organizations, users, API keys and sandboxes in each resting state through the public and admin APIs, under a name prefix, and `npm run clean` deletes exactly those. Seeded users get a generated password, kept only in the git-ignored state. - `npm run loop` signs in through the login page, visits every console route, runs the operator flows (create and delete an organization, a user, an API key, and a sandbox through its lifecycle), and records a screenshot, console and page errors, failed and slow requests, error toasts and its own checks for every step, in a JSON summary. - `--compare` (or `npm run compare`) compares two runs by error type and exits 1 when the later run, say a branch, has one the earlier run, say main, does not. - `npm run capture` records pages as a video, and a GIF with ffmpeg, for a pull request description. The README covers the fast loop (`next dev` against the port-forwarded backends), the Tilt-image loop, comparing a branch with main, and using runs as evidence. The Kind runbook, the installation guide and the space-validation local-development reference point to it, and the plugin version is bumped for the reference change. Playwright, TypeScript and the Node types are the package's only dependencies, pinned exactly, outside ui/'s dependencies and image. The scripts run with Node's type stripping. Its node_modules and output are git-ignored and kept out of root-context Docker builds. It is a development tool, not a CI gate; the typed space_smoke console scenarios are a separate follow-up.

- seed, clean, and loop runs that create things refuse an API or console URL that is not loopback unless --allow-remote is passed, and print the target they act on. - clean deletes exactly what seed.json records: the seed and the loop now record every object as soon as it exists. It waits for each sandbox delete operation and counts anything but success as a failure, refuses to delete an org that still holds a child org, user, API key or live sandbox it did not record, reports what it could not delete, keeps it recorded, and exits 1. - Seeding looks every object up by its prefixed name before creating it, scoped to its org, so a rerun after an interruption duplicates nothing; it fails loudly when an org has too many sandboxes for the admin list to tell, and checks every recorded fixture by ID. The loop's first step does the same. Org display names carry the prefix, so two prefixes can share a stack. - The API-key step hides the new key's secret before its dialog opens and checks that it stays hidden, and the recorder replaces anything shaped like a key in what it writes. - Each event belongs to the step that issued its request; a response that lands later is a note, not an event of the next step. - Comparisons keep the step in failed checks and exceptions, and list an error type as "not covered", not resolved, when the later run did not run every step it appeared in. - Development warnings are classified as React 19 and antd print them (format strings with `%s` placeholders, "Warning: …"), and the README says so, along with the loop's scope: development evidence, not a pass/fail gate; smoke tests stay typed space_smoke scenarios.

Every mutating run now reads the cluster row's id through the console's /admin and /auth proxies and through the API, and refuses when they differ or differ from the seed state. The state is per prefix, records each create before it is submitted (with the sandbox idempotency key) and every operation, and clean waits for those operations before it forgets anything. Clean keeps organizations unless --delete-orgs is given, since deleting one cascades to its keys, credentials and grants. The API-key step replaces the secret in the create response with page.route, so the browser never holds a live key. Seeding fails when a fixture cannot be set up and verifies the full set, --only and --skip take exact step names or tags instead of regular expressions, and a late response body is checked against its step again after it is read.

The API deletes finished operations after a week, and the loop treated the resulting 404 as an error, so seed and clean failed for good. A vanished operation is now recorded as gone and the object is judged by its ID; every outcome is recorded once seen and not asked for again. An unanswered seed create is resubmitted with its recorded request, unchanged, so a later --template cannot turn its key into a conflict. A browser sandbox create is marked as submitted from page.route before it leaves, and clean keeps one that left with no answer and no sandbox of its name as unresolved, unless --forget-unresolved is given. Browser lifecycle and delete operations are recorded as they arrive. The API-key step routes every POST without following redirects, replaces any secret in an answer, and clears the page, fails, and deletes a --video recording when a secret could have reached the page another way. Clean now deletes the orgs loop runs created and keeps only the seeded ones, and a state from a recreated stack is refused with its remedy.

A sandbox create the API refused before admitting it (a 4xx other than 409) left no operation, so every rerun replayed the same bad body and clean never dropped it. It is now recorded as rejected: a rerun makes a new create with the current options and a new key, and clean drops the intent; the browser create handler records the same. A 409 stays unresolved, replayed unchanged with its key. When the API-key step clears the page for a possible live key, the flag is set before it navigates, the run stops after that step, and the browser is closed and the recording deleted in a finally path. The README says an interrupted run can still leave a raw recording.

The seeder journaled create intents, replayed unanswered creates with their recorded keys, classified 409 and 422 answers, kept unresolved browser creates and recorded every operation outcome, including ones the API had since garbage-collected. The README already requires a disposable stack, so recovery is now to delete the seed state and reseed, or recreate the stack, and the README and errors say so. Each object is still recorded as soon as the API answers its create, and clean still deletes only what was recorded, waits for each delete and exits 1 on failures. The stack identity checks, the per-prefix state, the org handling and the API-key secret routing are unchanged.

The README told contributors to add a loop step for every behavior a change introduces and keep it if it failed before and passed after, which is regression-test guidance. Behavior checks that need no browser belong in typed space_smoke scenarios; loop steps cover what only a browser shows. The loop stays development and review evidence, not a pass/fail gate.

Under infra/space/kind/ every edit to the tool selected the six-shard Kind smoke, while CI ran neither its type check nor its unit tests. It now lives in scripts/console-loop/, which ci/kind-pull-paths.txt does not match and ui/'s build does not read, and the SPACE Guide job, the pull job that already sets up Node, runs its npm ci, typecheck and tests with a separate npm cache. The Kind README no longer links it, so this change touches no Kind path; docs/installation.md and the space-validation reference point at the new location.

The console loop's pointer in the space-validation reference changes shipped plugin content. main already moved the plugin to 0.2.45 for its own skill change, which absorbed this branch's identical bump on rebase.

The SPACE Guide step for the console loop used an empty npm cache of its own, so a required job fetched every package from the registry on each run, and a registry outage would fail every pull request. The tool's lockfile is now one of the npm cache family's inputs, the trim keeps the packages of every lockfile it is given, and the step uses the job's restored cache.

A create's answer names the sandbox while its operation can still be queued, before the sandbox exists. clean treated that first 404 as gone and dropped the record, so the sandbox appeared later untracked, and a reseed in that window created a second one and replaced the record by name. Each sandbox create's operation is now recorded with its answer. clean and the reseed wait, up to ten minutes, for a recorded create operation to finish before they read a missing sandbox again; past that the record stays and the sandbox counts as a failure. An absent sandbox is still never deleted.

github-actions Bot added the triage:pending Waiting for a maintainer to triage label Oct 31, 2026

github-advanced-security Bot found potential problems Oct 31, 2026

View reviewed changes

Comment thread infra/space/kind/console-loop/src/loop.ts Fixed

Comment thread infra/space/kind/console-loop/src/loop.ts Fixed

ac-pplx marked this pull request as ready for review November 1, 2026 06:52

kumarpallav-pplx requested review from kumarpallav-pplx and nikhilgupta-pplx November 1, 2026 06:53

zibi-pplx previously approved these changes Nov 1, 2026

View reviewed changes

ac-pplx changed the title ~~kind: add a browser verification loop for the console~~ scripts: add a browser verification loop for the admin console Nov 2, 2026

ac-pplx added 11 commits November 2, 2026 04:52

agents: bump the pplx-space plugin to 0.2.46

9ca2f9a

The console loop's pointer in the space-validation reference changes shipped plugin content. main already moved the plugin to 0.2.45 for its own skill change, which absorbed this branch's identical bump on rebase.

ac-pplx force-pushed the ac/kind-console-loop branch from b1766a2 to 5d27ebe Compare November 2, 2026 05:17

kumarpallav-pplx removed their request for review November 3, 2026 02:33

nikhilbirmiwal-pplx approved these changes Nov 3, 2026

View reviewed changes

ac-pplx added this pull request to the merge queue Nov 3, 2026

Merged via the queue into main with commit 42a9900 Nov 3, 2026
23 checks passed

ac-pplx deleted the ac/kind-console-loop branch November 3, 2026 20:57

Navigation Menu

Repository navigation

scripts: add a browser verification loop for the admin console - #1272

scripts: add a browser verification loop for the admin console#1272
ac-pplx merged 11 commits into
mainfrom
ac/kind-console-loop

ac-pplx commented Oct 31, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 31, 2026

Uh oh!

Uh oh!

Uh oh!

zibi-pplx left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ac-pplx commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Visual confirmation

Severity and risk

Not covered

Uh oh!

github-actions Bot commented Oct 31, 2026

Uh oh!

Uh oh!

Uh oh!

zibi-pplx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ac-pplx commented Oct 31, 2026 •

edited

Loading