Skip to content

Let an agent see a frame of the animation it wrote - #1501

Merged
milanb17 merged 5 commits into
mb/scene-card-formsfrom
mb/elated-solomon-90e5dd
Nov 4, 2026
Merged

milanb17 merged 5 commits into
mb/scene-card-formsfrom
mb/elated-solomon-90e5dd

Conversation

@milanb17

@milanb17 milanb17 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1498 (the clip block and the reference animations live on that stack, so this is based on mb/scene-card-forms).

After publishing an animation an agent had no way to see what it drew: session_get says only whether the script ran, so agents fell back to driving the screen. session_render({ sessionId, targetId, frame }) shows one frame of an animation (or a clip at one marker; omitted, the first) on the whiteboard open in Desktop and returns it as a PNG image, with the error the stage reported, if any. The PNG is also written to <home>/animation-renders/<session>/<block>/<key>.png and named in the result, for agents that read files rather than images (whiteboard api session_render prints the JSON without the base64). Step 5 of "Work in files" in animation.md tells agents to render each frame they changed after publishing, instead of taking screenshots.

What the agent gets back, from a scene animation and a clip published through whiteboard api into a dev Desktop:

An animation at its third frame: the API has asked Postgres and the cache missed

The same animation at its last frame: the client has its order and the cache is filled

A clip at its second marker: the request chip sits under the API, the progress line at the marker

{"sessionId":"2d67401e-…","targetId":"block-1","type":"animation","frame":{"key":"query","title":"Postgres answers","position":2,"count":4},"clipped":false,"error":null,"file":"…/animation-renders/2d67401e-…/block-1/query.png"}

Over MCP the same call answers an image content block (the PNG) ahead of that text.

How a frame is rendered: Desktop captures its own canvas

Two ways were weighed. (A) The attached Desktop shows the frame on the open board and captures the block's rectangle with Electron's capturePage. (B) The review server renders the stage headless. This PR does A:

  • A frame as the reader sees it exists only in Desktop. The code peeks and embedded blocks a stage shows are native Monaco editors and canvas components that Desktop's host bridge lays over the stage; a headless renderer would have to rebuild that bridge or draw something else.
  • The headless server has no browser. whiteboard server start with an attached Desktop renders through the same relay session_open uses; without one the route answers 409.
  • The costs, and how they show: the board must be open in Desktop (the error says to call session_open); the window must be on screen (a covered or minimized window paints nothing and throttles its timers, so the canvas fails fast with a message saying so rather than waiting out the relay); the reader sees the frame change; a figure taller than the window is captured as far as it fits, with clipped: true.

The pieces

  • Protocol: verb renderAnimation {reviewId, targetId, frame}; ReviewCanvasHandle.renderAnimation and ReviewRenderedFrame.
  • Canvas: animation/stage-renderer.ts keeps the stage blocks on a board by id. StageBlock.show(key) waits for the stage to boot and paint in, switches to a scenario that has the frame when the one showing does not, picks the stop with the new PickedBy "render" (playback stops; a clip pauses), waits for settled or the clip's marker, scrolls the figure into view, waits for it to hold still, and reports its rectangle in CSS px. The stage port now notes the frame or marker last asked for, and the block notes what the stage reports showing, so a render waits until they agree.
  • Fork: reviewCanvasPart.ts answers the verb for the board it shows (main window only), scales the rectangle by the window's zoom and calls hostService.getScreenshot(rect, { format: "png", maxEdge: 1600 }). The native host's getScreenshot gained those two options; JPEG stays the default, and an empty capture answers undefined.
  • Server: review-api/animation-render.ts holds the query and result schemas and renderTarget, which checks the frame against the block's frames or markers (naming them when it is wrong) and picks the first scenario's first frame when omitted. GET /:id/render in http.ts; relayReviewCallbacks.render in review-server-core.ts. The /control/result bound is 16 MiB so the PNG can ride back.
  • Agent boundary: the tool is listed with animation authoring (session_checkout beside it); public-tools.ts writes the file; mcp.ts returns the image block; agent-cli.ts keeps the base64 out of stdout.

Verification

  • Node: animation-render.test.ts (target resolution, the route with a stub Desktop, the tool listing, the file and the image split), render-relay.test.ts (the relay callback: result parsing, Desktop's refusal and a missing Desktop as 409s), plus the existing authoring, public-tools, agent-client and authoring-capabilities tests.
  • Browser: stage-renderer.browser.test.tsx mounts the retry-backoff reference, renders a3fail (only in its second scenario), checks the reported rectangle against the figure and that Playwright's capture of it is a non-empty PNG; lands a clip on a marker at rest; reports unknown stops and blocks. The clip, paint, blocks and references suites pass (100 tests).
  • Desktop: pnpm desktop:build and desktop:run under an isolated DEV_REVIEW_HOME, animations enabled, a scene animation and a clip published through whiteboard api. session_render returned the captures above, exact to the figure, for frames in either direction and for markers, and the errors for a frame the block lacks and for a hidden window. Not checked: dark theme, a zoomed window, secondary windows.
  • pnpm lint, pnpm format, both typechecks and the fork's typecheck-client are clean.

The screenshots live on pr-assets/render-animation-frames; delete that branch with this PR.

After publishing an animation an agent had no way to look at it short of
driving the screen: session_get said only whether the script ran without
an error. session_render shows one frame of an animation, or a clip at one
marker, on the whiteboard open in Desktop and returns it as a PNG image,
with the error the stage reported, if any; the PNG is also written beside
the animation drafts and named in the result, for agents that read files
rather than images. The animation instructions tell agents to render each
frame they changed after publishing, instead of taking screenshots.

Desktop captures the frame from its own canvas rather than a headless
renderer drawing it: the code peeks and embedded blocks a frame shows are
Desktop's own native editors and canvas components laid over the stage,
so only Desktop can draw a frame as the reader sees it, and the headless
server reaches it through the same relay session_open uses. The canvas
asks the block to show the stop at rest (switching scenario when the frame
needs one, pausing playback, waiting for the stage to settle, paint in and
hold still in view) and reports where its figure is; the canvas part
captures that rectangle with Electron's capturePage, now able to answer a
PNG scaled to at most 1600 pixels on its longest edge. The board must be
open in Desktop and its window on screen; the result says so when it is
not, instead of waiting out the relay.

Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683
Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b
Agent-Session: b315a4f8-1333-461c-9be6-366ceb231b7f
Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad
Agent-Session: b5f9f841-35c5-4ec1-95a8-6485f64d254b
Agent-Session: 32e129b9-586e-4fd2-8961-e7224c5978ad
…ted-solomon-90e5dd

Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683
Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b
Agent-Session: b315a4f8-1333-461c-9be6-366ceb231b7f
Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad
Agent-Session: b5f9f841-35c5-4ec1-95a8-6485f64d254b
Agent-Session: 32e129b9-586e-4fd2-8961-e7224c5978ad

# Conflicts:
#	packages/review/app/src/animation/animation-block.tsx
Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683
Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b
Agent-Session: b315a4f8-1333-461c-9be6-366ceb231b7f
Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad
Agent-Session: b5f9f841-35c5-4ec1-95a8-6485f64d254b
Agent-Session: 32e129b9-586e-4fd2-8961-e7224c5978ad
@milanb17
milanb17 added this pull request to stack #1505 November 4, 2026 02:36
…d into mb/elated-solomon-90e5dd

Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b
Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683
Agent-Session: ffed49d6-ca86-4940-b93c-e8eb7e0de440
Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad

# Conflicts:
#	packages/review/src/server/review-server-core.ts
@milanb17
milanb17 merged commit 3e8da1e into main Nov 4, 2026
3 checks passed
@milanb17
milanb17 deleted the mb/elated-solomon-90e5dd branch November 4, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants