Muster's own computer use on macOS: helper, MCP server and agent cursor

docs/spec/clauses/09-browser-and-previews.md

@@ -23,6 +23,10 @@ Boundaries:
 | [MUS-SPEC-09-465C](09-browser-and-previews.md#mus-spec-09-465c) | A lost connection retries read-only browser tools with backoff, never repeats a click, retries a refused connection for any tool, and gives up after three retries. | [`MCP server: connection resets retry read-only tools with backoff but never repeat a click`](../../../packages/agent-app/tests/browser-route-matrix.test.ts) |
 | [MUS-SPEC-09-465D](09-browser-and-previews.md#mus-spec-09-465d) | The idle sweep leaves a tab an agent tool just used awake for three minutes, then freezes it. | [`agent use thaws a hidden page; the sweeper refreezes it once idle`](../../../packages/agent-app/tests/browser-freeze.test.ts) |
 | [MUS-SPEC-09-564A](09-browser-and-previews.md#mus-spec-09-564a) | A download name with a right-to-left override keeps its real .exe extension and holds no direction control characters. | [`a downloaded file name cannot use text-direction controls to disguise its extension`](../../../packages/agent-app/tests/browser-download-name.test.ts) |
+| [MUS-SPEC-09-721A](09-browser-and-previews.md#mus-spec-09-721a) | When an agent clicks a button in a watched tab, an orange cursor glides there first and the click lands under its tip. | [`the agent cursor glides to a button and the press lands under it`](../../../packages/agent-app/tests/agent-cursor.test.ts) |
+| [MUS-SPEC-09-721B](09-browser-and-previews.md#mus-spec-09-721b) | Typing shows a ring around the field; after 1.6 s of quiet the cursor fades out and comes back from the same spot. | [`typing rings the field, and the cursor fades once the agent goes quiet, then returns where it was`](../../../packages/agent-app/tests/agent-cursor.test.ts) |
+| [MUS-SPEC-09-721C](09-browser-and-previews.md#mus-spec-09-721c) | Screenshots the agent takes never include the cursor, and a tab nobody can see gets no cursor at all. | [`the model never sees its own cursor, and a page nobody watches pays nothing`](../../../packages/agent-app/tests/agent-cursor.test.ts) |
+| [MUS-SPEC-09-721D](09-browser-and-previews.md#mus-spec-09-721d) | With Reduce motion on, the cursor moves to each target instantly. | [`with Reduce motion on, the cursor jumps instead of gliding`](../../../packages/agent-app/tests/agent-cursor.test.ts) |
 
 ### Page background
 
@@ -119,3 +123,49 @@ Boundaries:
 - Environment: Node test runner; no network, no provider.
 - Known gaps: Calls the name function used by the download handler, not a real download in the browser tab.
 - Evidence: UNVERIFIED
+
+### Agent cursor
+
+<a id="mus-spec-09-721a"></a>
+#### MUS-SPEC-09-721A: The agent cursor glides to the control and the press lands under it
+
+- Precondition: An in-app browser tab is open and visible, showing a page with two buttons far apart.
+- Action: The agent clicks the near button, then the far button.
+- Expected result: An orange "Muster" cursor appears at the near button, then glides to the far button in 140–520 ms depending on distance, with a press ring on each click. Both clicks register on the page, and the cursor tip ends at the centre of the far button.
+- Test: `packages/agent-app/tests/agent-cursor.test.ts::the agent cursor glides to a button and the press lands under it`
+- Environment: Node test runner driving a real headless Chrome over the DevTools protocol; skipped locally when Chrome is not installed.
+- Known gaps: Drives the cursor script in a plain Chrome page, not inside the Electron browser pane.
+- Evidence: UNVERIFIED
+
+<a id="mus-spec-09-721b"></a>
+#### MUS-SPEC-09-721B: Typing rings the field; the cursor fades when the agent goes quiet
+
+- Precondition: A watched tab shows a text field.
+- Action: The agent types into the field, then does nothing for two seconds, then clicks again.
+- Expected result: A halo ring circles the field while text goes in. After about 1.6 s idle the cursor fades to transparent over about half a second and stops catching anything. The next action brings it back from where it faded.
+- Test: `packages/agent-app/tests/agent-cursor.test.ts::typing rings the field, and the cursor fades once the agent goes quiet, then returns where it was`
+- Environment: Node test runner driving a real headless Chrome over the DevTools protocol; skipped locally when Chrome is not installed.
+- Known gaps: Drives the cursor script in a plain Chrome page, not inside the Electron browser pane.
+- Evidence: UNVERIFIED
+
+<a id="mus-spec-09-721c"></a>
+#### MUS-SPEC-09-721C: The model never sees its own cursor, and unwatched pages pay nothing
+
+- Precondition: A watched tab with the cursor showing, and a second tab that is not on screen.
+- Action: The agent takes a screenshot of the watched tab, then clicks in the hidden tab.
+- Expected result: The screenshot has no cursor pixels, and the cursor reappears afterwards. The hidden tab never mounts a cursor element, and its click costs no extra time.
+- Test: `packages/agent-app/tests/agent-cursor.test.ts::the model never sees its own cursor, and a page nobody watches pays nothing`
+- Environment: Node test runner driving a real headless Chrome over the DevTools protocol; skipped locally when Chrome is not installed.
+- Known gaps: Drives the cursor script in a plain Chrome page, not inside the Electron browser pane.
+- Evidence: UNVERIFIED
+
+<a id="mus-spec-09-721d"></a>
+#### MUS-SPEC-09-721D: Reduce motion makes the cursor jump instead of glide
+
+- Precondition: The system asks for reduced motion.
+- Action: The agent clicks two buttons far apart.
+- Expected result: The cursor jumps straight to each target with no glide, and the click still lands under it.
+- Test: `packages/agent-app/tests/agent-cursor.test.ts::with Reduce motion on, the cursor jumps instead of gliding`
+- Environment: Node test runner driving a real headless Chrome over the DevTools protocol; skipped locally when Chrome is not installed.
+- Known gaps: Drives the cursor script in a plain Chrome page, not inside the Electron browser pane.
+- Evidence: UNVERIFIED

docs/spec/clauses/14-agents-skills-plugins-and-automations.md

@@ -27,6 +27,9 @@ Boundaries:
 | [MUS-SPEC-14-651B](14-agents-skills-plugins-and-automations.md#mus-spec-14-651b) | Two agents asked the same thing get "(2)" and "(3)"; a prompt with no readable task falls back to "Agent N". | [`names stay unique within a chat, and an unreadable prompt falls back to Agent N plus its first distinctive words`](../../../packages/agent-app/tests/subagent-names-651.test.ts) |
 | [MUS-SPEC-14-651C](14-agents-skills-plugins-and-automations.md#mus-spec-14-651c) | The transcript announces three differently named sub-agents started together and each finished once, even though every prompt opens the same way. | [`three parallel sub-agents with the same location clause show three distinct rows in the transcript`](../../../packages/agent-app/tests/subagent-names-651.test.ts) |
 | [MUS-SPEC-14-651E](14-agents-skills-plugins-and-automations.md#mus-spec-14-651e) | With three finished agents the panel lists three different names and its header reads "0 running 3 completed", not "0 completed 3 reported back". | [`three finished sub-agents with one location clause list three distinct names and read three completed`](../../../packages/agent-app/tests/subagent-components.tsx) |
+| [MUS-SPEC-14-721E](14-agents-skills-plugins-and-automations.md#mus-spec-14-721e) | A desktop app is driven with no other app's plugin installed: reading it lists its controls, a click answers with only the lines that changed (a changed element once, as its new line), element numbers stay valid across steps and an unknown number is refused. | [`MUS-SPEC-14-721E: an agent drives a desktop app with Muster alone and each step answers with only what changed`](../../../packages/agent-app/tests/computer-mcp.test.ts) |
+| [MUS-SPEC-14-721F](14-agents-skills-plugins-and-automations.md#mus-spec-14-721f) | In a read-only chat the click and type tools are not offered, a click is refused, and the app is unchanged. | [`MUS-SPEC-14-721F: a read-only chat can look at apps but not change them`](../../../packages/agent-app/tests/computer-mcp.test.ts) |
+| [MUS-SPEC-14-721G](14-agents-skills-plugins-and-automations.md#mus-spec-14-721g) | Every agent chat gets the muster_computer server on Codex and Claude Code, a Full access router chat still runs on its own route, and without the helper the old note stays. | [`MUS-SPEC-14-721G: every agent chat gets Muster computer use on every MCP route, without forcing router chats onto Codex`](../../../packages/agent-app/tests/computer-mcp.test.ts) |
 
 ### Claude Code background sub-agents
 
@@ -154,3 +157,38 @@ Boundaries:
 - Environment: DOM suite built with esbuild and run with Node and linkedom; the real panel component and store; the bridge is a fixture.
 - Known gaps: The card still tells "Reported back" from "Completed" on its own badge; only the header folds them.
 - Evidence: UNVERIFIED
+
+### Muster computer use (#721)
+
+<a id="mus-spec-14-721e"></a>
+#### MUS-SPEC-14-721E: An agent drives a desktop app with Muster alone, one small answer per step
+
+- Precondition: macOS with Accessibility allowed for the runner. No other desktop app's computer-use plugin is involved. A fixture app with a notes field, an "Add one" button and a "Count 0" label is built from `tests/fixtures/computer-use/fixture-app.swift`.
+- Action: Over stdio, start the shipped `dist/main/computer-mcp.cjs` with the shipped `dist/main/muster-computer` helper. Read the app, click "Add one" by element number, type into Notes, click again, then click element 9999.
+- Expected result: The server names itself muster_computer and lists nine tools. Reading the app shows the button and "Count 0". The click answers within 3 s, after the app settles, with "Changed:" followed only by the label's new line ("Count 1"). It does not repeat the old value or the unchanged button line. The transcript names the step "Clicked “Add one” in MusterFixture", not an element number or the app's path. The typed text shows in the next answer. The second click with the same number gives "Count 2". Element 9999 is refused with an error.
+- Test: `packages/agent-app/tests/computer-mcp.test.ts::MUS-SPEC-14-721E: an agent drives a desktop app with Muster alone and each step answers with only what changed`
+- Environment: Node test runner on macOS with a real AppKit app and the real helper. Skipped when Accessibility is not granted, unless `MUSTER_REQUIRE_COMPUTER_E2E=1`.
+- Known gaps: Windows and Linux backends are not built yet. CI macOS runners have no Accessibility grant, so this runs on a developer Mac.
+- Evidence: UNVERIFIED
+
+<a id="mus-spec-14-721f"></a>
+#### MUS-SPEC-14-721F: A read-only chat can look at apps but not change them
+
+- Precondition: The same fixture app and shipped server as 721E, started with `MUSTER_COMPUTER_READ_ONLY=1` (what a read-only chat gets).
+- Action: List the tools, read the app, then call click on "Add one".
+- Expected result: click and type_text are not in the tool list. The click is refused with a message that the chat is read-only. Reading the app again still shows "Count 0".
+- Test: `packages/agent-app/tests/computer-mcp.test.ts::MUS-SPEC-14-721F: a read-only chat can look at apps but not change them`
+- Environment: As 721E.
+- Known gaps: None.
+- Evidence: UNVERIFIED
+
+<a id="mus-spec-14-721g"></a>
+#### MUS-SPEC-14-721G: Every agent chat gets Muster computer use, on every MCP route
+
+- Precondition: A computer-use launcher file exists, as main writes it on macOS when the helper ships.
+- Action: Build the turn options for a workspace agent chat and a read-only one, pass them through the Claude Code route, check whether a Full access router chat may stay on its route, and build them once more with the launcher missing.
+- Expected result: The Codex options register `mcp_servers.muster_computer` with the chat id and the read-only flag (0, then 1). The notes name muster_computer and no longer send the model to another app's plugin. The Claude Code route receives the server. A Full access router chat is still allowed on its own route. A muster_computer tool call shows as computer use in the transcript. Without the launcher the previous note is kept.
+- Test: `packages/agent-app/tests/computer-mcp.test.ts::MUS-SPEC-14-721G: every agent chat gets Muster computer use on every MCP route, without forcing router chats onto Codex`
+- Environment: Node test runner; no desktop needed.
+- Known gaps: OpenCode and Grok get the server through the same overrides but are not checked here.
+- Evidence: UNVERIFIED

packages/agent-app/CHANGELOG.md

@@ -8,6 +8,7 @@ All notable changes to Muster Agent. Each `## <version>` section becomes the not
 - Claude chats on your own router no longer reset to a summary when the router has lost the saved thinking for an earlier tool call. Muster resends that part of the history as plain text, so the chat keeps its full history and thread and carries on. Starting fresh is now a last resort.
 - Chats on your coding-client sign-ins and profiles no longer show up in your other coding apps. Muster now keeps its own threads in its data folder while sharing your sign-in, profiles, skills and plugins; existing chats continue where they left off.
 - Switching a chat to a model on another provider or account no longer loses the conversation. The first message on the new model now carries a summary of the earlier messages, as forks and compaction already do, and the chat says so.
+- Computer use no longer needs another desktop app installed. Muster ships its own computer-use helper for macOS: every agent chat, on Codex, Claude Code or your own router, can read an app's controls, click, type, press keys and scroll. Each step answers with only what changed on screen, so runs are quicker and use less context; read-only chats can look but not act. An orange Muster cursor shows where the agent is working, in the browser and in desktop apps, and fades out when it stops.
 
 ## 0.3.15
 

packages/agent-app/electron-builder.yml

@@ -36,6 +36,8 @@ asarUnpack:
   - node_modules/node-pty/**
   - dist/runtime/resources/**
   - dist/main/browser-mcp.cjs
+  - dist/main/computer-mcp.cjs
+  - dist/main/muster-computer
   - dist/main/*.node
   - dist/renderer/THIRD-PARTY-LICENSES.txt
 # Chromium/Electron locales: the app UI is English only; Chromium strings fall back to en-US.

packages/agent-app/scripts/build-computer-helper.mjs

@@ -0,0 +1,21 @@
+import { existsSync, mkdirSync, statSync } from 'node:fs';
+import { execFileSync } from 'node:child_process';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const root = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
+const source = path.join(root, 'src', 'native', 'computer-use-macos.swift');
+const output = path.join(root, 'dist', 'main', 'muster-computer');
+
+if (process.platform !== 'darwin') {
+  console.log('Skipping the computer-use helper: macOS is required.');
+  process.exit(0);
+}
+if (!existsSync(source)) throw new Error('Computer-use helper source is missing.');
+if (existsSync(output) && statSync(output).mtimeMs >= statSync(source).mtimeMs) {
+  console.log('Computer-use helper is up to date.');
+  process.exit(0);
+}
+mkdirSync(path.dirname(output), { recursive: true });
+execFileSync(process.env.SWIFTC || 'swiftc', ['-O', '-o', output, source], { cwd: root, stdio: 'inherit' });
+console.log(`Built ${path.relative(root, output)}`);

packages/agent-app/scripts/build.mjs

@@ -106,6 +106,7 @@ const builds = [
   },
   // The muster_browser MCP server Codex spawns per chat (Electron as node); it only forwards to main's local bridge.
   { ...common, entryPoints: [src('main', 'agent-tools', 'browser-mcp.ts')], outfile: dist('main', 'browser-mcp.cjs'), platform: 'node', format: 'cjs', target: 'node22' },
+  { ...common, entryPoints: [src('main', 'agent-tools', 'computer-mcp.ts')], outfile: dist('main', 'computer-mcp.cjs'), platform: 'node', format: 'cjs', target: 'node22' },
   {
     ...common,
     entryPoints: [src('preload', 'index.ts')],
@@ -184,6 +185,7 @@ function copyRendererStatic() {
 }
 
 execFileSync(process.execPath,[path.join(root,'scripts/build-quick-look.mjs')],{stdio:'inherit'});
+execFileSync(process.execPath,[path.join(root,'scripts/build-computer-helper.mjs')],{stdio:'inherit'});
 copyRendererStatic();
 mkdirSync(dist('runtime', 'resources'), {recursive: true});
 // One generic launcher: the route (profile file or config.toml provider) comes from the user's own Codex config.