Skip to content

Latest commit

 

History

History
85 lines (51 loc) · 11 KB

app.md

File metadata and controls

85 lines (51 loc) · 11 KB

Desktop Tools app

Desktop Tools.app (dev.githubnext.desktop-tools) is a menu bar host for agents that don't embed the library, such as Claude Code or Codex through the MCP server. It holds its own Accessibility, Screen Recording and Keyboard input grants, requested from its menu while any is missing, and serves ~/.local/state/desktop-tools/desktop.sock. It links desktop-client there to its signed client. Only one copy runs for each state folder. See Keeping the socket served. Show agent cursor, on by default, draws each agent's own pointer for each click, scroll, drag, hover and press, so several agents' cursors move independently. Each agent has its own color: orange for Claude, blue for Codex, and for any other agent a color picked from its name, the same every time. Highlight agent's window, also on by default, outlines the window the agent is observing or acting on and labels it with the agent and app, such as "Claude is using TextEdit". See input.

Keeping the socket served

Agents find the app only through desktop.sock, so the app and its client both work to keep it served:

  • When the app can't start serving, such as while a copy that is quitting still holds the socket, its menu's status line shows why and it retries after 1, 2, 4… seconds, up to 30.
  • While serving, the app watches the socket. If it is removed or replaced, the app publishes it again at once and notes it in its status line and its log.
  • Opening the app while it runs makes it check its socket at once, and retry a failed start without waiting.
  • When a call finds no app serving the socket, the client opens the app in the background, which starts it or makes it check its socket, and waits up to 10 seconds for it to serve. Nothing reached the app the first time, so the call then runs once. Only when that fails does the call fail, with HOST_NOT_RUNNING or HOST_NOT_SERVING and what the client found. Reconnecting the MCP server never helps: the app's desktop-tools-mcp reconnects on the next call after the app restarts.

The client does the same from a shell. Both commands default to the socket beside the desktop-client link they are run through:

~/.local/state/desktop-tools/desktop-client status   # whether the socket serves, and the app's path, pid and bundle ID
~/.local/state/desktop-tools/desktop-client repair   # open the app if the socket doesn't serve, and wait until it does

Each prints JSON. reach is serving, refused (the socket is there but nothing listens) or missing. repair exits 1 with HOST_NOT_RUNNING or HOST_NOT_SERVING when the socket still doesn't serve.

Install

Download the notarized DMG from the latest release, drag Desktop Tools to Applications, and open it. Grant each permission from its menu.

Anything running as the user could use those grants, so the app asks before serving each agent (see approved agents). Grant them only on machines where that is acceptable, and quit the app when it isn't needed.

Updates

The app keeps itself up to date from this repository's GitHub Releases, which anyone can read without signing in.

  • Update Automatically, on by default, checks a minute after launch and then once a day (a failed check is retried an hour later). A newer release is downloaded and checked in the background, then installed once agents have run no operation for 10 minutes, input isn't paused and no Desktop Tools prompt is open. Turn it off and the app only checks when you ask.
  • Check for Updates… checks at once. It says when the app is up to date or why a check failed; when there is a newer release it downloads it and asks before installing (Install and Relaunch or Later). Below the item the menu shows the running version and when it last checked, or why the last update failed.
  • An update that is downloaded but not installed yet shows as Install Desktop Tools version and Relaunch, and is also installed when you quit the app, ready for the next launch.

Installing stops serving new calls, lets the calls already running finish (at most 30 seconds), replaces the app in place in one rename, and opens the new version through LaunchServices once the old one has exited. Agents reconnect on their next call: the client waits for the app to serve again, as in Keeping the socket served. Accessibility, Screen Recording and Keyboard input stay granted, and approved agents and settings are kept, because the new app has the same bundle ID and signature requirement.

Nothing is installed unless the release's disk image and the app inside it are signed with Developer ID by team 8S43JJMN6B, notarized, and pass Gatekeeper's assessment, and the app has the bundle ID dev.githubnext.desktop-tools, satisfies the running app's own designated requirement, and is the newer version the release names. The app must be able to write to the folder it is installed in, such as Applications; otherwise the update fails and names the release page to download it from. Each check and update is noted in the log.

The development app (Desktop Tools Dev) and any build not signed by that team never update themselves and have no update items in their menu.

To test a release build against another release list, open it with DESKTOP_TOOLS_UPDATE_FEED set to a URL that returns the same JSON as GitHub's list releases API, such as a file: URL; the app passes it on when it relaunches, as it does DESKTOP_TOOLS_STATE. Updates from it are checked the same way.

Approved agents

The first time an agent calls, the app asks Allow name to control this Mac?, naming who signed it and where it runs from, with Allow and Don't Allow. Until you answer, and after Don't Allow, every request from that agent is refused with CALLER_NOT_APPROVED, listings and screen captures included; the agent is told to ask you, and is never left waiting. The answer is remembered in callers.json in the state folder (~/.local/state/desktop-tools), readable only by you, and the app asks about each agent once. Updating from 0.13.1 or earlier, each agent you use is asked about once.

The agent is the program that ran the MCP server, skipping node, npx and shells: Claude Code is claude-code, and a server run from a terminal is that terminal app. Signed programs are recognized by their signature, so an update keeps its approval; unsigned ones by their path. A team counts only when Apple issued the certificate to it.

An agent run as a background job, such as claude --bg, has no terminal or app above it: launchd adopted its processes. Its agent is then the nearest program above the MCP server that is signed by a developer team, such as Claude Code, and the prompt says it is a program, not an app: Allow the program name to control this Mac? Allowing it covers every agent that program runs, in a terminal or in the background, since both are the same signed program. When no app and no team-signed program runs the agent, nothing can be approved, so every request is refused with CALLER_NOT_APPROVED and no prompt, naming the processes it found; run the agent from a terminal app or a signed program instead.

A background job doesn't load your shell's startup files, so a version manager such as nvm doesn't put node on its PATH. Set the MCP server up with the app's desktop-tools-mcp, or give an npx or node setup by absolute path, such as the output of which npx; a bare node fails with ENOENT: Executable not found in $PATH. Approved Agents in the menu lists every agent with who signed it and its path: revoke an allowed one, allow one you refused or forget it so it's asked about again, or answer one still waiting.

The app's own prompts, menus and windows answer only to you. Agent input aimed at any Desktop Tools process, including the app, its client and another build of it, is refused with SELF_TARGET_REFUSED, and so is every agent input while one of its prompts or menus is open. Agents can still observe them.

Open Log in the menu opens activity.log in the state folder, in Console unless you chose another app for .log files. Each line has the time and one operation an agent ran:

  • the operation, such as Click, Key, Type, Capture or Read clipboard;
  • the target app and window title;
  • the agent, as named under Approved agents, and its pid;
  • the outcome: completed, unknown, refused, or paused for a request refused because input was paused.

Every desktop change is logged, and so are screen captures and clipboard reads. Turning on an app's web accessibility for a read is logged once per launch of that app. Listings and snapshot bookkeeping are not. Only the operation and its target are kept, never typed text, clipboard contents or values. The log also notes when the app starts, with the socket it serves, and when you pause input, or allow, refuse or forget an agent. The file is readable only by you; past 1 MiB it moves to activity.log.1 and a new one starts.

The menu shows a Grant item only for a permission the app is missing. Pause All Input sits just above Quit.

Always Allow Agent-Supplied Passwords, off by default, lets desktop_fill_password fill a password an agent passes in without asking you each time. Keychain items still need macOS's own approval.

Pause All Input stops agents from changing anything until you turn it off again. While paused, the app refuses every click, key, typing, scroll, drag, menu command, app or window change, dialog action and clipboard write before it is sent. The refusal tells the agent to ask you to resume. Agents can still observe: listings, the accessibility tree, screen captures and clipboard reads keep working. So does turning on a Chromium or Electron app's web accessibility before a read, since it only lets the app's web content be read and sends no input (see tools). The menu bar icon becomes a pause sign while input is paused. Only this menu item changes it; no client can pause or resume, and agent input can't reach the menu. Pausing isn't saved, so the app starts resumed.

Build

node dist/app.js --out build --identity "Developer ID Application: … (TEAM)" --release

assemble(out, scratch?) and sign(app, identity, release) in dist/app.js are what the release workflow uses. assemble(out, scratch, true) builds the development app instead, and sign takes the client's identifier from the bundle's.

Development

Run development builds with npm run dev-app, described in Install and build. It installs one Desktop Tools Dev.app (dev.githubnext.desktop-tools.dev) at a fixed path and serves ~/.local/state/desktop-tools-dev/desktop.sock, so it never relinks the installed app's ~/.local/state/desktop-tools/desktop-client or takes its socket. Point the MCP server at it:

DESKTOP_TOOLS_STATE=~/.local/state/desktop-tools-dev node dist/mcp.js