Skip to content

Run the agent computer on a BoxCompute VM - #190

Open
farhan-helmy wants to merge 2 commits into
CopilotKit:mainfrom
farhan-helmy:feat/boxcompute-computer
Open

farhan-helmy wants to merge 2 commits into
CopilotKit:mainfrom
farhan-helmy:feat/boxcompute-computer

Conversation

@farhan-helmy

@farhan-helmy farhan-helmy commented Oct 20, 2026 •

Copy link
Copy Markdown

What changed

Adds COMPUTER_PROVIDER=boxcompute, a second opt-in desktop provider beside e2b-desktop. Docker stays the default, and the Docker and E2B paths behave as before.

With it, each owner gets one BoxCompute VM. That is an Ubuntu VM with a minimal Wayland desktop (sway), Google Chrome, Python 3, git and outbound internet. It is matched by deployment and owner metadata, as with E2B. Stop stops the VM: every process ends and /workspace stays on its disk.

  • Terminal and files. These reuse the existing lease, receipts, idempotency and files.py contract: the 30 second limit, exit code 124 on timeout, the output caps, symlink refusals and response shape. BoxCompute's exec API takes no stdin, so the command travels in an environment variable and file operations use BoxCompute's file API. A file over 8 MB is written in parts and joined in the VM.
  • use_desktop keeps its actions. Each one becomes one BoxCompute desktop call whose response carries the screenshot taken after it.
    • Keys map from xdotool names and combos.
    • The VM's keyboard is US ASCII, so other text is refused before anything is typed. The agent is told to write such text to a file with a command instead.
    • Input whose outcome BoxCompute reports as unknown is recorded interrupted, a fresh screenshot is taken, and the input is never sent again.
  • Desktop tab and live chat card. These open a BoxCompute viewer link, a random hostname serving a noVNC page. It is embedded with the same iframe sandbox as the E2B stream.
  • Desktop as a capability. The screen, action schema and command wrapper move from computer-e2b-desktop.ts to a neutral computer-desktop.ts, and E2B re-exports them. The service, tools, conversation step limit and mobile app now check desktop on the computer snapshot instead of the e2b-desktop provider name. E2B behaviour is unchanged.
  • Credentials. The BoxCompute API key stays on the API host and task worker. It is never passed into the VM or to the client.
  • Config.
    • Requires BOXCOMPUTE_API_KEY and a unique COMPUTER_DEPLOYMENT_ID, because VMs are matched across the whole account.
    • BOXCOMPUTE_WORKSPACE_ID is optional. By default a workspace named OpenMuse is created once.
    • BOXCOMPUTE_API_URL is optional.
  • Docs.
    • docs/COMPUTER.md has a setup section and how the provider differs from Docker.
    • SECURITY.md covers the threat model: internet egress, root in the VM, and the viewer link as a bearer secret.

Fail-closed behaviour mirrors E2B:

  • The server refuses to attach when the metadata does not match exactly, when more than one VM matches, or when the VM is not a large, network-enabled VM.
  • VMs are created with autoResume: false. A command, file operation or desktop action on a stopped computer is refused (SANDBOX_STOPPED) instead of resuming it; only Start wakes it.

Disclosure: I work on BoxCompute. Desktop control, metadata matching and autoResume were added to BoxCompute's public API this week for this integration.

Verification

pnpm lint                         # pass
pnpm typecheck                    # pass
pnpm test                         # 512 pass, incl. 29 new: 28 provider tests against an in-memory API fake, 1 config test
pnpm build:server                 # pass
pnpm build:web                    # pass
pnpm build:ios                    # pass
pnpm build:android                # pass
pnpm --dir apps/worker typecheck  # pass
BOXCOMPUTE_API_KEY=... pnpm test:computer:boxcompute   # pass, 1 test in 331s on this commit against a real VM

The live smoke test runs against a real BoxCompute VM with a random deployment ID and deletes every VM it created. It covers:

  • start, commands, and Chrome launched from a command
  • typed input reaching a focused page
  • desktop actions with screenshots
  • viewer link reuse
  • runaway output
  • a near-10 MB file import and export
  • the 30 second timeout
  • Stop, after which commands and desktop actions are refused instead of resuming the VM

The first live runs found five provider bugs that the in-memory fake could not catch. All are fixed in the second commit:

  • the execute result is wrapped in result
  • Stop answers 409 until an accepted stop settles
  • the VM image captures PNG only
  • the overflow notice was lost under the shared output cap
  • reads did not survive a dropped connection during a BoxCompute deploy

They also found a BoxCompute bug: after Stop and Start, chrome would not launch because of a stale profile lock. That is fixed on the BoxCompute side (0.84.1, live). The unmodified test then passed end to end: VM and desktop started in 38 s, the PDF imported in 22 s, and the VM and desktop resumed in 41 s.

Integration limits

  • Needs a BoxCompute account and API key. Running VMs are billed by BoxCompute, and a stopped VM keeps its disk and counts toward the account's storage until it is deleted.
  • Not yet checked by hand in the web app.
  • Native iOS and Android binaries were not run; only the platform exports are built.
  • pnpm test:browser and the browser-container suite were not run, because the browser worker is untouched.

🤖 Generated with Claude Code

farhan-helmy and others added 2 commits October 20, 2026 14:46
Adds COMPUTER_PROVIDER=boxcompute, a second opt-in desktop provider beside
e2b-desktop. Each owner gets one large BoxCompute VM, matched by metadata,
with a Wayland desktop and Chrome; terminal, files, use_desktop, the
Desktop tab and the chat card work as for E2B. Docker and E2B behaviour
is unchanged.

The neutral desktop pieces (screen, action schema, command wrapper) move
to computer-desktop.ts, and the service, tools, conversation and mobile
app check a desktop capability instead of the E2B provider name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live smoke test found what the in-memory fake could not:

- execute reads the result BoxCompute wraps in `result`
- Stop retries the 409 BoxCompute answers until an accepted stop has
  settled, for about 90 seconds
- screenshots are PNG, because BoxCompute's VM image does not capture
  JPEG
- stderr is kept before stdout under the shared output cap, so the
  overflow notice survives a runaway command
- reads and whole-file uploads ride out a dropped connection or a
  gateway error, as during a BoxCompute deploy; commands and input are
  never resent

The smoke test now serves its typing page as HTML that fills the
window, checks for Chrome with a pattern that cannot match itself,
probes the running command through BoxCompute instead of the lease that
command holds, and cleans up its VM even when a request drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@farhan-helmy
farhan-helmy marked this pull request as ready for review October 20, 2026 14:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant